Training on a subset of data for few epochs and then proceeding to the next subset for few epochs and so on? My training data is very huge and it's impossible to load all of it at once even into main memory. So I'm loading a few blocks (subset) of data and training till convergence, then proceeding to next subset and training till convergence and so on. Is it the right approach ?

The model performance kind of remains the same even when training on a new subset of data.

Is this method fundamentally wrong, why ?

Training until convergence on a subset of data and starting again on another subset is not a good idea.

Gradients of loss will have high variance over your batches and so optimizing over it will not be useful (you basically start at a far new point on your loss for the next subset).

Instead, the good way to go in batch-training methods is to iteratively make training steps on differents subsets of your data until a global convergence.
Your may hear about an epoch, which consists of training steps on subsets until the overall data has been seen.
You see how convergence is over the epochs and not over one subset.


How to deal with mixed type features? I'm working on a dataset with more than 2000 features. Most of the features contain both numerical values and categorical features. For example, in the feature represents how long has the user been living in the current address, the value could be numbers or some letters which mean the value cannot be obtained due to some reasons.
I don't know how to process these features. If they are just pure numerical or categorical values, things will much easier. But since they are mixed, I'm really confusing. Could anyone give me some advice?
Update: I may not express clearly that it is not a dataset include both numerical features and categorical features. I mean in one feature, there are both numerical values and categorical values.
For example: (here M, C, T mean that because of different reasons, no exact values can be found)
TOTAL INCOME
3000
5000
M
8000
C
4000
T

The best will be to have domain knowledge and understand how this feature affects the target and what will be the best way to encode it:
First way: treat them all as a categorical feature with high cardinality.
Second way: split them into two columns, one categorical and one numerical. And then treat them separately.
Without domain knowledge in this case I cant think of on something better. Answering this question would help: Is the number really a number? Why can I have numbers and strings in the same column? How will this information might help my model?


Best way to combine two similar document I have f.ex.: two news-articles that report the same event. However, these two text are similar BUT not the same. I would like to combine these two texts creating one text that contains only the most "relevant" information.

I was thinking to check the texts paragraph wise on its "information value" and then only combine the paragraphs that are the most "relevant".

Any suggestions on an approach how to do this? Any research papers about this topic?

Your description could corresponds to the NLP task of summarization. This is an active field of research: https://scholar.google.com/scholar?q=text+summarization

A much simpler option is to only extract sentences from both: in this case the goal is not to produce a text which reads like a story, just an enumeration of sentences.

Even in this case you will have to define how to measure "information value", this is not easy afaik.


Logistic Regression or regression SVM for probability of outcome I am working on a prediction question: what's the percentage of Y = 1 using a number of features?

The output Y values I have for training are in binary.  In this case, should the prediction be treated as classification or regression?
Would logistic regression that returns the probability be suited or the probability of SVM, e.g., this?

Applying regression methods would render useless, as we usually feed binary (or multiple) categories as target variable in classification problems. However, if you have somehow got probability values for each sample, you can perform regression as well - I can't think of such a rare case, though.

If you are looking for prediction probabilities, Logistic Regression is the best call. Having some experience with sklearn, I can tell that SVM, by default, does not compute probabilities. If you wish to obtain those prediction probabilities, SVM uses another set of algorithms, which cause serious slowdown in process and extra computational cost.


Tools suitable for semi-automatic video labeling? So far I have been using labelme to label objects in videos I use for training, but it is quite time consuming.

Are there good tools to help with that?

I was thinking about a tool where I label some frames, and the tool would then match the labelled areas in the next frames. Note that a simple interpolation would not be enough.

You could use some tracking lib. Adrian Rosebrock gives nice overview on his blog.


How to binary encode multi-valued categorical variable from Pandas dataframe? Suppose we have the following dataframe with multiple values for a certain column:

    categories
0 - ["A", "B"]
1 - ["B", "C", "D"]
2 - ["B", "D"]

How can we get a table like this?

   "A"  "B"  "C"  "D"
0 - 1    1    0    0
1 - 0    1    1    1
2 - 0    1    0    1

Note: I don't necessarily need a new dataframe, I'm wondering how to transform such DataFrames to a format more suitable for machine learning.

If [0, 1, 2] are numerical labels and is not the index, then pandas.DataFrame.pivot_table works:

In []:
data = pd.DataFrame.from_records(
    [[0, 'A'], [0, 'B'], [1, 'B'], [1, 'C'], [1, 'D'], [2, 'B'], [2, 'D']],
    columns=['number_label', 'category'])
data.pivot_table(index=['number_label'], columns=['category'], aggfunc=[len], fill_value=0)

Out[]:
              len
category      A      B      C      D
number_label
0             1      1      0      0
1             0      1      1      1
2             0      1      0      1

This blog post was helpful.

If [0, 1, 2] is the index, then collections.Counter is useful:

In []:
data2 = pd.DataFrame.from_dict(
    {'categories': {0: ['A', 'B'], 1: ['B', 'C', 'D'], 2:['B', 'D']}})
data3 = data2['categories'].apply(collections.Counter)
pd.DataFrame.from_records(data3).fillna(value=0)

Out[]:
       A      B      C      D
0      1      1      0      0
1      0      1      1      1
2      0      1      0      1


How to provide classified feature to a neural network Let say I have a feature that may have one of 4 values, 1,2,3,4. I want to provide it as a NN input, what is proper way to do that?

I can map it like 1 -> -1.0 | 2 -> -0.3 | 3 -> 0.3 | 4 -> 1.0 , or something similar to have mean of 0.0 and std near 1.0. But in this example 1 is much different than 4 compared to difference of 3 and 4, and I don't want such discrimination cos 1 and 4 are equally different to me as 3 and 4.

Another way is to have 4 features where each of them relates to a class 1,2,3,4 and has value of 1 if initial feature has value that matches it, and has value of 0 if it doesn't match it.

Like this 1 -> [1,0,0,0], 2 -> [0,1,0,0], 3 -> [0,0,1,0], 4 -> [0,0,0,1]
But I don't like here the fact that this feature gets to much weight, especially if it has many classes.

I was thinking to make separated layer just for this feature, do you have some better solution?

In your problem, the label is a categorical variable (you cannot infer relation between classes just from the label value) and not ordinal (value shows relation/distance between classes).

The solution that you propose:

  1 -> [1,0,0,0], 2 -> [0,1,0,0], 3 -> [0,0,1,0], 4 -> [0,0,0,1]

is called One-Hot Encoding. This is one of the most popular ways of encoding the classes during preprocessing, in order to feed them to a classifier, thus I recommend you of doing so.

You mention that you are afraid that the

  feature gets to much weight

because of the number of samples in the dataset. This is called class imbalance. A way of circumventing it is to pre-weight your samples for your classifier, please take a look at this method implementation provided by sklearn package.


What is the proper train data format in LSTM? I want to train a model to detect wrong word using in sentence.

I have 1 million sentences(word base or char base) with different length. Each position(word or char) has a label to indicate it is wrong or correct .
If a sentence length is  10,  I want LSTM learn 2 words surrounding current word ( so each word become 5 word) . So I need transform the sentence to [10, 5] .
But the first word(doesn't have left words) and  right word (doesn't have right words) , so I need pad them.
Then use word2vec transform this sentence to  [10, 5, embedding_size] , same as other sentences. (i.e each word transform to 5 word embeddings)
trainning (feed model by [5, embedding_size] * bacth_size )

I saw a lot LSTM input format and finally work out above flow.
Is my steps right ?

Actually I am still confusing with LSTM input format.

I have seen

Some pad all sentence to same length
Some split sentence into word blocks with same size (like I do above).
others ....

I am trying to use code to generate error samples, and mix with correct samples for lstm training.

For example:

corrects  = [
    ['How', 'are', 'you', '!'],          # [1, 1, 1, 1]
    ['Fine', ',', 'thank', 'you', '.'],   #  [1, 1, 1, 1, 1]
    ['Do', 'you', 'have', 'meal', '?'],   #  [1, 1, 1, 1, 1]
    ...
]

wrongs  = [
    ['How', 'were', 'you', '!'],            # [1, 0, 1, 1]
    ['Find', ',', 'thank', 'you', '.'],    # [0, 1, 1, 1, 1]
    ['Did', 'you', 'have', 'meal', '.'],    #  [0, 1, 1, 1, 0]
    ...
]

I come up with several thoughts

Because lstm can learn current word's network weight with pre words. So directly pass sentence to lstm model, don't transform from [sentence_len, embedding_size] to [sentence_len, step_len, embedding_size] (step_len means present a word with it surround words).
Previous sentence's last word may be affacted by latter sentence's first few word, so It better chain all sentence . (How to BiLSTM in this situation ?)
Use feature like [sentence_len, step_len, embedding_size], doesn't need BiLSTM any more ? I think it still help here.

There are too many thoughts in my brain, hard to make it clear. So I post question here.

[sentence_len, embedding_size] to [sentence_len, step_len, embedding_size] (step_len means present a word with it surround words).

Use step_len=1, just the current word. after squeezing the tensor, its shape will be  [batch_size,sentence_len,embedding_size].

  Previous sentence's last word may be affected by latter sentence's first few words.

Use padding, append the last 3 to 5 words of previous sentences at the start of the current sentences, do not backpropagate for the errors of those padded words, by masking their loss. as for the case of first sentences of paragraphs, padded with a simple <pad> token.

  doesn't need BiLSTM anymore? I think it still helps here.

Using BiLSTM is a must. if you want to limit step_len to be 1, by the way, you are computing the conditional probability of Y being true (correct/not correct) conditioned on the whole context (sentence). as you are feeding the sentence at once, the loss will be the summation over all single losses.

Side note:
To batch the training examples, you also need to pad the end of sentences with the three to five words of the next ones. use <pad> for sentences ending the paragraph, don't permit their contributions to the model (adjusting the weight), mask your padded examples to have zero loss. because their labels are just noise.


Is gradient descent slower for finite differences? In gradient descent, we updated each parameter $\theta_i$ in the direction which minimizes a function $f(\theta_1,\theta_2,\dots,\theta_N)$ by doing $$\theta_1 \leftarrow \theta_1 - \alpha \frac{\partial f}{\partial \theta_1}(\theta_1,\theta_2,\dots,\theta_N)$$
$$\theta_2 \leftarrow \theta_2 - \alpha \frac{\partial f}{\partial \theta_2}(\theta_1,\theta_2,\dots,\theta_N)$$
$$\vdots$$
$$\theta_N \leftarrow \theta_N - \alpha \frac{\partial f}{\partial \theta_N}(\theta_1,\theta_2,\dots,\theta_N).$$

If we have $N$ parameters, then it will involve $N$ evaluations.

If possible, obviously, we want to use this analytical form of the gradient.

But we could, of course, approximate it using finite differences,
$$\theta_1 \leftarrow \theta_1 - \alpha \frac{f(\theta_1+\varepsilon,\theta_2,\dots,\theta_N)-f(\theta_1,\theta_2,\dots,\theta_N)}{\varepsilon}$$
$$\theta_2 \leftarrow \theta_2 - \alpha \frac{f(\theta_1,\theta_2+\varepsilon,\dots,\theta_N)-f(\theta_1,\theta_2,\dots,\theta_N)}{\varepsilon}$$
$$\vdots$$
$$\theta_N \leftarrow \theta_N - \alpha \frac{f(\theta_1,\theta_2,\dots,\theta_N+\varepsilon)-f(\theta_1,\theta_2,\dots,\theta_N)}{\varepsilon}$$

This involves $N+1$ evaluations, which is negligible if $N$ is large, as is the case with deep neural networks.

Obviously, the analytical gradient is preferable since it is more accurate, and it avoids numerical problems due to $\varepsilon$ being too large or too small.

My question is: I have heard people claim that using gradient descent with the analytical derivative is faster than approximating the derivatives. Is this true? To me, it sounds like it only avoids one extra evaluation.

EDIT: Practical example:

Let's say we have
$$y=\sigma(ax)$$

$$z=by$$
where $a$ and $b$ are the parameters we want to optimize.

Gradients

Using the gradients, we would have $\nabla=\left(\frac{\partial z}{\partial a},\frac{\partial z}{\partial b} \right) = \left(bx\sigma(ax)(1-\sigma(ax)), \sigma(ax)\right)$

(I hope I didn't make a mistake there. $\frac{\partial z}{\partial a}=\frac{\partial z}{\partial y}\frac{\partial y}{\partial a}=bx\sigma'(ax)=bx\sigma(ax)(1-a\sigma(ax))$.)

For each back-propagation, we would need two evaluations for $a$ and for $b$: $\nabla(a,b)$, so that we can do...

These are the computations required:

$a \leftarrow a-\alpha bx\sigma(ax)(1-\sigma(ax))$

$b \leftarrow b-\alpha \sigma(ax)$

Finite differences

We have the differences $$\Delta=\left((z(a+\varepsilon,b)-z(a,b))/\varepsilon, (z(a,b+\varepsilon)-z(a,b))/\varepsilon \right)$$

$z(a,b)=\color{blue}{b\sigma(ax)}$

$z(a+\varepsilon,b) = \color{blue}{b\sigma(ax)} + b\sigma(\epsilon x)$

$z(a,b+\varepsilon) = \color{blue}{b\sigma(ax)} + \varepsilon\sigma(ax)$

These are the computations required:

$a \leftarrow a-\alpha b\sigma(\varepsilon x)/\varepsilon$

$b \leftarrow b-\alpha \sigma(ax)$

The terms in blue disppear in the graph because they are subtracted by themselves.

(In response to the comments)

Consider logistic regression with labels +1/-1, where the function is $f(x) = \sum log(1 + \exp(-y * (x \theta)))$ . Taking finite differences requires computing $\sum log(1 + \exp(-y * (x (\theta + \epsilon_n))))$ (where $\epsilon_n$ is a small value for only one variable) for each variable (in each evaluation, you require plugging in the current values of all other variables too). The analytical solution is $\sum residual \theta$ - you only require plugging in the current values twice. You can keep a sum of $\theta x$ to save calculations (still more calculations with the differences approach), but as your function gets more non-linear, there’s less that you can precompute between variables – e.g. if you have 3 hidden units, changing the first unit will not let you reuse computations for the others.

In terms of big-oh notation, if you are looking at number of evaluations and you consider the gradient evaluation as $O(n)$ (keep in mind vectorization makes a difference though), I guess you won’t see much difference, but if you look at others aspects you’ll see that the analytical solution requires fewer calculations.

In addition, optimization techniques such as L-BFGS will not work if you don’t use precise gradients.


K means library designed for large dataset in python I find a k means library for large dataset in R (https://www.r-bloggers.com/k-means-clustering-on-big-data/) , may I know is there any similar library in python?

http://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html

Although k-means is not to hard to code.


How to print accuracy in each fold of Validation Dataset? and assign fold number to each row in the dataframe? How to print the accuracy in each fold of a Validation Dataset ? And how can I assign the fold number to each row in the dataframe?

classifier = RandomForestRegressor(n_jobs = -1, criterion='mse')
clf = GridSearchCV(classifier, param_grid = tunedParameters,cv=10)
all_accuracies = cross_val_score(classifier, X=X_train, y=y_train, cv=10)
print(all_accuracies)

You're mixing up GridSearchCV and cross_val_score; you should only need to run one of them.

GridSearchCV will search through your hyperparameter space, for each combination using cross-validation and producing a score.  You can access these scores through the attribute cv_results_.

cross_val_score has no hyperparameter search; it just scores using cross-validation.  The output is a list of the individual fold scores.

If you've already used GridSearchCV, there's probably no reason to use cross_val_score.  (After hyperparameter searching, you've seen and used all the data in that set, so the scores in cv_results_ are biased as would be the scores out of cross_val_score; if you need an unbiased estimate of performance, you'll need another test set (or nested cross-validation in the first place).)

If you want to keep track of which samples go in which fold, I think you need to use a cross-validation generator or iterable for the cv parameter instead of an integer.  Then you can use that generator/iterable to also tell you which samples are in which fold.


How to build Non Linear Vector auto regression using Neural Nets I have worked on VAR ( vector Auto regression), which forecasts multiple output values(continuous) when there is linear relationship between all variables. I want to use Neural nets to do so,sothat I can include non linear relation among variables also.

Is there any specific type of NN available?

If not how can I implement it using NN?

The type of artificial neural nets (ANN) you are looking for is ANN with cycles which is widely known as recurrent neural networks(RNN). As you have mentioned Vector Auto regression (VAR) I would recommend you to check out this paper which compares VAR to RNN on two different tasks pertaining to two different real world datasets.

One specific type of RNN is Long short term memory (LSTM). It is particularly good at learning over long sequences. Although RNN is also perfectly capable of that theoretically, but with little empirical success. So, in recent years LSTM has gained huge popularity and has been very successful over sequence learning tasks. I would recommend this book which provides a fair bit of understanding of ANN with cycles and the tasks they are being applied to.

For implementation of RNN there are many APIs available. Keras provides a high level API for the same.

For modelling multivariate time series with LSTM check out this tutorial.

You can also have a look at this question that I answered on cross validated.


Recommendations for statistical models given my dataset I'm having a problem when using the k-fold cross-validation with the Random Forest method. One of the outputs is the error "Error in randomForest.default(x, y, mtry = param$mtry, ...) : Need at least two classes to do classification." However, I already have two classes to do the classification, which are "Normal" and "Failure". When posting this question at https://stackoverflow.com/questions/60643415/error-in-randomforest-defaultx-y-mtry-parammtry-need-at-least-two?noredirect=1#comment107290940_60643415 was recommended to me that I ask for "recommendations for statistical models given my data and your prediction/estimation/modeling needs".

Someone could help me?

The R script:

library(caret)
library(randomForest)

data_failures <- read.csv('OUTPUT.csv', header = TRUE, sep = ",", stringsAsFactors = TRUE)

train.control <- trainControl(method = "cv", number = 10)
model <- train(Period_1 ~., data = data_failures, method = "rf",
                                   trControl = train.control)
print(model)

print(class(str(data_failures)))

The ouput:

Random Forest

112 samples
 11 predictor
  2 classes: 'Failure', 'Normal'

No pre-processing
Resampling: Cross-Validated (10 fold)
Summary of sample sizes: 101, 101, 101, 101, 101, 101, ...
Resampling results across tuning parameters:

  mtry  Accuracy  Kappa
   2    1         NaN
   6    1         NaN
  11    1         NaN

Accuracy was used to select the optimal model using the largest value.
The final value used for the model was mtry = 2.
'data.frame':   112 obs. of  12 variables:
 $ Period_1 : Factor w/ 2 levels "Failure","Normal": 2 2 2 2 2 2 2 2 2 2 ...
 $ Period_2 : Factor w/ 2 levels "Failure","Normal": 2 2 2 1 2 2 2 2 2 1 ...
 $ Period_3 : Factor w/ 2 levels "Failure","Normal": 2 2 1 2 2 2 2 2 2 2 ...
 $ Period_4 : Factor w/ 2 levels "Failure","Normal": 2 2 2 2 2 2 2 2 2 2 ...
 $ Period_5 : Factor w/ 2 levels "Failure","Normal": 2 2 2 2 2 2 2 2 2 2 ...
 $ Period_6 : Factor w/ 2 levels "Failure","Normal": 2 2 2 2 2 2 2 2 2 2 ...
 $ Period_7 : Factor w/ 2 levels "Failure","Normal": 2 2 2 2 2 2 2 2 2 2 ...
 $ Period_8 : Factor w/ 2 levels "Failure","Normal": 2 2 2 1 2 2 2 2 2 2 ...
 $ Period_9 : Factor w/ 2 levels "Failure","Normal": 2 2 2 2 2 2 2 2 2 2 ...
 $ Period_10: Factor w/ 2 levels "Failure","Normal": 2 2 2 2 2 2 2 2 2 2 ...
 $ Period_11: Factor w/ 2 levels "Failure","Normal": 2 2 2 2 2 2 2 2 2 2 ...
 $ Period_12: Factor w/ 2 levels "Failure","Normal": 2 2 2 2 2 2 2 2 2 2 ...
[1] "NULL"
Warning messages:
1: model fit failed for Fold08: mtry= 2 Error in randomForest.default(x, y, mtry = param$mtry, ...) :
  Need at least two classes to do classification.

2: model fit failed for Fold08: mtry= 6 Error in randomForest.default(x, y, mtry = param$mtry, ...) :
  Need at least two classes to do classification.

3: model fit failed for Fold08: mtry=11 Error in randomForest.default(x, y, mtry = param$mtry, ...) :
  Need at least two classes to do classification.

4: In nominalTrainWorkflow(x = x, y = y, wts = weights, info = trainInfo,  :
  There were missing values in resampled performance measures.

A sample of the data:

    Period_1 Period_2 Period_3 Period_4 Period_5 Period_6 Period_7 Period_8
1     Normal   Normal   Normal   Normal   Normal   Normal   Normal   Normal
2     Normal   Normal   Normal   Normal   Normal   Normal   Normal   Normal
3     Normal   Normal  Failure   Normal   Normal   Normal   Normal   Normal
4     Normal  Failure   Normal   Normal   Normal   Normal   Normal  Failure
5     Normal   Normal   Normal   Normal   Normal   Normal   Normal   Normal
6     Normal   Normal   Normal   Normal   Normal   Normal   Normal   Normal
7     Normal   Normal   Normal   Normal   Normal   Normal   Normal   Normal
8     Normal   Normal   Normal   Normal   Normal   Normal   Normal   Normal
9     Normal   Normal   Normal   Normal   Normal   Normal   Normal   Normal
10    Normal  Failure   Normal   Normal   Normal   Normal   Normal   Normal
```

My guess is that either your Failure/Normal class is a lot lesser than the other. As such, for a certain (i.e. nth) fold, there only exists instances of one class. You can try doing oversampling the under-represented class to prevent this, or try doing a stratified K-Fold so that each fold will have occurrences of both classes.


How to cache GridSearchCV optimizer result in Google Colab? I have to execute GridSearchCV() cell every time I reload the page and it takes a lot of time. Is there a caching mechanism which stores the GridSearchCV result so that I can use it without executing the cell on the reload?

Code:

 neigh_clf = KNeighborsClassifier()
 grid_search = GridSearchCV(neigh_clf, param_grid, cv=5,verbose=3,n_jobs=-1)
 grid_search.fit(X_train, y_train)
 grid_search.best_params_

I am trying to save grid_search object here so that I can retrieve best_params_ value after reload

We can save the trained model or any other file via Google Colaboratory.

How I'm using it?

  I have mapped my Google Drive with Google Colaboratory notebook and
  saved trained model as a pickle file in it. You can create a file and save your data in it.

How to integrate Google Drive with Google Colaboratory notebook?

#Add and execute below mentioned line of code in Google colaboratory notebook cell.
from google.colab import drive
drive.mount('ndrive')
#Once you execute these two lines, it will ask you to authorize it. Please follow the steps as visible on the Google Colaboratory Notebook.

That's it :).

  Few important points:

  Now you can run usual linux command to browse the directories.
  By default it mapped to content folder and you can create sub directories in it.
  You can visit these sub-directories via Google-Drive application as well.


Modeling the influence of events order on probability The case is to model if the sequence of events influences the probability of binary target variable. We have for example five different events which occur in time (event: A,B,C,D,E). They can occur in order from 1 to 5. I would like to check if the order of their occurrence influences the target variable.

My first idea was to convert the time of occurrence into numbers from 1 to 5 and then for example use logistic regression.

Do You know any other practices?
Any whitepapers and ideas will be helpful.

If the order in which the events appears matters, consider using a recurrent neural network. The setup that you propose is invariant to event ordering, whereas in a RNN the events are fed in in sequential order.


handling missing data in pandas python I have to impute missing values in the column named <Age> with the mean of the nearest available values above and below in the column <Age>.

If the Age column had values in order

NA,7,6,NA,7,8,NA,NA,NA,10,5,NA,NA,5,9,9,12,8,6,NA,NA

After imputation, column should look like
 7,7,6,6.5,7,8,9,9,9,10,5,5,5,5,9,9,12,8,6,6,6

I know about filling value with mean, mode, or median forward filling, backward filling but not this one. I am new to this field and a student. Any help would be appreciated.

Thanks


Ranking problem and imbalanced dataset I know about the problems that imbalanced dataset will cause when we are working on classification problems. And I know the solution for that including undersampling and oversampling.
I have to work on a Ranking problem(Ranking hotels and evaluate based on NDCG50 score this link), and the dataset is extremely imbalanced. However, the example I saw on the internet use the dataset as it is and pass it to train_test_split without oversampling/undersampling.
I am kind of confused if that is true in the Ranking problems in which the imbalanced data does not matter and we do not need to fix this before passing the data to the model?
And if that is the case why?
Thanks

You are completely right, imbalance of labels does have an impact on ranking problems and people are using techniques to counter it.
The example in your notebook applies list-wise gradient boosting. Since pairwise ranking can be made list-wise by injecting the NDCG into the gradient, I will focus on pair-wise rank loss for the argument. I will base myself on this paper (https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/MSR-TR-2010-82.pdf).
$C = -\bar{P}_{ij}$log$P_{ij} - (1 - \bar{P}_{ij})$log$(1 - P_{ij})$
with, $P_{ij}\equiv P(U_{i}\rhd U_{j})\equiv {1\over{1 + e^{-\sigma(s_{i} - s_{j})}}}$ and $\bar{P}_{ij} = {1\over2}(1 + S_{ij})$ for $S_{ij}$ being either $0$ or $1$.
This is actually just a classification problem, with 0 being article i being less relevance than article j and 1 in the opposite case.
Imagine that now you are working with queries which have a lot of matching documents but only a couple of documents have been tagged as relevant. Often such sparse tagging does not mean that ONLY these documents were relevant, but only is caused by the limitation of the estimation of relevance (https://www.cs.cornell.edu/people/tj/publications/joachims_etal_05a.pdf).
Hence, it is not uncommon to down-sample high rated documents.
Another reason for applying imbalance methods such as reweighting labels, is for example bias due to position (see for example https://ciir-publications.cs.umass.edu/getpdf.php?id=1297). The loss is reweighted based on the observed position of the documents when their relevance was tagged.


How to professionally spell library names such as "scikit learn"? For example for inclusion in a CV. This is mostly a question of whether one should use capital letters or not.
Usually I search the library to find how the library name is spelled on the official page.
For scikit-learn it is spelled without capital letter, while most of the other letter has some capital letter at the beginning (and often inside as well). Still it doesn't feel right next to other capitalised names.
Should I still use a capital "Scikit-Learn" or am I overthinking this?

Scikit-learn has a citation guide: https://scikit-learn.org/stable/about.html#citing-scikit-learn
They spell it with a uppercase "S" and lowercase "l":

Scikit-learn: Machine Learning in Python


Can we combine multiple K-Means Models as a single model? I have a NLP problem statement where I use a Word2Vec embedding pre-trained model to convert key text to vectors and then on a set of terms run k-means clustering to get a final model for certain k
For various sets of terms, I would develop a different model, which I would store to disk.
My question is, in case there is a new term, which I wish to classify as to which cluster should it point to from all the models can I follow the following approach?

Load all models to memory and get their cluster centers.
get the vector of the new term based on the same pre-trained model as before.
get distance from each cluster center to the new vector and whichever is nearest can be considered as the winning cluster

I would like to know what could be the possible drawbacks of such an approach.
My assumption is that since the vector space is same as defined by the pre-trained model, therefore the cluster centers would be in the same space.

I agree with your assumption, the vector space is the same so I don't see any major problem with this approach.
Still this approach might cause some more subtle bias, depending on the differences between the models (sets of terms, number of clusters). I could imagine the following problems happening:

if there is a big difference in number of clusters between models, a model which has more clusters is more likely to contain the closest match, simply because it has more centroids. This might favour the most precise clusters (this might actually be a good thing, depends).
if there are many models sometimes there might be many close centroids across the models, and this would probably make the selection of the closest among them almost random: the exact position of a centroid is significant with respect to other centroids within the same model, not necessarily with respect to other centroids outside the model.


Intuition behind the entropy definition The definition of information entropy is defined below:

This looks fine but I got no intuition why it is defined this way. Could any one share their ideas on this? Thanks!

Pls. refer Pattern recognition and Machine Learning for details - section 1.6: Information Theory.

Left graph is Information(events) vs Probability. And the right graph is Expected Information/(Uncertainity) vs Probability.

If we look carefully at the left graph; the variation at either extremes of the curve dies down, representing less change. Translated in current context, it means not much Information flows-in at extremes.

Staying with the left curve, it's somewhere in the middle (45-degree, from vertex) where there's relatively more "meat" in the sense that more change happens/ translated, more information flows-in.

Same thing is shown by the right curve: at two extremes, Change/ Expected Information is low at extremes, and highest at centre.

A crude Graph using Excel:


Using keras tokenizer on Spark DataFrame. Column is not iterable I am having trouble using the Tokenizer

The code is

from keras.preprocessing.text import Tokenizer
from keras.preprocessing.sequence import pad_sequences

tokenizer = Tokenizer()
tokenizer.fit_on_texts(df.descricao_despesa)

X = tokenizer.texts_to_sequences(df.descricao_despesa)

maxlen = 100

X= pad_sequences(df.descricao_despesa, padding='post', maxlen=maxlen)

I tried to cast column descricao_despesa to String but still doest work.

If i try to cast to StringType i receive

from pyspark.sql.functions import col
df = df.withColumn("descricao_despesa", col("descricao_despesa").cast('StringType'))

ParseException: u'\nDataType stringtype() is not supported.(line 1, pos 0)\n\n== SQL ==\nStringType\n^^^\n

I am using spark 2.0 and Python 2.7

I'm not sure that one can use Keras on Spark to process data in parallel (using multiple workers).

But if your data can fit in RAM on a single node, that you can easily create a Pandas DataFrame from a Spark DataFrame and pass that DF to Keras. You can also convert result back to Spark DF if you need.

Spark DF --> Pandas DF:

pdf = df.toPandas() # df - is a Spark DataFrame

Pandas DF --> Spark DF:

sdf = spark.createDataFrame(pdf)


Format of data in SQL for machine learning I am a beginner at Machine Learning and am starting out on a ML project. I have a large chunk of the source material and have started extracting the data from it to be stored in SQL (initial test with SQLite, but that is going to be insufficient for production).

The question that I am now facing that I can't find any kind of answer to is to what extend to preprocess the data that I store for best performance?

For example, ML methods are usually bad at handling categories and need them to be more like 0/1 values in a lot of columns showing the category rather than the category as a string in a single column. As I have many different such cases for a single row it would mean a lot of extra columns in SQL to achieve this preprocessing. I will also be using different ML methods like regression and classification on the data so exact preprocessing requirementsmight be hard to predict.

The data consists of %, times, categories, string labels and more. I will have to do some additional processing after retrieval from database regardless of how much preprocessing I am doing beforehand as some preprocessing is just not feasible (or even possible) to store completely prepared in SQL. % is of course easy, but when to do what for many of the other forms of data still eludes me.

Setup is single machine for daily data retrieval (small updates), data extraction and storage, modelling (unknown update interval) and predictions (multiple daily). Since I will be using many different models and aggregate prediction results I am very keen to have high performance without having to go nuts about it. I work in Python but can shift c/Java-like language if significant gains can be shown. Current estimate is for around 10 million data points, but that could easily be 10 times that number when broken down into categories.

As I am new at this I think it fair that you ask for clarifications if I have left out anything of relevance in determining what the best format for data in the SQL should be. I realise that performance is not as clear as desired, but I don't know what the bottleneck is going to be. Small investments like some additional RAM is not really a bottleneck compared to the need for an additional machine to run some portion of the process.

Disclaimer: I'm not at all expert about deploying big ML systems in production. This answer is only based on my experience with many different ML problems and datasets

My humble advice would be not to try to design the format of the data before having a quite precise idea of what kind of ML process is going to be applied. There is no "one size fits all" in ML and there's a real risk that by starting with the format of the data, you will end up with something which turns out to be completely inappropriate for the task.

Start with local experiments instead:

Use a small subset of the data at first.
Design some simple problems of the same kind as the real ones you plan to do eventually.
Vary as many aspects as possible of the experiments: preprocessing, learning algorithms, parameters, size of the data, etc. Move progressively to more realistic tasks and amount of data.
Evaluate the advantages/disadvantage of the different methods/setups, then select a range of target setups
Finally design everything including the data format based on these target setups.

Following this logic during the experimental stage you can just export your data in any format convenient for whatever framework you're testing. it's only at the end of the experimental stage that you design the production system, e.g. SQL server etc.


Detect blur image using ssdmobilenet and tensorflowlite I have clear images of cards vs blurry images of card. My task is to capture photo when the image is not blurry, as you can see from the description I need this code to run in real time on android device.
I have done some background reserarch on this topic 'Identify blurry image'. And found out few interesting solutions.

Apply opencv transforms such as laplace or sobel filter. The blurry image will have less edges. And then using techniques such as SVM to find out which image has less edges
Use other opencv transforms similar to sobel to get edges of images and then find image with less edges.

Although these transforms produce good output. They are badly slow. I need something which has speed similar to tflite object detection using android. Taking this logic in my mind my obvious step was to annotate images 1500(blurred cards) vs 1500(non blurry cards) and retrain ssdmobilenet model using tensorflow object detection api.
My dataset
(clear)

(blur)

However when I exported the trained model to android I completely messy output, it appears as though the model has not learned anything from the data. My question is Is this problem solvable using object detection api as I mentioned above ? if yes where am I going wrong ? If no what are the fastest alternatives to detect blur in real time

One idea is to use a shallow (a handful of layers) CNN. Deep CNN's are good at detecting objects in images. Shallow networks focus on lower-level features, such as edges and textures and colors. So, my suggestion is to train a simpler network on a larger training set by training it on image patches rather than on full cards. If the majority of your patches are classifed as blurry, then you classify your whole card as blurry.


Extracting the code from Keras Suppose I write a program in Keras for MNIST data set. I used TensorFlow as my backend in Keras. Is it possible in any way that I can extract the backend code used for TensorFlow that has been used by Keras. if yes, please provide an example.

If you are looking for a way to compile code written for the Keras API to code only using the Tensorflow API, there is no ready-to-use solution for that. And I cannot see any good reason why someone would want to do that in the first place...

You can simply look at the Keras backend source code for Tensorflow and rewrite your entire implementation. But once again I don't see any valid reason that would make this worth it.


Stuck implementing k means for big and small dogs - dodgy results my algortithm isnt working. The code seems to make sense and everything but im just not convinced with the results. I fell like the centroids should be amongst the data more, sort of central, but they arent doing that no matter how many iterations i set the algorithm too. Could someone run my code and give me some pointers? Ive labeled the code so you can understand it.
Some main points:

k = 2; 2 centroids
centroid1 = Dark red, so datapoints in this cluster are light red
centroid 2 = Dark green, so datapoints in this cluster are light green
dogs are either 'big' or 'small' depending on their height and weight
The euclidean distance is used to decide on which cluster to belong to
Raw/main data stored in dictionary labeled 'x'; they are moved about into the dictionaries 'cluster1' and 'cluster2' depending on their distance to the centroids

code:

from matplotlib import pyplot as plt
import random, math

plt.title("Big Dogs and Small Dogs")
plt.ylabel("Height (cm)")
plt.xlabel("Weight (kg)")

def euclideandis(x, y, a, b):   # Formula for euclidean distance
    return math.sqrt((x-a)**2 + (x-a)**2)

x = {4.4:31,
     3.2:19,
     4.6:32,        # Data- key = weight, value = height
     4:25,
     4.1:29,
     2.90:17,
     2.9:11}

centroid1 = [[random.uniform(min(x.keys()), max(x.keys())), random.uniform(max(x.values()), min(x.values()))]]   # create random centroids that are between the datapoints
centroid2 = [[random.uniform(min(x.keys()), max(x.keys())), random.uniform(max(x.values()), min(x.values()))]]

for j in xrange(1):   # keep updating colours, position of centroids
    cluster1 = {}       # Everything in cluster 1 is closest to the dark red spot, therefore gets included in this dict, and gets scattered in light red
    cluster2 = {}       # Everything in cluster 2 is closest to the dark green spot, therefore gets included in this dict, and gets scattered in light green
    for key in x:
        temp1 = 0       # works out euclidean distance for all data points in x, then compares them
        temp2 = 0
        temp1 = euclideandis(key, x[key], centroid1[0][0], centroid1[0][1]) # Dis betw data point, red centroid
        temp2 = euclideandis(key, x[key], centroid2[0][0], centroid2[0][1]) # Dis betw data point, green centroid
        if temp1 < temp2:
            cluster1[key] = x[key]      # if the euclidean distance between datapoint and red spot,
                                        # smaller than the euclidean distance between datapoint and green spot,
                                        # add the point to the red cluster, else add to green cluster
        else:
            cluster2[key] = x[key]

    centroid1 = [[0, 0]]        # Centroids reset as they will be changed
    centroid2 = [[0, 0]]

    iterable = 0
    for key in cluster1:        # works out mean coordinates of each cluster and changes the centroids coordinates to this
        iterable = iterable + key
    centroid1[0][0] = iterable/len(cluster1)
    iterable = 0
    for key in cluster1:
        iterable = iterable + cluster1[key]
    centroid1[0][1] = iterable/len(cluster1)
    iterable = 0
    for key in cluster2:
        iterable = iterable + key
    centroid2[0][0] = iterable/len(cluster2)
    iterable = 0
    for key in cluster2:
        iterable = iterable + cluster2[key]
    centroid2[0][1] = iterable/len(cluster2)

plt.scatter(cluster1.keys(), cluster1.values(), color = "red")      # scatters everythning
plt.scatter(cluster2.keys(), cluster2.values(), color = "lime")
plt.scatter(centroid1[0][0], centroid1[0][1], color = "maroon")
plt.scatter(centroid2[0][0], centroid1[0][1], color = "green")
plt.show()

To sum up, there isn't a clear error with my program, only I'm not really convinced about the results

(above) why after 100k iterations arent the centroids in the middle of the datapoints? The red one is but the green one isnt

The centroid is likely correct, you have a display error

The line

plt.scatter(centroid2[0][0], centroid1[0][1], color = "green")

should be

plt.scatter(centroid2[0][0], centroid2[0][1], color = "green")

That's just one of those things that happens when you implement an algorithm from scratch in order to learn it . . . I bet you've spent hours looking at the top part of your script :-)


Why are optimization algorithms slower at critical points? I just found the animation below from Alec Radford's presentation:

As visible, all algorithms are considerably slowed down at saddle point (where derivative is 0) and quicken up once they get out of it. Regular SGD itself is simply stuck at the saddle point.

Why is this happening? Isn't the "movement speed" constant value that is dependent on the learning rate?

For example, weight for each point on regular SGD algorithm would be:

$$w_{t+1}=w_t-v*\frac{\partial L}{\partial w}$$

where $v$ is a learning rate and $L$ is a loss function.

In short, why are all optimization algorithms slowed down by the saddle point even though step size is constant value? Shouldn't a movement speed be constantly same?

In that simulation, the movement speed is a proxy for step size. The step size is a function of learning rate (v) and the approximate gradient of the function at that point ($\frac{\partial L}{\partial w}$). The learning rate can be constant. However, the approximate gradient is not constant. The approximate gradient is typically smaller closer to a critical point (i.e., the valley starts to gradually level out). Thus, the computed update value is smaller and the movement speed slows down.


About 1M rows of data. Should I restrict myself to few columns as well? I'm trying to build a predictive model from about 1 million rows of data. My goal is to predict a certain numerical value.

I have the intuition that I should use very few numerical binary columns so I don't get data points that are too separated, a.k.a., the curse of dimensionality. Is this true? Besides, is it the same for numeric columns?

Did you try it?

In general, I believe the curse of dimensionality is way overrated. As a rule of thumb, the curse of dimensionality refers to having more columns than rows. I doubt that you have 1M columns...

A robust algorithm, such as Logistic Regression, can often deal with sparse data. I sometimes get the feeling that this algorithm doesn't get the credit it deserves. Especially in production (due to high bias and low variance), I prefer it whenever possible. Nothing worse than an algorithm that fails in production...


Why 100% accuracy on test data is not good? I was asked this question in an interview and wasn’t able to give a satisfactory answer not only upto the interviewers' expectations but of my own as well. The question was as above only, he later gave an example as if why if my model predicted the prices of oil of tomorrow 100% accurately why that might be bad or why having a model 100% accurate bad or is it? Is there something in the question or is there a deeper explanation?

I see two ways to go:
1 - There is an error
2 - There is no error.

1 - Look for the error

You probably have commited Data Leakage.  You have added the target in one of the features and the model found out.
The validation is not right, you have a time series and you have done random validation.
Your test has only a few instances or it is unique.
The test is repeated from the train.

2 - There is no error

If the prediction is right and you have 100% accuracy, then no need to do Machine Learning. Open the model find where is taking the decision and don't do machine learning, do classical modeling.

For example if your model is a decision tree, just plot it or print it and get the decision and put them yourself.

This sometimes happens when modeling a previous developed algorithm. The new ML model is able to learn what was going on before.


Clause type classification We would like identify similar text (clauses) on a contract based on a trained corpus.

For instance:

Contract - small sample

NOW, THEREFORE, the parties hereto mutually agree as follows:

1. Scope of Services. The CONTRACTOR shall, in a proper and satisfactory manner as determined by OHA, provide all the goods and services set forth in Attachment – S1,
which is hereby made a part of this Contract.

2. Time of Performance. The performance required of the CONTRACTOR under this Contract shall be completed in accordance with the Time Schedule set forth in Attachment – S2, which is hereby made a part of this Contract.

3. Compensation. The CONTRACTOR shall be compensated according to the Compensation provision set forth in Attachment – S3, which is hereby made a part of this Contract.

4. Standards of Conduct Declaration. The Standards of Conduct Declaration of the CONTRACTOR is attached and is made a part of this Contract.

Trained clauses

We already have classified a few clauses from previous contracts. For instance:

#time_of_performance =
[
    "Under this contract the performance required to be completed in accordance with the a predefined schedule."
,
    "The completion of each phase of the project will be used to define the performance of this contract"
,
    etc.
]

Where #time_of_performance is the classification for these clauses.

Expected result

Given the contract and the trained set, we would like to get parts/ranges of the texts and its classifications:

#time_of_performance = ["2. Time of Performance. The performance required of the CONTRACTOR under this Contract shall be completed in accordance with the Time Schedule set forth in Attachment – S2, which is hereby made a part of this Contract."]

Is there a known approach for this problem or a recommended processing pipeline?

Not sure where you are in terms of prior knowledge, but this blog post might get you started.


Is RL applied to animal dispersion a valid approach? I have an agent which has a medium-sized, discrete set of actions $A$: $10<|A|<100$. The actions can be taken over an infinite horizon of 1 second per timestep $t$. The world is essentially pictures from a static camera and the states $S$ are the amount of animals detected on the picture (assuming perfect detection for simplicity). We can safely say that $max(S) < 200$. Each action $a$ is meant to lower the amount of detections because of dispersion or fear from the animals. The reward is the difference of detections from $s_{-1}$ to $s$.

Admittedly I am only a beginner in RL, so I haven't seen much more than MDP's. I believe that this isn't a Markov problem since the states aren't independent (past actions having an impact on the outcome of the current action). That being said, I'm wondering if there's a specific RL algorithm for this setup or if RL even is the right way to go ?

Admittedly I am only a beginner in RL, so I haven't seen much more than MDP's. I believe that this isn't a Markov problem since the states aren't independent (past actions having an impact on the outcome of the current action).

If there are enough hidden variables, then this could be a real problem for you. A policy maps states to actions - and searching for an optimal policy will always map the same action to the same state. If this is some kind of automated scarecrow system, then the animals are likely to become habituated to the "best" action.

Two ways around that:

If the habituation is slow enough, you might get away with treating the environment as having the simple state that you suggest and have an agent which constantly learns and adapts to changes in what will trigger animals to leave. This would be an environment with non-stationary dynamics (the same action in the same state will, over time, drift in terms of its expected reward and next state). For a RL agent, this could just be a matter of sticking with a relatively high exploration rate and learning rate.
If the habituation is fast or adaptive, then you have to include some memory of recent actions used in the state representation. This will make the state space much larger, but is unavoidable. You could try to keep that memory inside the agent - using methods that work with Partially Observable MDPs (POMDPs) - it doesn't really make the problem any easier, but you may prefer that representation.

  The reward is the difference of detections from $s_{-1}$ to $s$.

You need to re-think the reward system if habituation is fast/adaptive, as it will fail to differentiate between good and bad policies. All policies will end up with some distribution of detections, and all policies will have a mean total reward of 0 as a result. Your reward scheme will average zero in the long term for random behaviour, and also will average zero if the number of creatures remains at 200 the whole time, or is zero the whole time. Presumably having zero creatures would be ideal, whilst having 200 would be terrible - you need to be able to differentiate between those two scenarios.

You want to minimise the number of creatures in the area consistently, so a simple reward scheme is just negative of the number of visible animals, per time step. Maybe scale this - e.g. divide by 100, or by some assumed background rate where you would get -1 reward per time step if the number of animals is the same on average as if the agent took no action. You could measure this average using the camera over a few days when the agent is not present or simply not active. It doesn't need to be super accurate, teh scaling is just for convenience - you could think of an agent that gets a mean reward of -0.2 per time step is five times better than having no agent at all, whilst an agent that scores -1.5 per time step might be attracting creatures for all you know!

  That being said, I'm wondering if there's a specific RL algorithm for this setup or if RL even is the right way to go ?

The problem does seem like a good match to RL. There are other ways of searching for good policy functions, such as genetic algorithms, that might also apply. However, you will still need to equip the agent with either continuous learning so it can re-find the best action as it changes, or with a memory of recent actions, depending on the speed of habituation. You may even need both as smart animals like birds or mammals can adapt in long and short term in different ways.


Working with HPC clusters In my university, we have an HPC computing cluster. I use the cluster to train classifiers and so on. So, usually, to send a job to the cluster, (e.g. python scikit-learn script), I need to write a Bash script that contains (among others) a command like qsub script.py.

However, I find this process very very frustrating. Usually what happens is that I write the python script on my laptop and then I login to the server and update the SVN repository, so I get the same python script there. Then I write that Bash script or edit it, so I can run the bash script.

As you see this is really frustrating since, for every little update for the python script, I need to do many steps to have it executed at the computing cluster. Of course the task gets even more complicated when I have to put the data on the server and use the datasets' path on the server.

I'm sure many people here are using computing clusters for their data science tasks. I just want to know how you guys manage sending the jobs to the clusters?

Ask your grid administrator to add your local machine as a "submit host", and install SGE (which we assume you are using, you don't actually say) so then you can qsub from your machine.

OR....

Use emacs, then you can edit on your HPC via emacs's "tramp" ssh-connection facilities, and keep a shell open in another emacs window. You don't say what editor/operating system you like to use. You can even configure emacs to save a file in two places, so you could save to your local machine for running tests and to the HPC file system simultaneously for big jobs.


Using TF-IDF for feature extraction in Sentiment Analysis I am working on sentiment analysis for twitter data, for which I have used Vader to get an approximation of sentiment for a tweet. Along with, I have used TF-IDF for feature extraction. These feature words I am using to train and test a Random Forest model. In my dataset, there are around 3K plus tweets from which I extracted around 570 unique feature words using TF-IDF. And all these features I have used for training the Random Forest Model.

My query is in regards to the usage of this trained model in the real world. What if the new tweets which this model has never seen do not have feature words that I have used for training, will the model fail to make correct predictions for them (in my case there are only 3 possible predictions viz. positive, negative and neutral) correctly for them? If yes, then how should I handle this scenario?

Please let me know in case I am missing something or doing something wrong here.

TF-IDF is a vectorization technique used to convert documents (a single tweet in your case is a document) to vectors. After you train the TF-IDF model, the only words/vocabulary it has learnt, would be from the set of documents (aka corpus, the entire set of 3k tweets).

Since you mentioned that there were 570 unique feature words after TF-IDF, that would be the vocabulary your model has learnt. If you give this model a document with words that are present in its vocabulary, it will successfully vectorize it. However, if one or more words in your new document are not present in the model's vocabulary, those words won't be included in the vectorization at all. The words in a new sentence will be given weights only if the model had encountered those words in its training. In other words, every sentence will be vectorized w.r.t the model's vocabulary.

Ex -

Model's vocabulary - a, big, hat, have, I, mat

Input - "I have a big mat"

Vector - [ sequence of tfidf weights in the order of the model's vocabulary. If word in vocab is not present in the sentence, weight assigned is 0 ]

Input - "I have a dog"

Vector - [ sequence of tfidf weights calculated for all the words in the vocabulary, in the same order as the vocabulary ]

Since "dog" wasn't a part of the vocabulary, it's not included in the vectorization.

  What if the new tweets which this model has never seen do not have
  feature words that I have used for training, will the model fail to
  make correct predictions for them (in my case there are only 3
  possible predictions viz. positive, negative and neutral) correctly
  for them?

I can't say much about the final prediction, but your feature vectors from TF-IDF can be way off, if you're expecting it to vectorize documents with a large number of words that the model hasn't trained on. Improper vectorization can affect your prediction accuracy.

  If yes, then how should I handle this scenario

My suggestion would be to consider the following -

Pre-process the train data thoroughly. Removing punctuation, substituting abbreviations by their full forms and other steps can help the TF-IDF model train well.
If gathering more data is an option, try training the model on a large set of
tweets that fall in the context of your prediction. The more the vocabulary, the better equipped the model is, to vectorize new unseen documents.
Try using a pre-trained model like BERT.


How much data warrants building a pipeline? If I'm doing simple aggregation dashboards, what's the minimum amount of data which justifies building a full blown data processing pipeline? Is it reasonable to build a complicated pipeline with Mysql -> Hadoop -> Redshift and then analytics and visualizations on top of redshift if my database only has about 10million transactions per month. Would it be better to just run something like ReDash visualizations on top of my unaggregated data in MySql?

You should only complicate it as much as necessary to meet your requirements.

What kind of aggregations are you doing? Most DBs can handle basic counts pretty easily. 10 million in a month sounds like something a large MySQL instance could handle.
How quickly does everything have to update? If the users only expect the summarized data to be updated monthly, you have a lot of time. If you have to update it more often, it might make sense to have MySQL collecting it but Redshift presenting it. Redshift's analytical focus and columnar storage means you may not need pre-aggregated data until your volume gets much higher than what would require aggregation in MySQL. I was using a 4 node Redshift cluster with the smallest instance types and it could do a count across >300 million rows in 17 seconds.

Can you not build aggregate tables in MySQL? What is making you want to put Hadoop in between MySQL and Redshift?


Which metric should I use for classifying an imbalace data with fewer labels for the negative class? From reading, I understand that when we have fewer positive class labels, it is better to use precision or recall as the evaluation metric. Which metric should I use when we have fewer negative samples?
I'm looking for an approach other than switching the labels.
Problem setting: I'm developing parametrized fragility functions for predicting damage to a structure (for example trees). An example of fragiltiy function is here The fragility function will estimate the probability of exceeding a damage state given some parameters (say wind load). The damage state can be expressed in terms of damage ratio (0-1 with 1 being fully damaged). Now, we are interested in estimating the probability of exceeding a damage ratio given features. To elaborate, the probability of any damage would be P(Damage_ratio>0.0|features). Logistic regression can be used to learn this curve from data after categorizing 0-1 continuous damage ratio to damaged (- class)/no damage (+ class) for a particular threshold. Now, as we move from the threshold from 0 to 1, the dataset transforms from imbalanced data dominated by damaged cases to a balanced state and then finally to another imbalanced data dominated by non-damage case.
Now, when learning the model, 'AUR-ROC' performs really well when the data is balanced. Precision performs well when the data is imbalanced with more no-damage case (P(Damage_ratio>0.1|features)). These metrics don't do well for the case with few negative case (P(Damage_ratio>0.9|features)). I tried switching the label with very limited success. Are there any other 'metrics' that perform well in an imbalanced data setting?

The names of the classes don't matter, you might as well call them class A and class B. In binary classification the typical choice is to evaluate using precision, recall and F1-score. There are other options, but that depends on the task.
Assuming you choose F1-score, the choice of which class you select as the "positive" class for evaluation also depends on the task. Usually it's recommended to use the minority class because it's the most challenging one for the classifier.
The only problem here is the possible confusion of calling a class "negative" and calculating F1-score using it as the "positive" class, but that's just a naming issue. You could easily add this point to the explanations, or avoid any confusion by calling your classes A and B for instance.


# of iterations in Restricted Boltzmann Machine (RBM) I have a training set, I provide it (consider a data from training set) to the visible layer. Then the normal process is followed, i.e. Positive Phase-> Negative Phase-> Reconstruction of weights, bias units take place. Does it end here or should I have to iterate with the present visible unit value again (the process follows for few fixed iterations) ? Because it is not properly stated anywhere whether with each data of training set, it has to iterated again and again or with each data of training set.

When you are unsure about how something (e.g. RBMs) should really be implemented, it is often useful to look at the code of others. G. Hinton himself has published a MATLAB-script (here) which demonstrates the training of an RBM. There, you can see that for each mini-batch, he does the positive phase, then the negative phase, and finally updates the weights - and that's it. So he doesn't iterate between the visible and hidden states.

However, this is not the full truth: for the weight updates we need to know the probability $p(v,h)$. This is very complicated to calculate, as it would contain a sum over all possible states of the RBM. There is a "mathematical trick" called Gibbs sampling: it allows us to iterate back and forth between visible and hidden units to calculate this probability $p(v,h)$.

But: for the result to be correct, we have to iterate forever, which is not really practical. So what Hinton proposed is to iterate for only 1 step instead (this is $CD_1$), so he only goes back-and-forth once.

But, you can also iterate any number of times $k$, which is denoted by $CD_k$.
While for Hinton's $CD_1$, you would do

visible --> hidden --> visible

For a $CD_3$, you would iterate from visible to hidden and back, three times:

visible --> hidden --> visible --> hidden --> visible --> hidden --> visible

And just to be clear: you run this iteration for every data point in the training set. Actually, you usually make so-called mini-batches of maybe 10 data points, which you run at the same time. Then you calculate the average weight update from this batch. But then, you do this iteration again, and again, and again, until you have finished your training.


Should I pad zero or mean value in a convolution neural network? What about reflective padding? In convolution layers, sometimes you need to pad some (usually 1 or 2 pixel) 0s at the edges of the original image before applying your convolution kernel (filter).

However, 0 has different meanings.

In the case of 0 - 255 gray scale images, 0 means black.

In the case of normalized images ( minus mean then divided by std), 0 is the mean value.

Should I always pad 0, or should I sometimes pad mean value ?

If in some cases I should pad mean value, what are those cases?

Edit:

And when should I use reflective padding?

Further more, is reflective padding always better than zero padding?

Mean, zero, duplication (i.e., constant extrapolation), reflection, and symmetric padding are all valid and widely used methods of padding for conv layers. To my knowledge, there is no systemic study that says one is better than the other in all cases. In other words, one is not always better than any other. (Ideally, one would vary the padding type as a hyper-parameter I suppose.) I think intuitively that reflection and symmetric padding alters the local image structure and global statistics the least. (Clearly, the zeros from zero padding on the boundary are not truly part of the image, and the network has to learn this). However, these have different problems with "realism": for instance, reflection padding will "duplicate" a chair leg on the boundary if the padding is large enough, which may present its own problems depending on the task.

Overall, I think it doesn't matter too much, but it depends on the task and setup. For instance, in generative modelling and image-to-image translation, reflection padding avoid some artifacts on the boundary, as noted in the CycleGAN paper, for example.

An interesting recent paper is Partial Convolution based Padding  by Liu et al, where they sidestep this issue by essentially having the convolution completely ignore the boundary. This is a partial convolution in the sense that it ignores the part of the filter that reaches outside of the image. The overall improvements seem to be relatively marginal, except that it does seem to help significantly on the boundary, as one would expect. See also this earlier work doing a similar thing in the segmentation context and this work which applies partial convolutions to inpainting. This approach to avoiding padding is seemingly well-principled.

TL;DR: test different paddings and see which is best for your architecture/task (also check the literature). If that's too much work, I default to reflect padding. If you want to get fancy or worry about boundary effects, use partial convolutions.


Are ontologies and the Semantic Web dead? Is the Semantic Web dead? Are ontologies dead?

I am developing a work plan for my thesis about "A knowledge base through a set ontology for interest groups around wetlands". I have been researching and developing ontologies for it but I am still unclear about many things. What is the modeling language for ontologies?

Which methodology for ontologies is better? OTK or METHONTOLOGY?

Is there any program that does  as does

Cratilo is a software for analyzing of textual corpora and for extraction of specific terms of the domain of study (it is developed by professors Jorge Antonio Mejia, Francisco Javier Alvarez and John Albeiro Sánchez, Institute of Philosophy the University of Antioquia). It enables lexical analysis of texts, identifying the words that appear their frequency and location in the text. Through a process of recognition, Cratylus identifies all the words in the text and builds a database becomes the draft analysis of the work. Are there other similar tools?

Can the terms found by Cratilo be used to create a knowledge base?

What are the existing open semantic frameworks that can be used for such things?

Is there software that automatically creates RDF, OWL, and XML? How does Tails work? Jena? Sesame?

(While I think that your question is usually not regarded as a good question for this site, I think that it should be preserved because almost everyone new to the field is similarly confused at first and it's hard to find straight, sober, and balanced information about it.)

Is the Semantic Web dead? Are ontologies dead?

Many people moved away from the Semantic Web. On the other hand, there are still many using it. There's always been a lot of confusion about where its value lies. There are use cases that genuinely benefit from the semantic web technologies but they are quite few and far between.

Linked Open Data

That's partially why the subfield of Linked Open Data was created. You can see Linked Data as a more pragmatic stripped down version of the Semantic Web (or as a necessary enabler of the grander semantic web vision).

Linked Data, while more pragmatic, still uses ontologies. It is just not so uptight about using OWL and designing your ontologies first and in a very formal way.

A knowledge base such as Freebase (now being replaced by Wikidata) doesn't even build on Semantic Web technologies (it's always been available also in the form of RDF dumps though). On the other hand, even Freebase builds on concepts similar to those in the Semantic Web and it does have a form of ontologies.

Software to generate ontologies

There is no software that would create high-quality ontologies automatically for you. At least not in the sense of OWL/first-order logic ontologies. On the other hand, many kinds of software, including Cratilo mentioned by you, can help you to build a lower step in the semantic spectrum such as a glossary or maybe even a folksonomy which can later be transformed into more of a taxonomy or ontology by other algorithms. There might be even software to create fullblown OWL ontologies but I think that's still rather an active area of research.

Ontologies and methodologies

As always, it depends on what exactly your needs and goals are. In fields such as biomedicine and life sciences in general, they create complex formal ontologies because they have data with highly varied structure and ontologies help them keep things organized and usable. In contrast, an eshop might be better off sticking with normal relational database modelling and adding only global identifiers (URIs) in the spirit of Linked Data if they want to build a proper knowledge graph later.

Even if you need ontologies you might want to skip the methodologies first. Especially if you are really new to the field and have little idea what your needs are.

Jena, Sesame, ...

Jena, Sesame, Virtuoso, etc. are triplestores - they are used to store and query RDF. Most ontologies can be represented in RDF. Even those written in OWL. RDF has the semantic part, RDF/S, which allows you to formulate some basic ontologies. That might well be enough for start. Even RDF/S can get confusing and convoluted when you start to think about blank nodes, named graphs (are they fixed? are they dynamic?), etc.

While I think that your question is usually not regarded as a good question for this site, I think that it should be preserved because almost everyone new to the field is similarly confused at first and it's hard to find straight, sober, and balanced information about it.


How to add confidence to model's prediction? I am a newbie in ML working on a time series prediction project. The objective is to predict the future outcome of a time series (int valued, with different upper bound, think of it as different sized parking lot availability prediction) based on its historical value.

I'm currently using a regression approach using slide windows algorithm. I tried different ML models and they seem to be working okay(better than my baseline at least).

Now I'm trying to add confidence to my prediction, something like "I have 95% confidence that the outcome would be 2". I'm thinking about using the prediction mean squared error as a metric.

The problem is,

Is it feasible to assume the prediction error follows gaussian distribution and add confidence based on that?
What distribution should I use for highly discrete state space? For example, when there is only 4 possible states {0,1,2,3}, and I predicted 2.5 with mse 1, how can I distribute the possibility over those states?

Any advice on the general model architecture and confidence set up will be appreciated!

It is common to say the error term follows a standard Guassian distribution. If you assume that to be true, then your squared errors follow a Chi-squared distribution:

  In probability theory and statistics, the chi-squared distribution (also chi-square or χ2-distribution) with k degrees of freedom is the distribution of a sum of the squares of k independent standard normal random variables.

Have a look here for some ideas about how to implement a quasi-confidence metric, based on your (mean squared) errors. It assumes the errors follow a chi-aquared distribution and then uses the normalised RMSE to define a set of confidence boundaries for a given confidence level,  $\alpha$, as follows:

$$
\left[\sqrt{\frac{n}{\chi_{1-\frac{\alpha}{2},n}^{2}}}RMSE,\sqrt{\frac{n}{\chi_{\frac{\alpha}{2},n}^{2}}}RMSE\right]
$$

See the link for the steps involved. Here is the coded simulation taken from that post, with some added comments (requires python 3):

from scipy import stats
import numpy as np

s = 3                           # a constant to scale the random distribution
n = 4                           # number of samples/states per prediction
alpha = 0.05                    # confidence interval

# distribution with confidence intervals ɑ = 0.05
c1, c2 = stats.chi2.ppf([alpha/2, 1-alpha/2], n)

# we will take this many samples (this pre-allocates the y-vector)
y = np.zeros(50000)

# Loop over each sample and record the result mean sample
# This would be your prediction vector - here it is random noise
for i in range(len(y)):
    y[i] = np.sqrt(np.mean((np.random.randn(n)*s)**2))

# Use the chi-squared distributed confidence intervals to see when predictions fall
# finds percentage of samples that are inside the confidence interval
conf = mean((sqrt(n/c2)*y < s) & (sqrt(n/c1)*y > s))

print("1-alpha={:2f}".format(conf))

Here is another answer on CrossValidated, which gives more information around the area.

Additionally, if you assume your predictions lie within a Gaussian distribution, you could use the variance of your predictions as the confidence (welcome to Bayesian learning!).

There are packages that will help you do this, such as BayesOptimization. There are lots of examples on that webpage. Essentially, you will be able to make predictions and automatically get robust estimates for condifence... and some cool plots to show where your model is quite sure, and where it isn't:


What is the difference between a "cell" and a "layer" within neural networks? So I understand what "layers" are.  If you have 5 layers in your model, your data basically gets transformed 5 times via 5 activation functions.  The number of "neurons" within a layer dictate how many outputs a layer creates.
So what are "cells"?  I never understood where "cells" come into play.  Are they a collection of layers?
Per Wiki: https://en.wikipedia.org/wiki/Long_short-term_memory
If the orange are layers, then I would imagine each has a bunch of neurons.  So a cell is a collection of layers and yellow stuff?  I'm having trouble understanding where this "cell" fits into an overall NN architecture.  I am used to the pictures with input layer -> hidden layer -> output layer.  So where would the "cell" occur?

In short
A cell is in essence a forward neural network consisting of neurons and so $cell_{RNN}\neq neuron_{NN}$.
Rationale
In the context of recurrent neural networks, a layer consists of cells e.g. LSTM cells. Although they seem to position similarly in terms of architecture (figure), there is a fundamental difference between an RNN_cell and a NN_neuron.

The number of RNN cells is generally equal to the sequence length (e.g. number of words in a sentence) or embedding output dimension (figure below). Conversely, the number of neurons in forward neural networks can generally be of any number.
Each RNN_cell acts much line a NN itself. Specifically in LSTMs, each cell consists of hidden state $W_{hidden}$ and cell state $W_{cell}$ matrices. These are used to store weights and perform transformations in the various gates (forget, input, output) of the LSTM cell.

In essence, the transformations that take place in each cell are matrix multiplications of $W_{hidden}$ and $W_{cell}$ much like those in a forward NN. And so, the size of $W_{hidden}$ and $W_{cell}$ cannot be but the number of neurons of that NN which essentially determine the memory of the LSTM cells and model itself.


artificial neural network for vehicular speed prediction based on volume I an trying to develop a neural network for vehicular speed prediction. my question is how can i train neural network with many input output combination of data?

I think LSTM based RNN network having series of image data with a speed label, can be useful for training such model and you can try if you get some data set. Please do share your result as well.


Decision Tree - Preprocessing for very sparse features How do we pre-process data for very sparse features for a decision tree?

From this Turi documentation for decision trees

It mentioned this:

  Why chose decision trees? Different kinds of models have different
  advantages. The decision tree model is very good at handling tabular
  data with numerical features, or categorical features with fewer than
  hundreds of categories. Unlike linear models, decision trees are able
  to capture non-linear interaction between the features and the target.

  One important note is that tree based models are not designed to work
  with very sparse features. When dealing with sparse input data (e.g.
  categorical features with large dimension), we can either pre-process
  the sparse features to generate numerical statistics, or switch to a
  linear model, which is better suited for such scenarios.

If we have a categorical column that is very sparse (many 0's), then for that category what type of numerical statistics can we transform for each row?

There are many feature engineering options for sparse categorical data. A couple of common options:

Remove features - Drop columns that are sparse
Feature reduction - One example is singular value decomposition (SVD)
Reduce number of categories - Using domain knowledge, group together related low frequency categories
Use sparse representation - only stores nonzero elements


How to get vector representations(or embeddings) of time series? Even if a time series is constructed up of numbers only, finding abstract fixed-dim vector representation would be interesting for classification/clustering purposes. As we can learn & find abstract representations/embeddings of text/images, can we do something similar on Time series? Finding such ways would result in better clustering & related tasks instead of traditional ways using some statistical measures like Pearson correlation etc. All thoughts are welcome.

Maybe the framework of Neural Processes could be interesting here? It defines a  family of functions parameterized by a neural network. The parameters could serve as your embeddings, eventually after projecting into a lower dimension. See the paper Attentive Neural Processes and the preceding papers cited therein.


Scikit-learn OneHotEncoder effect on feature selection If I need to run feature selection on my dataset isn't it problematic to use OneHotEncoder? Couldn't it then decide to remove a one of the encoding columns? How should I deal with this? Thank you.

Yes it would be possible that it happens. It means that this event happening has no importance for the target.

Imagine a categorical feature with a lot of categories(high cardinality). Maybe only one of them does not have any influence in the target so if you do feature selection this feature might have a high chance of getting dropped. So this is normal thing, just make sure that it doesnt contain any information at any point of the dataset.


Fittting histograms with combination of an unknown number of Gaussians I have many data in the form of one dimensional histogram, to give an example consider the data at  http://pastebin.com/embed_js.php?i=1mNRuEHZ

I expect that these data are obtained from a pdf composed of a sum of Gaussian distributions but with their actual number being unknown.

I want then to fit these data with a model given by a sum of Gaussians

$$
f_N(x) = \sum_{k=1}^N c_k \exp \left[ - \frac{(x-a_k)^2}{2 b_k^2} \right]
$$

where $a_k$, $b_k$, $c_k$ and $N$ are in principle parameters to be fitted.

The problem can be viewed as a problem of model selection between different $f_N$ for all the possible values of $N$.

My idea is then to make an usual fit by least squares at fixed $N$ and then compare the results for different $N$ via some statistical measure of the quality of the fit, as e.g. Akaike information criterion (AIC).
Is this an acceptable test?

For the data of the previous example (that I generated as a test from 3 gaussians) I will obtain

 N        AIC
 -------------
 1      +568.1
 2      +557.4
 3      -446.6
 4      -443.5
 5      -442.7

As you see, the AIC decreases quickly till $N=3$, where it starts to increase again very slowly (due to overfitting). So it is clear that $N=3$ is the best choice here.

I mention that in some cases I have more data histograms extracted from the same pdf, so I could do some cross validation. Although this always a good test, the availability of my data changes from sample to sample, so I'd prefer to have a criterion for the single histogram (if the quality of the test is not too much different).

You're almost done with your results.
Try to substitute the AIM for the Likelihood of the sum of Gaussian with your data, and as soon as your Likelihood value stabilizes and doesn't increase with the number of Gaussians you've found the right value for your N.

You can even try to look for the EM algorithm for the estimation of the Mixture of gaussians that fit your data.

The value of L = f(N) increases as a log function and the so called "knee" is where your optimal value of N lies on!


Looking for similar items in a large data set I have a large database of people and I want to show a small number of people who are similar to each person in the database. So if one of the people was Wolfgang Mozart I would want to show Beethoven, Haydn, Schubert etc based upon they are "similar" in that they are roughly contemporary with the same occupation (i.e. Composer). For each person, I have a short occupation which usually contains the nationality (e.g. Austrian composer, American actor, British politician) and dates of birth/death.

Is there a recognised algorithm or technique to do something like this? I don't have a training set as I haven't manually produced any examples of what I want to see, so I don't think machine learning is appropriate.

If I had to write something right now I'd probably just do a string comparison and order so the closest contemporaries (i.e. dates of death) come first.

But is there a better way?

Consider the R package poLCA. Given a set of people with attributes that have categorical values, it can cluster the set into a fixed number of groups. As Anony-Mousse says above, it's not quite the same problem. Similarity would be to find people that are close to the one in the query. Clustering is close to answering that question for everybody.

The name of the package gives some deeper insight into those differences. LCA is for latent class analysis. The idea is that there is some characteristic of the people in the dataset that can be used to group them into clusters, but it's not explicit in the variables you have. It's latent.


After performing data Augmentation on tf.data.Dataset, should i MERGE it with original tf.data.Dataset? Kind of a silly question, but I read that data Augmentation can be used in order to solve problem of small datasets. In my case, I've got a dataset with 5 different classes and around 2k examples per class. After performing various image transformations, do I merge the augmented dataset with original dataset into a new, bigger (around 20k image) dataset, or did I misunderstood this procedure?

Yes, use all the data you can.

The goal behind data augmentation is to indeed create more data points from a smaller set of instances. Therefore, you definitely want to keep your original data in there, since especially, this is the only "pure" data you have.


Why is this series of house prices being thrown into one bin with plt.hist? I have a series of houseprices that looks like this:

0    410000
1    610000
2    579950
3    559950
4    404950
Name: price, dtype: int64

And using plt.hist(df.price) I get the below histogram, which is not useful. What is causing this? Am I missing something here? I want a range of prices to be binned together and their frequencies to be plotted.

Can't reproduce:

>>> housing = pd.DataFrame([410000, 610000, 579950, 559950, 404950],
                           columns=['price'], dtype=int)
>>> housing.price
0    410000
1    610000
2    579950
3    559950
4    404950
Name: price, dtype: int64

>>> plt.hist(housing.price)

Resolution

You have an outlier at 1e8 while most observations fall into 1e7 range.


How to model manufacturing shift data with irregular production times? Problem Setting:

Let's say there are three shifts a day in a manufacturing plant. The plant suffers from irregular power supply thereby affecting in how it is functioning in time and the shift's production. Therefore, you have the manufacturing process going on for a while and then pauses due to power outage(or lunch break) to be resumed when the power is back on. This on/off pattern of power is pretty much inconsistent and unpredictable.

Objective:

By mining data of the past history, I want to come up with a model that given a point in a day of a shift I want it to be able to forecast the production for the rest of the shift(possibly accounting for the likelihood of experiencing power outages).

I was hoping for some perspective and idea in how to go about making this problem a machine learning problem and some advise in picking techniques amenable to the problem.

From the information at hand, you could break this down into two problems -

Predicting the production for the shift, and
Finding the probability of breakdown during the shift

For (1) you could either go down the time-series route (ARIMA, Box-Jenkins, Exponential smoothening) or the regression route (provided you have good features)

For (2) you could build a logistic regression model that gives you a probability score

Possible features you could look into are

Time of day, week, month and year
Time between breakdowns
Available labor
Electricity Board data?


How to combine two different embeddings in the best way possible? I have two models which are giving two books embedding
Ml_model_a => book1_embedding [ 1, 200 ]

Ml_model_b => book2_embedding [ 1, 200 ]

I am building a third model which will take these two different embeddings to tell me which book to choose.
Now my final layer is the classification between 0,1 ( which book to choose ). How to learn these embeddings best way possible to classify better?
What I have tried yet :

If I am averaging those embeddings and then sending to one model, then embeddings are losing a lot of information, so I am using concatenation method.
But it's not classifying well, Is there any other model, a technique I could use to enhance the capability of learning to book embedding and predict which book to take?

I would say that the best thing to do here is to concatenate both embeddings and use the concatenated vector as an input for your binary classification model without using the norm -> you loose way too much information, i.e.:
final_layer = concatenate [ book_1_attention, book_2_attention]

I reckon that concatenating without norm will increase the dimension (but that's potentially a problem for which you have solutions - such as regularisation)


Define value of a centroid I want to choose a clustering algorithm for which I can define the value of centroids (and not only the number of them).

Which algorithm should I look at?

My understanding of k-means is that I can specify 'K' (the number of centroids) but not the value itself.

Instead, I have a given number of centroids that I want to "test". Some of them will be indeed the actual centroid, whilst others will have only few to none values and therefore I will drop them.
However, I don't want to create new values for the centroids.

If you want to give the means, then it's not clustering anymore.

The name-giving part of k-means is computing the mean.

Instead, you are doing nearest neighbor classification. Your predefined means are your "training" data.


How do multiple linear neurons together allow for nonlinearity in a neural network? As I understand it, the point of architecting multiple layers in a neural network is so that you can have non-linearity represented in your deep network.

For example, this answer says:  "To learn non-linear decision boundaries when classifying the output, multiple neurons are required."

When I watch online tutorials and whatnot, I see networks described as in the screenshot below. In cases like this, I see a series of linear classifiers:

We have a multiply, add, ReLu, multiply and add, all in series.

From studying math, I know that a composite function made out of linear functions is itself linear.

So how do you coax non-linearity out of multiple linear functions?

The phase

"To learn non-linear decision boundaries when classifying the output, multiple neurons are required."

is NOT correct. More precisely, it should be:

"To learn non-linear decision boundaries when classifying the output, we need a non-linear activation function."

To understand why, imagine you have a network with many layers and nodes (multiple neurons in your question). If you don't have a non-linear activation function such as ReLu or sigmoid, your network is just a linear combination of bias and weights. Your network won't be useful for classifying non-linear decision boundary. But if your inputs can be linearly separable, you don't need neutral network...

That's why all neutral networks almost always have a non-linear activation function. ReLu is the most popular, but there are other possibilites. When you pipe up a dozen of non-linear outputs like in neutral network, your network will be able to classify a non-linear decision boundary. The more your have, the better it can perform (but also easier for overfitting).


Advise on making predictions given collection of dimensions and corresponding probabilities I am a CS graduate but am very new to data science. I could use some expert advise/insight on a problem I am trying to solve. I've been through the titanic tutorial on gaggle.com which I think was helpful but my problem is a bit different.

I am trying to predict diabetes risk based upon Age, Sex...and other factors given this data: http://www.healthindicators.gov/Indicators/Diabetes-new-cases-per-1000_555/Profile/ClassicData

The data gives new cases people per 1,000 people for each dimension (Age, Sex...etc). What I would like to do is devise a way to predict, given a list of dimensions (Age, Sex...etc) a probability factor for a new diagnosis.

So far my strategy is to load this data into R and use some package to create a decision tree, similar to what I saw in the titanic example on kaggle.com, then feed in a dimension list. However, I am a bit overwhelmed. Any direction on what I should be studying, packages/methods/examples would be helpful.

Aggregate Data

Since you're only given aggregate data, and not individual examples, machine learning techniques like decision trees won't really help you much. Those algorithms gain a lot of traction by looking at correlations within a single example. For instance, the increase in risk from being both obese and over 40 might be much higher than the sum of the individual risks of being obese or over 40 (i.e. the effect is greater than the sum of its parts). Aggregate data loses this information.

The Bayesian Approach

On the bright side, though, using aggregate data like this is fairly straightforward, but requires some probability theory. If $D$ is whether the person has diabetes and $F_1,\ldots,F_n$ are the factors from that link you provided, and if I'm doing my math correctly, we can use the formula:
$$ \text{Prob}(D\ |\ F_1,\ldots,F_n) \propto \frac{\prod_{k=1}^n \text{Prob}(D\ |\ F_k)}{\text{Prob}(D)^{n-1}} $$
(The proof for this is an extension of the one found here). This assumes that the factors $F_1,\ldots,F_n$ are conditionally independent given $D$, though that's usually reasonable. To calculate the probabilities, compute the outputs for $D=\text{Diabetes}$ and $\neg D=\text{No diabetes}$ and divide them both by their sum so that they add to 1.

Example

Suppose we had a married, 48-year-old male. Looking at the 2010-2012 data, 0.73% of all people get diabetes ($\text{Prob}(D) = 0.73\%$), 0.77% of married people get diabetes ($\text{Prob}(D\ |\ F_1)$$= 0.77\%$), 1.02% of people age 45-54 get diabetes ($\text{Prob}(D\ |\ F_2) = 1.02\%$), and 0.70% of males get diabetes ($\text{Prob}(D\ |\ F_3) = 0.70\%$). This gives us the unnormalized probabilities:
$$ \begin{align*} P(D\ |\ F_1,F_2,F_3) &= \frac{(0.77\%)(1.02\%)(0.70\%)}{(0.73\%)^2} &= 0.0103 \\
P(\neg D\ |\ F_1,F_2,F_3) &= \frac{(99.23\%)(98.98\%)(99.30\%)}{(99.27\%)^2} &= 0.9897 \end{align*}$$
After normalizing these to add to one (which they already do in this case), we get a 1.03% chance of this person getting diabetes, and a 98.97% chance for them not getting diabetes.


Measuring quality of answers from QnA systems I am having a question answering system which is using Seq2Seq kind of architecture. Actually it is a transformer architecture. When a question is asked it gives startposition and endposition of answer along with their logits.
The answer is formed by choosing the best logits span and final probability is calculated by summing the start and end logits.

Now the problem is, I have multiple answer and many times the good answer is at 2nd or 3rd place (after sorting on the result of sum of start and end probability).  Is there any metric in search engine science using which I can rank the best answers?

Followings have been tried:

cosine similarity between question words and answers  -  This works many times but  fails when question semantic meaning is complex
TFIDF - gives good score but fails when there is synonym in answers rather than matching word.
gensim semantic similarity - fails badly.
BLUE score and new BERTF1Score also tried

Few terms I heard of but I doubt if these work, like Mean Reciprocal Rank which I think gives search quality rather than answer quality and also the correct response is required to calculate MRR (Please correct if I am wrong). Or the PageRank which is not valid in my case as the answer semantic meaning is preferred in QnA rather than the document popularity.

Kindly suggest other metrics which search engines generally use to rank the answers.

The ranking of the answers is part of the ML process, i.e. a system should be trained to rank the answers according to their relevance. Heuristic measures such as the ones mentioned in your question may offer decent approximations, but as you noticed they are very limited.

You may be interested in datasets and methods used in shared tasks about QA, for instance https://mrqa.github.io/shared.


Using the whole dataset for testing (not validation) in case of small datasets for an object detection task I created a small dataset to train an object detector. The class frequency is more or less balanced, however I defined some additional attributes with environmental conditions for each image, which results in a rather uneven distribution of classes depending on the viewed attribute (e.g. class X occurring with attributes A and B).
I used a typical training/validation split and the loss curve lets me conclude that no over-fitting is occurring. I am aware that in general for testing unseen images with a realistic feature distribution should be used, however due to the small size of the dataset, splitting off additional 10 or 20 % for testing or randomly drawing examples would result in certain combinations of classes and attributes not or only barely occurring in the test dataset and manually selecting examples would be very time-consuming.
I am therefore now wondering, if it reasonable to use the whole dataset (train+val) for testing and calculation of performance metrics as a compromise since I see no signs over-fitting?

Considering that you have not used a cross-validation strategy, you could try to use the LOOCV (Leave One Out Cross Validation) strategy, so you have several splits (as many as samples considered in your training dataset) and still leave some samples as a final never-seen-before validation set.
This is costly from a performance perspective, but interesting for small datasets as in your case. Another option, in case the dataset is not really small, is usin k-fold cross-validation (with k being 5, 10...).
A post with practical info can be found here and a related topic discussion can be found here