(Python Basic) more elegant way of creating a dictionary Is there a more elegant way to write a code like this?

my_dic = {
    'Model':['Apple', 'Banana', 'Pineapple', 'Melon', 'Orange', 'Grape'],
    'AAA':[
        method1(y, y_Apple),
        method1(y, y_Banana),
        method1(y, y_Pineapple),
        method1(y, y_Melon),
        method1(y, y_Orange),
        method1(y, y_Grape)]
    ,'BBB':[
        method2(y, y_Apple),
        method2(y, y_Banana),
        method2(y, y_Pineapple),
        method2(y, y_Melon),
        method2(y, y_Orange),
        method2(y, y_Grape)]
    ,'CCC':[
        method3(y,y_Apple),
        method3(y, y_Banana),
        method3(y, y_Pineapple),
        method3(y, y_Melon),
        method3(y, y_Orange),
        method3(y, y_Grape)]
    ,'DDD':[
        method4(y, y_Apple),
        method4(y, y_Banana),
        method4(y, y_Pineapple),
        method4(y, y_Melon),
        method4(y, y_Orange),
        method4(y, y_Grape)]
    ,'EEE':[
        method5(y, y_Apple),
        method5(y, y_Banana),
        method5(y, y_Pineapple),
        method5(y, y_Melon),
        method5(y, y_Orange),
        method5(y, y_Grape)]
}

You can try:

my_dic = dict()

my_dic['Model'] = ['Apple', 'Banana', 'Pineapple', 'Melon', 'Orange', 'Grape']
y_list = [y_Apple, y_Banana, y_Pineapple, y_Melon, y_Orange, y_Grape]
keys = zip(['AAA', 'BBB', 'CCC', 'DDD', 'EEE'], ['method1', 'method2', 'method3', 'method4', 'method5'])
func  = lambda F, a, b: eval(F)(a,b)

for name, method in keys:
    my_dic[name] = [ func(method, y, y2) for y2 in y_list]


Could I use logloss as classifier performance measures when "Test & Score"? I try orange today, and want to use logloss as the classifier performance measures when "Test & Score", but I don't know how to do that.

Hover on the scoring header and use right click. This should open a window where you can select logloss as well.


standardize dataset with numerical and dummy features I have a dataset with both numerical and categorical features (variables), I converted all the categorical variables in dummies, then I split the train and test data.

Now I am at the step where I want to standardize the features before fitting the model.

Should I apply the standardisation to all the features or only to the numerical ones?
In this case, is it preferable to use the MinMax scaler on a range 0 to 1?

1) Dummy variables doesn't need to be standardized, just numerical ones, but if you use MaxMin scaler you can pass both numerical and dummy variables because this scaler doesn't change the values of dummy variables, try to apply the equation to dummy variables to see.

$$ z_i=\frac{x_i-\min(x)}{\max(x)-\min(x)} $$

2) That's a trick question and the response is: it depends. If you have some outlier in your data MaxMin doesn't fit well, because this outlier will be replaced by 0 or 1 and the rest of data will be confined in a very restricted range of values, in this case you should use a normal scaler.


Feature Engineering of mixed data type column I have a data set in which I have to predict the price of a building. Among many features there is a feature called Availability which has two type values like :

"Ready To Move" and a date (e.g. "20-Jan", "18-May", etc.)

For example:

Availability           Price
------------           -----
Ready To Move           55
18-Apr                  38

How would you engineer this column to find its relationship with price and to draw other conclusions.

You have several choices. Among them try these:

Discretize your target and use chi-square association measure or Cramer's V to see the association between variables.
The other way around i.e. convert availability labels to numerics and simply try different association analysis methods like Correlation (Pearson, Spearman, Kendall) or Mutual Information.

PS: Please note that Spearman or Kendall Rank Correlations can also be applied to your categorical variables as they are ordered (dates).


Which optimization technique to use with noisy and slow black-box system? I have a complex physical system that depends on many continuous input parameters (let's say 10 important ones) and produces an output that I can boil down to one continuous figure of merit. I want to optimize this system and find the set of parameters that produces the best output.

The challenges are the following:

The system is sufficiently complicated that I cannot predict the behavior well enough to build a useful model. I have to treat it as a black box and do all tests with the actual system.
The input parameters are generally correlated. So, optimizing the parameters individually is not enough.
The system is slow to react. When changing the parameters it takes some time until an equilibrium state is reached again, depending also on the magnitude of the change.
The system is noisy. Even when not touching the system the output figure of merit will fluctuate strongly. I want to optimize the long-term average of the figure of merit, but in order to get a good estimate of that I have to measure for some time (many minutes if I wanted to bring the noise down to a negligible level).

On the other hand, the system is relatively well behaved when it comes to the shape of the function to optimize:

I know that I'm already close to the global optimum and there are probably no local optima to get stuck in.
I expect no sudden changes of the figure of merit. The gradients vary by not more than maybe two orders of magnitude over the permissible parameter space.

Question: What is a good technique to apply to this problem? Naïve approaches like a simple grid search are prohibitively time-intensive.

It seems like there will be a trade-off between long averaging and dealing with noisy data. So I would either need an algorithm that makes as few queries as possible, or one that's very robust to noise.

Are you a petroleum engineer by any chance? This sounds very like the sort of optimization you have to perform in oilfield waterflood operations; you have to allocate your limited budget of injection water and/or lift gas to maximize oil production, with the additional complication that the optimal set point changes as the reservoir depletes and/or wells get drilled or abandoned.

The oilfield case has the useful feature that the overall objective function (field oil flowrate) is a simple sum of well flow rates. Each production well is generally only sensitive (in the short term!) to a small local subset of the inputs (gas lift rate in the production well, and water injection rate in near-offset water injection wells).

I would generalise the solution employed in the oilfield as follows:

Regularly perturb the inputs
Measure the affected local outputs
Maintain a simple linear or low-order polynomial model of the system and its parts
Use the model to optimize day to day
Keep good records to ensure that the model is up to date with reality
Monitor local and global model fidelity to decide what to perturb and recalibrate next

This works because the small perturbations that you keep on making (for model calibration purposes) are of short duration and local effect, so they don't take the overall system very far away from the optimum. The trade-off between keeping the system optimized, and "de-optimizing" it to calibrate the model, works out favorably. The system reacts on a timescale of days to months, and the measurements generally take a day or two to accomplish.

Try Googling waterflood optimization and gaslift optimization for some more detailed pointers, although a lot of the trade literature is paywalled.

Looking forward to learning what approaches have been taken in other fields with this kind of problem.


Predicting probability from scikit-learn SVC decision_function with decision_function_shape='ovo' I have a multiclass SVM classifier with labels 'A', 'B', 'C', 'D'.

This is the code I'm running:

>>>print clf.predict([predict_this])
['A']
>>>print clf.decision_function([predict_this])
[[ 185.23220833   43.62763596  180.83305074  -93.58628288   62.51448055  173.43335293]]

How can I use the output of decision function to predict the class (A/B/C/D) with the highest probability and if possible, it's value? I have visited https://stackoverflow.com/a/20114601/7760998 but it is for binary classifiers and could not find a good resource which explains the output of decision_function for multiclass classifiers with shape ovo (one-vs-one).

Edit:

The above example is for class 'A'. For another input the classifier predicted 'C' and gave the following result in decision_function

[[ 96.42193513 -11.13296606 111.47424538 -88.5356536 44.29272494 141.0069203 ]]

For another different input which the classifier predicted as 'C' gave the following result from decision_function,

[[ 290.54180354 -133.93467605  116.37068951 -392.32251314 -130.84421412   284.87653043]]

Had it been ovr (one-vs-rest), it would become easier by selecting the one with higher value, but in ovo (one-vs-one) there are (n * (n - 1)) / 2 values in the resulting list.

How to deduce which class would be selected based on the decision function?

Your link has sufficient resources, so let's go through:

  When you call decision_function(), you get the output from each of the pairwise classifiers (n*(n-1)/2 numbers total). See pages 127 and 128 of "Support Vector Machines for Pattern Classification".

Click on the "page 127 and 128" link (not shown here, but in the Stackoverflow answer). You should see:

Python's SVM implementation uses one-vs-one. That's exactly what the book is talking about.
For each pairwise comparison, we measure the decision function
The decision function is the just the regular binary SVM decision boundary

What does that to do with your question?

clf.decision_function() will give you the $D$ for each pairwise comparison
The class with the most votes win

For instance,

  [[ 96.42193513 -11.13296606 111.47424538 -88.5356536 44.29272494 141.0069203 ]]

is comparing:

  [AB, AC, AD, BC, BD, CD]

We label each of them by the sign. We get:

  [A, C, A, C, B, C]

For instance, 96.42193513 is positive and thus A is the label for AB.

Now we have three C, C would be your prediction. If you repeat my procedure for the other two examples, you will get Python's prediction. Try it!


sklearn SimpleImputer too slow for categorical data represented as string values I have a data set with categorical features represented as string values and I want to fill-in missing values in it. I’ve tried to use sklearn’s SimpleImputer but it takes too much time to fulfill the task as compared to pandas. Both methods produce the same output.

Here is the code to reproduce the behavior on a synthetic data:

import numpy as np
import pandas as pd
from sklearn.impute import SimpleImputer

lst = np.array(['a', 'b', np.nan], dtype='object')
arr = np.random.choice(lst, size=(10**6,1), p=[0.6, 0.3, 0.1])
ser = pd.Series(arr.ravel())

Using SimpleImputer:

%%time
imp = SimpleImputer(missing_values=np.nan, strategy='most_frequent')
imp.fit_transform(arr)

Wall time: 13 s

Using pandas:

%%time
ser.fillna(value=ser.mode()[0])

Wall time: 64.8 ms

Things get even worse when string values are longer (e.g. ‘abc’ instead of one letter ‘a’). For numerical data pandas still outperform sklearn, but the difference is not that huge.

What am I doing wrong?

Searching the source code of Sklearn for SimpleImputer (with strategy= "most_frequent"), the most frequent value is calculated within a loop in python, therefore that is the part of code that is so slow. In the source code of SimpleImputer there is also the comment that explains why they do not use the scipy.stats.mstats.mode, which is mush faster:

  scipy.stats.mstats.mode cannot be used because it will no work properly if the first element is masked and if its frequency is equal to the frequency of the most frequent valid element. See https://github.com/scipy/scipy/issues/2636

So if you want to use the SimpleImputer with this strategy, a faster way would be to use the  "constant" strategy and pass the most frequent value by yourself (ser.mode()[0]) then the time is almost the same:

t0 = time.time()
imp = SimpleImputer(missing_values=np.nan, strategy='most_frequent')
imp.fit_transform(arr)
print('Simple Imputer (Most Frequent) Time Elapsed:', time.time()-t0)

t0 = time.time()
imp = SimpleImputer(missing_values=np.nan, strategy='constant', fill_value=ser.mode()[0])
imp.fit_transform(arr)
print('Simple Imputer (Constant) Time Elapsed:', time.time()-t0)

t0 = time.time()
ser.fillna(value=ser.mode()[0])
print('Pandas Time Elapsed:', time.time()-t0)

And the time elapsed for each strategy:

Simple Imputer (Most Frequent) Time Elapsed: 14.320188045501709
Simple Imputer (Constant) Time Elapsed: 0.052472829818725586
Pandas Time Elapsed: 0.04726815223693848


Ambiguity in Perceptron loss function (C. Bishop vs F. Rosenblatt) Bishop's Perceptron loss

On one hand, it is stated in equation 4.54 of Chris Bishop's book (pattern recognition and machine learning) that the loss function of perceptron algorithm is given by:

$${E_p}(\mathbf{w}) =  - \sum\limits_{n \in M} {{\mathbf{w}^T}{\phi _n}} {t_n}$$

where $M$ denotes the set of all misclassified data points.

  Original Perceptron loss

On the other hand, The loss function used in the original perceptron paper written by Frank Rosenblatt is given by (wikipedia):

$${1 \over s}\sum\limits_{j = 1}^s {\left| {{d_j} - {y_j}\left( t \right)} \right|} $$
which when translated to the notation of Bishop's book is given by:

$${1 \over N}\sum\limits_{n = 1}^N {\left| {{t_n} - {\mathbf{w}^T}{\phi _n}} \right|} $$
where $N$ denotes the set of all data points.

  My question

My question is that why Bishop's version of Perceptron loss is different from the original paper? Considering Bishop's book as a highly recognized book in machine learning field, can we call it Bishop's Perceptron?!

  Scikit-learn's implementation

By the way, it seems that Scikit-learn uses Bishop's version of Perceptron loss (Scikit-learn documentation). It is apparent from the following formula and figure:

-np.minimum(xx, 0)

which for one sample reduces to:
$$ - \min \left( {0,{\mathbf{w}^T}{\phi _n}{t_n}} \right)$$

I managed to find the  Bishop's version by unearthing! the Rosenblatt's 1962 Principles of neurodynamics, Page 110 book, so the Wikipedia's version must be the alternative one.

It is worth noting that the book also has a chapter on error back-propagation (page 292), which resembles the Wikipedia version, but I think it is not exactly the same.


p-value and effect size Is it correct to say that the lower the p-value is the higher is the difference between the two means of the two groups in t-test?
For example, if I apply the t-test between two groups of measurements A and B and then to two groups B and C and I find that in the first case the p-value is lower than the second case, could one of the possible interpretations be that the difference between the means of group A and B is greater than the difference between the means of group C and D?

No.
What you refer to (the difference between the means of group A and B) is actually the effect size, and it has absolutely nothing to do with the p-values.
The situation is nicely summarized in the (highly recommended) paper Using Effect Size—or Why the P Value Is Not Enough (emphasis mine):

Why Report Effect Sizes?
The effect size is the main finding of a quantitative study. While a
P value can inform the reader whether an effect exists, the P value will not reveal the size of the effect. In reporting and
interpreting studies, both the substantive significance (effect size)
and statistical significance (P value) are essential results to be
reported.
Why Isn't the P Value Enough?
Statistical significance is the probability that the observed
difference between two groups is due to chance. If the P value is
larger than the alpha level chosen (eg, .05), any observed difference
is assumed to be explained by sampling variability. With a
sufficiently large sample, a statistical test will almost always
demonstrate a significant difference, unless there is no effect
whatsoever, that is, when the effect size is exactly zero; yet very
small differences, even if significant, are often meaningless. Thus,
reporting only the significant P value for an analysis is not
adequate for readers to fully understand the results.

In other words, the p-value reflects our confidence that the effect indeed exists (and it's not due to chance), but it says absolutely nothing about its magnitude (size).
In fact, the practice of focusing on the p-values instead of the effect size has been the source of much controversy and the subject of fierce criticism lately; see the (again, highly recommended) book The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives.
The following threads at Cross Validated may also be useful:

P value as a measure of effect size?
Are effect sizes really superior to p-values?


Text classification: Choosing training and testing set for each category I am very new to machine learning.
I have a text classification problem in hand. I have a tagged dataset of around 750 documents( short texts), categorized manually into 16 buckets. I want to train a classifier on this data. I know that there should be a training set and a test set (an option could be 80-20 ). In my understanding, this should be for the complete set( 80% of my 750 documents- training, 20% of 750 documents - testing ).
1. They should be randomly generated or is there some condition for category? ie. if category A constitutes 60%,category B 5%, C 7% etc. how to choose the training set?

The most commonly used option is 2/3rd of the data as training and 1/3rd as testing sets respectively. What kind of software or tool are you using for this categorization process? For your classifier to work efficiently at the end of all the hard work, have you considered cross validating it using a multiple fold cross validation before using and assigning the training and testing sets? As in for the skewed data categories, try finding instances for the classes/ categories which have fewer samples (~10%).


How to create a score for a SWOT analysis (strengths, weaknesses, opportunities, and threats)? I'm developing a participatory social environmental diagnostic. To do this, I'm using primary (qualitative data from interviews with stakeholders) and secondary data (local socioeconomic data).

From this data, I distribute the identified factors in a SWOT matrix - identifying opportunities, strengths, weaknesses and threats that could afftect the local community. After that, each factor is given a scale from 0 to 2, being:

0 - Null or small impact
1 - Medium impact
2 - High impact

In this way, I have  this kind of data:

Ok. Now I build a scale with this formula:

= (((strengths + opportunities) - (weaknesses + threats)) / ((strengths + opportunities) + (weaknesses + threats)) * 2

OBS: basically opportunities and strengths are positive values and threats and weaknesses are negative, so I subtract the negative values from the positive ones. The division by the sum of all values are made to have the statistical universe (population, in this case the maximum value). The multiplication by 2 are made to have the possibility to create a range from -200% to 200%, but it could be -100% to 100% too ;)

Now I have a scale ranging from -200% to 200% that I interpret in this way:

Very unfavorable (-200% to -100%)
Unfavorable (-100% to -29%)
Balance (-30% to 30%)
Favorable (31% to 99%)
Very favorable (100% to 200%)

OBS: This scale was used at this website (https://en.luz.vc/template/swot-analysis-excel-tool-template/)

Now, what I need is to convert this to 0 to 10 scale so I apply this formula:

I've discover this  formula here: (https://stats.stackexchange.com/questions/25894/changing-the-scale-of-a-variable-to-0-100)

Finally, my question is how can I achieve this score (from 0 to 10) right from the beginning, without the need to make the conversion from the first scale?

OBS: The first scale give a percentage (-200% to 200%) and the second one a decimal number (0 to 10).

Warm regards ;)

Let's suppose you have a variable $x_{old} \in [-1,1]$. If you would like to transform that to $ x_{new} \in [-2,2]$ then you could just apply the transformation formula you posted:

With $min_{old}=-1, max_{old}=1, min_{new}=-2, max_{new}=2$ you would get $$\frac{max_{new}-min_{new}}{max_{old}-min_{old}} (v-min_{old}) + min_{new}
= \frac{2-(-2)}{1+1} (v+1) + (-2) = \frac{4}{2} (v+1) -2\\= 2v+2-2=2v$$

Now if you compare that to your formula

= (((strengths + opportunities) - (weaknesses + threats)) /
 ((strengths + opportunities) + (weaknesses + threats)) * 2

then you can see that it is exactly what you did here by multiplying with 2 at the very end. You transformed your first scale from $[-1,1]$ to $[-2,2]$. Therefore, if you want it to be in the interval $[0,10]$ just apply a different transformation at the very beginning when deriving the first scale:

$$\frac{max_{new}-min_{new}}{max_{old}-min_{old}} (v-min_{old}) + min_{new}
= \frac{10-0}{1+1} (v+1) + 0 = 5v +5$$

Which means your calculation for the first scale needs to be

= (((strengths + opportunities) - (weaknesses + threats)) /
 ((strengths + opportunities) + (weaknesses + threats)) * 5 + 5

(instead of scaling it to $[-2,2]$).

Intuitively I like to think of it like this: you "stretch" the interval from $[-1,1]$ to $[-5,5]$ by multiplying with 5 ("making it 5 times larger"). And then, since you want it to be positive and start at $0$, you "shift" it "upwards" by adding 5.


How to predict a group of events using machine learning? Whether any machine learning model can dynamically predict different types of stages it would go through for completing a process.

Welcome to the site!

Assumptions before answering your question:

The Target variable is a categorical(each stage is considered as a separate category).
You have the past data of the customers whose application got rejected(if hasn't reached the last stage)/successful(reach last stage) at the respective level. For Example, customer A applied and got his application rejected at stage-2, No Legal).
You have some demographics of the customer to predict the stage at which the application might get rejected.
You have good data(no outliers, well engineered features, good understanding on data etc).

Now if the above assumptions are met, then you can apply Machine Learning algorithms which are used for doing multi-class classification.

Applicable Algorithms:

Neural Networks
Random Forest
K nearest Neighbor
Decision Tress
Support Vector Machines etc.

Finally, you can apply Ensemble model by combining results of the 2 or more models.

For measuring the accuracy you can use Confusion Matrix(By diving the data into Test: 30% of data and Train: 70% of data).


One-hot encode multi-class multi-label sequences Suppose I want to build a timeseries where each timestep is represented by a categorical array: the encoded sequences look like [[2, 0, 5],[3, 1, 4],..] and each entry has a different number of possible values (categories).
For example the first entry has 0-3 values, the second 0-1 and so on...

I want to train an LSTM model in order to predict the next timestep. So I defined a one hot encoding of each entry by means of the maximum number of classes:

For example [2, 0, 5] becomes

[[0. 0. 1. 0. 0. 0.],
[1. 0. 0. 0. 0. 0.],
[0. 0. 0. 0. 0. 1.]]

Unfortunately this kind of representation raises the error

  ValueError: Invalid shape for y: (1, 3, 5)

I have three questions:

Is it possible to pass a 3d y target to Keras?
Should I define a single one-hot encoding which combines all the possible triplets of categorical values instead? The problem is that in this case I would lose the correlation between the occurrences of the labels in the same category, because each possible combination of labels would become independent from the other ones.
Should I only one-hot encode the target y or also the input X?

You may need to try cat2vec which converts categorical features into vector representation using Word2Vec approach. Check also this link for multi-feature inputs into LSTM.

For the target y, one-hot is a better technique for NN-based models.


Possibility of working on KDDCup data in local system I'm trying to apply classification algorithms to KDD Cup 2012 track2 data using R
http://www.kddcup2012.org/c/kddcup2012-track2

It seems not possible to work with this 10GB training data on my local system with 4GB RAM.
Can anyone work on this data using this kind of a local system ? Or is using a cluster the norm ?
It would be great if anyone could provide me with any guidance on how to get started with working on a cluster and the normally used type of cluster for such tasks

I think that you have, at least, the following major options for your data analysis scenario:

Use big data-enabling R packages on your local system. You can find most of them via the corresponding CRAN Task View that I reference in this answer (see point #3).
Use the same packages on a public cloud infrastructure, such as Amazon Web Services (AWS) EC2. If your analysis is non-critical and tolerant to potential restarts, consider using AWS Spot Instances, as their pricing allows for significant financial savings.
Use the above mention public cloud option with R standard platform, but on more powerful instances (for example, on AWS you can opt for memory-optimized EC2 instances or general purpose on-demand instances with more memory).

In some cases, it is possible to tune a local system (or a cloud on-demand instance) to enable R to work with big(ger) data sets. For some help in this regard, see my relevant answer.

For both above-mentioned cloud (AWS) options, you can find more convenient to use R-focused pre-built VM images. See my relevant answer for details. You may also find useful this excellent comprehensive list of big data frameworks.


Decision tree classifier prediction changes from one run of the model to the next I'm running a very basic gender ['male', 'female'] classifier using the sklearn DecisionTreeClassifier based on [height, weight, shoe size] in a Jupyter notebook.

The prediction changes from male to female for the same input as I keep running the model.

I don't understand how that's possible. Shouldn't the build of the model be completely deterministic and therefore output the same prediction each time for a specific input?

Here's my code:

X = [[190, 90, 43], [165, 65, 38], [170, 70, 39], [160, 56, 36], [190, 88, 45],
[164, 63, 37]]
Y = ['male', 'female', 'male', 'female', 'male', 'female']

clf = tree.DecisionTreeClassifier()
clf.fit(X, Y)

print(clf.predict([[200, 70, 37]]))

While this is a duplicate and the suggested link answers your question, for learning purposes I would like to suggest that you plot your DecisionTree every time you have a new run to see by yourself what happens behind the scenes:

from sklearn.tree import DecisionTreeClassifier
from sklearn.externals.six import StringIO

from IPython.display import Image
from sklearn.tree import export_graphviz
import pydotplus
dot_data = StringIO()

X = [[190, 90, 43], [165, 65, 38], [170, 70, 39], [160, 56, 36], [190, 88, 45], [164, 63, 37]]
Y = ['male', 'female', 'male', 'female', 'male', 'female']

clf = DecisionTreeClassifier()
clf.fit(X, Y)

export_graphviz(clf, out_file=dot_data,
                filled=True, rounded=True,
                special_characters=True)

graph = pydotplus.graph_from_dot_data(dot_data.getvalue())
Image(graph.create_png())

Once you may get:

where splitting is done on X0 feature, where as next run you may get:

where splitting is done on X1 feature.

If you wanna reproduce your result at every run, you can use random_state = a_random_number and then you should expect to get the same result every time as the same tree has been constructed every time!


How would I classify this variable? I am learning about the difference between categorical, ordinal and numerical variables.
From what I understand:

Categorical variables have 2+ categories without any intrinsic order.
Ordinal variables have clear ordering but the intervals are not necessarily equally spaced.
Numerical variables are ordered with equally-spaced intervals.

I am stuck on trying to classify these two situations:
1. What level of commitment are you prepared for?

Multiple times per week
Once per week
A couple times per month
No more than once per month
Level of commitment is not a factor.

2. What level of commitment are you prepared for?

9-11 times per month
6-8 times per month
3-5 times per month
0-2 times per month
Level of commitment is not a factor.

My guess would be #1 is ordinal and #2 is numerical. However, the "Level of commitment is not a factor" is throwing me off. Can anyone help clarify this for me?

The responses to both of those questions can be treated as categorical, as although they mention numbers you can’t fully encode the information in them as a number. You could obviously number them, e.g. “I’ll call 9-11 times per month response 1”, but the number 1 doesn’t reflect the value of the response.
These types of categorical “buckets” representing a range of values are often used when the value we want to measure is continuous and the range of possible values is large.
If you wanted numerical variables you could recode the responses. For example you could represent answers to question 2 as two numerical variables, min_times and max_times.


Filtering detail table based on selection in summary table in Orange 3.11 I'm rather new to Orange, so apologies in advance if I'm missing something obvious or am trying to fit a square peg in a round hole.

I have two files, Summary and Detail, that I'm importing into an Orange workflow. Summary has one row per Contact, and Detail has multiple rows per Contact, each row corresponding to a particular action they've taken. Contacts are identified in both files by Contact ID.

I've done a bunch of analysis to identify Contacts in Summary that I'm interested in using Orange's machine learning algorithms. I'd like to be able to step through each row in Summary (that is, each Contact) to see what actions in Detail that the Contact took.

I thought I'd be able to use the Merge Data widget to do this by hooking up Selected Data from Summary as the Data and Data from Detail as the Extra Data using a Find Matching Rows join type. However, since each unique Contact ID appears multiple times in Detail, I'm not able to select it in the Merge Data widget.

I know it's nonsensical, but I did try to wire Data from Detail as the Data and Selected Data from Summary as the Extra Data with the only Merge Data option that worked (Append columns from Extra Data). This showed me all the data from Detail, but appended with columns from Summary when the appropriate row was selected in Summary. Based on my understanding of Merge Data, this is predicted but (in my use case) unhelpful behavior.

In database terms, I'm sort of trying to set up an inner join with a one-to-many relationship between Summary and Detail and, by selecting a row in Summary, to be able to see the related rows in Detail.

So, the bottom line is, is this sort of behavior doable in Orange, or am I trying to paint a house with a chainsaw? Is there another widget that would be more appropriate for the task? (I also tried the Select Rows widget, but couldn't figure out how to populate the filters based on the row that I had selected in Summary.)

A workflow like this might do the trick, although it is static (does not adjust to varying actions).

Load the two files, Summary and Details.
I suppose "Details" has columns such as "Action" and some value. In my example, Details will contain "Subject" and "Grade" for 3 highschool subjects.
"Select columns" is optional (slim down unused columns)
You need a "Select rows" for each subject/action, filtering that specific one.
Every "Edit Domain" is used to rename the generic feature name to a specific on (like "Grade" to "Grade_Math").
Results are then merged step-wise to the Summary (which in my example contains students, your summary contains some other people).

If the number of actions is limited, the time needed is reasonable.

If this is a one-time job, you might as well save the output (CSV) and use that in a different project.

If it needs to be more dynamic and adaptive, you'll have to dig into Python and use the Python code widget to create a list of actions, create a list of columns based on that, and then populate those with matching values.


How to prefer no choice instead of bad choice with sklearn decision tree I'm using sklearn decision trees to classify documents in two possible types "type1" and "type2".

I've isolated few features that seem pertinent and tried to combine them manually to evaluate the result of a model.
When classifying the documents manually i use the following outcomes:

type 1
type 2
unknown

Then I give the same features to a decision tree. The results are worse in that case because it always tries to classify documents in one of the categories "type1" or "type2" but is unable to classify documents in "unknown"

Is it possible to configure a sklearn decision tree in a way that in case of high uncertainty the document won't be classified instead of selecting a category that is likely to be wrong?

TL;DR
You can train your classifier as usual and threshold on the prediction probability.
Scikit-learn
You won't get that out of the box, but you can use this simple class to do just that!
from sklearn.tree import DecisionTreeClassifier
from numpy import argmax, max

class MyClassifier(DecisionTreeClassifier):

    unknown_class = 'unknown'

    def __init__(self, no_class_threshold=0.75, **kwargs):
        self.no_class_threshold = no_class_threshold
        super().__init__(**kwargs)

    def predict(self, X):
        preds = self.predict_proba(X)
        y_pred = [self.classes_[i] if v > self.no_class_threshold else self.unknown_class
                  for i, v in zip(argmax(preds, axis=1), max(preds, axis=1))]
        return y_pred

Just use this class as you were using the DecisionTreeClassifier before. You can pass all the parameters from DecisionTreeClassifier plus an extra one: no_class_threshold.
The no_class_threshold works in the following way:
IF
    prediction_probability > no_class_threshold
THEN
    output predicted class
ELSE
    output 'unknown'

Changing classifier
You might not want to stay with the DecisionsTreeClassfier, and instead, you might want to experiment with other classifiers. You can do this by inheriting from another Scikit-learn class. Simple as that!
For example:
from sklearn.svm import SVC

class MyClassifier(SVC):
    [...]

Choosing the no_class_threshold
You can:

Choose it manually, to a degree that you feel confident
Use hyper-parameter tuning to learn it, if you have data points labelled as 'unknown'.

Example
Here is a script you can run to test all the above.
from numpy import argmax, max
from sklearn.svm import SVC

class MyClassifier(SVC):
    unknown_class = 'unknown'

    def __init__(self, no_class_threshold=0.75, **kwargs):
        self.no_class_threshold = no_class_threshold
        super().__init__(**kwargs)

    def predict(self, X):
        preds = self.predict_proba(X)
        y_pred = [self.classes_[i] if v > self.no_class_threshold else self.unknown_class
                  for i, v in zip(argmax(preds, axis=1), max(preds, axis=1))]
        return y_pred

if __name__ == '__main__':
    X = [[1, 1, 1],
         [1, 0, 1],
         [0, 0, 0],
         [0, 1, 0],
         [0, 1, 1]]
    y = ['class1', 'class1', 'class0', 'class0', 'class0']

    clf = MyClassifier(no_class_threshold=0.55, probability=True, C=1)

    clf.fit(X, y)

    X2 = [[1, 1, 0],
          [0, 1, 1],
          [0, 0, 1]]
    pred = clf.predict(X2)
    print(pred)


Why my loss is negative while training SAE? I am using loss='binary_crossentropy' here is my code:

I tried to increase number of training image and Epoch ,but that did not help me.

input_img = Input(shape=(28, 28, 1))

x = Convolution2D(16, 3, 3, activation='relu', border_mode='same')(input_img)
x = MaxPooling2D((2, 2), border_mode='same')(x)
x = Convolution2D(8, 3, 3, activation='relu', border_mode='same')(x)
x = MaxPooling2D((2, 2), border_mode='same')(x)
x = Convolution2D(8, 3, 3, activation='relu', border_mode='same')(x)

encoded = MaxPooling2D((2, 2), border_mode='same')(x)

x = Convolution2D(8, 3, 3, activation='relu', border_mode='same')(encoded)
x = UpSampling2D((2, 2))(x)
x = Convolution2D(8, 3, 3, activation='relu', border_mode='same')(x)
x = UpSampling2D((2, 2))(x)
x = Convolution2D(16, 3, 3, activation='relu', border_mode='valid')(x)
x = UpSampling2D((2, 2))(x)

decoded = Convolution2D(1, 3, 3, activation='sigmoid', border_mode='same')(x)

autoencoder = Model(input_img, decoded)
autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')
autoencoder.fit(x_train, x_train, nb_epoch=10, batch_size=500,
               shuffle=True, validation_data=(x_test, x_test), verbose=1)

Use a linear output and mean squared error loss, assuming you are predicting normalised pixel intensity values.

Cross-entropy over sigmoid output layer activations can do odd things when the values are not strictly in $\{0,1\}$, depending on implementation.


Why don't we transpose $\delta^{l+1}$ in back propagation? Using this neural network as an example:

The weight matrices are then
$$ W_0=[2\times4], W_1=[4\times4], W_2=[4\times2]$$
To find the error for the last layer, we use
$$ \delta^{[2]} = \nabla C \odot \sigma'(z^{[2]})$$
which makes sense. This will produce a $[1\times 2]$ vector. But to find the error in the next layer, we use
$$ \delta^{[1]} = (W_2^T\delta^{[2]})\odot \sigma'(z^{[1]}) $$
This appears to try to multiply a $[4\times2]$ matrix and a $[1\times 2]$ matrix together, which is illegal. Am I just wrong about how the layers are represented? Should $z^{[n]}$ really be a $[l\times 1]$ vector? That doesn't really make sense to me, because it would be multiplied by an $[l\times m]$ matrix as the feed-forward continues. Do we just always represent $\delta^{[n]}$ as a $[l\times 1]$ vector, and the formula doesn't mention this as it's common knowledge?
What am I missing here?
(Note: these formulae are based on this book)


What is the meaning of "probability distribution of p(x)" of something uncountable? I'm studying VAE and new to both of the neural network and the statistic.

After some researches, I could understand the rough concept of VAE.
But what makes me confused is, the meaning of probability distribution p(x) itself.
When the x is an image data, what is the meaning of "probability distribution of the image"? and what is "probability distribution of the latent space"?

When I learned the probability distribution in school, x in p(x) was always something numerable(the value of dice, the number of apples, ..), and so I could get some value like p(x=1).
But I couldn't understand the meaning of p('image data'), p('latent space').
Although many websites explains the concept and logical flow of VAE magnificently, I'm stuck by the lack of my knowledge.

Anyone help me?

Let me preface this answer by saying I am not an expert on variational autoencoders, but I think your conceptual gaps don't have anything to do with autoencoders.

First, it is possible to specify a probability distribution over categorical outcomes or continuous outcomes.  For example, a sample from a normal distribution is a continuous outcome, and the normal distribution describes the distribution of outcomes (or, equivalently, the relative likelihood of outcomes with various values).  When you sample from a normal distribution, the probability of getting any particular numerical value (like 0, or 3.14) is infinitely small.  Because of this, people typically talk about the probability density of a continuous distribution -- which can be interpreted as the likelihood of a sample being from a given (small) region.  The probability value is called a probability density because it represents probability mass divided by space (or volume).

Second, it is common to represent images as a vector of numbers. For a grayscale image, this might be 1 numeric value per pixel (it's slightly more complicated for color images).  You can think of the vector representation of an image as a point in space.  If you have a bunch of images, you can talk about the density of images in this space. Some regions of space have a lot of 'image points', and some have very few.  You can approximate this density by fitting a high-dimensional probability distribution to the image points in your dataset -- like fitting a smooth curve to a binned histogram of image counts.  Because of this, each image maps to a specific value for the fitted probability density.  That means you can talk about p(image) -- it really means, if we look 'near' the vector representation of this image in our vector space, how many images are there, relatively speaking? Some images are in high-density regions and some are in low-density regions, which will reflected as high and low values for p(image).


Metrics for Name Entity Recognition Working on a NER project, I have been facing the problem of evaluating my model during training. I cannot be using the accuracy metrics or f1 score or any other metrics to evaluate my model on runtime as this always leads to high values due to a number of O (other) tags in the dataset.

As a first start I would recommend to use Precision and Recall.

You can also produce stats on the different types of errors:

entity not found (false negative, or Type I error). Also you can get the false negative rate by dividing by the total number of true entities. This would be 1-recall.
entity found where it shouldn't be (false positive or Type II error). Also you can get the false positive rate by dividing by the total number of returned entities. This would be 1-precision.
entity found but system got wrong number of tokens (e.g. York instead of New York). You can also get the "token boundary error rate" by dividing by the total number of true entities.

So the first two error rates I mentioned don't give any extra info other than precision and recall but the third allows you to check how accurately you are getting entity boundaries.

If you are doing entity disambiguation too, you can produce additional metrics by only counting a success if the entity was correctly resolved.

You can also use the F-score (thanks Erwan) which combines precision and recall. However you need to be careful because you have a large class imbalance of O-tokens vs entity tokens, and the F-score gives equal importance to precision and recall. You also have the options of weighted F-scores such as F2 score.

Other option: ROC curve and AUC

Finally I would add that if your model outputs a score or probability rather than a binary classification, it means you can tweak its sensitivity by changing the threshold (do you want to count all entities with score>0.5 as entities, or score>0.75, for example). This naturally leads to the possibility of calculating the false positive and false negative rates for every possible threshold choice. You can plot it on a graph and you have a ROC curve. This is a nice graphical way to see the performance of your model and exactly how it misclassifies instances. The area under the ROC curve is itself a nice metric called AUC. A classifier that is as good as random has AUC=0.5, a classifier that classifies everything wrong has AUC=0, and a perfect classifier has AUC=1. I normally plot the ROC curve as well as generating metrics, Sklearn and most stats packages have a function to do it.


Is there a way to get the sample weights that were used to fit a scikit-learn estimator? Many sklearn estimators support two weighting schemes:

Per-class weights: given when creating the estimator object (e.g. clf = DecisionTreeClassifier(class_weight={'Cat':0.4, 'Dog':0.6})
Per-sample weights: given when fitting a created object (e.g. clf.fit(X, y, sample_weight=[0.1, 0.1, 0.8])

For an already fitted estimator, I can get the per-class weights with clf.get_params()['class_weight'].

But what's the right way (if any) to get, given a fitted estimator, the per-sample weights that were used?

A quick search of the source code, e.g. RandomForestClassifier's, doesn't find anywhere that sample_weight gets saved as a class attribute. I suspect that was a conscious decision: the sample weights are directly tied to the dataset, which also doesn't get saved for later use; that's why sample_weight appears in the fit method rather than the class instantiation.


Database/warehouse for storing 128D feature vectors I use openface to extract feature vectors in 128D. I need to find a suitable database engine to store these vectors for future comparisons (calculating the euclidean distance between newly extracted features and those stored in the database).

Here is an example of what I'm trying to do:

I use OpenFace to extract the face representation, this is a 128D feature vector.
I then calculate the euclidean distance between this vector and those stored in the database, returning the vector that has the lowest distance (less than 0.9) as a match.
If no vector stored in the database matches this criteria, I store the newly extracted features as a new entry in my dataset.

What is a good database engine to achieve this?

If you really need to do that (I will argue it is not a good idea), with Postgres you can store an array type and write a stored procedure for new item insertion.  This stored procedure can do whatever distance checking you wish, such as checking the distance of the new vector against all others in the database, before storage.

I would argue against this design because I suspect the criteria for uniqueness* could easily change over time.  I think it'd be a better idea to store all of the vectors except exact matches.  Then, create another table which uses the definition of uniqueness.  Creating this table would be handled on the application side.  If your definition of uniqueness changes, no problem, just make a new table.  You could even compare several different definitions to see how your results are sensitive to it.  If you do it this way, Cassandra is a great database choice.  It's specifically designed for denormalized data storage (you have the same data stored in different forms or variations, so that your application gets exactly what it needs without further computation).

*In your post you stated that a similarity of < 0.9 would result in storing a new vector.  That's what I mean by criteria for uniqueness.


error trying to build a histogram using r with the rStudio application Using ggplot2 I am attempting to create a histogram.  I have a column that is full on the continents.  I need to add all the continents which I attempted to do with the aggregate function.

data <- aggregate(country$continent,country["continent"], sum)
hist(data)

Error in Summary.factor(c(1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,  :
  ‘sum’ not meaningful for factors

The following code :

data.frame(table(country$continent))

will create a dataframe where one column is the continent and the other is the number of countries in the continent.


Modeling the Price Movement- What analysis should be used I am trying to model the price of a hotel as the check-in date arrives. I have a data set which looks like-

For e.g- if I am looking at the booking date of Dec 31st, I would want to analyze the movement of price for the hotels from 31st October onwards and model this movement with given variables-City, Star_rating, Accomodation_type, Chain_hotel, and Time before check-in.

I started with checking which of the variable(Feature) was more important in deciding the price of the hotel on a given day and applied multiway ANOVA. However, I am still not confident on my analysis and I am really confused that what could be the best way to model such a problem.

Thanks in advance for your help.

The question is if you really want to treat this as a time series problem. I say this because there may be no obvious/persistent autocorrelation (meaning that $y$ is contingent on $y_{t-1}$).

My take (not knowing the data) would be that each hotel has an own unobserved "identity" (like location, reputation etc) apart from star rating. Bookings (and thus prices) may also be highly contingent on time. So you could give a fixed-effects model a try. The idea is that you model this in a way like:

$$y=\alpha + \beta X+\gamma Z + \theta t + u.$$

Where $X$ are observed hotel characteristics, $Z$ are hotel fixed-effects (i.e. one indicator/dummy per hotel), $t$ is time (day of year or so), and $u$ is the error term.

In terms of model interpretation, $X$ are important (standard confounders), where $Z$ are not so relevant (and often omitted in FE regression) since $Z$ is only "one intercept per hotel". If you are interested in modeling the time aspect, clever encoding of $t$ is key. It is hard to tell how this could work out without knowing the data. There are two general options, a) dummy encoding (one dummy/indicator per time step) or continuous treatment (calender day?), maybe with a lagged component (price yesterday). In the latter case, you go in the direction of dynamic panel, which can be a little challanging.

You can try OLS first since it is very efficient in terms of computation. In order to get a better fit you may also try boosting, e.g. based on LightGBM. Here is a minimal example for boosting regression (but no fixed-effect model, just a normal one).


Accuracy stagnated while training notMNIST data I am a beginner in machine learning. I have built a logistic classifier in Python using TensorFlow to train on notMNIST dataset. My code is as such:

weights = tf.Variable(tf.truncated_normal(shape = [784, 10]))
bias = tf.Variable(tf.zeros(shape = [10]))
logits = tf.matmul(features, weights) + bias
prediction = tf.nn.softmax(logits)
cross_entropy = -tf.reduce_sum(labels * tf.log(prediction), reduction_indices=1)
loss = tf.reduce_mean(cross_entropy)

train_feed_dict = {features: train_features, labels: train_labels}
valid_feed_dict = {features: valid_features, labels: valid_labels}
test_feed_dict = {features: test_features, labels: test_labels}

is_correct_prediction = tf.equal(tf.argmax(prediction, 1), tf.argmax(labels, 1))
accuracy = tf.reduce_mean(tf.cast(is_correct_prediction, tf.float32))

epochs = 5
batch_size = 50
learning_rate = 0.1

optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)

validation_accuracy = 0.0

with tf.Session() as session:

    session.run(tf.global_variables_initializer())
    batch_count = int(math.ceil(len(train_features)/batch_size))

    for epoch_i in range(epochs):

        for i in range(batch_count):

            session.run(optimizer, feed_dict = train_feed_dict)
            print(session.run(accuracy, feed_dict = train_feed_dict))

However, the problem is that while the training loss is decreasing continuously, the accuracy wavers initially, and then finally stagnates (at around 0.062). I am not able to understand what's wrong with the code. Any help would be appreciated. Thanks.

Use tf.softmax_cross_entropy_with_logits function instead of writing it by yourself because tf.softmax_cross_entropy_with_logits is more computationally stable.

Try this,
loss = tf.reduce_mean(tf.softmax_cross_entropy_with_logits(logits))

Refrence: https://www.tensorflow.org/api_docs/python/tf/nn/softmax_cross_entropy_with_logits


What are the best sites to display your data science skills & projects? What are the best sites to display your data science skills & projects? Obviously there's kaggle, but any others that recruiters look at?

LinkedIn
Github
Personal Website (of course)
Coderbits (not popular, but you can display your profile link on the resume/LinkedIn)
Kaggle


choosing classifiers For what I read the 5x2cv t test is

"a procedure for comparing the performance of two models (classifiers or regressors) that was proposed by Dietterich to address shortcomings in other methods such as the resampled paired t test and the k-fold cross-validated paired t test"

.

I am currently making some experiments with an unbalanced data set, which I balanced using SCUT and trained a set of different classifiers. The problem is a multiclass with three different classes to choose. I am applying a Multilayer Perceptron, a Decision Tree and a Random Forest and the results, after a 10-fold cross validation, are the following:

Multilayer Perceptron: 0.95 acc

Decision Tree: 0.93 acc

Random Forest: 0.935 acc

When I apply the 5x2cv t test I got the following results:

MLP and DT:

t-statistic 4.75

p-value 0.005

So, if I assume that for the tests I will have a value of alpha of 0.05 for rejecting the null hypothesis, which is that both algorithms perform well with the same database, then from the p-value I got I could reject the null hypothesis. This would mean that both models do not perform equally well, so it would be better to use MLP over the DT because of its higher accuracy.

When I do the same with the MLP and the RF I got the following results:

t-statistic: 2.46

p-value: 0.055

Here, I suppose that I can conclude that using MLP or RF for the current dataset is almost the same, because I failed to reject the null hypothesis. The question that I have here is if should I choose the RF even though the accuracy was lower?

The comparison with the DT and RF has the following values:

t-statistic: -2.49

p-value: 0.054

So I can reject the null hypothesis and said that there is difference between the use of the DT over the RF.

Are my conclusions correct?

Thanks

See, your conclusions appear to be correct, your statistical results explain your conclusions , i would say statistically speaking even with a P-value of 0.055 you still have an error rate of 15 to 25% that you may be accepting a wrong null hypothesis and maybe your alternate hypothesis is right , so combining knowledge of statistics with data science i will suggest you to use catboost(if you have some categorical features in your dataset it will give you a strong baseline prediction before you start tuning it) otherwise random forest is good any day and if you have a small dataset (e.g. less than 1500 instances in your training data) i would suggest use random forest any day over MLP because MLP based classifiers need more data to classify better with less data they tend to become bias towards your training data.But if you have a large enough dataset you can actually use MLP based classifier try some basuc feature selection techniques such as RFECV or feature_importanece_ function you can use contained in every model after you train it . And if at the end of the day you are still confused to which model to use i would suggest use all of them stack them and you will have something better.Making these decisions solely on the basis of statistics is tough because statistics puts everything into an abstract form that is a whole dataset being represented by 1 number thus it is always prone to certain errors thus i always prefer to combine my data science experiments with my statistical results and then only conclude about my results.Also while using statistical analysis a lot of things depend on your dataset i.e. is it big enough to draw meaningful conclusions out of it what might be the sampling error and many more things .Hope this helps.


How to optimize for time correlated hidden function - the magical candy machine Let's assume that we have this magical candy machine which takes candies as input and delivers again candies as output.

For any given time t, it picks a random function which is strictly
increasing up to a point such as f(2) = 6 here, and then it strictly
decreases. It likes to be challenging but if you become greedy it
punishes you like most of the stuff in life.

  f(1) = 5

  f(2) = 6

  f(3) = 4

  f(4) = 2

  f(100) = 0

The tricky point is that this function is changing all the time, but
still highly correlated with time. So f() will be similar between
t(1) and t(2), but very different between t(1) and t(100).

I want to write a program to maximize my candies using this magical candy machine. I know the fundamentals of ML but I'm not sure which approach would fit best here. Any ideas?

Note: You can only play only once every minute.

Thought #1

From the short description you give, and assuming you have some data (or can synthesise it), I would have a go at training a Hidden Markov Model. Have a look here for an awesome visual primer. Intuitively, HMMs do what you describe.

There are some observables:

how much candy you put in the machine,
how much comes out, and
which timestep we are in

There are certain states (here just discrete states):

amount of dispensed candy goes up (strictly increasing)
amount of dispensed candy goes down (strictly decreasing)

Lastly, there is a transition: between these state, given the inputs. These are the observables, as far as you are concerned.

With HMMs, we are however assuming that there is more than meets the eye; there are some hiddens states (latent states), which are unobservable to use. The model keeps track of these and uses them in conjunction with the observables list above to decide how to act - it has the full mapping function. Below is a schematic diagram of such a model:

This just shows several states, along with the probabilities of transitioning to the other states, or stating in the same state. Check out the source for a detailed description of the diagram. One can already imagine sketching this drawing for your problem.

For more on the theory of HMMs, I recommend this text based walkthrough by Jeff Bilmes, and this YouTube series by mathematicalmonk (chapter 14 of the playlist). If you go this route, you might consider trying out the Pomegranate library (in Python).

The final logic of deciding what to do, what action to take, is something you could either hard-code or use a model to take it for you. For example, a hard-coded approach would be something like:

if current_state == 'normal':
    take_lots_of_candy()
elif current_state == 'unsure':
    take_some_candy()
elif current_State == 'dangerous':
    pause_taking_candy()
else:
    catch_other_states()

For a model-based approach, you could expand further on the ideas outlines below.

Thought #2

Of course, you are in essence just mapping some input to some outputs. Almost any machine learning algorithm will have a good go at it and - depending on the complexity of the internal workings of your magical machine - will be better or worse. Algorithms such as boosting (e.g. via gradient descent) generally require you to put some kind of information into the model e.g. by specifying a regression equation, mapping inputs to outputs. A good implementation is the mboost package in R. There is a great tutorial to help decide what might be important.

Thought #3

Coming back to the point of just mapping inputs to outputs: a simple Multi-Layer Perceptron (MLP) from Scikit-Learn - a.k.a. feed-forward neural network - should theoretically offer the flexibility to approximate any function you might throw at it (with a few assumptions).
You could also implement this using one of the various deep learning frameworks, should your MLP just not cut the mustard.

Final thoughts

I would lastly like to highlight that your final model will only likely be as good as the data you feed it, and that while it may work well on your synthesised training data, there are no guarantees of generalisation, without putting more thought into the specific problem and specific distributions from which your data are sampled.

If you go the way of Markov models, you could perhaps then use their state predictions as inputs to further models; thus creating an ensemble.

If you want to get really fancy and more towards current state-of-the-art models in sequence analysis, you could venture into the world of Recurrent Neural Networks, e.g. utilising LSTM cells, which maintain an internal state representation, including new information and selectively discarding old information. This might be a good way to take all your points into consideration in one large model. The catch here would be that you generally require vast amounts of data to train such a model.


deriving the gradient of batch normalization I'm trying to figure out the gradient of batch norm wrt x for backprop, but I get stuck in what I will call 'the triangle of (gradient) death'.

I present to you the triangle of death (in red), in the context of a computation graph for the batch norm equation:

The problem is that applying chain rule to the triangle results in 1 - 1 = 0, which kills the gradient.

My guess is I am messing up one or more of:

the derivatives.
the relationships I think the graph implies with respect to the nodes.
the graph representation I chose for BN as a composition of functions.

but I'm not sure where my error(s) is/are.

Your derivative computation is correct, so I think your understanding of what BN does is slightly off. If you want a more thorough proof that your computation graph is correct, you can backpropagate from $\bar{x} = x-\mu$ using the partial derivatives with respect to each input in the batch, i.e. $\frac{\partial\bar{x}_i}{\partial x_j}$ from $\bar{x}_i = x_i - \mu$. There are two cases to calculate: when $i=j$:

$$
\frac{\partial\bar{x}_i}{\partial x_i} = \frac{\partial x_i}{\partial x_i} - \frac{\partial\mu}{\partial x_i} = 1 - \frac{1}{B}
$$

And when $i \neq j$:

$$
\frac{\partial\bar{x}_i}{\partial x_j} = \frac{\partial x_i}{\partial x_j} - \frac{\partial\mu}{\partial x_j} = -\frac{1}{B}
$$

of which there are $B-1$ cases. Therefore:

$$\begin{align}
\frac{\partial\bar{x}}{\partial x} &= \sum_i^B \sum_j^B \frac{\partial\bar{x}_i}{\partial x_j} \\
 &= \sum_i^B 1 - \frac{1}{B} - \frac{B-1}{B} \\
 &= \sum_i^B 0 \\
\frac{\partial\bar{x}}{\partial x} &= 0
\end{align}$$

Note also that $\frac{\partial\mu}{x}=1$, as you also correctly pointed out. What this means is that $\frac{\partial\text{BN}}{\partial x}$ is entirely dependent on the demeaned inputs ($\bar{x}=x-\mu$) and not the original inputs themselves; more importantly, backpropagation through a BN layer is invariant to the scale of the inputs. This directly results in some of the observed benefits of using BN, such as a smaller dependence on the initialization scheme and easier/faster training.


Comparative Analysis of two sets of data I have 2 sets of data which consist of marks of 60 students in a particular subject in year 1 and year 2 respectively.

Year 1 : 86, 76, 87, 67, 89, 95.... so on (60 students)

Year 2 : 82, 67, 99, 77, 65, 78.... so on (60 students)

I want to compare these two sets of marks and find if year 2 students performed better than year 1  students.

What kind of techniques are appropriate for this problem? I was told that the statistical techniques like ANOVA, hypothesis testing techniques are too common for this problem and I was asked to try something with Machine learning.

Does machine learning fit in the picture? If yes, what kind of algorithms could be used?

No, this is the classical application of statistical test. You should consider the problem in the scope of two-sample test. Machine learning does not fit in the picture.


anaconda navigator not start direct without source command when i run following command
 source ~/anaconda3/bin/activate root
it is work fine. I don't want to run this command every time. give me any solution.

You can write script (.sh) for that.

STARTPATH=$(pwd)
cd ~/anaconda3/bin
case $1 in
    -j|--jupyter)
        shift
        source activate
        jupyter-lab
        ;;
    -n|--navigator)
        shift
        source activate
        anaconda-navigator
        ;;
esac
source deactivate
cd $STARTPATH

Then you can run the script by ./ or you can put the script in /usr/local/bin and make it executable chmod +x *yourfile*.sh.

After that you should be able to type in your console: *yourfile* -n and this will launch anaconda navigator.


What are the benefits of having ML in js? What are the benefits of having ML in JavaScript I.e. the deeplearn.js (now tensorflow) stuff, as opposed to implementing the ML steps in a python backend?

There are a lot of services that offer free or very cheap hosting of static websites. If you are able to implement your ML model in JS this allows you to deploy your product/app/whatever easily and with low cost. In comparison, requiring a backend server running your model is harder to setup and maintain, in addition to costing more.


Data Engineering Stack - collect, transform and visualize geospatial data I'm making a side project, where I collect geospatial data by web scrapping and from OSM API. I've started with simple Java application, however, I would like to make it as a data flow, purely for learning purposes.

Unfortunately, my knowledge about tools, and mostly connecting them, is, well, low.

What is my goal?
As a final result I want to visualize scrapped geospatial points on the map with the roads connecting them(from OSM).

Current flow:
In standalone Java application I'm scrapping the data for geospatial points. There is a client consuming the OSM API for needed data.

What I think it might be useful:
Use Apache Spark for collecting and transforming the data. Then use somehow GeoSpark, or Geotrellis, and Zeppelin to visualize the data. I was also thinking about using ES + Kibana for geodata, but it looks like the Zeppelin is enough.
I feel comfortable to work with Java, then Scala.

What do you think? Are there any better tools I can use? Did I miss anything?

You can do this, waaay to easier to what you are currently doing.

For the data scraping, use whatever makes you happy. In my case I will use Uipath or just python, depending on the complexity. But this is up to you, you just want some dataset in a format that suits you.

Once you have your data, you want to visualize that. This is a classical data science task. I am from the python army so I will suggest going to python. If you are good with Java the transition will be like slicing hot butter.

There are some good libraries that will help you greatly.
Here I suggest some packages that I have used in the past and they will help you greatly.

Folium
Geoplot
Intro to folium (again)

In my case, I will probably stick to Foliumm since there is plenty of code to re-use on the internet and it is a piece of cake.

This is my personal opinion on what I will do. There might be other tools and languages...


How does transform work? I was looking at the source codes of MinMaxScaler on Github. I know that when you fit a preprocessing class to a dataset, it takes the data and prepares it for transformation.
Let's say, I fitted MinMaxScaler to X_train and transformed it. But how does transform work when I use another dataset, let's say X_test? When you call transform(), does it replace the datasets in the use?

Models are trained on training data, and evaluated on test data based on assumption that unseen test data also comes from the same distribution with training data. So when you calculated statistics for training data, based on assumption that test data also comes from the same distribution you should apply same transformations to test data. You should fit MinMaxScaler to your training data, and then use this scaler to transform both tranining data and test data. There are also issues about data leakage, take a look at that: StandardScaler before and after splitting data
For transformations, fit method extract relevant statistics(min, max value for min-max scaling, mean, std for standardization) from the provided data, and transform method transforms each feature individually based on extracted statistics.


Finding the perfect algorithm for realtime optimizing of content I am looking for an algorithm that allows me the following:

I have a webpage and I want to randomly show one "content" from a list of contents on that webpage depending on who (visitor) sees the webpage. I know my visitors' demographic features like age, gender, locale. I assume that visitors with similar demographics have a similar taste regarding content on my website. I also know that they liked my content when they share it in the end.

What I have:

A list of available contents, let's say: red, green, blue, purple,
violet
A constant stream of events of visitors with specific
demographic data that share a content

What I want:

First of all, all contents should be displayed completely randomly.
Each user should randomly get one content without any preference
As soon as the first user with a specific demographic shares the content
they got I want other visitors with similar demographic features that
end up on my webpage to see this specific content with a higher
probability.

So basically I want a self optimizing system that learns in realtime.

This sounds like a classic use for a contextual bandit solver.

In essence you can run a simple online model (pretty much any regression model, or even a simple classifier like logistic regression if your reward signal is binary success/fail such as in your case) that learns to associate your demographic data with expected reward from each possible action - for you the reward can simply be 1 for a share link created or 0 for no share link.

Whilst the model is learning, you select the next action according to predicted reward from the model. There are choices between different workable strategies. For instance you could use an $\epsilon$-greedy approach: Pick the action with maximum predicted expected reward (or randomly choose between shared maximum values), but sometimes - with probability $\epsilon$ - you choose random content. There are other approaches and options that you can discover by researching contextual bandits and the simpler multi-armed bandit problems.

As an example, you could use a logistic regression model to predict expected reward from user demographics, with one such model per possible action. For a version that picks evenly to start, but prefers items that have been shared more over time, you can use a Boltzmann distribution (also called Gibbs distribution) using the predicted rewards as the inverse "energies" for the actions, and lowering the temperature as you collect more data. You can also initialise the weights of your model to predict a small but optimistic positive reward to start with to encourage early exploration. Whenever a user views your page, you pick the action to take based on the predicted rewards, and afterwards take the user response (share or not share) as feedback to update the one model associated with that action.

In the above example, the logistic regression learning rate, temperature scheme and starting reward are hyper-parameters of your model, and you use them to trade off responsiveness to individual events versus long-term accuracy for selecting the best action.


Job Recommendation Engine My girlfriend has recently been struggling with finding a new job, so I thought I'd make a website to help her out. The basic idea is that she'll be shown a list of jobs, rate her interest, and then a hopefully interesting job will be emailed to her every day. So far I have the frontend worked out, but I'm struggling with the engine. I've looked into different kinds of recommendation engines, and it seems like a content based engine is the way to go since she will probably be the only user.
With that said, feature selection is problematic because job categorizations kind of unrealistic. For instance, I could have job categories like "administrative" or "analytical", but those would be hard to define, especially if I was analyzing the job description for certain keywords.

Sorry about the wall of text. My basic question is how should I go about selecting features for this job recommendation engine?

Thanks for the help.

The basic ways to make a recommendation engine are :

Link a profile to similar ones
Extract characteristics of a profile, and match them with characteristics of, here, jobs
A mix of the two solutions above

Either way, you need data, labels, characteristics extractors...


is it wrong to use average='weighted' when having only 2 classes? In the book 'Text Analytics with Python', the author provides model_evaluation_utils.py
In the code of the .py he does:
metrics.precision_score(true_labels, predicted_labels,average='weighted')

I have two questions regarding it:
1- per the documentation average='weighted' shall only be used when having more than 2 classes, right? Why is he using average='weighted' when using only 2 classes?
2- Why do I get different results when I run?
print('Recall:', metrics.recall_score(test_sentiments,predicted_sentiments,pos_label='positive'))
print('Recall:', metrics.recall_score(test_sentiments,predicted_sentiments,average='weighted'))

(I only have 2 classes in the data)

ps: I think that by using average='weighted' he is giving the wrong result, because the code doesn't know which one is the positive class, here is a like to the code of model_evaluation_utils.py https://github.com/dipanjanS/practical-machine-learning-with-python/blob/master/notebooks/Ch05_Building_Tuning_and_Deploying_Models/model_evaluation_utils.py

As you already know, a precision score (or recall, or f-score) is for a single class, and in the function the argument pos_label says which class.
Now I'm going to guess that when pos_label is not provided and instead average is provided the function probably calculates the metric for every class and then returns the average of these values.
A weighted average can be calculated with any number of classes, and since no pre-defined weights are provided we can reasonably assume that the function takes the proportion of the two classes as weights. So the result of:
metrics.recall_score(test_sentiments,predicted_sentiments,average='weighted')

is probably the weigthed average (by proportion of instances) of:
metrics.recall_score(test_sentiments,predicted_sentiments,pos_label='positive')
metrics.recall_score(test_sentiments,predicted_sentiments,pos_label='negative')

If I'm not mistaken, this is equivalent to the micro-average recall.


Pandas to_sql insert only some columns I am new using pandas.
I am asking about how to insert specific columns using to_sql.

This is my explanation.

I have a database with a table datasiswa with columns:

id: as int, with autoincrement, as primary key
name: string
age: string

And I have an excel file with header name and age.

How to use to_sql to insert in fields name and age only, without ids.

You'll need to perform two steps:

You get ride of the id column in the pandas dataframe:

df=df.drop(columns=['id'])

Add the index=False param: to the .to_sql() method:

selected.to_sql('dataiswa', engine, index=False)


Using Tensorflow model for prediction I am new to tensorflow. I have manged to train and validate a CNN, saved the session through the Saver object into a CPKT file and loaded it back. Now I'd like to use the trained model in order to check how it performs against a photo I took myself. I can't figure out how to do it.

Look at this blog. Mainly you have saved operations as a part of your computational graph. So after you load your model, you can restore the session and call the predict operation that you created for training and validating your data, and run it on the new data hy feeding into the feed_dict. Make sure it is in the same format and same shape as your training data. I am adding a small snippet of code, maybe that will help.

checkpoint_file=tf.train.latest_checkpoint(checkpoint_directory)
graph=tf.Graph()

 with graph.as_default():
     session_conf = tf.ConfigProto(allow_safe_placement=True, log_device_placement =False)
     sess = tf.Session(config = session_conf)
     with sess.as_default():
          saver =     tf.train.import_meta_graph("{}.meta".format(checkpoint_file))
          saver.restore(sess,checkpoint_file)
          input = graph.get_operation_by_name("input").outputs[0]
          prediction=graph.get_operation_by_name("prediction").outputs[0]
          #newdata=put your data here
          print sess.run(prediction,feed_dict={input:newdata})

To use operation_by_name, you will have to name the operations and variables in the original model. Secondly, in the feed dict you will have to give all the parameters needed to perform the operation(prediction here), so if you are using dropout you will have to give that too. There might be more simpler ways of doing this. This is the way I did it.


Using SMOTENC in a pipeline I am trying to figure out the appropriate way to build a pipeline to train a model which includes using the SMOTENC algorithm:

Given that the N-Nearest Neighbors algorithm and Euclidian distance are used, should the data by normalized (Scale input vectors individually to unit norm). Prior to applying SMOTENC in the pipeline?

Can the algorithm handle missing values? If data imputation and outlier removal based on median and percentiles values are performed prior to SMOTENC rather than after it, wouldn’t this bias the imputation/percentiles?

Can SMOTENC be applied after one-hot encoding and defining the numerical binary columns as categorical features?

When the pipeline is included in a cross validation schema, will the data balancing only be applied to the imbalanced training fold or also for the test fold?

Here is how my pipeline currently looks like:
from imblearn.pipeline import Pipeline as Pipeline_imb
from imblearn.over_sampling import SMOTENC

categorical_features_bool = [True, True, ……. False, False]
smt = SMOTENC(categorical_features =categorical_features_bool,
                random_state=RANDOM_STATE_GRID,
                k_neighbors=10
                ,n_jobs=-1
                     )

preprocess_pipeline = ColumnTransformer(
        transformers=[
            ('Winsorize', FunctionTransformer(winsorize, validate=False,
                                              kw_args={'limits':[0, 0.02],'inplace':False,'axis':0}),
             ['feat_1,'Feat_2']),

            ('num_impute', SimpleImputer(strategy='median', add_indicator=True) ,
             ['feat_10,'Feat_15']),
        ], remainder='passthrough', #passthough features not listed
        n_jobs=-1,
        verbose = False
    )

Model = LogisticRegression()

model_pipeline = Pipeline_imb([
            ('preprocessing', preprocess_pipeline),
            ('smt', smt),
            ('Std', StandardScaler()),
            ('classifier', Model)
            ])

The usual normalisation for Euclidian distance is NOT to scale each input to unit length, but to scale each column to mean 0 and variance 1. The scaling of each data is possible but it is not common.

I dont know

The whole point of SMOTENC is not to do the one-hot encoding. One hot encoding is a way to transform categorical data into numeric data (on multiple dimensions) for algorithms that cannot deal with categorical data. SO my suggestion is not to convert the categorical columns and let SMOTENC deal with them

The Pipeline in imblearn does the right thing - it only applies the oversampling (of other imbalance strategies) on the training set not on the test set. See this question in StackOverflow: https://stackoverflow.com/questions/63520908/does-imblearn-pipeline-turn-off-sampling-for-testing


Data Augmentation recommended pipeline I want to perform image classification using Keras and a dataset made of 50 classes. At the moment, I have only 7 images per class and I need to perform data augmentation in order to train the model and obtain acceptable accuracy values.

I am using the ImageDataGenerator class from keras which is recommended for image augmentation on the fly (during training). Since the classification is performing badly, I was wondering if it would be necessary to perform offline augmentation, i.e, enlarge the dataset before the training, because I honestly think that 7 is far from being a reasonable number of images per class.

Is it a common practice to perform both types of augmentation (before and during the training)? I am planning to use some 3rd party software or tools like imgaug to enlarge the dataset first and save the augmented images to disk and only then perform real-time augmentation with ImageDataGenerator class.

In conclusion, the flow would be similar to this:

Image pre-processing and offline data augmentation => enlarge the original dataset
Training with real-time augmentation => Load the dataset and use ImageDataGenerator

What do you think?

Thank you.

Out of the two pipelines you mentioned, I'd recommend the second (i.e. real-time augmentation). This is better than the first, because by performing random augmentations the network sees different images at each epoch.

I'd recommend imgaug, which is a python library for performing data augmentation. I've found it very helpful as it can work with keras' ImageDataGenerator very well. The way can do this is:

from imgaug import augmenters as iaa

seq = iaa.Sequential([...])  # list of desired augmentors

ig = ImageDataGenerator(preprocessing_function=seq.augment_image)  # pass this as the preprocessing function

gen = ig.flow_from_directory(data_dir)  # nothing else changes with the generator

One final note I'd like to make is that $7 \cdot 50 = 350$ images are very few for deep learning, even with augmentation. Make sure you use a pre-trained network or else you will have a serious overfitting problem.


Clustering combining numeric features and weekday & hour cyclic features The question is strictly related to
What is a good way to transform Cyclic Ordinal attributes?
and
Ways to deal with longitude/latitude feature

They presented a very clear answer about the approach to normalise hour variable and latlong variables.

The addition of this question is about how to approach normalization having both weekday and hour features. I wonder this because they are different granular level of the same information, hence, strictly related.

The objective is to run a k-means algorithm combining the normalised weekday and hour features with other traditional numeric features.

The essential transformation in @AN6U5's answer ist done in the following lines:

  df['hourfloat']=df.hour+df.minute/60.0
  df['x']=np.sin(2.np.pidf.hourfloat/24.)
  df['y']=np.cos(2.np.pidf.hourfloat/24.)

in the first line he transforms minutes into hours by dividing them by 60
so for example 20 Minutes are converted to 0.3333 hours

After that, in line 2 and 3 he converts this float number from polar coordinates to cartesian coordinates (https://en.wikipedia.org/wiki/Polar_coordinate_system)

So chaning this from hour to weekday you just need to adapt the first line.
Imagining a a clock where 00:00 is Monday, followed by Tuesday (clockwise), and so on ... you need to convert hour into weekday (for simplicity I assume weekday has the values 0-7). So first you divide your hour by 24 which transforms it into days and then further you divide it again by 7 which gives you a float number in weeks. Then add your weekday to the hour and then proceed with line 2 and 3 just as given, except that you correct the 24 to 7.

As a formula: hour/(24*7)+weekday = weekfloat
I haven't tried it out by myself but I think this should do it.

Alternatively, when you have two cyclic features you could transform weekday and hour into spherical coordinates. This would leave you with three coordinates, x,y and z but also preserves the 'closeness' within one feature itself.


Tool to Label Images for Supervised Classification I have a couple thousand photos of whales taken from drones and I'm planning to build a simple binary classifier to run on these and future images to see if they contain a whale. I'd like to label specific pixels within the image as whale (1) or not whale (0) and I'm at a loss for a good free tool to do this. I'm imagining something where I can just zoom in and manually draw an outline around a whale but I'm open to other options. Does anyone know of a good way to do this?

Thanks!

If you are specifically looking to outline the whales, seems like FastAnnotationTool could work:
https://github.com/christopher5106/FastAnnotationTool

Other options here:
https://en.wikipedia.org/wiki/List_of_Manual_Image_Annotation_Tools

When had to annotate many images for a project, I built a fairly simple MATLAB gui that displayed images.  I cycled through each image and if I clicked on a point on the image, that image was annotated and the point where I clicked was saved in a corresponding .mat file (.csv would have been better). It worked... mostly.  I'd guess between python and OpenCV, this could be done easily without MATLAB.


Find matching text from a text column This is my first time to use Data Analytics tool to figure out a solution to a problem.
I have a table with following columns Person ID, Person Name, Note ID, Note (notes is a free form text where a call representative can enter their comments) Insert User. I have been given list of key phrases to be identified from the Note column. These key phrases are in a sentence format.
Example:
I have to find exact or a similar matching phrase from those notes. The format of my final report would look something like this

I looked into couple of articles related to text matching which suggested options such as

fuzzywuzzy, Doc2vec, Difflib, python-levenshtein

It is all so confusing. Even if I get a starter to pick the best suitable option, I can may be take it from there. Any suggestions?
Thank you so much!

The problem you are suggesting is a text summarization problem. It can be of two types abstractive - understanding the text distribution and then producing a summary, extractive - extracting specific words from the text to produce a summary. Clearly, your problem falls in the latter category. Also, since you have the labels to be generated it is a supervised text summarization problem. Take a look at this paper regarding algorithms for the same. It suggests two solutions for a similar problem, you'll of course have to modify them a bit and see which one works best for you. Good luck.


Methodology for driving score(behavior) I am an intern at mobility data company and a Master's candidate in Statistics. I am researching about driving score which is based on a driver's driving habit.
We have trip data which contains the distance, quick acceleration(seconds), quick stop, steering angles, and so on.
I have read some related articles and papers but some of them contain the skills that I cannot handle such as genetic programming.

I would like to know which ML skills could be used for this unsupervised learning problem(maybe?). Below is an example of our data set.

Driver ID | Trip time | Distance | Harsh acceleration | Quick Stop | ...

1              60 mins     1 mile       180 seconds         7 times  ...

2             30 mins      0.3 mile    10 seconds           2 times ...

My goal is making a driving score based on the dataset for each driver. The scale does not matter. It could be 0-100 scale or classifiers such as poor, bad, normal, good, perfect.
The problem I'm struggling with is that I have to create the target value(driving score). I guess that unsupervised learning could be a hint but I am not pretty sure about it.
I welcome any kind of advice or source! Thank you so much in advance.

Since you do not have labelled data, you'll have to use a lot of domain knowledge. Also, predicting a class (poor, bad, normal, good, perfect) might be easier than predicting a score (0-100). The approach you could use is:

Apply clustering on your dataset.
Try to find the optimal number of clusters for your dataset using the Elbow Method. Suppose you get 3 optimal clusters.
Now comes the part for applying domain knowledge. Try to analyse the clusters by picking up individual examples from the cluster and using domain knowledge identify which category (here, good, bad, average) you can classify them into.
After analysing some examples from the clusters, you can assign the label you get to the entire clusters as the examples within a cluster are similar.

The above method makes some assumptions but can be an interesting experiment for your dataset.


Pretrained handwritten OCR model I've been looking around for pretrained models dedicated to handwritten OCR. So far I've found very little. Could you please share, if you know any?
I find tesseract hard to parse anything that isn't arial and perfectly captured.

Discover open-source deep learning code and pretrained models at Model Zoo

These are pre-trained sources available in the Github.

Handwritten Text Recognition with TensorFlow

More
Handwriting OCR
Handwritten Text Recognition (OCR) with MXNet Gluon

Some Helpful Resources:

Handwriting recognition and language modeling with MXNet Gluon
Handwriting OCR: Line segmentation with Gluon


Is this Neo4j comparison to RDBMS execution time correct? Background: Following is from the book Graph Databases, which covers a performance test mentioned in the book Neo4j in Action:

  Relationships in a graph naturally form paths. Querying, or
  traversing, the graph involves following paths. Because of the
  fundamentally path-oriented nature of the datamodel, the majority of
  path-based graph database operations are highly aligned with the way
  in which the data is laid out, making them extremely efficient. In
  their book Neo4j in Action, Partner and Vukotic perform an experiment
  using a relational store and Neo4j.

  The comparison shows that the graph database is substantially quicker
  for connected data than a relational store.Partner and Vukotic’s
  experiment seeks to find friends-of-friends in a social network, to a
  maximum depth of five. Given any two persons chosen at random, is
  there a path that connects them which is at most five relationships
  long? For a social network containing 1,000,000 people, each with
  approximately 50 friends, the results strongly suggest that graph
  databases are the best choice for connected data, as we see in Table
  2-1.

  Table 2-1. Finding extended friends in a relational database versus efficient finding in Neo4j

Depth   RDBMS Execution time (s)    Neo4j Execution time (s)     Records returned
2       0.016                       0.01                         ~2500
3       30.267                      0.168                        ~110,000
4       1543.505                    1.359                        ~600,000
5       Unfinished                  2.132                        ~800,000

  At depth two (friends-of-friends) both the relational database and the graph database perform well enough for us to consider using them in an online system. While the Neo4j query runs in two-thirds the time of the relational one, an end-user would barely notice the the difference in milliseconds between the two. By the time we reach depth three (friend-of-friend-of-friend), however, it’s clear that the relational database can no longer deal with the query in a reasonable timeframe: the thirty seconds it takes to complete would be completely unacceptable for an online system. In contrast, Neo4j’s response time remains relatively flat: just a fraction of a second to perform the query—definitely quick enough for an online system.

  At depth four the relational database exhibits crippling latency,
  making it practically useless for an online system. Neo4j’s timings
  have deteriorated a little too, but the latency here is at the
  periphery of being acceptable for a responsive online system. Finally,
  at depth five, the relational database simply takes too long to
  complete the query. Neo4j, in contrast, returns a result in around two
  seconds. At depth five, it transpires almost the entire network is our
  friend: for many real-world use cases, we’d likely trim the results,
  and the timings.

Questions are:

Is this a reasonable test to emulate what one might except to find in a social network? (Meaning do real social networks normally have nodes with approximately 50 friends for example; seems like the "rich get richer" model would be more natural for social networks, though might be wrong.)
Regardless of the naturalness of the emulation, is there any reason to believe the results are off, or unreproducible?

Looking at this document called Anatomy of Facebook I note that the median is 100. Looking at the cumulative function plot I can bet that the average is higher, near 200. So 50 seems to not be the best number here. However I think that this is not the main issue here.

The main issue is the lack of information on how the database was used.

It seems reasonable that a data storage designed specially for graph structures to be more efficient than traditional RDBMs. However, even if the RDBMs are not in the latest trends as a data storage of choice, these systems evolved continuously in a race with the data set dimensions. There are various types of possible designs, various ways of indexing data, improvements related with concurrency and so on.

To conclude I think that regarding reproducibility, the study lack a proper description of how the database schema was designed. I do not expect that a database to dominate on such king of interrogations, however I would expect that with a well-tuned design the differences to not be such massive.


Train classifier on synthetic images to recognise real images I am trying to train a classifier (let's say to classify an object X or not X). But I don't have too much real images of object X which I want to classify. So I made some synthetic images of my own using some photo editing tools, but the problem is after feeding synthetic data my classifier doesn't recognise the real object. And since I had like 2-3 images of real object so I can't feed it more real data.
Is there any solution on how to approach such problem with synthetic data to recognise real ones?

Neural networks require a lot of data for training. This is one of their largest drawbacks, and is very hard to get around this.

Using artificial data is not a solution. We don't really understand what the network is learning and we can't tell if it learned anything that is particular to the synthetic images.

What you can do is to use some sort of data augmentation. Rotate, scale, translate your images many times, and create a larger dataset.

Since it's such a small dataset, and if more data is really not possible to obtain, you should consider other approaches to help the classification. Either manually design some features to help a bit the classifier or do the classification with more traditional methods.


Plot of two different matrices in R I'm trying to plot two different size matrices using one graph (in R), but can't manage to do so.
I tried using matplot and the regular plot, but it didn't work.

Does anyone know how to plot it?

(Assuming this is a similar question to what was posted on Cross Validated, but was closed): You can merge the data from both matrices while adding a variable that specifies the origin (group), then plot them together in ggplot2:

library( ggplot2 )

N1 = 19
N2 = 17
M  = 10

m1 = matrix( rnorm(N1*M,mean=0,sd=1), N1, M)
m2 = matrix( rnorm(N2*M,mean=0,sd=1), N2, M)

y     = c( as.vector( t( m1 ) ),  as.vector( t( m2 ) )  )
x     = c( rep(1:10, each = N1 ), rep(1:10, each = N2 ) )
group = c( rep( '1', N1 * M ),    rep( '2', N2 * M )    )

df = data.frame( state = x, value = y, group = group )

ggplot( df, aes( x = state, y = value, colour = group ) ) +
    geom_point() +
    ggtitle( "State values in group 1 and 2" ) +
    labs( x = "State", y = "Value" ) +
    scale_x_continuous( breaks = seq(10) )

This is the result:


What is the convergence criteria of a semi-supervised learning algorithm? I would like to know when to stop doing semi supervision? For example, if I learn a classifier from a small dataset and then use it to label a pool of unlabelled dataset. I then use the newly labelled data to train my classifier again. How long should this process be continued?

Most of the semi-supervised methods are heuristics and more or less are modifications of the standard supervised learning algorithms, where you are trying to take into account unlabeled data considering a small dataset of labeled data. If the data from these two datasets do not follow the same distribution, then you have to look for transfer learning methods.

I guess that you are asking for a stopping criterion for iterative methods/approaches of semi-supervised learning, otherwise you do not need a stopping criterion. It depends on the approach. You could define a threshold that reflects the confidence that the unlabeled data could be labeled correctly and use them as pseudo-labeled instances with the ones that are already labeled for training an accurate classifier. Another option could be to define a maximum number of iterations.


Directly proportional Trend in training and cross validation curves In continuation with a question already asked, I tried the same curve on a different dataset I found. My model is a simple Logistic regression curve with OnevsRest Classifier. But the graph I obtained this time was different.

What could be a possible reason for this? As explained in the other answer, if it takes less data to fit better, shouldn't the accuracy decrease here as well?

These learning curves seem unusual - the training accuracy should start high and get lower as more samples are added.

I suspect that your data is sequential (it has some kind of time dependency). sklearn's learning_curve function does not seem to shuffle the data (should it?), so the training accuracy can change/increase once new structures appear in the data over time.

Here's a notebook trying to reproduce the effect:
https://gist.github.com/stmax82/79b744877b0a482f8739d372c4777e0d

Two images from the notebook:

The learning curves on the shuffled data look like that (as expected):

While the learning curves on the original / sequential data look like that (unusual because training accuracy is rising over time):

That's just my try at explaining your learning curves. It might be something completely different... try shuffling your data to find out.


Machine Learning Models for Non-Imaging Medical Clinical Data I am a beginner in machine learning. Since machine learning is a wide area, I would like keep my focus on clinical data.

When I search for machine learning samples in healthcare, most of the examples are related to interpreting medical images like retina image recognizing retinopathy. What are the clinical relationships modeled in ML?

Some of the non-imaging examples are

ECG (https://www.researchgate.net/publication/324457712_A_deep_learning_approach_for_ECG-based_heartbeat_classification_for_arrhythmia_detection   ,  https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5805987/)
Arrhythmia (https://archive.ics.uci.edu/ml/datasets/arrhythmia , http://ecg.mit.edu/time-series/)
Diabetes  https://www.sciencedirect.com/science/article/pii/S2405959518304624
Genotyping : https://www.mdpi.com/2073-4425/10/1/18/pdf , https://www.biorxiv.org/content/10.1101/241414v1