Is the PNY NVIDIA Quadro RTX 4000 a good GPU for Machine Learning on Linux? As a web developer, I am growing increasingly interested in data science/machine learning, enough that I have decided to build a lab at home.

I have discovered the Quadro RTX 4000, and am wondering how well it would run ML frameworks on Ubuntu Linux. Are the correct drivers available on Linux so that this card can take advantage of ML frameworks?

LINUX X64 (AMD64/EM64T) DISPLAY DRIVER

This is the only driver that I could find, but it is a "Display Driver", so I am not sure if that enables ML frameworks to use this GPU for acceleration. Will it work for Intel based processors?

Any guidance would be greatly appreciated.

You seem to be looking at the latest Quatro 4000, which has the following compute rating:

You can find the complete list here for all Nvidia GPUs.

While it seems to have an impressive score of 7.5 (the same as the RTX 20180ti), the main draw back the memory of 8Gb. This is definitely enough to get started with ML/DL and will allow you to do many things. However, memory is often the thing that will slow you down and limit your models.

The reason is that a large model will require large number of parameters. Take a look at the following table (models included in Keras), where you can see the number of parameters each model requires:

The issue is that the more parameters you have, the more memory you need and so the smaller the batch size you are able to use during training. There are many arguments for larger vs. smaller batch sizes - but having less memory will force you to still to smaller batch sizes when using large models.

It seems from Nvidia's marketing, that the Quadro product line is more aimed towards creative developers (films/image editing etc.), whereas the Geforce collection is for gaming an AI. This highlights that Quadro is not necessarily optimised for fast computation.


Filling in missing values Suppose I have a train and test data set and both of them contain missing values. Can I join both of them and then do fillna.(alldata.mean()) or will that cause data leakage. What would be a better way because doing them separately would have different mean for same column.

This is a controversial subject with no clear best answer and myriad options, some of which are model-specific. You can drop them, replace them with extreme values, interpolate, replace with median, impute with nearest neighbors, etc. In all cases besides dropping them altogether, you make assumptions and create data where no data exists.  I can't tell you the best approach, but I can advise you to try multiple approaches and hope the end results are robust to choice.


High RMSE and MAE and low MAPE I have used a few regression models on the same dataset and obtained error metrics for them as shown below,

The RMSE(Root Mean Squared Error) and MAE(Mean Absolute Error) for model A is lower than that of model B where the R2 score is higher in model A. According to my knowledge this means that model A provides better predictions than model B. But when considering the MAPE (Mean Absolute Percentage Error) model B seems to have a lower value than model A. I would really appreciate it if someone could explain why it is so. Thanks in advance.

The reason is the wider range of your output variable. Consider the following two cases,

Real value was 99, prediction is 101
Real value was 5520, prediction is 5522

In both cases, the absolute error is 2, but relative error in first case is much larger (2% - 2/101) than second case (0.035% 2/5520). Absolute and relative metrics are measuring different aspects of the prediction. So one model is not better than the other in absolute sense (pun intended).

Which metric to value depends on your application. When the outcome range is wide (probably your case) and skewed, relative error measurements are better than absolute error measurements.


Use of Correlation Map in Machine Learning I would like to know the use of correaltion map in machine leraning. For example, if there are 2 features with high correaltion, should either of the features be removed before appying the algorithm or it depends on every data set. Any explanation would be highly helpful. Thanks in advance.

It depends. A high correlation between two features suggests that they represents almost the same information. For some problems like clustering, it is always useful to remove redundant features while some algorithm like Gradient Boosting in xgboost is not affected at all by such features. So, it all depends on what you want to do with your data set.

As per my opinion, if your dataset has too many features, then I would suggest to check the correlation between those features and apply PCA to reduce the dimensionality of your dataset especially if you are doing tasks like clustering or regression.


Upper bound on 'relatedness'? We have ~100 answers to a questionnaire with five questions (Q5). Independently from that, we have about 50, somewhat overlapping, features describing the people who answers the questions (F50). After having thrown an impressive amount of 'black box' regression models at trying to predict any of the 5 answers from the 50 features, we are approaching the conclusion that the features are just completely orthogonal to the topic of the questionnaire.

This is interesting, and a little surprising, and it could be fun to try to 'prove'. Does anyone know of a measure or method for which we could argue that if

'X does not produce any predictive value in Q5 when applied to F50'

then

'the causal relationship between F50 and Q5 is weaker than C'

Could some flavour of multivariate mutual information be a way forward?

I hope the question makes sense. It seems like it would be generally interesting.

Very interesting question, it's always hard to prove a negative.
I have a vague idea but I really don't know if it's worth anything or even applicable to this problem, so please take it with a grain of salt!

The idea is to use randomness and multiple samples in order to compare the result of predicting from random noise vs. actual data X: if the result from X isn't significantly better than from random noise, then you have proved that X doesn't have any predictive power. Of course this relies on the assumption that the model used to predict is sensible enough.

I have seen methods using this idea but I don't remember the details unfortunately. The only directions I can give are these:

the impostor method for authorship verification, which is based on a somewhat similar idea
I've seen some works using the binomial test to formally test whether the outcome from a method X is significantly different from a random baseline.

Sorry for the lack of details, I hope this helps.


BERT uses WordPiece, RoBERTa uses BPE In the original BERT paper, section 'A.2 Pre-training Procedure', it is mentioned:

The LM masking is applied after WordPiece tokenization with a uniform masking rate of 15%, and no special consideration given to partial word pieces.

And in the RoBERTa paper, section '4.4 Text Encoding' it is mentioned:

The original BERT implementation (Devlin et al., 2019) uses a
character-level BPE vocabulary of size 30K, which is learned after
preprocessing the input with heuristic tokenization rules.

I appreciate if someone can clarify why in the RoBERTa paper it is said that BERT uses BPE?

BPE and word pieces are fairly equivalent, with only minimal differences. In practical terms, their main difference is that BPE places the @@ at the end of tokens while wordpieces place the ## at the beginning.
Therefore, I understand that the authors of RoBERTa take the liberty of using BPE and wordpieces interchangeably.


Neural Networks: How to prepare real world data to detect low probability events? I have a real world data set of credit borrowers (50,000 records). The set contains categories such as Married, Single, Divorced, etc. as well as continuous data such as Income, Age, etc. Some records are incomplete or contain outliers. The dependent variable is Defaulted / Good (0,1). We are trying to train a neural network to predict defaults based on the training data. I have experience with neural networks and used them for sample data with great results, however, I never had to normalize noisy real world data.

Any thoughts what I should keep in mind in respect to:
 - How to normalize the categories. Can I assign an indexed number? Should I tried to stratify them?
- How to deal with missing data. Assign 0 ?
- How to deal with the fact that defaults are only about 5 % of the data set. What transfer function would be useful to predict these low probabilities.
- Basically any other real world data advice is very much appreciated.

Thanks in advance!

You bring up a number of good questions here Ans. I will do my best to cover each of them in turn.  It isn't an exhaustive treatment but hopefully it helps...

1. How to normalize the categories.

First, assess whether your categorical variables can be considered zero variance (e.g. all records possessing one category only) or near zero variance (vast majority of records belonging to very few categories). Create a basic frequency distribution to identify this.

While it doesn't matter as much in Neural contexts per se, it is a good idea to consider filtering low variance variables from your model. Just be careful as eliminating near zero variance variables may have you throwing out the baby with the bath water.

You've worked with Neural Nets before so you know that you need to convert categories to numeric values.  A good question to ask is whether a given categorical value is ordinal in nature (e.g. on a Likert scale of 1 to 5) and whether you want to preserve ordinality. This pushes you into an area such as that described by Pinto da Costa and Cardoso (https://www.researchgate.net/publication/221112186_Classification_of_Ordinal_Data_Using_Neural_Networks).

2. How to deal with missing data.

Assuming that you are talking about missing continuous values, you will want to impute these numeric values based collectively on the values that are present across the entire attribute.  There are a number of approaches to use here but something important to keep in mind is dispersion.  In a basic sense, if your attribute is skewed by outliers, you will want to steer clear of a mean-based calculation and go with a median-based approach.

3. How to deal with the fact that 'Defaults' are ~5% of the data.

This isn't an immediate concern given that you have 2500 examples of what denotes a 'Default' (in a 50,000 dataset). A sane approach here would be to ensure that you are using a K-fold cross validation scheme (say with 10 folds) to ensure that you are truly randomizing training vs test.  This will help protect you against overfitting.  Again, this is pretty high level guidance, but it is prudent.

Extra Credit:
I haven't gotten into other standard tactics such as normalization of your continuous attributes, but you may want to get up to speed there to better generalize your model (while also developing a deeper understanding of the dynamics at play in the data). This could point you to trying out other algos, etc.


LSTM model bad forecasts I tried to implement LSTM model for time-series prediction. Below is my trial code. This code runs without error.
metrics = ['mean_squared_error', 'mean_absolute_error', 'mean_absolute_percentage_error']

# define model
model = Sequential()
model.add(LSTM(100, activation='relu', input_shape=(n_past, 6), return_sequences=True))
model.add(LSTM(100, activation='relu', return_sequences=True))
model.add(LSTM(50, activation='relu'))
model.add(Dense(2, activation='relu'))
model.compile(optimizer='adam', loss='mse', metrics=metrics)
model.summary()

)
The forecast correctly predicts the peaks, but the constant values should be 0 and you cannot predict it.
However, prediction is extremely poor. How to improve the predictin? Do you have any ideas to improve it?
Any ideas for improving prediction by re-designing architecture and/or layers?

Your loss function estimates the conditional expectation and the predictions mainly hover around the mean of the time series. I would argue that predictions are good, but the problem is poorly designed. One approach would be to partition the problem into two subproblems: probability of your series being $\mathbb{P}(Y>0)$ and predicting the conditional expectation by training only on non-zero values $\mathbb{E}[Y|Y>0]$. The final prediction would then be $\mathbb{P}(Y>0)\mathbb{E}[Y|Y>0]$


Transform a skewed distribution into a Gaussian distribution I have a skewed distribution that looks like this:

How can I transform it to a Gaussian distribution? The values represent ranks, so modifying the values does not cause information loss as long as the order of values remains the same. I'm doing this to experiment if different distributions change the behavior of my ML models.
I'm working with Python/NumPy/Pandas/scikit-learn.
Edit: I should clarify that I have a lot of features and I'm looking to automatically transform all feature distributions. I was able to find a reasonable transformation for a single feature with a lot of experimentation, but it doesn't generalize to other features:
normalize(np.log(0.30 + original)).
** here would be image i.stack.imgur.com/uzorK.jpg but I don't have enough rep to post more than 2 images **
normalize(np.log(0.17 + another_feature_distribution)).

In this image the purple bars represent the original distribution of another feature, green bars represent the transformed distribution. No matter how much I tweak the constant, I don't get the high green bar on the left extreme to disappear. Also, I don't have time to manually find a formula for each feature. Not sure if these are bell-shaped enough anyway?

You can do a log transformation on your data with the help of numpy log functionality as shown below :

log_data = np.log(data)

This will transform the data into a normal distribution. Moreover, you can also try Box-Cox transformation which calculates the best power transformation of the data that reduces skewness although a simpler approach which can work in most cases would be applying the natural logarithm. More details about Box-Cox transformation can be found here and here


Please let me know if I am on the right track to being an NLP Expert I am a college student(rising senior) and became interested in Natural Language Processing last semester. I decided to focus on studying this area this summer and become skilled in this area. I wanted to get some advice for studying this particular subject.

Right now, I am taking Andrew Ng's Machine Learning course on Coursera to get a sense of how Machine Learning works. After finishing this course, I am planning to take Standford's CS224n NLP course on Youtube and do its class activities. I am assuming AWS and Tenserflow is also important since they are included as topics of CS224n.

I want to know if my summer plan sounds reasonable. If not, could you please give me an advice of how to make a better plan. If this sounds reasonable, it would be great if you can add more or specify which part is particularly important in these areas.

Here are a couple of general NLP resources:

How To Do Things With Words by Peter Norvig
NLP sections of Intro to Artificial Intelligence from Udacity
Neural Network Methods in Natural Language Processing by Yoav Goldberg & Graeme Hirst


Efficiently Sending Two Series to a Function For Strings with an application to String Matching (Dice Coefficient) I am using a Dice Coefficient based function to calculate the similarity of two strings:

def dice_coefficient(a,b):
    try:
        if not len(a) or not len(b): return 0.0
    except:
        return 0.0
    if a == b: return 1.0
    if len(a) == 1 or len(b) == 1: return 0.0
    a_bigram_list = [a[i:i+2] for i in range(len(a)-1)]
    b_bigram_list = [b[i:i+2] for i in range(len(b)-1)]
    a_bigram_list.sort()
    b_bigram_list.sort()
    lena = len(a_bigram_list)
    lenb = len(b_bigram_list)
    matches = i = j = 0
    while (i < lena and j < lenb):
        if a_bigram_list[i] == b_bigram_list[j]:
            matches += 2
            i += 1
            j += 1
        elif a_bigram_list[i] < b_bigram_list[j]:
            i += 1
        else:
            j += 1
    score = float(matches)/float(lena + lenb)
    return score

However, I am trying to evaluate the best match out of a large possible list, and i want to use list comprehension/map/vectorize the function calls for a whole series of strings to be matched to make this computationally efficient. However, I am having difficult getting the run time into a reasonable ballpark for even medium sized series (10K-100K elements).

I want to send two input series into/through the function, and then get the best possible match from all candidates on dflist1 against a second series: dflist2 . Ideally, but not necessarily, the return would be another series in the dflist1 dataframe return the best possible score also. I have an implementation of this working (below), but it's incredibly slow. Is it also possible to parrelelize this? I think this would be a hugely valueable problem to solve as it would perform the same function that reconcile csv currently does.

dflist1 = pd.read_csv('\\list1.csv', header = 0,encoding = "ISO-8859-1")
dflist2 = pd.read_csv('\\list2.csv', header = 0,error_bad_lines=False)
dflist1['Best Match'] = 'NA'
dflist1['Best Score'] = '0'
d = []
start = time.time()
for index, row in dflist1.iterrows():
    d=[dice_coefficient(dflist1['MasterList'][index],dflist2['TargetList'][indexx]) for indexx,rows in dflist2.itertuples()]
    dflist1['Best Match'][index]=dflist2['TargetList'][d.index(max(d))]
    dflist1['Best Score'][index]=max(d)
    print('Finished '+str(index)+' out of '+str(len(dflist1.index))+' matches after '+str(round(time.time() - start))+' seconds.')

Any help would be appreciated very much!

Your function does a lot of pythonic data crunching. In these cases numba can be useful.
In the below code I split your function into two: sorting and scoring. I then converted your bigrams from strings to integers (to comply with numba datatypes) and decorated the scoring subfunction with numba's @autojit.
from numba import autojit
import numpy as np

def dice_coefficient(a,b):
    try:
        if not len(a) or not len(b): return 0.0
    except:
        return 0.0
    if a == b: return 1.0
    if len(a) == 1 or len(b) == 1: return 0.0
    a_bigram_list = [a[i:i+2] for i in range(len(a)-1)]
    b_bigram_list = [b[i:i+2] for i in range(len(b)-1)]

    a_bigram_list.sort()
    b_bigram_list.sort()

    lena = len(a_bigram_list)
    lenb = len(b_bigram_list)
    matches = i = j = 0
    while (i < lena and j < lenb):
        if a_bigram_list[i] == b_bigram_list[j]:
            matches += 2
            i += 1
            j += 1
        elif a_bigram_list[i] < b_bigram_list[j]:
            i += 1
        else:
            j += 1
    score = float(matches)/float(lena + lenb)
    return score

def dice_coefficient_new(a,b):
    try:
        if not len(a) or not len(b): return 0.0
    except:
        return 0.0
    if a == b: return 1.0
    if len(a) == 1 or len(b) == 1: return 0.0

    a_bigram_list, b_bigram_list = dice_coefficient_sorting(a,b)
    score = dice_coefficient_scoring(a_bigram_list,b_bigram_list)

    return score

def dice_coefficient_sorting(a,b):

    a = np.array([ord(i) for i in a])
    b = np.array([ord(i) for i in b])

    a_bigram_list = 256*a[:-1]+a[1:]
    b_bigram_list = 256*b[:-1]+b[1:]

    a_bigram_list.sort()
    b_bigram_list.sort()

    return a_bigram_list,b_bigram_list

@autojit(nopython=True)
def dice_coefficient_scoring(a_bigram_list,b_bigram_list):

    lena = len(a_bigram_list)
    lenb = len(b_bigram_list)
    matches = i = j = 0
    while (i < lena and j < lenb):
        if a_bigram_list[i] == b_bigram_list[j]:
            matches += 2
            i += 1
            j += 1
        elif a_bigram_list[i] < b_bigram_list[j]:
            i += 1
        else:
            j += 1
    score = float(matches)/float(lena + lenb)

    return score

Let's time it:
N = np.power(10,5)

a = ''.join([str(unichr(i)) for i in np.random.randint(97,123,N)])
b = ''.join([str(unichr(i)) for i in np.random.randint(97,123,N)])

%timeit dice_coefficient(a,b)
%timeit dice_coefficient_new(a,b)

Output:
1 loop, best of 3: 204 ms per loop
10 loops, best of 3: 52.9 ms per loop

So for 100K elements you get a speedup of 4x!
For further optimisation you could parallelise your global for loop (for example also using numba or multiprocessing).
(Note: I edited my rushed first answer which didn't work)


Extract or subset hundreds of columns from a data frame I need to extract many columns from a dataset.
I have a very large csv file with thousands of columns and rows, and I read it into R using:

mydata <- read.csv(file = "file.csv",header = TRUE,sep = ",",row.names = 1)

Each column is a gene name. I know how to extract specific columns from my R data.frame by using the basic code like this:

dataset[ , "GeneName1", "GeneName2"]

But my question is, how do I pull hundreds of gene names? Too many to type in? They are listed in a txt file.
I'm new, so please go easy on jargon and abbreviations.

Hopefully I've understood your question correctly.

Assuming your text file looks like this?

GeneName1
GeneName2

You can read that in using the readLines() function:

cols <- readLines("name_of_text_file")

Which returns cols as a vector of those names:

> cols
[1] "GeneName1" "GeneName2"

Which can then be used to subset the data frame as per your example:

mydata[ , cols]


Regressor to estimate probability from samples What is a good way to estimate probabilities in a dataset of samples? Each sample is a measurement, that is usually 1 or 0. The goal is to calculate probabilies based on all feature rows.

Simple example dataset:

    feature   label
    dog       1
    dog       0
    dog       0
    dog       0
    cat       1
    cat       0

Expected Output:

    feature  result
    dog      0.25
    cat      0.5

The real dataset has around 50 features.

Assume that X are features and y - labels

#Put algorithm as you wish
from sklearn.ensemble import RandomForestClassifier
clf = RandomForestClassifier(max_depth=2, random_state=0)
clf.fit(X_train, y_train )

clf.predict_proba(X_test) # there you will get probabilities of the class


Overfitting in an unsupervised technique I am trying to understand if over-fitting can happen in an unsupervised technique like kmeans clustering. Could someone help me understand if and how this would happen?

Thanks.

I'm not sure if this is valid but how about two trivial clustering examples:

Every object belongs to cluster which contain only this object. So for example if you would like to cluster N cars, there will be N clusters - one for each car.
On the other hand there could be case when algorithm will pick one cluster which will contain all elements inside it - one cluster with all N cars.

Those will be valid clusters but obviously they will not give you any useful information.


Multivariate Time-Series Clustering I have a streaming data along with timestamp dataset that looks like this:
1.png
Timestamp can be inclusive of "seconds" too, but the data may or may not change every second. it depends on the previous values(rows i.e data which came earlier w.r.t time).
Column1, Column2 .... ColumnN correspond to the variables (they change over time) and "Label" shows the different samples. You can assume that the values tend to decrease over time for a particular label.
Labels A1,B1,C1.........A2,... M labels.
Note : Values of timeNew of a Label depends on values of  timeOld of that Label and Labels belong to its cluster.
I need to group Labels with similar behavior over time together (e.g. Label A1 and Label C1 should be put in the same cluster and B1,D2 may fall into same cluster over time as they tend to behave similar over time).
I thought of using DTW and get the similarity of each Label with respect to other Labels. but not sure, how to proceed when i have N Columns.
To be precise, i need to group Labels based on their similarities (Column1 .. ColumnN) over time and group them.
Once i group them when new data comes in i should be able to predict the values(Column1.. ColumnN) for a Label based on the previously seen data(can be just minutes closer to the current prediction) and the values associated with the Labels in its cluster and predict it accordingly.

Aleksandr Blekh's answer in this older question provides a lot of interesting reading material for time-series clustering methods and examples. Also, I include below some interesting reading material for calculating similarity among multivariate time-series (the latest 2 are quite old but I think they are very interesting):

An approach on the use of DTW with multivariate time-series (the paper actual refers to classification but you might want to use the idea and adjust it for clustering)
A paper on clustering of time-series
A PCA-based similarity measure for multivariate time-series
A review on feature extraction and pattern recognition methods in time-series data

Before proceeding with any method, I believe it is important to spend some time to think of the following:

Try to select the right step for your input data (e.g. if the time-steps are per second, the time-series might be too long and unnecessarily detailed for this job, while hourly data might catch the patterns better).
Seasonality might be interesting to take under consideration: e.g. if the time-series are hourly and last for several days/months, there might be some daily/monthly seasonality. In this case, you might want to calculate and compare the average day/month from each time-series (if so, you will also need to decide whether weekdays and weekends should be averaged all together or treated in a different way).
Depending on what you are trying to find out, you need to decide
whether Dynamic time warping (DTW) is useful for you. For example, if
2 time-series have the exact same pattern but one of the 2 has a
time-delay, should they still belong in the same cluster? (and how
small/big time-delay is acceptable to put them in the same cluster?)


True Negatives for Named Entity Recognition Are True Negatives always zero for a Named Entity Recognition task because TN in NER would mean a not entity being classified as Not entity?
Actual Entity
[Microsoft Corp.] CEO [Steve Ballmer] announced the release of [Windows7] today
Model prediction
[Microsoft Corp.] [CEO] [Steve] Ballmer announced the release of Windows7 [today]
What will be the TNs in the above example?

By definition, true negative is non-entity tokens in the text, agreed by gold standard. So;
TN: 5 = {announced, the, release, of}
But, in practice the number of true negative tokens are quite high and this heavily dominates the metrics, and any measure using TN (like accuracy) becomes highly controversial. Instead, preferred better metrics:

Recall: TP/(TP+FN) - coverage in all correct entities, and
Precision: TP/(TP+FP) - how reliable for a positive result
F1 Score (harmonic mean of precision and recall) can be used as a balanced metrics.


How to model user's buying behavior on Amazon? For our final course project in Data Science, we proposed the following-

  Give the Amazon Reviews Dataset, we plan to come up with an algorithm (thats roughly based on Personalized PageRank) that determines a strategic position for placing ads on Amazon. For example, there are millions of products on Amazon. And the dataset gives you an idea of what products are related, what products were brought together, viewed together etc. ( We can construct a graph with this info of also viewed and also bought ) It also gives you the reviews associated with each product over 14 years. Using all these info, we will rate/rank products on Amazon. Now, you are a vendor on Amazon who wants to improve traffic to their product page. Our algorithm helps you identify strategic positions in the graph where you can place your ad so that you can derive max traffic.

Now, our Professor's question is, how will you validate your algorithm without real users? We said-

  We can model a fixed set of users. Some users follow also_bought and also_viewed links to the third hop more often than first or fifth hop. There users' behavior is normally distributed. Some other users hardly navigate beyond the first hop. This set of users' behavior is exponentially distributed.

Our Professor's said - Whatever distribution the users follow, users are navigating using links for similar products. Your ranking algorithm also considers similarity b/w 2 products to rank products. So using this validation algorithm is kinda cheating. Come with some other user behavior, something more realistic and orthogonal to the algorithm.

Any ideas on how to model the users' behavior? I am happy to provide more details about the algo.

How will you validate your algorithm?

Rather than trying to answer the second question, consider that your answer to the first question might need revision...

What methods have you used for validating learning methods throughout your data science class?  First you want to define a specific set of numerical metrics to assess the success or failure of your model. Second, what methods can you use to create a very realistic testing population (more realistic than modeling the population)?  The first hint I will give is that the Amazon Reviews Dataset is very large, so your data is very amendable to this method.  The second hint I will give is that this method is likely the one that you have used in 95% of the supervised learning problems that you have worked on in class...

Hope this helps... I will edit this if need be based on comments added by the OP, but don't want to provide the solution right away in order to elicit some organic thought e.g. since this is a class problem and the Prof is also trying to help you come up with the right solution on your own.


Why does OPTICS use the core-distance as a minimum for the reachability distance? The OPTICS clustering algorithm defines

$$\text{core-dist}_{\varepsilon,MinPts}(p)=\begin{cases}\text{UNDEFINED} & \text{if } |N_\varepsilon(p)| < MinPts\\ MinPts\text{-th smallest distance to } N_\varepsilon(p) & \text{otherwise}\end{cases}$$

and

$$\text{reachability-dist}_{\varepsilon,MinPts}(o,p) = \begin{cases}\text{UNDEFINED} & \text{if } |N_\varepsilon(p)| < MinPts\\ \max(\text{core-dist}_{\varepsilon,MinPts}(p), \text{dist}(p,o)) & \text{otherwise}\end{cases}$$

why isn't it simply

$$\text{reachability-dist}_{\varepsilon,MinPts}(o,p) = \begin{cases}\text{UNDEFINED} & \text{if } |N_\varepsilon(p)| < MinPts\\ \text{dist}(p,o) & \text{otherwise}\end{cases}$$

There are a few ways of looking at this. To make this easier I'm going to work describe things in terms of HDBSCAN*, a succesor to OPTICS. HDBSCAN* is essentially the same as OPTICS by removes the $\varepsilon$ parameter (or, if you like, fixes $\varepsilon = \infty$). HDBSCAN* also has a different approach for cluster extraction, but that's not overly relevant here. Thus under HDBSCAN* we define

$\text{core-dist}_\mathrm{MinPts}(p) = \text{Distance the the MinPts nearest neighbor of } p$

$\text{mutual-reachability-dist}_\mathrm{MinPts}(o, p) = \\ \qquad\max(\text{core-dist}_\mathrm{MinPts}(p), \text{core-dist}_\mathrm{MinPts}(o), \mathrm{dist}(p,o))$

much the same as OPTICS, but now we don't require any undefined values (we also include the core-distance of $o$ to ensure that mutual reachability is a metric).

One can view both OPTICS and HDBSCAN* as an extension of DBSCAN. DBSCAN takes two parameters MinPts and $\epsilon$ (note that this is a different use of epsilon than OPTICS -- if we focus on HDBSCAN* which has no such parameter we can avoid confusion: epsilon will refer only to the parameter used in DBSCAN).

HDBSCAN* generates a complete hierarchy of clusterings for a range of possible $\epsilon$ values, and thus for any fixed $\epsilon$ value you want the clustering at that level in the hierarchy to be the clustering DBSCAN would give for that $\epsilon$ value (OPTICS works similarly, but simply constrains the range of value that $\epsilon$ may take). To be in a cluster in DBSCAN you must be a core-point (I'm going to ignore border points for now); that is, the point must have at least MinPts other points within a ball of radius $\epsilon$ of itself. If a point isn't a core-point it is noise. The HDBSCAN* (and OPTICS*) reachability-distance captures this distinction by ensuring a point is not joined into a cluster until the DBSCAN $\epsilon$ value is such that the point is within the relevant distance of the other points in the cluster and the point is a core-point at that DBSCAN $\epsilon$ value. The alternative reachability-distance you suggest covers the first part, but not the second: noise points would be included in clusters as long as they were close enough -- and by transitive chaining this could result in clusters merging together prematurely via noise points.

An alternative way of looking at this is by comparing with other algorithms like Robust Single Linkage (which also produces a cluster hierarchy). Robust Single Linkage aims to improve single linkage clustering by making it more robust to noise. In single linkage at each distance-scale $r$ you form a graph by joining points with an edge if they are within $r$ of each other, and then the clustering at level $r$ of the hierarchy is simply the connected components of that graph. The problem is that single linkage is very susceptible to noise -- a point or two in the wrong place can join clusters together when perhaps they shouldn't be. Robust Single Linkage attempts to fix this by requiring a point to have at least $k$ neighbors within $\alpha r$ of it  before adding any edges to it. This discounts sparse points until larger distance-scales and makes the approach more robust to noise. It can be demonstrated that this approach provides convergence to the level set tree of the PDF that generated the data -- i.e. it does a good job of fixing the noise issues! If we fix $\alpha = 1$ and $k = \mathrm{MinPts}$ then we recover HDBSCAN*; if we fix $r \leq \varepsilon$ then we get essentially OPTICS. Again we are using core-distance to discount noise, and make the clustering more noise robust.


Best python library for neural networks I'm using Neural Networks to solve different Machine learning problems. I'm using Python and pybrain but this library is almost discontinued. Are there other good alternatives in Python?

UPDATE: the landscape has changed quite a bit since I answered this question in July '14, and some new players have entered the space.  In particular, I would recommend checking out:

TensorFlow
Blocks
Lasagne
Keras
Deepy
Nolearn
NeuPy

They each have their strengths and weaknesses, so give them all a go and see which best suits your use case.  Although I would have recommended using PyLearn2 a year ago, the community is no longer active so I would recommend looking elsewhere.  My original response to the answer is included below but is largely irrelevant at this point.

PyLearn2 is generally considered the library of choice for neural networks and deep learning in python.  It's designed for easy scientific experimentation rather than ease of use, so the learning curve is rather steep, but if you take your time and follow the tutorials I think you'll be happy with the functionality it provides.  Everything from standard Multilayer Perceptrons to Restricted Boltzmann Machines to Convolutional Nets to Autoencoders is provided.  There's great GPU support and everything is built on top of Theano, so performance is typically quite good.  The source for PyLearn2 is available on github.

Be aware that PyLearn2 has the opposite problem of PyBrain at the moment -- rather than being abandoned, PyLearn2 is under active development and is subject to frequent changes.


Can we generate huge dataset with Generative Adversarial Networks I'm dealing with a problem where I couldn't find enough dataset(images) to feed into my deep neural network for training.

I was so inspired by the paper Generative Adversarial Text to Image Synthesis published by Scott Reed et al. on Generative Adversarial Networks.

I was curious to know that, can I use available small dataset as an input to a GAN model and generate a much bigger dataset to deal with deeper network models?

Will it be good enough?

This is unlikely to add much beyond your direct data collection efforts.

The quality of current GAN outputs (as of 2017) will not be high enough. The images produced by a GAN are typically small and can have unusual/ambiguous details and odd distortions. In the paper you linked, the images generated by the system from a sentence have believable blocks of colour given the subject matter, but without the sentence priming you what to expect most of them are not recognisable as any specific subject.

GANs with a less ambitious purpose than generating images from sentences (which is despite my criticism above, a truly remarkable feat IMO) should produce closer to photo-realistic images. But their scope will be less and probably not include your desired image type. Also, typically the output size is small e.g. 64x64 or 128x128*, and there are still enough distortions and ambiguities that original ground truth photos would be far preferable.

The GAN is itself limited by training library available - it will not do well if you attempt to generate images outside of the scope of its training data. The results shown in the research paper of course focus on the domain supplied by the training data. But you cannot just feed any sentence into this model and expect a result that would be useful elsewhere.

If you find a GAN that has been trained on a suitable data set for your problem, then you are most likely better off trying to source the same data directly for your project.

If you are facing a problem with limited ground truth data, then maybe a better approach to using a GAN would be to use a pre-trained classifier such as VGG-19 or Inception v5, replace the last few fully-connected layers, and fine tune it on your data. Here is an example of doing that using Keras library in Python - other examples can be found with searches like "fine tune CNN image classifier".

* State-of-the art GANs have got better since I posted this answer. A research team at Nvidia has had remarkable success creating 1024x1024 photo-realistic images. However, this does not change the other points in my answer. GANs are not a reliable source of images for image classification tasks, except maybe for sub-tasks of whatever the GAN has already been trained on and is able to generate conditionally (or maybe more trivially, to provide source data for "other" categories in classifiers).


Input shape of an Xception CNN model I have a Keras Xception based model for gesture recognition. The accuracy of the model is around 60-70% for classifying 7 different gestures. The training dataset consists of 320x240 and 640x480 pixel images. Currently, I'm leaving the input_shape parameter of the model equal to default value of the Xception model in Keras, which is (299, 299, 3). I assume under the hood the network is rescaling all inputs to 299x299 pixels, which probably isn't a good approach.
My questions are:

Is the Xception model somehow optimized for the 299x299 image size? That is to say, should I aim to crop/pad the input to 299x299 pixels rather than change the model's configuration?
All usage examples I've seen so far have input width = height. Is there a reason to prefer square images?
If I don't use cropping/padding, I have two options for the input shape: rescale all images to 640x480 in a preprocessing step, or rescale all to 320x240. Is the 640x480 option likely to result in much better accuracy?

For your first question, yes, it is optimized for that size since the original paper for Xception used 299x299 size. But, you can use other sizes. You should resize your images to 299x299 that would be the best.
For your second question, the reason height = width because in the network, the convolutional filters which are used are square (3x3 filters). The reason for using square filter in computer vision is we assume that the features in image are symmetric most of the times (exception being text where the information is more on the vertical dimension than horizontal, 1x2 filters are used there).
For your third question, go for the smaller size, because if you rescale the smaller images to bigger ones, you add useless information into image since it is derived from the smaller image itself. Also, you create a model with more parameters.


How propagate the error delta in backpropagation in convolutional neural networks (CNN)? My CNN has the following structure:

Output neurons: 10
Input matrix (I): 28x28
Convolutional layer (C): 3
feature maps with a 5x5 kernel (output dimension is 3x24x24)
Max pooling layer (MP): size 2x2 (ouput dimension is 3x12x12)
Fully connected layer (FC): 432x10 (3*12*12=432 max pooling layer flattened and vectorized)

After making the forward pass, I calculate the error delta in the output layer as:

$\delta^L = (a^L-y) \odot \sigma'(z^L) (1)$

Being $a^L$ the predicted value and $z^L$ the dot product of the weights, plus the biases.

I calculate the error deltas for the next layers with:

$\delta^l = ((w^{l+1})^T \delta^{l+1}) \odot \sigma'(z^l) (2)$

And derivative of the error w.r.t. the weights being

$\frac{\partial C}{\partial w^l_{jk}} = a^{l-1}_k \delta^l_j (3)$

I'm able to update the weights (and biases) of $FC$ with no problem. At this point, error delta $\delta$ is 10x1.

For calculating the error delta for $MP$ , I find the dot product of $FC$ and the error delta itself, as defined in equation 2. That gives me an error delta of 432x1. Because there are no parameters in this layer, and the flattening and vectorization, I just need to follow the reverse process and reshape it to 3x12x12, being that the error in $MP$.

To find the error delta for $C$, I upsample the error delta following the reverse process of the max pooling ending with a 3x24x24 delta. Finding the hadamard product of each of those matrixes with each of the $σ′$ of the feature maps gives me the error delta for $C$.

But now, how am I supposed to update the kernels, if they're 5x5, and I is 28x28? $I$ have the error delta for the layer, but I don't know how to update the weights with it. Also for the bias, as it's a single value for the whole feature set.

So you are correct that the principle of backpropagation is to do the reverse of the operations. The same is true about the convolutional layer.
The forward pass of the convolutional layer can be expressed by
$x_{i, j}^l = \sum_m \sum_n w_{m,n}^l o_{i+m, j+n}^{l-1} + b_{i, j}^l$.
Where $m$ and $n$ is the shape of the convolutional kernel that you will pass over your input image and $w$ is the associated weight for that kernel. $o$ is the input features and $x$ is the resulting value represented by their respective layers $l-1$ and $l$.
For backpropagation we will want to compute $\frac{\partial x}{\partial w}$.
$\frac{\partial x^l_{i, j}}{\partial w^l_{m', n'}} = \frac{\partial}{\partial w^l_{m', n'}} (\sum_m \sum_n w_{m,n}^l o_{i+m, j+n}^{l-1} + b_{i, j}^l)$.
By expanding the summation we end up observing that the derivative will only be non-zero when $m=m'$ and $n=n'$. We then get
$\frac{\partial x^l_{i, j}}{\partial w^l_{m', n'}} = o^{l-1}_{i+m', j+n'}$.
We can then put this result into the overall error term we have calculated.


how can i leverage NLP features like SRL, LSA, POS, NER, entity type , relation type with deep learning to find semantic similarity of texts? I have texts similar to the ones below, and I want to find semantic similarity between these texts and the intent.

  1) what are the different steps to resign ?

  1) procedure to resign ?

  1) what are the process do i have to follow to resign ?

  1) what are the common schemes followed for resignation by any company !!

  2) to whom i need to contact after i resign ?

  3) i did resign today.

  4) if you misbehave again , i'll resign !!

  4) your rude behavior made me to resign.

  5) different company have different policy on resignation !!

To tackle above problem i found " Quoras Model to handle Duplicate Questions " interesting . so i thought let's try this but with small changes. and changes is instead doing binary classification , let's do multi class classification. so i was trying to do multi class classification on a "Quora Data Set". my intention was to classify each question with their duplicate questions into one class on a vector space using LSTM, CNN models. LSTM worked if sequence remain unchanged. but as you know questions structure can changes keeping intent same (or we can call them as duplicate questions.). so to handle such variation of question i tried with CNN. CNN worked for some small data set but for large data set it become very sensitive or overlapping.

So i am thinking " can i leverage NLP with deep learning to find intent and semantic relation ?"

I have POS tags, NER, SRL, LST, entity type, relation type, etc features. How can i leverage these features of NLP in deep learning to achieve a state-of-the-art result?

There is a paper, "When Are Tree Structures Necessary for Deep Learning of Representations?", that uses parse tree as input for recursive neural network model. Is there any similar work? Can any one give me data sets used in this paper?

TL;DR:

Represent words as word vectors. Then add extra dimensions to the word vectors. In these extra dimensions, include POS, NER, etc. features in a numeric form.

Longer Version:

Say you have a word2vec/Glove model with each word represented by a 100 dimensional vector. Additionally, you have features like POS, NER, etc. for each word.
Instead of representing each word with just 100 dimensions, represent it with 100 + n dimensions and fill these n slots with the nlp features.

This will give the vectors an additional meaning which you have defined manually.

Suggestions

You could benefit from using other models like Doc2Vec/Sent2Vec which can represent a whole sentence as vector. With these vectors you could query similarity of sentences.

So say you've trained it on your data, just query :

model.most_similar_cosmul(SENT_42)

where SENT_42 represents "to whom i need to contact after i resign ?",
you will get a list of sentences similar to it.

I suggest you use gensim's doc2vec model in python.


Does white balance correction improve the performance of visual object detection? Visual object detecion aims to locate and recognize objects of interest in images.

I wonder, does white balance correction contribute to better performance of visual object detection? Are there any related studies or papers?

Thanks in advance for your suggestions!

I don't know about any papers, though making your objects more distinct should definitely help your model to learn proper filters. One thing to keep in mind though is your production data. Will you be also feeding images with poor exposure, white balance, contrast? If so, you need to train your model on such data. As far as I know, this repository produces by far most possible augmentations to images.


Good t-SNE or bad t-SNE? I have used tsne to visualise a large dataset and it has produced the following graph. I need help interpreting it, as I have never seen a tsne graph like this before!

I am aware that not much overlapping data is a good thing, but I have rarely seen more than a few clusters. Has something gone wrong here? Or are there just a lot of clusters in this dataset.

Thanks. P

t-SNE is notoriously fickle.  I would advise you read this short article How to Use t-SNE Effectively before you invest too much time trying to interpret the plots.  In case you don't read it, you'll find that your t-SNE plot will change considerably based on your iterations, and perplexity.  You'll find that distance between clusters aren't accurate and that cluster sizes aren't accurate too.  t-SNE is a cool data reduction too, but often takes multiple runs to get a "good" plot.  Laurens van der Maatan even suggest that running multiple iterations and picking the one with the lowest KL-divergence is perfectly okay.  This is due to the fact that t-SNE is non-convex so you won't necessarily find a global minimum.


How can I monitor the usage of NVLink connections? If I'm running a Keras model on some Nvidia GPUs which are connected via NVLink, how can I monitor the usage of the NVLink connections?

I want to be sure the NVLink connections are being used and see at what rate.

nvidia-smi nvlink

This command shows various information about nvlink including usage. If nvlink connections are utilized, usage should go up during training.

nvidia-smi nvlink -h

Shows available performance counters on present cards.

"NVLink Usage Counters" section in this tutorial shows how to see if data is being transferred across nvlink.

https://blog.exxactcorp.com/exploring-nvidia-nvlink-nvidia-smi-commands/


How to discretize a numerival value with predefined ranges in Weka? I have imported a csv file into Weka. One of the features has a value with minimum 0 and maximum 160. Now, I want to discretise that value into three ranges as you can see below:

Less than 6 > L

More than 6 and less than 20 > M

More than 20 > H

How can i do that?

Discretising or binning, very common. There is a filter for that in Weka. You'll find it under weka.filters.unsupervised.attribute.Discretise. It is in the GUI too.

You can find the documentation under: https://weka.sourceforge.io/doc.dev/weka/filters/unsupervised/attribute/Discretize.html


Interpreting lasso logistic regression feature coefficients in multiclass problem I have a dataset with a large number of text features, where the target variable has three classes.

I have encoded the features using tf-idf. This has resulted in a dataset with an extremely large number of features.

I have tried performing classification using lasso logistic regression (in sklearn, LogisticRegression(penalty='l1')), which automatically sets some features' coefficients to 0 and thus performs some kind of feature selection for me.

The sklearn implementation of LogisticRegression() performs multiclass classification using 'one-versus-rest'. That is to say, it trains three separate classifiers which predict one particular class against the other two, and then assigns each data point to the class whose predictor gave the highest probability to.

The coefficients of some of the features from my dataset were set to $0$ in all three classifiers. Others, however, are set to non-zero numbers in all three.

Is it correct to think of those with coefficients set to $0$ by all three classifiers as uninformative, in the sense that they add no new information given the presence of all the others?

Does it make sense to compare the magnitude of the non-zero coefficients, across the three classifiers, in order to find out which features are most informative?

Welcome to the site!

I think you are right that feature weights = 0 across all three classifiers mean that the features are not predictive of the outcome.

In terms of comparing magnitudes, I'm not sure the absolute values are really comparable - each model is trying to predict a different target.

However, you could rank the coefficients in some way and then compare the rank of the coefficient across the three models. You would need to choose a robust way to rank the raw coefficients since they would range from positive to negative - e.g. a strongly negative coefficient such as -2 is probably more predictive than a barely positive coefficient of 0.1.


Network Size in neural network What are the limitations of having too many hidden units in the Neural Network
Does it take more memory or takes longer time to train the model

The number of hidden units is a hyperparameter which increases the model capacity. Therefore, increasing it not only increases time and memory requirements but can also lead to overfitting (also see section 11.4.1 in the Deep Learning Book).


'list' object has no attribute 'values' when we are using append in python Here I have a dataset with three inputs. Here I generated y value using append. After the append I got the output like this:

 y.append(rec.iloc[0]['y'])

Then I tried to develop neural network model with these values. Before that I tried to scale the y value.
my code:

y =y.values().astype(int)
scaler_y = preprocessing.MinMaxScaler(feature_range =(0, 1))
y = np.array(y).reshape([-1, 1])
y = scaler_y.fit_transform(y)

Then I got an error :

AttributeError                            Traceback (most recent call last)
<ipython-input-254-2ec9d2fcbffd> in <module>()
----> 1 y = y.values().astype(int)

 AttributeError: 'list' object has no attribute 'values'

Can anyone help me to solve this error?

It is basically what the error message says. The problem is this:

y =y.values().astype(int)

y is a list and lists do not have a method values() (but dictionaries and DataFrames do).

If you would like to convert y to a list of integers you can use list comprehension:

y = [int(x) for x in y]

Or alternatively use map (but I'd prefer the list comprehension):

y = list(map(int, y))

Since this is actually a coding related question you might want to consider posting on https://stackoverflow.com next time.


How to calculate the evaluation metrics (i.e., F1 score) in leave one subject out cv when a subject belongs to single class only I have dataset of 10 subjects. the  dataset has 4 classess. 0,1,2 and 3. The distribution of  classes are not same. For example subject 1 does not have 1,2 and 3. It belongs to zeros class. currently I am evaluating the model using leave one subject out.In such scenario how to calculate the evaluation metrics like F1 score. I  tried the sklearn classification report but for  some subjects there are  only one class. shall I consider the macro F1 score?

You should not use summary statistics if you only have 10 data points. With that few datapoints, the summary statistics estimates will not be robust.


Using multiple machine learning algorithms together I'm kinda new to machine learning and wanted to know if we could use multiple machine learning algorithms, for example, SVM and backpropagation together to solve a particular problem.

You can train multiple machine learning models with same data and based on accuracy and confusion_matrix response you can decide which one to use.

In theory, you don't need to combine/merge two different ML Models (you can enhance your pre-processing)
but if you still want to use different models there are two approaches:

Ensemble (available with three categories - a) Bagging b) Boosting c) Stacking
Hybrid (This approach allow users to create own models (or use existing) and combine them for better prediction)

Note: You need to be careful with individual algorithm response before combining them together :)


Which algorithm to use to predict the duration of some task I have a data set of 36K processes each with 7 features. Each set represents a task and I know how long it took to complete each of the tasks. I want to build a model which will be able to predict the duration of the future tasks.

I've only learned about Decision Tree (DT) and tried to apply it to my problem. The resulting accuracy score is 0.03. I believe DT is not suitable because the time is continuous and DT is meant for categorization.

Which algorithm is suitable for duration prediction?

My environment: Python with sklearn, if that matters.

What others have said is accurate, that you need to build a regression model of some sort. Depending on the scale of the duration of your task, you will have to model it slightly differently.

In fact, there's an entire class of models that try to predict duration. These are called survival models. Here's a Python library on survival analysis.

But these models are fairly academic. A typical hack is to use Gamma, Poison, or Log-Normal regression which works out nicely because these models predict non-negative values.


Is there an intuitive interpretation of precision always higher than recall? I have a multiclass-classifier whose macro-precision is always greater than macro-recall. I suppose it means false negatives outnumber false positives in general. Is there an intuitive interpretation about this?

Generally lower recall means that the system is too strict, i.e. it predicts an instance as positive only when it has clear indications in the features that it's indeed a positive. As a consequence, it misses the true positive instances for which the indications are not so clear.
But when looking at macro-recall it's more complex: it depends primarily on how many classes there are and their distribution. In case of imbalance a very common problem is that the system predicts the majority class too often, causing a low recall for the minority class(es) and therefore a lower macro-recall. The only way to check for this is to look at the performance of individual classes.


Synchronizing Multiple Cameras in Autonomous Driving Please forgive the naivity of this question, it's just due to lack of experience.

It goes without saying that self-driving cars have up to 8 cameras and more that do various vision related tasks:

object detection and localization,
2D to 3D depth perception,
semantic segmentation,
and probably more that I have yet to learn about.

My question is on the synchronization of these cameras:

I am assuming there's a hardware sync to ensure all inputs are captured at the "exact same" time.  Is that true?

Is there a way to account in the model / protocol of capture for the fact that the inputs might have some delay in between them?

Even though this question makes me sound very novice, feel free to discuss advanced topics. I have just started reading this paper on depth perception using a single camera and am able to understand it.

This paper from Texas Instruments pretty much outlines the entire problem and current state of the art.


plotting histogram together with variance I have a small piece of code that print values gotten from a csv file into a histogram. This is all done using matplotlib library

dev_x= X     #this is where my integers are stored
plt.hist(dev_x, bins=7)

#var=np.var(dev_x)
#print(np.mean(dev_x))
#plt.axhline(var)

plt.xlabel('Values')
plt.ylabel('Quatity')
plt.title('Graph')

When the following lines are commented, I get the result

However, when I uncomment these lines

var=np.var(dev_x)
print(np.mean(dev_x))
plt.axhline(var)

I get only this graph

The code aims to show the histagram of the data and then the mean and the variance horizontal .

The problem is that you are plotting two things of different scale on the same axis.
Your variance appears to be 6*10^9 while your counts are in the hundreds. Therefore the bins are so small you can no longer see them.

You should probably use an vline instead of an hline to get the line on the x-axis

plt.axvline(var)

Here is a sample:

import seaborn as sns
import matplotlib.pyplot as plt
diamonds = sns.load_dataset('diamonds')
plt.hist(diamonds['carat'])
plt.axvline(np.var(diamonds['carat']), color='red')
plt.show()

If you really want to do this on the y axis you should use a second y axis.

Change this

plt.axhline(var)

to this

ax2 = plt.gca().twinx()
ax2.axhline(var)

Check the documentation for more info

Otherwise consider using multiple plots since it might be really confusing to have a line with a different scale on a histogram.


How to combine two logistic regression models trained on different set of data? My data has a hierarchy structure - meaning that there is an N class at level 1 and an M class at level M. After training both models separately with a different set of data (both are Logistic regression but can be changed) I want to predict to N + M classes.
Currently, I am solving it by collecting all data before training and adding examples from level 2 to level 1 results in (N + 1) classes (+ 1 meaning all examples from level 2) and if the model at level 1 predicts that "+1" class then a send the query to the second model. But collecting the examples before training is costly and also the training set is unbalanced after collecting. Pseudocode:
clf1 = LogisticRegression()
clf1.fit(X1 + X2, Y1 + ["ID of +1 class"] * len(Y2))
clf2 = LogisticRegression()
clf2.fit(X2, Y2)
if cl1.predict(X_test_example) == ["ID of +1 class"]:
    clf2.precict(X_test_example)

But I need to preserve this hierarchical structure so the merging the two models directly is not possible. Is there a way to combine two outputs of classification models (in a form of the probability distribution) to get meaningful output are avoid problems with the overconfidence of models? Pseudocode:
clf1 = LogisticRegression()
clf1.fit(X1, Y1)
clf2 = LogisticRegression()
clf2.fit(X2, Y2)
combined_clf = clf1 + clf2

I can think of two options, there are probably more:

Train a binary classification "top model" which predicts whether the instance belongs to  model 1 or model 2. The two final models only need to be trained with their respective data and only predict their respective N or M classes. The performance of the top model is crucial in this option.
Train an open set classification model. It's a bit similar to what you're doing currently by adding a class "other model" to the first model, except that "other" is not a regular class and you don't need to train the model with instances from this class. This is related to one-class classification and requires a specific training method which can handle open-set classification.


How to choose dimension of Keras embedding layer? Looking for some guidelines to choose dimension of Keras word embedding layer. For example in a simplified movie review classification code:
# NN layer params

MAX_LEN = 100        # Max length of a review text
VOCAB_SIZE = 10000   # Number of words in vocabulary
EMBEDDING_DIMS = 50  # Embedding dimension - number of components in word embedding vector

text_model = tf.keras.Sequential([
    tf.keras.layers.Embedding(input_length=MAX_LEN,input_dim=VOCAB_SIZE,
                              output_dim=EMBEDDING_DIMS),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(6, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid')
])
text_model.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])

Embedding vector has 50 components in this example. Trained on 17500 and tested on 5625 reviews this model reports:
              precision    recall  f1-score   support

           0       0.87      0.86      0.87      2802
           1       0.87      0.88      0.87      2823

    accuracy                           0.87      5625
   macro avg       0.87      0.87      0.87      5625
weighted avg       0.87      0.87      0.87      5625

With 10 and even 2 dimensions I get similar values in classification report!
Then what guiding principle really works when choosing word embedding vector dimension? When select 50, 10,100, 200, ... etc. dimensions?

There is no "Right" answer to this question , but you should take in mind the following guidelines:

Embedding layer is a compression of the input, when the layer is smaller , you compress more and lose more data. When the layer is bigger you compress less and potentially overfit your input dataset to this layer making it useless.

The larger vocabulary you have you want better representation of it - make the layer larger.

If you have very sparse documents relatively to the vocabulary, then you want to "get rid" of unnecessary and noisy words - you should compress more - make the embedding smaller.


Thought process behind data analysis I am looking for books/tutorials that help you gain the insight into the thought process behind data analysis.

Most of the books I've read were mostly documentation - the author shows you a function and some data that he applies that function to. They also show how to use graphs and how histogram/boxplots etc. work. They go through popular libraries like numpy, pandas and how to use them.

I am interested in how people on (for example) Kaggle in their kernels on Titanic get their ideas from, on what to apply to datasets. Those people know which columns to plot as a function of each other, when to plot histograms, when to plot density functions and so on.

I am somewhat experienced in Machine Learning. It's pretty obvious which algorithm can be applied to what situation. Data exploring seems like a very ambiguous exercise with many solutions/ideas.

Another way to put is: Where to get the ideas for data exploration from?

I guess it is more about "What would be the most informative (and easy readable) way to visualize one specific problem?". Having a specific question in mind you can explore galleries of the libraries with their examples, for instance,  seaborn. You could also explore another kernels to see how people visualized it. For some question there are pretty trivial solutions. Other questions can be visualized in very many different ways but what counts is how readable and informative your plot is.

In case of titanic imagine three questions that you want to visualize.

What if survival rate depends somehow on the age and/or sex?

For this you might want to choose a density or histogram with having two curves on the plot: for survival and non-survival. Maybe you want to split the plot into two plots for female and male.

What if survival rate depends on a Pclass? Or maybe Emarked as well? And maybe sex plays also an important role?

I am a fan of factor plots as you can introduce lots of information into it and it is quite understandable from the first sight. So, you see that 1st and 2nd class had a higher survival rate comparing to the 3rd class. (The next question can arise: For each embark what is the relative proportion of the pclass? What would be the better visualization?)

What if a survival rate depends on the pclass and age? Additionally you could see what is the age distribution of each class.

Violin plots might be good for that.


Choosing the size of Character Embedding for Language Generation models I am working on a character-based Language Generator, loosely based on this tutorial on the TensorFlow 2.0 website. Following the example, I am using an Embedding() layer to generate character embeddings: I want my model to generate text character by character.

My vocabulary counts 86 unique characters. What embedding size should I choose?

Should I always choose an embedding size that is shorter than the size of vocabulary? The embedding size in the example above is much larger than the vocabulary size, I can't understand how this can build an effective model (but apparently it does, if it's an official tutorial, and if anyone can explain me why it'd be much appreciated).

EDIT:

Another thing I find puzzling is: when we generate word embeddings is because we want a dense representation of a word meaning. Does it make sense to make it larger than the actual one-hot encoded vectors we started with?

There is a theoretical lower bound for embedding dimension

I would urge you to read this paper, but the gist of it is dimension could be chosen based on corpus statistics

GLOVE paper discussed embedding, check page 7 for graphs. What I want to say with this reference is that you can treat it as hyperparameter and find your optimal value.

EDIT: Here is my personal/borrowed from google rule of thumb. Embedding vector dimension should be the 4th root of the number of categories is start with that, and then I play around it. Read this toward the end when they explain their embedding. Why COULD (it must not) make sence: What is BOW rather than one hot encoding of your n-grams.

Does it make sence to make it larger? it depends. On one hand you are right if we make it too big we loose the distributed representation property of the word embedding matrix, on the other hand it works in praxis.


Which supervised learning algorithms are available for matching? I'm working on a non-profit where we try to help potential university applicants by matching them with alumni that want to share their experience/wisdom and, at the moment, it is happening manually. So I'll have two tables, one with students and one with alumni (they may have some features in common, but not necessarily all of them)

$\begin{array}{|l|c|c|} \text{Name} & \text{Gender} & \text{Height} \\ \hline \text{Kathy} & F & 165 \\ \hline \text{Tommy} & M & 182 \\ \hline \text{Ruth} & F & 163 \\ \hline ... & ... & ... \\ \end{array}$ $\begin{array}{|l|c|c|} \text{Name} & \text{Gender} & \text{Weight} \\ \hline \text{Miss Lucy} & F & 65 \\ \hline \text{Miss Geraldine} & F & 70 \\ \hline \text{Miss Emily} & F & 60 \\ \hline ... & ... & ... \\ \end{array}$

Currently, we are manually matching the members of table 1 with those in table 2. We will also collect information after the match ("Was it a good match? Please rate it on a scale from 1 to 10"). So it will look something like this:
$$
\begin{array}{|l|l|c|}
\text{Person #1} & \text{Person #2} & \text{Match?} \\ \hline
\text{Ruth} & \text{Miss Lucy} & N \\ \hline
\text{Tommy} & \text{Miss Emily} & Y \\ \hline
\text{Kathy} & \text{Miss Geraldine} & N \\ \hline
\text{Ruth} & \text{Miss Emily} & N \\ \hline
... & ... & ... \\
\end{array}$$

I would like to use a learning algorithm for this process. I know a little bit of machine learning, but I am still very much a novice (so it's also an opportunity for me to learn more about it), but I can't wrap my head around how you would do this kind of supervised learning when you have two sets both of which have multiple features. What sort of matching algorithms are available to do this? (Also, I prefer to work in R)

(By the way, I would be grateful if you could just point me in the right direction and I'll try to read about it and solve it myself. Also, I know how deeply frustrating it is to see questions that have already been answered -- if this is case, please don't hesitate to let me know without answering the question. I have already tried to search for various strings on Google and StackExchange, but mostly find lecture slides on graph theory that don't seem to be what I'm looking for (although it may just be because it's a bit over my head). Many thanks!)

You can try to frame this problem as a recommender systems situation. Where you have your users (prospective students) and items (alumni) and want to recommend to the users one item.

It's not a perfect fit as you want just one item for each user and you don't have previous match data for each user. However you could investigate this idea a bit further. I'm applying these techniques to the recruitment problem, I'm matching users with job offers and I'm having some success.

Try to read a bit about recommender systems, to start I recommend chapter 9 of mining massive data sets, it's really introductory, but gives a good overview of the most common techniques.


What are suitable datasets for univariate time series forecasting with RNNs, LGBM, TBATS, SARIMA models (topic, frequency, sources)? I am currently looking for a suitable dataset (univariate time series) for short-term forecasting using lag features or moving windows of lag features to employ models like LSTM, GRU, SARIMA, LGBM, TBATS and ETS. I have been able to successfully implement the models for a hourly traffic flow dataset using no external features, but I have tried 5 different datasets since then and the simple models (ETS, ARIMA) give nonsensical point forecasts so I don't really know how to find a second suitable dataset. What can i look out for when trying to find a useful dataset if I don't want to leave it up to chance?
So far I have also explored daily Air Quality AQI data, hourly water meter consumption data and monthly electricity demand data but the point forecasts of the simpler models were never useful.
Or can it just be the case that the data were simply too random to predict?
Any advice will be much appreciated.

A dataset I used when checking the ARIMA methodology is the one you can find here about monthly air passengers history data since 1949. It shows both trend and seasonality, and it is nice to play with searching for hyperparameters (p, d, q) when implementing ARIMA.
You can find below the results I got sometime ago as a quick check (being blue the true data and red the predictions):


Why is hard for neural machine translation model to learn rare words? i'm kinda new with neural machine translation. I've read some papers, authors usually limit the size of vocabulary by replace rare words by unk token. In this paper, they said that "...NMT model cannot learn the translation of rare words...". I want to understand why is hard for NMT mode to learn rare words and also the impact of word counts to NMT model. Thanks

Rare words are not a problem only for NMT, they are a problem for MT in general. The reason is simple: in order to accurately translate a word in any particular context, the model needs to see as many examples as possible during the training stage. By definition the training data contains very few occurrences of rare words (especially hapax words which occur only once), so the model doesn't have enough information to learn their translation properly.


How to set weight in Weighted Kernel K-Means? The objective function of kernel K-means is

$$      \sum\limits_{c=1}^k \sum\nolimits_{a_i \in c}   w_i \Vert \phi(a_i)- m_c \Vert^2 \ $$

where

$$ m_c = \frac{\sum\nolimits_{a_i \in c }w_i\phi(a_i)}{\sum\nolimits_{a_i \in c  }w_i}  $$
I need to know how to determine wi

These weights should be introduced by a user. With a weight you tell the K-means algorithm, that one feature is more important than the other.

  [0] These might represent a measure of importance, a frequency count, or some other information. The intent is that a point with a weight of 5.0 is twice as "important" as a point with a weight of 2.5, for instance. This gives rise to the "weighted" K-Means problem.

[0] http://people.sc.fsu.edu/~jburkardt/m_src/kmeans/kmeans.html


Policy gradient - and auto-differentiation (Pytorch/Tensorflow) In policy gradient, we have something like this:

Is my understanding correct that if I apply log cross-entropy on the last layer, the gradient will be automatically calculated as per formula above?

Yes, just take the cross-entropy loss of the last layer and take the gradient with respect to it. The actual action will be the target.

For example, in PyTorch: apply CrossEntropyLoss on the last layer (no need of doing softmax, as it is done implicitly by this function)


Filtering spam from retrieved data I once heard that filtering spam by using blacklists is not a good approach, since some user searching for entries in your dataset may be looking for particular information from the sources blocked. Also it'd become a burden to continuously validate the current state of each spammer blocked, checking if the site/domain still disseminate spam data.

Considering that any approach must be efficient and scalable, so as to support filtering on very large datasets, what are the strategies available to get rid of spam in a non-biased manner?

Edit: if possible, any example of strategy, even if just the intuition behind it, would be very welcome along with the answer.

Spam filtering, especially in email, has been revolutionized by neural networks, here are a couple papers that provide good reading on the subject:

On Neural Networks And The Future Of Spam
A. C. Cosoi, M. S. Vlad, V. Sgarciu
http://ceai.srait.ro/index.php/ceai/article/viewFile/18/8

Intelligent Word-Based Spam Filter Detection Using
Multi-Neural Networks
Ann Nosseir, Khaled Nagati and Islam Taj-Eddin
http://www.ijcsi.org/papers/IJCSI-10-2-1-17-21.pdf

Spam Detection using Adaptive Neural Networks: Adaptive Resonance Theory
David Ndumiyana, Richard Gotora, and Tarisai Mupamombe
http://onlineresearchjournals.org/JPESR/pdf/2013/apr/Ndumiyana%20et%20al.pdf

EDIT:
The basic intuition behind using a neural network to help with spam filtering is by providing a weight to terms based on how often they are associated with spam.

Neural networks can be trained most quickly in a supervised -- you explicitly provide the classification of the sentence in the training set -- environment.  Without going into the nitty gritty the basic idea can be illustrated with these sentences:

Text = "How is the loss of the Viagra patent going to affect Pfizer", Spam = false
Text = "Cheap Viagra Buy Now", Spam = true
Text = "Online pharmacy Viagra Cialis Lipitor", Spam = true

For a two stage neural network, the first stage will calculate the likelihood of spam based off of if the word exists in the sentence.  So from our example:

viagra => 66%
buy => 100%
Pfizer => 0%
etc..

Then for the second stage the results in the first stage are used as variables in the second stage:

viagra & buy => 100%
Pfizer & viagra=> 0%

This basic idea is run for many of the permutations of the all the words in your training data.  The end results once trained is basically just an equation that based of the context of the words in the sentence can assign a probability of being spam.  Set spamminess threshold, and filter out any data higher then said threshold.


Please enlighten me with Platt's SMO algorithm (for SVM) From A_Roadmap_to_SVM_SMO.pdf, pg 12.

(source: postimg.org)
Assume I am using linear kernel, how will I be able to get both the first and second inner product?
My guess, inner product of datapoint with datapoint j labelled class A for the first inner product of the equation and inner product of datapoint j with datapoints labelled class B for second inner product?

Your understanding is correct. The point is that equation (8)
$$y_i(<\textbf{w}, \phi_i> + b) - 1 = 0$$
is not exactly an equation, but a system of equations, one for each $i$ index of the support vectors (those for each $0<\alpha_i<C$.

The point is that you cannot compute $b$ during the optimization of the dual problem since it does not matter for optimization, you have to go back and compute $b$ from all the other equations you have (one possible way is (8)).

Vapnick suggestion is to not use only one of those equations, but two of them, specifically one support vector for a negative observation and one for a positive observation. In other words two support vectors which have opposite signs for $y_i$.

Let's name $A$ the index of one support vector and $B$ the index of a suport vector of opposite side, baiscally you select from the system of equations at (8) only two of them. Evaluate both of them and take the mean.

From:
$$y_A(<\textbf{w},\phi_A>+b)=1$$
$$y_B(<\textbf{w}, \phi_B>+b)=1$$
We get:
$$b_A=\frac{1}{y_A}-<\textbf{w},\phi_A>$$
$$b_B=\frac{1}{y_B}-<\textbf{w},\phi_B>$$
Where $b_A$ and $b_B$ are two estimations, then the mean is
$$b = (b_A+b_B)/2 = -\frac{1}{2}(<\textbf{w},\phi_A>+<\textbf{w},\phi_B>)=-\frac{1}{2}\sum_{i=1}^{n}y_i\alpha_i(<\phi(x_i),\phi(x_A)>+<\phi(x_i),\phi(x_B)>)$$


What is the term for data that is too sparse to represent the underlying data model? I am giving a presentation on Data Science, and I want to talk about the idea that data that is not "big" enough is a big barrier for Machiene Learning. Looking online, there are concepts like overfitting and underfitting, but I am more looking to talk about data that, even if fitted optimally, would still not actually be a good model for the system.

Is there a good term to use for this?

Small sample size is probably the concept you're looking for. A common failure in statistics is trying to draw conclusions from data that isn't "big" enough to accurately represent the underlying distribution.

It's worth noting that "small data" increases your chances of overfitting - which is another way of saying that your model is weak to outliers and noise. It's also worth noting that bootstrapping, simulation, and duplication might help (but not always).

There may be some other niche term for this, but if you relate it to basic stats there's a high probability everyone in the room will understand what you're talking about.


Algorithm for document retrieval in QA system I am working with question answering and machine reading comprehension system. I want to match questions and documents (around 100,000 docs) in database. I've used tf-idf but it accuracy is about 55% and I need to be at least 80%. Could you give me some advice?

I am still not quite sure how you are solving in the problem with tf-idf for a QA system. However, there have been lots of improvements that has been done in the QA domain over the years, with the usage of deep learning for natural language processing.

I would urge you to look at the following approaches that might help you to reach the accuracies you are looking for:

Bidaf Model that was used to solve SQuAD dataset.
BERT model that was used to solve the SQuAD dataset.

You have multiple open source implementations to train the network and use it for prediction. You could  also learn about them in the papers published by the creators of these networks.


Optimal parameter estimation for a classifier with multiple parameters The image on the left shows a standard ROC curve formed by sweeping a single threshold and recording the corresponding True Positive Rate (TPR) and False Positive Rate (FPR).

The image on the right shows my problem setup where there are 3 parameters, and for each, we have only 2 choices. Together, it produces 8 points as depicted on the graph. In practice, I intend to have thousands of possible combinations of 100s of parameters, but the concept remains the same in this down-scaled case.

I intend to find 2 things here:

Determine the optimum parameter(s) for the given data
Provide an overall performance score for all combinations of parameters

In the case of the ROC curve on the left, this is done easily using the following methods:

Optimal parameter: Maximal difference of TPR and FPR with a cost component (I believe it is called the J-statistic?)
Overall performance: Area under the curve (the shaded portion in the graph)

However, for my case in the image on the right, I do not know if the methods I have chosen are the standard principled methods that are normally used.

Optimal parameter set: Same maximal difference of TPR and FPR

  Parameter score = TPR - FPR * cost_ratio

Overall performance: Average of all "parameter scores"

I have found a lot of reference material for the ROC curve with a single threshold and while there are other techniques available to determine the performance, the ones mentioned in this question is definitely considered a standard approach. I found no such reading material for the scenario presented on the right.

Bottomline, the question here is two-fold: (1) Provide methods to evaluate the optimal parameter set and overall performance in my problem scenario, (2) Provide reference that claims the suggested methods to be a standard approach for the given scenario.

The answer depends a lot on what you want to achieve:

Please future reader's of your machine learning paper OR
Solve a use-case that matters to some business

In first case, you have to scan this audience's publications, see what they use, and use the same. There isn't a perfect metric, since actually it depends on the business' cost-ratio.

In the second case, all you need is the cost-matrix and the result follows naturally. Optimizing anything which isn't the cost-matrix is a rather academical exercise. The cost-matrix has one parameter and depends really on the business. No reference is going to solve this decision for you. The discussion between performance metrics is missing the point. There is only one answer once you manage to squeeze out a cost-matrix from your business stakeholders. In a business anything can be estimated in terms of money. It only gets tricky, if it more complex than a single class weight.

So the answers are:

Parameter performance: calculate cost; Overall performance: take max(), or to be more precise correct for testing multiple hypotheses, too
There isn't any, unless you find one which proves that optimizing a metric optimizes for all possible cost matrices simultaneously.

In that sense, ROCAUC is actually a pretty meaningless artifical metric, despite being popular. There is an interpretation which relies on the fitted model's probabilities being correctly estimated, but if you want to get real, only cost matrices matter. Model the real world impact of you model and you'll get answers!

PS: I personally consider TPR-FPR the best bet in absence of a defined cost matrix.


One-Class classification I am currently using an one class classification svm and I am trying to boost the classification results by employing more than one svm-occ with varying gamma parameters and combine these decisions with majority voting. But the results are not that satisfactory , since with a careful choice of gamma parameter I can get better results with just one-class svm. So , I was just wondering if I can employ also another one-class classifier e.g. SVDD and then combine its decisons with svm-occ . But the problem is my project is being developed in python enviroment and I can't find an implementation of SVDD with sklearn. So is there any suggestions which classifiers to use and if an according python implementation exists or how to combine them to obtain in overall better classification results ? (**I am keeping the nu parameter on svm-occ fixed)

Ummm... SVM is a linear model, so if you average the predictions of several models with different regularization, what you get is the same result of another model which uses a different regularization.
Regarding the intent behind your question: there's more methods than SVM and SVDD. You can check the PyOD package for ensemble models for outlier detection.


Export pandas dataframe to a nested dictionary from multiple columns Its a similar question to

Export pandas to dictionary by combining multiple row values

But in this case I want something different.

from pandas import DataFrame

df = DataFrame([
           ['A', 123, 1],
           ['B', 345, 5],
           ['C', 712, 4],
           ['B', 768, 2],
           ['B', 768, 3],
           ['A', 123, 9],
           ['C', 178, 6],
           ['C', 178, 5],
           ['A', 321, 3]],
           columns=['maingroup', 'subgroup', 'selectedCol'])

I'd want the output to be:

{
 'A': {'123':[1, 9], '321':[3]},
 'B': {'345':[5], '768':[2, 3]},
 'C': {'712':[4], '178':[6, 5]}
}

Using dict comprehension with nested groupby:

d = {k: f.groupby('subgroup')['selectedCol'].apply(list).to_dict()
     for k, f in df.groupby('maingroup')}

Output:

{'A': {123: [1, 9], 321: [3]},
 'B': {345: [5], 768: [2, 3]},
 'C': {178: [6, 5], 712: [4]}}


Understand how to simulate a statistics This solution describes how to simulate statistics to find a confidence interval.
A journalist called 1000 people in town to ask who will they be voting for out of candidates A and B.
The observed value came out to be 511 votes for A and 489 votes for B.
this makes us think that candidate A will win.
But we need to know if this sample is truly representative of the underlying population distribution. To find this, we simulate this poll 1000 times through below python function.

def sample(A,n=1000):
    return pd.DataFrame({'vote': np.where(np.random.rand(n) < A,'A','B')})

s = sample(0.51,n=1000)

dist = pd.DataFrame([sample(0.51).vote.value_counts(normalize=True) for i in range(1000)])

what I cannot understand is, what is the significance of parameter A in the function definition.

Is it trying to simulate a sample where A occurs 51% times?
If I am just trying to random samples from a population, why am I relying on 0.51 to do so?

The simulation aims at quantifying the uncertainty of the poll. Candidate A wins win with 51% of the votes. How likely is this?

The code simulates this poll (1000 times as you see in the range function) so that we get the probability of A winning the poll. You can calculate it with e.g.
dist.loc[dist.A>dist.B]['A'].count()/len(dist). You'll find here that A wins with a confidence of ~ 73%. With this you can answer questions like "How big should the size of polled population ($n$) be to get a confidence of 95%?"


Self-designed objective for linear regression learning A multiple linear regression is to use several predictor variables to predict the outcome of a response variable, like the following relationship:

$y_{i}=\beta_{1}x_{i1}+...+\beta_{p}x_{ip}+\epsilon_{i},   i=1,...,n$

I understand the typical objective to learn the $\beta$ paramters is least-squares, which means to minimize the sum of the sqaure of $\epsilon_{i}$. Now I want other kinds of objective, for example to maximize the Shannon entropy of the sequences of $\epsilon$ (or other self-specified objective). I googled towards this direction but no luck. I am wondering if there is any problem (and tool to solve it if possible) I can look into to do that?

Thank you for your help.

This question seems really old, it may be helpful to new visitors.

Yes, we can define any objective function and use gradient descent to minimize it. For that the objective function need to have some properties like

It should be convex function else we'll stuck at local minima or may never find the minimum value
It must be differentiable on every point and must have non zero derivative. Else gradient descent will not make progress and get stuck

If cost function have these properties then we can use tensorflow (or other library) to create graph to calculate the cost and then to calculate its gradient using auto-diff algorithm and use gradient descent to minimize it.

Auto-diff basically breaks calculation of cost function into elementary arithmetic computation like addition, multiplication, division etc and use chain rule to find derivative.


Using random forest to select important variables & then putting into logistic regression? I was wondering does it make sense to use random forest to select most important variables then put into logistic regression for prediction? I think that it might not make sense because what's important for random forest might not be important for logistic regression?

There are many factors which are underlying the 'importaces' of featues obtained from random forest.
For instance, features with more number of categories (unique values if its a numerical feature) would be more likely to get find splits; making it more important feature.
Having all features with same number of categories is not a common scenario.
So, even though the features identified are likey to be the best predictors (in logistic regressions etc.), one need to take the inherent biases in the randome forest algorithm into consideration while utilizing the 'important' features. Therefore whether it would make sense to use the important features from random forest model into logistic regression would depend on if any and how much the importances are biased.
Source for the information and further useful details on the biases in estimation of feature importances by random forest algorithm: Bias in random forest variable importance measures: Illustrations, sources and a solution Bias in random forest variable importance measures: Illustrations, sources and a solution


Is it possible to identify what data a neural net was trained on? For example, if there are 5 major data sets for a certain topic. Would it be possible to see if a deployed network was trained off of one them?

How would one approach this?

If you can see the loss for a specific dataset, then the most probable set would be the one with the lowest loss.


Predicting labels which are not independent of one another I want to make a multilabel classifier where the labels are dependent on one another. Concretely, I have a situation where:

If label A is true, then label B is also true.

If label B is true, label A could be true or false.

If label B is false, then label A is false.

How can I model this in a multilabel classifier?

In this case the easiest way is to use a simple single-label classifier with three labels:

A0B1
A1B1
A0B0

As long as the number of combinations is reasonable, that's what will give the best results while satisfying the constraints.

By the way "If label B is false, then label A is false" is logically implied by "if label A is true, then label B is also true". (contrapositive).


How to creat a plot for the accuracy of a model Iam pretty new to the whole topic so please dont be harsh. I know these may be simple questions but everybody has to start somewhere ^^

So I created (or more copied) my first little Model which predicts sons heights based on their fathers.

#Father Data
X=data['Father'].values[:,None]
X.shape

#According sons data
y=data.iloc[:,1].values
y.shape

#Spliting the data into test and train data
X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.2,random_state=0)

#Doing a linear regression
lm=LinearRegression()
lm.fit(X_train,y_train)

# save the model to disk
filename = 'Father_Son_Height_Model.pckl'
pickle.dump(lm, open(filename, 'wb'))

#Predicting the height of Sons
y_test=lm.predict(X_test)
print(y_test)

Now I wanted to create a plot that displays the accuracy of my model or how "good" it is.
Something like it is done here:
https://machinelearningmastery.com/how-to-calculate-precision-recall-f1-and-more-for-deep-learning-models/

But i cant quiet get it to work. This here Sems like I would work but what should I store in "model_history"?

plt.subplot(212)
plt.title('Accuracy')
plt.plot(model_history.history['acc'], label='train')
plt.plot(model_history.history['val_acc'], label='test')
plt.legend()
plt.show()

A easy to adapt tutorial-Link would be a geat help already.
Keras seems to be a thing but I would want to avoid yet another library if possible and sensible.

Your context is different than the one provided in the link. There, the author has made a neural network in Keras and has plotted the accuracy against the number of epochs. One epoch is when an entire dataset is passed both forward and backward through the neural network once. So, he is calculating accuracy after every epoch while the weights vary to fit data based on the loss function. (Thus, the accuracy increases as the number of epochs increases.)

In your case, you are performing a linear regression which fits the data and generates an equation. There is no feedback system. Accuracy here can be defined based on your need. In this case (Predicting sons height based on their father's), you can define accuracy as how accurate your predictions were. Take an error function like MAE (Mean absolute error). Lesser the error rate is more accurate is your model. MAE is an accuracy measure here.

For general classification tasks, accuracy is the number of instances you predicted correctly divided by the total number of instances. (In the link, author used default keras accuracy metric defined somewhat like this)


Demonstrating and visualizing cluster centroid updates by using the first four iterations in Python? I managed to create a dataset with 6 clusters and visualize it with the code below, now I would like to visualize demonstration of update of the cluster centroids in KMeans algorithm. This demonstration should include first four iterations by generating 2×2-axis figure
Here is my code:
# import statements
from sklearn.datasets import make_blobs
import numpy as np
import matplotlib.pyplot as plt
# create blobs
data = make_blobs(n_samples=200, n_features=6, centers=6, cluster_std=1.6, random_state=50)
# create np array for data points
points = data[0]
# create scatter plot
plt.scatter(data[0][:,0], data[0][:,1], c=data[1], cmap='jet',marker="+",label="Original Data")
plt.xlim(-15,15)
plt.ylim(-15,15)
plt.show()

How can I implement this algorithm? It has been asked before using R but, I would like to do it in python. Can you help me visualize the first 4 iterations?
Output should be like:

You can use the plt.scatter() and plt.subplots() to achieve this as follows:
import matplotlib.pyplot as plt
from sklearn.datasets import make_blobs
data = make_blobs(n_samples=200, n_features=8,
                           centers=6, cluster_std=1.8,random_state=101)

fig, ax = plt.subplots(nrows=2, ncols=2,figsize=(10,10))

from sklearn.cluster import KMeans
c=d=0
for i in range(4):
    ax[c,d].title.set_text(f"{i+1} iteration points:")
    kmeans = KMeans(n_clusters=6,random_state=0,max_iter=i+1)
    kmeans.fit(data[0])
    centroids=kmeans.cluster_centers_
    ax[c,d].scatter(data[0][:,0],data[0][:,1],c=data[1],cmap='brg')
    ax[c,d].scatter(kmeans.cluster_centers_[:, 0], kmeans.cluster_centers_[:, 1], s=200, c='black')
    d+=1
    if d==2:
        c+=1
        d=0

This will produce:


Could one algorithm fetch keywords from texts of different natural languages? I am new to practicing NLP and most topics related, but I want to make a program that can gather and extract data for me on its own.

To be more specific, I want to tell the program "I want more information on this topic(i.e heart attacks)", and then the program shall find, gather and extract meaningful texts on the topic from around the www.

I happen to live in Norway, which means that most interesting data will be in English, but I also want to fetch interesting data found in Norwegian.

One challenge is the differences in stop words. For instance, "are" and "and" are both stop words in English and subjects in Norwegian.
Other challenges are also likely to occur.

So my question is: Would I need to create separate algorithms for every natural language to be interpreted?

There are a few ways to deal with this issue.  Python has a package called NLTK which contains stop word lists for several languages (including English and Norwegian).  You can simply use this package, it's usage is as follows:

>>> from nltk.corpus import stopwords
>>> stop = stopwords.words('english')
>>> sentence = "this is a foo bar sentence"
>>> print [i for i in sentence.split() if i not in stop]
  ['foo', 'bar', 'sentence']

Alternatively, a method for automatically suppressing stop words is called tf-idf;  tf-idf is commonly used in search engines so that the most important words are promoted to the forefront.  In your case, I would suspect you'd want to have IDF scores for both English and Norwegian and apply only the appropriate one on a language to language basis.

http://en.wikipedia.org/wiki/Tf%E2%80%93idf