What is the difference between Okapi bm25 and NMSLIB? I was trying to make a search system and then I got to know about Okapi bm25 which is a ranking function like tf-idf. You can make an index of your corpus and later retrieve documents similar to your query.
I imported a python library rank_bm25 and created a search system and the results were satisfying.
Then I saw something called Non-metric space library. I understood that its a similarity search library much like kNN algorithm.
I saw an example where a guy was trying to make a smart search system using nmslib. He did the following things:-

tokenized the documents
pass the tokens into fastText model to create word vectors
then combined those word vectors with bm25 weights
then passed the combination into nmslib
performed the search.

If the above link does not opens the document just open it in incognito mode.
It was quite fast, but the results were not satisfying, I mean even if I was copy pasting any exact query from the doc, it was not returning that doc. But the search system that I made using rank_bm25 was giving great results. So the conclusion was
bm25 gave good results and nmslib gave faster results.
My questions are

How do they both (bm25, nmslib) differ?
How can I pass bm25 weights to nmslib to create a better and faster search engine?
In short, how can I combine the goodness of both bm25 and nmslib?

Note that I don't know nmslib and I'm not familiar with search optimization in general. However I know Okapi BM25 weighting.

How do they both (bm25, nmslib) differ?

These are two completely different things:

Okapi BM25 is a weighting scheme which has a better theoretical basis than the well known TFIDF weighting scheme. Both methods are intended to score words according to how "important" they are in the context of a document collection, mostly by giving more weight to words which appear rarely. As a weighting scheme, Okapi BM25 only provides a representation of the documents/queries, what you do with it is up to you.
nmslib is an optimized similarity search library. I assume that it takes as input any set of vectors for the documents and query. So one could provide them with vectors made of raw frequencies, TFIDF or anything else. What it does is just computing (as fast as possible) the most similar documents to a query, using whatever representation of documents is provided.

How can I pass bm25 weights to nmslib to create a better and faster search engine?

Since you mention that the results based on BM25 are satisfying, it means that the loss of quality is due to the nmslib search optimizations. There's no magic, the only way to make things fast is to do less comparisons, and sometimes that means discarding a potentially good candidate by mistake. So the problem is not about passing the BM25 weights, it's about understanding and tuning the parameters of nmslib: there are certainly parameters which allow the user to select an appropriate trade off between speed and quality.


Bayesian regularization vs dropout for basic ann Does it make sense conceptually to apply dropout to an artificial neutral network while also applying bayesian regularization?

On one hand I would think that technically this should work just fine, but on the other hand if bayesian regularization is like finding the MAP estimate given some prior shouldn't the effect just be that it creates random noise in your outcomes?

It actually makes perfect sense to use both. Gal et al. provided a nice theory on how to interpret dropout through a Bayesian lense. In a nutshell, if you use dropout + regularization you are implicitly minimizing the same loss as for a Bayesian Neural Network (BNN), where you learn the posterior distribution over the network weights given the training data.

You can think of the fully Bayesian approach as taking it one step further on the road from MLE over MAP to Bayesian regression, as the former two provide only point estimates, while the latter gives access to the entire posterior distribution.

If you approximate a BNN with dropout, the introduced noise will take the role of drawing samples from the posterior distribution. The regularization will take the role of a prior, which is the same interpretation you would give it in the MAP-context. I don't think, however, that the represented priors are identical.

Either way, the prior helps to regulate the "spread" of the posterior. If you choose a large regularization parameter, which corresponds to a narrow prior distribution, the model will reduce variance in the posterior distribution as well.

If you only need a simple ANN with dropout and regularization, you probably don't care too much about using the approximate posterior after training, but I think this is still a nice perspective on things.


Check Imputation Efficiency - How To Compare Data Frames? I try to evaluate several NA imputation methods with supervised approach: I clone my original data frame with no NAs, artifically insert NAs into the resulting Data Frame and apply imputations to the latter.

Now, I'd like to evaluate the imputations by comparing the imputed new DFs with the original one. I wonder what would be the best metod; is there any distance method for instance that I could apply to the original/imputed DF pairs?

(My DF contains only numeric data, but a solution that handles factor variables as well would be especially handy).

If we want to check the imputation efficiency, we should probably compute the performance for the imputed values only. Indeed, the performance depends on the percentage of missing values.


How to find the weights for weighted least squares regression? When we are doing weighted least squares how do we find the weights? Where ever I see tutorials are just using $w_i = \frac{1}{(sigma)i^2}$ and doing it with basic data. But I want to know how to find the weights for real data. Is it always the inverse of the square of variance?

No, the weight does not always has to be equal to the inverse of the square of variance. There is no universally accepted scheme. Different weighting schemes are used, such as $1/x$, $1/x^2$, $1/\sqrt{x}$, where $x$ are your data. The weighting scheme depends on many things: the distribution of your data, their meaning, and in which regions you want to fit them more closely (whether small data values should have lower error), etc.
This is a paper that compares different weighting schemes for data in chemistry: [1]. The introduction in that paper has references to many other papers discussing different weighting schemes.
[1] Ram B. Jain, Comparison of three weighting schemes in weighted regression analysis for use in a chemistry laboratory, Clinica Chimica Acta,
Volume 411, Issues 3–4, 2010, Pages 270-279, ISSN 0009-8981, https://doi.org/10.1016/j.cca.2009.11.021.


What is meant by Distributed for a gradient boosting library? I am checking out XGBoost documentation and it's stated that XGBoost is an optimized distributed gradient boosting library.

What is meant by distributed?

Have a nice day

It means that it can be run on a distributed system (i.e. on multiple networked computers).

From XGBoost's documentation:

  The same code runs on major distributed environment(Hadoop, SGE, MPI) and can solve problems beyond billions of examples. The most recent version integrates naturally with DataFlow frameworks(e.g. Flink and Spark).


Causal inference VS Active learning? Imagine we have some lists of features that are changing in time. Each row of the list corresponds to a sample (Change in space). I would like to know whether machine learning is able to determine the effect of each sample on another sample. For instance, the target value for the sample "S" is dependent on features of samples "S-4","S-3","S-2","S-1","S+1","S+2","S+3". I have seen something like Active learning and Causal Inference but still not sure each of which would be useful for my aim.
To elaborate more, imagine we have the picture below:

the red line is a result for one year and the blue one is next year. we have these results in an appropriate amount so in this manner we do not have problem. for the target that has been shown by the red circle and other samples, we have different features. But I am looking for an algorithm to tell me whether group 1 is affecting my target in the red circle point or group 2. To this aim, it is good to use Causal inference or Active Learning?

This is how the problem is formulated as a Causal Inference problem:
Take group 1 as the Control Group and group 2  the Treatment Group. Based on the observations ( "S-4","S-3","S-2","S-1","S+1","S+2","S+3"), a model is fit on the observations. How many models depend on the type of learning, S,T or X-learner.
Basically, what these learners do, is separately fit the features to the target (S), conditionally on the features originating from control or treatment.
Once you have fit the learners, a treatment effect is estimated.
You can see the treatment effect as the difference between the predictions on either group, given the knowledge restricted to the opposite group.
Here is a sample code from Uber's causalML
from causalml.inference.meta import XGBTRegressor

y, X, treatment = load_data()

xg = XGBTRegressor(random_state=42)
te, lb, ub = xg.estimate_ate(X, treatment, y)
print('Average Treatment Effect (XGBoost): {:.2f} ({:.2f}, {:.2f})'.format(te[0], lb[0], ub[0]))


How are non-restricted Boltzmann machines trained? Restricted Boltzmann machines are stochastic neural networks. The neurons form a complete bipartite graph of visible units and hidden units. The "restricted" is exactly the bipartite property: There may not be a connection between any two visible units and there may not be a connection between two hidden units.

Restricted Boltzmann machines are trained with Contrastive Divergence (CD-k, see A Practical Guide to Training
Restricted Boltzmann Machines).

Now I wonder: How are non-restricted Boltzmann Machines trained?

When I google for "Boltzmann Machine", I only finde RBMs.

I will apologize in advance for linking to the Wikipedia article, but I'm not sure I can do a better job explaining it.  https://en.wikipedia.org/wiki/Boltzmann_machine

(Will extract selected passages to prevent link rot shortly.)

One of the reasons we see so much talk about training RBMs and not much talk about the other is, by my recollection of a talk by Hinton, "Intractable, except for very small problems or limited cases."  The activation of a hidden node depends not only on the state of visible nodes, but on its compatriots. This means our clever contrastive divergence and gradient decent methods don't work!

But you didn't ask why it's difficult, you asked how to do it.

The short, short answer is simulated annealing.

The long short answer, we select an initial state for the network, then perform updates until the network reaches a global thermal equilibrium, then try and adjust the weights while reducing the temperature until our activations approximate a global minimum.  The process repeats until we're happy with the outcome.


Implementing the Trapezoid rule without the formula for the curve I know that if I have some function f(x) that describes a curve, I can approximate the area under the curve using the trapezoid rule as follows:
def auc(f, a, b, n):

    subinterval = (b - a) / n

    s = f(a) + f(b)
    i = 1
    while i < n:
        s += 2 * f(a + i * subinterval)
        i += 1

    return (subinterval / 2) * s

However, I am trying to implement the trapezoid rule to approximate the area under the ROC curve. I don't have the function f(x), but rather true positive rates and false positive rates at thresholds from 0 to 1 spaced by .01. I tried to implement the rule following this guide https://byjus.com/maths/trapezoidal-rule/ as so:
def roc_auc(tprs, fprs):

    y_sum = max(tprs) + min(tprs)
    for i in range(1, len(fprs)-1):
        y_sum += 2*tprs[i]

    interval = (max(fprs) - min(fprs)) / len(fprs)
    return ((interval / 2) * y_sum) / 100

However, when I test it against auc functions implemented in numpy, scikit, etc. I get different values than the one I calculate so I know I'm doing something wrong. Can anyone tell me where I'm going wrong?

You're assuming that the points are equally spaced along the fpr axis, which is generally not true.  See e.g. the "Uniform grid" vs "Nonuniform grid" sections of the wikipedia article.  You need something like
delta_xs = np.diff(fpr)
left_endpoints_y = tpr[:-1]
right_endpoints_y = tpr[1:]
trap_areas = 0.5 * (left_endpoints_y + right_endpoints_y) * delta_xs
area = trap_areas.sum()

(That's more verbose than it needs to be, probably not the most efficient, and I don't know what order your fpr/tpr lists are, so it'll need some finagling.)
Using thresholds at 0.01 spacing is a little unusual too: the ROC curve is represented by using every predicted probability as a threshold (together with $\pm\infty$, or a convention that $(0,0)$ and $(1,1)$ are on the curve).


When is feature transformation required? I was fitting machine learning models to clean data(Imputed missing values, removed unnecessary features etc). I didn't transform the features that are skewed. Before moving forward, I want to understand how important feature transformation is to fit data into a model. Any opinions?

(I know what happens in Random Forest, but unable to comprehend for other ML models)

Even if there are models that are robust w.r.t. feature transformations (like Random Forest), in general it's a good practice to transform the features in order to have better performance in a ML model.

For three reasons:

1) Numerical stability: computers cannot represent every number, because the electronic which make them exist deals with binaries (zeros and ones). So they use a representation based on Floating Point arithmetic. In practice, this means that the numerical behavior in the range [0.0, 1.0] is not the same of the range [1'000'000.0, 1'000'001.0]. So having two features that have very different scales can lead to numerical instability, and finally to a model unable to learn anything.

2) Control of the gradient: imagine that you have a feature that spans in a range [-1, 1], and another one that spans in a range [-1'000'000, 1'000'000]: the weights associated to the first feature are much more sensitive to small variations, and so their gradient will become much more variable in the direction described by that feature. This can lead to other instabilities: some values of learning rate (LR) can be too small for one feature (and so the convergence will be slow) but too big for the second feature (and so you jump over the optimal values). And so, at the end of the training process you will have a sub-optimal model.

3) control of the variance of the data: if you have skewed features, and you don't transform them, you risk that the model will simply ignore the elements in the tail of the distributions. And in some cases, the tails are much more informative than the bulk of the distributions.


Markov Decision Process representation I'm attempting to model a simple process using a Markov Decision Process.

Let $A$ be a set of $3$ actions : $ A \in \{b,s\}$.
$T(s,a,s')$ represents the probability of if in state $s$ , take action $a$ and end up in state $s'$

Notation for the MDP diagram is as follows :

Here is my MDP diagram which models 7 states:

The outgoing actions for each state sum to 1.

$T(1,b,2) = .7 $

$T(1,b,3) = .3 $

$T(1,s,4) = .9 $

$T(1,s,5) = .05 $

$T(1,s,6) = .05 $

I've tried to keep this as simple as possible to check my understanding. Are my representations & probabilities correct ?

Looks 'correct' to me, in the sense that it satisfies the requirements for being an MDP. Whether it models the underlying real-world problem correctly cannot be validated with the information given here.


Classifier with a single 1 value per year Let's say I want to find the probabilities of winning the best movie category in the Oscars. I know to following rules:

There is only 1 winner per year.
Logically, the sum of the predicted probabilities for each year should add up to be 1.

I can have a year feature in my classifier but that does not mean that 1 and 2 are enforced. If I use a simple classifier like logistic regression I also don't see how having the year as a feature is going to help.

My questions are:

Is there a way to enforce those rules in a model? Does it matter?
Should I just use a model without those rules and normalize the probabilities to 1 when I try to predict for a given year?

As @Emre noted in the comments, you will want to use the softmax function. After acquiring a set of scores per film, the function will squash the scores into the range [0,1] and the scores will add up to 1.

Here's a possible procedure:

Generate a feature set per image
Train a model on your feature set to output a score for a movie.  The score is a number in an arbitrary domain.
Repeat step 2 for all of your candidate movies to create a score vector.
Feed the score vector into the softmax function to squash the scores into the domain [0,1] and make them add to 1

For reference, the softmax function is defined:

$ \sigma(z_i) = \frac{e^{z_i}}{\displaystyle\sum^N_{n=1}e^{z_n}} $

And an example:

Our feature set will include the following: critic/audience ratings, revenue, cost, total tickets sold, etc.

Let's say you have historical data of the Oscars that includes the feature sets and a score that we will use for the ground truth metric.  You define the score.  For instance, if a movie was not even nominated for the Oscar award, it might have a score of zero, whereas an Oscar winning movie might have a score of 1.  A movie that was nominated but did not receive many votes might have a score of 0.50.

You train a model on your historical data so that, given a movie's feature set, it will output a score, similar to your training set.

Now you are considering three movies for prediction: Avengers Infinity War, Deadpool 2, and Venom.

You acquire their feature sets (the same categories that you used for training: critic/audience ratings, revenue, etc.)

Then you pass each feature set through your model and acquire a vector of scores:

\begin{array}{|l|c|}
\hline
Movie & Score\\
\hline
Avengers\ Infinity\ War & 0.98\\
Deadpool\ 2 & 0.82\\
Venom & 0.24\\
\hline
\end{array}

We can interpret the score results as a probability by using the softmax function:

The denominator of the function is given:

$ \displaystyle\sum^N_{n=1} e^{z_n} = e^{0.98} + e^{0.82} + e^{0.24} \ \ \ where: z=\{0.98,0.82,0.24\} $

We calculate the softmax of a given score $ x $ as such:

$ \Large\frac{e^x}{e^{0.98} + e^{0.82} + e^{0.24}} $

Therefore the softmax scores are:

\begin{array}{|l|c|c|}
\hline
Movie & Score & Softmax\\
\hline
Avengers\ Infinity\ War & 0.98 & 0.42932\\
Deadpool\ 2 & 0.82 & 0.36584\\
Venom & 0.24 & 0.20484\\
\hline
\end{array}

and we can see that:

$ 0.42932 + 0.36584 + 0.20484 = 1 $


Probability of Gaussian Naive Bayes How would I go about attaching a probability to the prediction outputted by a Gaussian Naive Bayes model ?
I'm asking because the predict_proba function U can use with sklearn's Gaussian Naive Bayes classifier only gives me zeros or one, which is not what I want.

I'm not familiar with sklearn NB methods but it's likely that these are actually the predicted probabilities. NB is known to often assign extreme probabilities, i.e. close to either 0 or 1. This issue implies that the probabilities predicted by NB are not really meaningful, i.e. they cannot (usually) be interpreted as "there is an x % chance that this instance belongs to this class" for example.


Does the Bishop book imply that a neuron feeds to itself in chapter 5.3? I just read Bishop's book Pattern Recognition and Machine Learning. I read the chapter 5.3 about backpropagation, and it said that, in a general feed-forward network, each unit computes a weighted sum of its inputs of the form $$a_j=\sum\limits_{i}w_{ji}z_i$$

Then the book says that the sum in the above equation transformed by the non-linear activation function $h(.)$ to give the activation $z_j$ of unit $j$ in the form $$z_j=h(a_j)$$

I think the notation is somehow akward: suppose I want to compute $a_2$, then
 $$a_2=w_{21}z_1+w_{22}z_2+\dots$$

Then does  $$a_2=w_{21}z_1+w_{22}h(a_2)+\dots$$ mean that the neuron $a_2$ is connected to itself?

The equations are only working for a given layer.

If you want to generalize, you need to rewrite them as, for example :
$$a^l_j=\sum\limits_{i}w^l_{ji}z^{l-1}_i + b^l_j$$


Need to prepare the data to Link Analysis project? I've a dataset with following schema:

Customer_ID - Unique ID
Product - ID of purchased product
Department - ID of the department that sells the product
Product_Type - The purchased product type
Date - The date of purchase
Quantity - The number of units purchased

I need to do a link analysis project to analyze some consumption patterns of the products and answer the following questions:

"If product B is purchased then customer will also take product A"

I will use Scala/Python to make the link analysis over the datasets but the  examples I have seen are dataset with direct links to the project of "Flight Data" in the schema is:

ID
Origin
Destination

My question is: there I need to prepare my dataset to make the Link Analysis (there exists some best practices to do this?) or can I analyze the dataset with that structure?

Many thanks! Sorry my inexperience on this topic!

When referencing code examples built to analyze phenomena that differs from what I am studying, I find it best to learn as much as I can about the procedures themselves.

In this case, I would look at the procedures defined in the example code you have found, run a few tests on a subset of your own data to make sure you know how the procedures are functioning, and then, if you are satisfied with the results, proceed to use the code to in your analysis (making sure to cite code sources where required or applicable).


City pollution data for data mining project I want to do my MSc Project on Data Mining. And I want to work on pollution in cities.
Are there accessible and useable pollution data (any type of pollution. e.g. air, water or forest pollution) in specific city or area?
If not, do you have any advice on how to get or search for them?
Thank you..

The second module of this free machine learning class revolves around working with geospatial pollution data for a river in the Netherlands:

https://www.edx.org/course/applied-machine-learning-microsoft-dat203-3x-1


looking for approaches to detecting outliers in individuals unequal sequential time series I am looking for approaches related to outlier detection in time series.

Example:
A person visits hospital overtime on multiple bases and there are some measurements made (bmi, blood_pressure, stress_level) at each occasion. Usually, the stress value will be the same for most of the individuals but somehow for one person increases continuously over time. So, the idea is to find such a person who are dissimilar to all others over time.

Some of the literature which I have already searched look to deal with time series on continuous data and some of the approaches involve similarity-based where an average threshold similarity is created(DTW and DTW-Adaptive) and with the nearest neighbor approaches a most dissimilar series whose distance greater than the threshold is identified. These series are usually long and are sensor observations.

The series which I am looking at is a short time series of discrete observations and may vary in length. I am looking for approaches that are usually suitable for them.

Could the multivariate data outlier detection algorithms be suitable over time or could be adapted over time?

Additionally, are there any outlier models which take time into account? (As I could not find anything on this in the literature)

An example:

Patient Time bmi blood_pressure stress
1       t1  v1  v2              v3
2       t1  v1  v2              v3
3       t1  v1  v2              v3
4       t1  v1  v2              v3
5       t1  v1  v2              v3
1       t2  v1  v2              v3
2       t2  v1  v2              v3
4       t2  v1  v2              v3
3       t2  v1  v2              v3
5       t2  v1  v2              v3

Some of the multivariate approaches already looked at are:
1. https://pyod.readthedocs.io/en/latest/ - Only holds for static data.
2. Shokoohi-Yekta, M., Hu, B., Jin, H., Wang, J., & Keogh, E. (2017). Generalizing DTW to the multi-dimensional case requires an adaptive approach. Data mining and knowledge discovery, 31(1), 1-31. (Time-series based approach but uses sensors)

I recently also thought, if I could include time itself as a feature to model outlier? Has anyone did this way or have any literature in these lines

Outlier detection doesn't sound like the most promising approach to me, as you have a model for the data.  Some ideas you could try: use a hypothesis test to check the hypothesis that the stress values fit iid Gaussian with pre-defined standard deviation and unknown mean; use linear regression to fit a line that predicts stress as a function of time, and see if the slope is greater than zero by a statistically significant amount; etc.


RL: Collecting States (training data) in real-life. Must use fixed timestep? I am using a Reinforcement Learning agent to play a 3D game, but have trouble with collecting the "current and next state" pairs.

To decide what action to perform, the network must perform a forward pass.

It performs forward pass in time $t$, but in the meantime the actual environment might have evolved significantly.  In computer games this means something like 10 frames or more have already passed (a varying amount).

The situation is worsened if I run, say 100 games at once on the same computer.

I don't have the ability to stop the game at each frame to do forwardprop. Anyway  it wouldn't be possible were I to train, say a real-life robot to walk.

Question:

Should I stick to a 'fixed timestep' approach, only asking to provide an action every 0.1 seconds? While it computes next action, I could pretend the network keeps outputting the most recent action for all the skipped frames. Good idea?

If that's the only option, then should I avoid at all costs situations where forward prop takes more than the 'fixed timestep'? (more than 0.1 sec in my case) So it's better to choose say, 0.2 seconds just to be safe.

Seems quite unreliable - is there a better way to do it?

Is there a paper that explores the alternatives? (I guess such a paper will be about real-life robot training)

Your "fixed timestep" idea is actually very similar to a common technique called frame skipping. Instead of waiting a fixed amount of time, agents wait a fixed number of frames $k$ before choosing a new action. In the meantime, they repeat their most recently chosen action.

Frame skipping was included as part of the Atari 2600 Arcade Learning Environment. It was also used in the foundational DQN paper. Common values of $k$ are 3, 4, and 5. The value chosen depended on the game being played, since different games had important events happen at different time resolutions. In these papers, frame skipping enabled training to happen roughly $k$ times faster. So this is definitely a valid technique to try.

I actually think this would generally be less of a concern in the robotics application. Forward propagation usually happens much more quickly than real-world event timescales. As an example, Stanford famously applies RL to fly small helicopters, which requires considerable precision.

Finally, if your forward propagation really is taking too long, you should consider a faster architecture. One approach would be just to make your neural net smaller. You might consider policy distillation for compressing a large, trained network into a smaller one. Also, make sure you're not using some ridiculously slow activation function like sigmoid or tanh. ReLU is the common choice if you don't need a bounded output for a given neuron. If you do, I recommend softsign.

If your time bottleneck is actually in action selection, due to a large action space and using a value network, you should seriously consider switching to a policy-based method (e.g. actor critic). This would help because sampling from a distribution over actions would potentially be much faster than the $\max$ operation involved in value-based methods. You can read more about this in Section 13.7 of Sutton and Barto's RL book.


Visualizing many one-to-many relationships I am trying to visualize the language tags on github repository data. I have names of 16k github repositories, and all the languages associated with each one of them. Below is a chord diagram I came up with.

However, I find that the chord diagram is not a very good representation of the data because

It does not show one-to-many relationships. For example, most of the repositories with JavaScript will have both html and css. This is represented as JavaScript-HTML and JavaScript-css, but not all together.
The size of the arc does not represent the number of repos in the dataset. For example, the number of repos that use JavaScript is about 6k in my dataset, however, the arc is about 17k. This is because of multiple languages in each repo. For example, if a repo has JavaScript, HTML, CSS and Python, the length of the arc would be 3.

Do you have any suggestions for a better visualization of this data, Thanks. This particular one is a d3 chord diagram, but the visualization can be done using any package (Python or JavaScript or R).

Here is the dataset if you are interested, do this query on Google BigQuery

SELECT
    sample_repos.repo_name,
    sample_repos.watch_count,
    languages.LANGUAGE
FROM
  `bigquery-public-data.github_repos.languages` languages
INNER JOIN
  `bigquery-public-data.github_repos.sample_repos` sample_repos
ON
  languages.repo_name = sample_repos.repo_name
WHERE sample_repos.repo_name IN (
  SELECT repo_name[OFFSET(0)]
  FROM `bigquery-public-data.github_repos.commits`)
ORDER BY sample_repos.watch_count DESC
LIMIT 16000

Without seeing the data itself, two visualisations come to mind. One option is to run hierarchical clustering on the repositories, and plot the results as a dendrogram with heatmap (or clustermap). The second would be to generate a force-directed graph of language co-occurrence across the repos.

Dendrogram with Heatmap

This plots all language co-occurrences, similar to an adjacency matrix; this clustermap produced with seaborn is an example of a dendrogram with heatmap:

It might be possible to gain insight on predominance/hierarchy from the dendrogram by considering which languages are seen more frequently with others, or not at all. The general co-occurrence of languages should be evident from the heatmap.

The advantage of this visualisation is that it should show relationships between languages more clearly than the chord. In particular, some insight might be gained from the 'hierarchy' of languages within each cluster generated. However, a drawback is that the number of repos is not easily represented.

Force-directed Graph

A visualisation that addresses the drawback of the previous method is the force-directed graph, where structure is derived by using "repulsive forces between nodes and attractive forces between adjacent nodes". This might show the numbers of repos involved more effectively, as nodes appearing more frequently with other nodes are weighted with greater 'mass':

The graph above by Mike Bostock is a great example with code and data, but for your use case it may be improved if language counts were represented by the size of node, similar to this visualisation by Christopher Manning. As an afterthought, this bubble chart might also serve if it could be reworked with edges connecting nodes to show where languages were used together in a repo.

The caveat is that without seeing the data I'm not exactly certain what is possible with this dataset. If you can wrangle the data into counts of which languages appear in each repo, and assume that all languages appearing in the same repo are co-occurrences, then these visualisations should be an option.


Keras: confusion matrix with predict_generator I'm not using ImageDataGenerator because i'm using hdf5 files. I used DataGenerator class (1) to feed data to the model.fit_generator. How can I get the confusion matrix given that I have batches of random data? I know how to get it with model.predict using sklearn.

(1) https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly#

loaded_model.predict_generator(generator=test_generator)

will give us a set of probabilities

y_true = test_generator.classes

will give us true labels

Because this is a binary classification problem, you have to find predicted labels. To do that you can use:

y_pred = probabilities > 0.5

Then we have true labels and predicted labels on the test dataset. So, the confusion matrix is given by:

font = {
'family': 'Times New Roman',
'size': 12
}
matplotlib.rc('font', **font)
mat = confusion_matrix(y_true, y_pred)
plot_confusion_matrix(conf_mat=mat, figsize=(8, 8), show_normed=False)


How do you distinguish between conversational text and possible news article? Context

When you receive messages in group chats, how do you detect if that message belongs to conversational dialogue or if it is a 'news' article (could be fake or real) that they are sharing?

Examples

Conversational dialogue: "Does anyone wanna have dinner today? I am pretty free this evening and I don't have dinner at home. Please let me know by 8pm!!"

'News' article: "New Japan’s growth has been helped by YouTube, which has made New Japan’s matches more accessible to an audience outside of Asia, said Dave Meltzer, publisher of the Wrestling Observer Newsletter, which has followed the sport since 1983. Capitalizing on this rise, New Japan launched an online streaming service — similar to the W.W.E. Network — in December 2014."

Question

Would you use rule-based matching or a classifier for this problem? (Assuming you already have a classifier for detecting if the news article is fake or real)

News sentences will have more unique tokens than normal conversations.
Conversations have more stop words than news articles.

I think you can use bert or normal wordvect classification to train a baseline model here.
I would play aroud the pipeline  of fake news classifier and news-conversation classifier. like passing the text to news classifier first and then passing it to news-conversation classifier. Try to mix and match to get the best results. Set some thresholds.


Extracting Features Using TensorFlow CNN I'm trying to extract features of set of images. I'm using CNN from this site.

Can anyone please tell me how to do feature extraction of images using CNN? I looked for various places. But nowhere it's clearly mention the feature extraction part.

CNN feature extraction in TensorFlow is now made easier using the tensorflow/models repository on Github. There are pre-trained VGG, ResNet, Inception and MobileNet models available here.

I have used the following wrapper for convenient feature extraction in TensorFlow. You can just provide the tool with a list of images. It will pre-process the images and extract the features by feed-forwarding through the network. You can specify the layer names that you want to extract and save them to HDF5 or pickle afterwards.

https://github.com/tomrunia/TF_FeatureExtraction


Industrial application(s) of LDA (latent Dirichlet allocation)? LDA ( Latent Dirichlet allocation) - is quite a popular topic in data-mining.

Question What are the industrial systems using LDA or may be some related models ?  (May be Google/Amazon/ ... ? )

PS

I would be also thankful for comments on:
https://mathoverflow.net/questions/304352/latent-dirichlet-allocation-math-words-digest

I can't answer for the systems in already existing companies, but I can definitely share an application of LDA in NLP.

Latent Dirichlet Allocation is a popular technique use for topic modelling in Natural Language Processing. The idea is to have corpus of natural langue text with lots of documents and the goal is to get the distribution of the words appearing in the corpus each (Distribution) being termed as a topic.

For visual analysis you can view this note book here. It makes use of the NLP library built for LDA, called pyLDAvis.

So lets say you have a corpus, and you want to build a tagger which labels the document(input text) into one of the N Classes, but you don't have labelled data. How do you start? One way is to use LDA to perform Topic Modelling, get the topics, then name the topics to N Classes. Label the data using LDA model, Use the loosely labeled data to build a classifier.


Date Extraction in Python I would like to extract all date information from a given document. Essentially, I guess this can be done with a lot of regexes:

2019-02-20
20.02.2019 ("German format")
02/2019 ("February 2019")
"tomorrow" (datetime.timedelta(days=1))
"yesterday" (datetime.timedelta(days=-1))

Is there a Python package / library which offers this already or do i have to write all of those regexes/logic myself?

I'm interested in Information Extraction from German and English texts. Mainly German, though.

Constraints

I don't have the complete dataset by now, but I have some idea about it:

10 years of interesting dates which could be in the dataset
I guess the interesting date types are: (1) 28.02.2019, (2) relative ones like "3 days ago" (3) 28/02/2019, (4) 02/28/2019 (5) 2019-02-28 (6) 2019/02/28 (7) 2019/28/02 (8) 28.2.2019 (9) 28.2 (10) ... -- all of which could have spaces in various places
I have millions of documents. Every document has around 20 sentences, I guess.
Most of the data is in German

Stanford CoreNLP has a very good implementation of NER for date/time.

https://nlp.stanford.edu/software/sutime.html   (demo: http://nlp.stanford.edu:8080/sutime/process)

Though this is written in Java, there are quite a few Python wrappers for this library (Such as : https://github.com/FraBle/python-sutime).  List of such libraries : https://stanfordnlp.github.io/CoreNLP/other-languages.html


Combining outputs of ridge regression models? I am facing an issue where I have 7 sets of different variables/columns/predictors.

I am trying to predict same target variable and I want to observe the importance/effect of all the sets according to their importance in an ordered manner. (I am trying to use ridge regression models for each of the 7 individual set as I want to keep all the variables and I want to combine the output of these 7 models, each set has more than 20 variables)

If you mean predictors by "variables", You may combine your 7 models by using "rbind or cbind" and save it into a data frame or data table. But, remember to first save your each of the 7 models into a datatable or dataframe.

About ranking, I agree with @Valentin you may want to make the predictions first, then rank it . However , if you want to take all the predictors into account there is no point to apply the ranking of all the sets. You will see their importance once you are done predicting.

Additional note: Glmnet is way more sensitive library/method when performing predictions, make sure all the columns while creating a model and doing a prediction are in the same order and have the same names. This caused me much trouble while predicting.
Hopefully, this might help you.


Input sequence ordering for LSTM network When training a LSTM network with time series data, I guess the order in which this data is fed matters, my question is how should this ordering be...

Let's take a time-series vector which will be the input for the LSTM:

\[ X = [x_0, x_{\small(-1\small)}, x_{\small(-2\small)},\ldots, x_{\small(-N\small)}]\]

with negative indexing indicating past values.

Which vector should be fed (theoretical case, not related to any API)?:

\[ X = [x_0, x_{\small(-1\small)}, x_{\small(-2\small)},\ldots, x_{\small(-N\small)}]\]

or

\[ X^r = [x_{\small(-N\small)},\ldots, x_{\small(-2\small)}, x_{\small(-1\small)}, x_0]\]

More precisely, what of these two orderings should be used in TensorFlow LSTM-Cells?

In order to make this decision, you have to think about what you want the representation to be passed to the next layer (or network output) to represent.

If you want the representation at (after) $t = 0$ to be passed, you should pass the arrays in the $X = [x_{-n} .. x_0]$ order. The LSTM cell will form a representation of the sequence, then, at $t = 0$.

In the reverse case, the representation will indicate the state at (before) $t = -n$

That's not to say that this is always a simple question for LSTM design- frequently in some domains such as NLP (text modeling), a bidirectional LSTM is used. This, basically, means both representations are implemented and used.


How to use GridSearchCV with RidgeClassifier I'm trying to use GridSearchCV with RidgeClassifier, but I'm getting this error:

My problem is regression type.

  IndexError: too many indices for array

I'm new to Machine Learning, please help me out.
This is the code I've been trying to implement:

from sklearn.grid_search import GridSearchCV
from sklearn.metrics import classification_report

tuned_parameters = [{'kernel': ['rbf'], 'gamma': [1e-3, 1e-4],
                     'C': [1, 10, 100, 1000]},
                    {'kernel': ['linear'], 'C': [1, 10, 100, 1000]}]
scores = ['precision', 'recall']
alphas = np.array([1,0.1,0.01,0.001,0.0001,0])
model = RidgeClassifier(normalize=True, random_state=100, tol=0.1)
for score in scores:
    clf = GridSearchCV(estimator=model, param_grid=dict(alpha=alphas))
    clf.fit(X, Y)
    print("Best parameters set found on development set:")
    print(clf.best_params_)
    for params, mean_score, scores in clf.grid_scores_:
        print("%0.3f (+/-%0.03f) for %r"
              % (mean_score, scores.std() * 2, params))

This is the complete error log:

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-59-c97c7e0fc6f3> in <module>()
     12 for score in scores:
     13     clf = GridSearchCV(estimator=model, param_grid=dict(alpha=alphas))
---> 14     clf.fit(X, Y)
     15     print("Best parameters set found on development set:")
     16     print(clf.best_params_)

/usr/local/lib/python2.7/dist-packages/sklearn/grid_search.pyc in fit(self, X, y)
    836
    837         """
--> 838         return self._fit(X, y, ParameterGrid(self.param_grid))
    839
    840

/usr/local/lib/python2.7/dist-packages/sklearn/grid_search.pyc in _fit(self, X, y, parameter_iterable)
    551                                  'of samples (%i) than data (X: %i samples)'
    552                                  % (len(y), n_samples))
--> 553         cv = check_cv(cv, X, y, classifier=is_classifier(estimator))
    554
    555         if self.verbose > 0:

/usr/local/lib/python2.7/dist-packages/sklearn/cross_validation.pyc in check_cv(cv, X, y, classifier)
   1833         if classifier:
   1834             if type_of_target(y) in ['binary', 'multiclass']:
-> 1835                 cv = StratifiedKFold(y, cv)
   1836             else:
   1837                 cv = KFold(_num_samples(y), cv)

/usr/local/lib/python2.7/dist-packages/sklearn/cross_validation.pyc in __init__(self, y, n_folds, shuffle, random_state)
    568         for test_fold_idx, per_label_splits in enumerate(zip(*per_label_cvs)):
    569             for label, (_, test_split) in zip(unique_labels, per_label_splits):
--> 570                 label_test_folds = test_folds[y == label]
    571                 # the test split can be too big because we used
    572                 # KFold(max(c, self.n_folds), self.n_folds) instead of

IndexError: too many indices for array

The problem is that Y has shape (15780,1). The shape of Y should show (15780,). The explanation for this is a bit long, so I just give here the links-

Difference between shapes (R,1) and (R,)

Solution for IndexError: too many indices for array


Extract 2 pieces of information from a string - what to use? First of all, I am a complete newbie in regard to data science and I am not asking for the complete solution but some guidance as to what I should read up to achieve my task (what algorithms, techniques etc are used to tackle similar problems).

I have different lists of strings which contain one or two useful pieces of information I would like to extract. In the following is an example I need to extract the bold and italic part from each line. This is just an example though, eventually I will need to end up with a process I can apply to different lists with different context. Here's a small sample from a list of 500:

50" Sony KDL 50W756CSAEP Smart LED Full HD
55" Samsung UE55JU6400 Smart LED HD
LG 55LF652V 55" SMART 3D FULL HD
HITACHI 55HGW69 55'' LED ULTRA SMART WIFI
TV 65" SAMSUNG UE65KS7500 4K LED Smart

In my full list I have already manually extracted the brand and model. So what I need now is a way to automate the process for a new list containing more brands and models. I thought I could go about this heuristically but since I am not just doing this for this type of data it won't work well.

So can someone give me some suggestions on a good way to go about it?

Thanks!

Maybe you can use Python with dictionaries.

You can set a group of word in the dictionary that could be filled each time with new words you find.

  To find new word (if the pattern remains the same as your example) you
  can grab the "brand" (Samsung) and the next word will be the model
  (UE65KS7500).

This is a good resource


Can anyone explain the reasoning behind this post? I am reading THIS blog post,
and I do not understand the logic behind this part:

Why is the relu here max(0, Xavg - X) ? And even so, it does not really explain the values obtained in the two situations where X1 = X2 = 0, X1 = 1 and X2 = 0, and X1 = X2 = 1.

Can someone please explain this in more detail?

"Weight of $-4$" seems to be the key thing you're missing.  Translating a little, the neuron's value is $\operatorname{relu}(5-4x_1-4x_2)$.

Oh, and there seems to be a typo in the post:
"Then, if X2 also gets switched to 0" should be "switched to 1"


How to Avoid rarely used discrete feature values in a dataset On Google's ML crash course it states:

  Good feature values should appear more than 5 or so times in a data
  set. Doing so enables a model to learn how this feature value relates
  to the label. That is, having many examples with the same discrete
  value gives the model a chance to see the feature in different
  settings, and in turn, determine when it's a good predictor for the
  label.

This make sense, but if that happens what should we do? For example, consider a dataset with street_name as feature and out 2000 rows, only 4 of them have street_name equal to "Learning St.".

Should we remove those rows containing rare feature values? Or what?

Any insights would be greatly appreciated.

Working with Categorical Features can raise a few challenges. For instance, you might encounter features with high cardinality in certain values or the other case (your case), features with rare categories.

The first thing you can consider is, what you will lose if you drop those rows. What extra info do you get with keeping the rare categories. Here you probably need the domain knowledge of the project to identify the impact of the specific data. If your dataset is small or you don't feel safe to remove those rows, then you should look for alternatives.

The second approach is to group all the rare categories in one new called "Rare" or "Other". So, whenever you have a record with that value, you replace it with the new one.

Finally, you can try either a Dimentionality Reduction such as PCA or a Feature Hashing where you will map multiple categories in one new value.


Should one hot vectors be scaled with numerical attributes In the case of having a combination of categorical and numerical Attributes, I usually convert the categorical attributes to one hot vectors. My question is do I leave those vectors as is and scale the numerical attributes through standardization/normalization, or should I scale the one hot vectors along with the numerical attributes?

Once converted to numerical form, models don't respond differently to columns of one-hot-encoded than they do to any other numerical data. So there is a clear precedent to normalise the {0,1} values if you are doing it for any reason to prepare other columns.

The effect of doing so will depend on the model class, and type of normalisation you apply, but I have noticed some (small) improvements when scaling to mean 0, std 1 for one-hot-encoded categorical data, when training neural networks.

It may make a difference too for model classes based on distance metrics.

Unfortunately, like most of these kind of choices, often you have to try both approaches and take the one with the best metric.


Identifying important interactions between features using machine learning Let's say I have a set of features: a, b, c, d, e, f. I'm now interested in identifying possible interactions between these features that best predict an outcome. For example, it could be that the features a, f, and the interactions a:b:g, d:f and c:e are the 5 most important factors that predict the outcome. It is not only important for an algorithm to account for feature interactions, but I also want to be able to identify these interactions.

How could I approach this problem with machine learning?

I think that it is important, in this situation, to ask yourself why you are using machine learning to detect the interactions. It feels a bit like data dredging. Using domain knowledge to think about what interactions would be likely or feasible given the phenomenon you are studying may well serve you well.

That said, you can fit a tree model using, for instance, R. This will allow you to detect complex and high order interactions. You can see this as an example on page 30 here. Note that I would probably use partykit for fitting the tree model because it allows you to use non-normal distributions. This method, though, works best if you have a very large data set and you may struggle to fit them in a linear model once you have found them (because they can only occur on one side of the tree).

An alternative method that addresses some of the limitations above is to use a technique called Additive Groves. These work on the principle of observing the relative performance of differently restricted tree models. Because interaction effects are not additive, this technique can allow for the identification of interactions.


Generating Synthetic Image to improve the performance of classifier I need some suggestion from experts. For my project work, I have been learning about Generative Adversarial Network.

I am trying to make a classifier (say CNN) to train with original CIFAR-10 dataset and then evaluate the performance.

After that, Generating more data of all classes using GAN and retrain the model and compare the performance.

But I don't know how feasible will the work be. How complex will be the work and also I want to know if the idea is acceptable as a project?

I think it could work well, if you try to reduce the false positive error. Generating images for negative examples shouldn't be to difficult. Generating images for positive examples on the other hand would require a good classifier first.


Creating a loss function with two minima? I'm using keras to build a model. The model has one output, and I want a loss similar to mse loss, but there are two values to predict, and I'm fine if the model predicts either one of them, so I want the loss function to look something like this

(p.s. the graph doesn't have to be symmetrical)

Does any function like this exist? If not, how do I make one

You basically don't want to have a loss like that in any case.

The objective of Machine Learning is define a loss function that is convex to find the minimum by approximating it with the given data.

Not having a convex loss function means that you will have multiple minima and won't be able to determine if the one you're in is a global minimum or just a local one so you won't know if it's optimal or not.

The only kind of Machine Learning without convex function is Deep Learning. And you see how complicated it is to work with it and having "good results" without understanding what is happening under the hood.

Deep Learning models are (most of the time) stochastically with batch gradient descent, that helps getting out of local minima, but that's not even easy.

We can't even prove mathematically that these systems are well-optimized.

So basically, if you want a good model, you can avoid creating such a monster loss :)


Loss function of linear regression How do we decide whether mean absolute error or mean square error is better for linear regression? Are there other loss functions that are commonly used for linear regression?

Generally, MAE is more robust to outliers, so if your data set has outliers, then you can use MAE. But if the outliers represent anomalies in data and it is important that you want to find these anomalies and report it, then we should use MSE. But, if the outliers are just the corrupt data that acts as noise in the data set, then you can use MAE.

RMSE is another very common loss function that can be used for the linear regression :


Should the bias value be added after convolution operation in CNNs? Should we add bias to each entry of the convolution then sum, or add bias once at end of calculating the convolution in CNNs?

Short answer: the bias is added once after the convolution has been calculated.

Long answer: discrete convolution that you see in CNNs is a linear function applied to pixel values in a small region of an image. The output of this linear function is then jammed through some nonlinearity (like ReLU). For a region $\mathbf{x}$ of size $i \times j$ of an image and a convolutional filter $\mathbf{k}$, and no bias term, this linear function $f$ would be defined as:

$$
f(\mathbf{x}, \mathbf{k}) = \mathbf{x}*\mathbf{k} = \sum_{i,j} k_{i,j} x_{i,j}
$$

Without a bias term, this linear function $f$ must go through the origin. In other words, if $\mathbf{x}$ or $\mathbf{k}$ is all zeroes, the output of $f$ will be zero as well. This may not be desirable, so we add a bias term $b$. This gives the model more flexibility by providing a value that is always added to the output of the convolution, regardless of the values of $\mathbf{x}$ and $\mathbf{k}$ -- in other words, it's the intercept value.

$$
f(\mathbf{x}, \mathbf{k}, b) = b + (\mathbf{x}*\mathbf{k}) = b + \sum_{i,j} k_{i,j} x_{i,j}
$$

If this value was added to each entry of the convolution, it would not achieve its purpose as $f$ would still necessarily go through the origin.


Classifying whether a comment or review is a complaint or appreciation of product and extracting the Topic? I need to classify whether a given review or comment is a complaint or appreciation. This is planned to be used in multiple places, product review pages of own site as well as facebook and twitter. Suggestions on how to approach please.

The Problems that are confusing me:

In FB/Twitter I don't know which product it is for, I need to extract that from text as well.
I need to extract the complaint/appreciation part and group similar ones together, (like good color reproduction and great clarity into just good display)
Articles(each document) will be differently sized.
Data availability is none, I will prepare data by going through our FB etc.

My initial thought was LSTM based classification, but point 3,4 make that hard. Even with 3,4 solved. How do I go about 1,2. I only have played with word2vec a bit and done some twitter sentiment analysis dummy projects. Point 1,2 seem Information Retrieval, Need pointers for that.

Your (basic) task is sentiment analysis, covered in many places.
There is a number of algorithms proven good for that, including LSTM but you need a good deal of data to train that (and compute power). Others is fastText - tool by Facebook - where you have already embeddings for a number of languages including EN.
But you need anyway 'ground truth' - samples of positive and negative posts / reviews - to train final classification model. Assuming you have standard EN reviews - meaning a lot of language typical to online reviews... - you can use some of existing datasets to train your model and boost results. Otherwise you would have to manually select representative (meaning the more the better...) samples and label them (pos/neg). E.g. there is yelp review dataset, actually a number of versions, one of them.
The other tasks - extracting product and complaint/appreciation part - are  entity extraction and topic modelling respectively, each a seprate story.
For entity (product) extraction, possibly you can just search the post for the names from your inventory. Assuming again EN language, beware only of plural versions. And you will need a good data structure / indexes for it to get results fast.
For topic modelling, there is LDA implementation in gensim. You will have to manually go through the topics and process them, depending on the exact scenario you want to use them, e.g. assign each a meaningful name, selecting meanwhile those that make sense for you, merging some.


How to solve this problem as Machine Learning? I have a time series data. I have to identify the point in red. I know how to solve this using statistics. But if I had to use Machine Learning to detect these points how can I solve this.

How can I modify this data in order to feed it to machine learning model. Data is just one column readings as show in plot. Any idea?

Generally, You can use anomaly detection to detect a sudden change in Timeseries data. SOM is useful for detect it. Its like you want to capture an anomly in continuous distribution of data.


Dimensions of Transformer - dmodel and depth Trying to understand the dimensions of the Multihead Attention component in Transformer referring the following tutorial https://www.tensorflow.org/tutorials/text/transformer#setup
There are 2 unknown dimensions - depth and d_model which I dont understand.
For example, if I fix the dimensions of the Q,K,V as 64 and the number_of_attention_heads as 8, and input_embedding as 512 , can anyone please explain what is depth and d_model?

d_model is the dimensionality of the representations used as input to the multi-head attention, which is the same as the dimensionality of the output. In the case of normal transformers, d_model is the same size as the embedding size (i.e. 512). This naming convention comes from the original Transformer paper.

depth is d_model divided by the number of attention heads (i.e. 512 / 8 = 64). This is the dimensionality used for the individual attention heads. In the tutorial you linked, you can find this as self.depth = d_model // self.num_heads. Each attention head projects the original representation into a smaller representation of size depth, then computes the attention, and then all the attention head results are concatenated together, so that the final dimensionality is again d_model. You can find more details on the individual computations in this other answer.

Note that the implementation of the multi-head attention in the tutorial is not a straightforward implementation from the original paper but it is equivalent: in the original paper, there are different matrices $W_i^Q, W_i^K, W_i^V$ for each attention head $i$, while in the implementation of the tutorial there are combined matrices $W^Q, W^K, W^V$ that compute the projection for all attention heads, which is then split into the separate heads by means of the function split_heads.


Spliting training data into multiple variables using R So right now I am trying to create multiple variables with training data, and in the process I have reached an error

  Error in eval(predvars, data, env) : object '1.band1' not found

which is a product of these lines:

for(i in 1:length(data_split)){
 assign(paste("fit.lda",i, sep = ""), train(class~., data=data_split[i],
 method="lda", metric=metric, trControl=control))}

is it something that I did wrong or is it something that can be fixed with another methodology

EDIT:
My dataset is a data frame which was created by the following lines

validation_index <- createDataPartition(valuetable$class, p=0.80, list=FALSE)
dataset <- valuetable[validation_index,]

where valuetable is a dataframe which containes the values of multiple rasters.

Now, I wanted to split my data so my train function can be processed faster, that is because my machine's specs are pretty low, thus allowing me to re-create the later the results that would take time using other methods.

Let me know if you want more clarification.

You didn't explained how is your dataset and its variables, so I'll put a generic example.

Let's say your dataset has a variable called SET, whose content is TRN for training and TST for test. To create a dataset containing only training set you have to use the following code:

trn.ds<- ds[ds$SET == 'TRN',]

If you need a test dataset, just adapt the code:

tst.ds<- ds[ds$SET == 'TST']

PS: It wasn't me who downvoted your question, I don't understand why people downvote almost every question here.


Limitations of Perceptron If you are allowed to choose the features by hand and if you use
  enough features, you can do almost anything.For binary input vectors,
  we can have a separate feature unit for each of the exponentially many
  binary vectors and so we can make any possible discrimination on
  binary input vectors.This type of table look-up won’t generalize.But
  once the hand-coded features have been determined, there are very
  strong limitations on what a perceptron can learn.

This is what Hinton explains in his Neural Networks course but I don't get the binary input example and why it is a table look-up type problem and why it won't generalize? What does he mean by hand generated features? I understand that perceptrons cannot classify non-linear data but I cannot relate this to his slide (slide 26). It would be nice if anybody explains this with proper example.

The slide explains a limitation which applies to any linear model. It would equally apply to linear regression for example.

  What does he mean by hand generated features?

This means any features generated by analysis of the problem. For instance if you wanted to categorise a building you might have its height and width. A hand generated feature could be deciding to multiply height by width to get floor area, because it looked like a good match to the problem.

  I don't get the binary input example and why it is a table look-up type problem and why it won't generalize?

A table look-up solution is just the logical extreme of this approach. If you have a really complex classification, and your raw features don't relate directly (as a linear multiple of the target), you can craft very specific manipulations of them that give just the right answer for each input example. Essentially this is the same as marking each example in your training data with the correct answer, which has the same structure, conceptually, as a table of input: desired output with one entry per example.

In fact this might generalize, but only exactly as well as the crafted features do. In practice, when you have a complex problem and sample data that only partially explains your target variable (i.e. in most data science scenarios), then generating derived features until you find some that explain the data is strongly related to overfitting.

From your comment:

  In his video lecture, he says "Suppose for example we have binary input vectors. And we create a separate feature unit that gets activated by exactly one of those binary input vectors. We'll need exponentially many feature units. But now we can make any possible discrimination on binary input vectors. So for binary input vectors, there's no limitation if you're willing to make enough feature units." 1.What feature? 2.Why are we creating this feature? And why adding exponential such features we can discriminate these vectors?

Here is an example of the scheme that Geoffrey Hinton describes. Say you have 4 binary features, associated with one target value and see the following data:

data 0 1 1 0 -> class 1
data 1 1 1 0 -> class 2
data 0 1 0 1 -> class 1
data 1 1 1 0 -> class 2
data 0 1 1 1 -> class 2
data 0 1 0 0 -> class 1

It is possible to get a perceptron to predict the correct output values by crafting features as follows:

data 0 1 1 0 -> features 1 0 0 0 0 -> class 1
data 1 1 1 0 -> features 0 1 0 0 0 -> class 2
data 0 1 0 1 -> features 0 0 1 0 0 -> class 1
data 1 1 1 0 -> features 0 1 0 0 0 -> class 2
data 0 1 1 1 -> features 0 0 0 1 0 -> class 2
data 0 1 0 0 -> features 0 0 0 0 1 -> class 1

Each unique set of original data gets a new one-hot-encoded category assigned. It is clear that ultimately if you had $n$ original features, you would need $2^n$ such derived categories - which is an exponential relationship to $n$.

Working like this, there is no generalisation possible, because any pattern you had not turned into a derived feature and learned the correct value for would not have any effect on the perceptron, it would just be encoded as all zeroes. However, it would learn to fit the training data very well, it could just associate each unique vector with a weight equal to the training output - this is effectively a table lookup.

The whole point of this description is to show that hand-crafted features to "fix" perceptrons are not a good strategy. Even though they can be made to work for training data, ultimately you would be fooling yourself.


Getting monthly revenue predictions for outlets I am often presented with a task of predicting monthly revenues of retail outlets. Say I have a training set of N outlets, each associated with a series of historical monthly revenues (target) and a set of features which are time-independent (e.g. location (and all associated geofeatures), outlet type, trading area etc.) Then I have a test set of M outlets for which I need to predict monthly revenues for a specified time period (e.g. Q1-Q3 2019).

The usual approach I take consists of adjusting historical revenues on the training set for trend, seasonality and inflation, then take the average for each outlet and use it as a target, train my regression model (could be anything but mostly LightGBM), get predictions for the test set. These predictions are, again, averages for each outlet adjusted for trend, seasonality and inflation, so I need to deadjust them to obtain monthly predictions. So basically the scheme is: monthly historical revenues -> adjusted average as a target -> adjusted average as a prediction -> reconstructed monthly predictions (hope this makes sense).

The whole approach is pretty tedious and feels unnatural. I wonder whether there are frameworks that would allow the use of historical monthly revenues "as is" as a target and produce predictions for the specified time range.

There is, of course, panel regression, but 1) my features don't change over time so the idea seems superficial and 2) gradient boosting in panel regressions isn't exactly a hot topic so implementations are rare (if any exist).

LTSTM models are very good at time-series data, so look into that for starters.

Additionally I'd like to ask why you need to adjust and deadjust your model at all?

Even simpler models like boosted trees, random forest or even a plain regression should give you good predictions by simply including trends, seasons, etc. as predictors.

Edit:

I try to be a bit more comprehensive in my answer. Your problem is actually a classical problem of sales prediction, especially for retail stores like supermarkets, etc.

A great resource to see solutions to this problem can be found here:

Kaggle Walmart Sales Forecast

You will find a lot of notebooks with great approaches and solutions (e.g. based on the aforementioned LTSTM model).

But let's also try to point out a basic solution right here:

*Data**
Let's assume we have data that looks basically like this:

Store  |  Month  |  Sales  |  Store_Size  |  ...
East   |  Jan    |  $0.9M  |  500         |  ...
East   |  Feb    |  $1.1M  |  500         |  ...
North  |  Jan    |  $1.5M  |  800         |  ...
North  |  Feb    |  $2.0M  |  800         |  ...

Some of the ... variables will be constant for each store (store info), some will be related to the date (seasonal) and some might vary for the store and the month (trends).

Problem

Trying to predict the sales of store West for March given all constant variables associate with that store but NOT the trend variables.

It is important to understand seasonal variables here! These are engineered features based on the month. E.g. we can easily compute the influence on sales for each month (e.g. trends like christmas sales, etc.) these are not trends because they are tied ONLY to months and not any other pattern.

Solution

Suppose we fit an XGB model to predict the sales, the simplest solution would be to simply only use features in the model that well be available at the point of prediction, so only seasonal and store info.

This might already deliver a very good model and might be enough for prediction. If you have no information on trends, you cannot add them to the model and therefore should not.

However you easily feature engineer trend data based on GAN or MICE models to augment your data.

Finally please understand: machine learning models work best if you simply throw all available data at them and do not get too cute with feature selection and engineering. XGB and similar models will identify trends by themselves and model the ideal relation between the other factors for you.

Degree of freedom, etc. really does not apply here.


Use the extra features for unsupervised learning. You might enjoy Vladimir Vapnik's take on this in the context of SVMs, which he calls privileged learning: Learning with Intelligent Teacher: Similarity Control and Knowledge Transfer


How to deal with time series of multi-source energy in order of classifcation? I would like to do classification of multi-source energy (wind/solar/teg) repersented in a time series data.
My questions are :
1- What are the most relevent feature that I should chose to do the classification (statistical ones (kurtoisis/means/ variance...) on each sliding window (for experimental purpose) or spectral ones (DWT/FFT)) and which feature selection/extraction method is the best in this case.
2-What is the best classification method should I chose?

Thank you

OK, let us say you have 24 data points per day, in a year you would have 8700~ data points. As an initial analysis you would likek to classify the data points in 2 classes, 'summer energy', 'winter energy'. Notice I am refering to data points and not time series. For this analysis, on way is not to take a time series view but a collection of datapoints view. Once you do that, it becomes a unsupervised classification problem. You could use K-Mean, where the value of K is 2. You could also use neural network based models like Adaptive Resonance Theory networks. As for feature selection, take all the features that you receive from the sensor. It could additionally help to center and scale the data.

In my opinion, if you want to classify the data into 2 classes, timeseries analysis is not immediately required, you could employ the method mentioned above and see how the results are.

EDIT 1:
As I undestand (and assume), you have 4 sensors placed at 4 points of a single entity, e.g. an office. Each sensor output per hour is one of your dimensions. Hour of the day is also your data dimension. As an example, if you have 4 sensors, let us call them A,B,C,D, then your data point will look like:

Datapoint 1::
Vector position 0: value of A sensor, 1.0
Vector position 1: value of B sensor, 1.5
Vector position 2: value of C sensor, 0.5
Vector position 3: value of D sensor, 3.5
Vector position 4: hour of day (24),  14

You will have 720 such data points for one month. You should apply any clustering algorithm on the data points.

Few suggested clustering algorithms (there are many more):
1) K-Mean with K value as 2
2) ANN based approach

EDIT 2:

You could explore the ANN approach, you could make a 3 layer feed-forward network with 4 inputs and 2 outputs. The inputs would be the data point I mantioned above, however, it would be best to normalize the values ( here are some techniques) between 0 and 1. The output in your case will be 2 neurons, one representing summer and another winter, the input vector would be the datapoint vector and the output a vector of length 2, eg [0,1] for winter ideal valu & [1,0] for summer ideal value.

In another approach you could use K- Nearest Neighbour, in this case data normalization may not be strictly (technicaly) required.

For ANN library, you could use R/Python/Java, reference to one Java library is here.

For KNN, there are many library options, one reference is here


Training is not stable with extreme class imbalance I'm dealing with a multi-class classification problem with around 30 categories.
This problem has a severe class imbalance:

Around 300 examples for the least common class.
Around 100k examples for the most common class.

I don't want the classification model to be dummy and predict the most common class for most of the examples, for this reason, I'm using class_weight='balanced' in my LogisticRegression from sklearn. However, in this case, the classes that the algorithm predicts are mostly the less frequent ones. I understand the model overfits them somehow, as it assigns every sample from these class a very high weight.
On the other hand, if I don't apply the class weights, the model predicts the most common categories.
Is there a way to solve this? Is there a way to ensure the model predicts approximately the same proportion of samples for each category?

There are probably many different strategies but it's a difficult problem when the imbalance is as severe as it is here.
Without any correction the model is likely to ignore the smallest classes, as you noticed. However forcing the class weight as if the data is balanced is certainly too strong a correction. A middle ground would be to resample the training set instances yourself before fitting the model: by trying different ways to undersample the large classes and/or oversample the small classes you should be able to find an optimal tradeoff between the two extremes (use a separate validation set to determine the optimal combination).

Is there a way to ensure the model predicts approximately the same proportion of samples for each category?

Maybe I misunderstood but this looks like a bad idea: if the true proportions are not equal then the model shouldn't predict equal proportions either. The ideal scenario is for the model to predict the correct label every time, which implies predicting the true proportion for every class.
It might also be useful to analyze the performance in simpler configurations, e.g. by picking a few "average size" classes and observing how well the classifier discriminates between them only. The harder it is for a classifier to predict correctly, the more it relies on basic class proportion since it doesn't know any better.


Generating ordinal data I would like to generate synthetic data which are ordinal, i.e. ordered, in Python.  But how would I do this?  What are the differences in generating ordinal data vs categorical data?

I'm reading the paper "Automatic Discovery of the Statistical Types of Variables in a Dataset," by Valera and Ghahramani.  In it, they write: "We account for categorical data by sampling a multinomial variable with $R$ categories, where the probability of the categories is sampled from a Dirichlet distribution....To account for ordinal observations, we first sample the first variable in our dataset from a uniform distribution in the interval $(0,R)$, which we randomly divide into $R$ categories that correspond to the ordinal variable in our dataset."

Can someone help me understand the latter part about generating ordinal data?  Thank you!

Ordinal data deals with categories, which themselves are ordered by meaning somehow, but we cannot strictly talk about the distance between the categories. In the example below, you can see that the ordinal data runs from awesome to terrible, with 5 categories explaining an entire ordinal range:

Having a look at the paper, it seems the define a uniform distribution $U(a, b)$ (probably continuous), which they then divide into some random categories, let's say 5, to follow the table above. So if we set the bound of the distribution, $a = 0$ and $b = 1$, we might randomly create five categories, such as:

Category :  a      b
----------------------
Awesome  : 0.00 - 0.09
Great    : 0.10 - 0.44
OK       : 0.45 - 0.46
Bad      : 0.47 - 0.59
Terrible : 0.60 - 1.00

We can see that they are ordinal, because they numerical values do signify some kind of meaningful order, but the size of the categories is not necessarily uniform. The authors generated them randomly.


Tools to perform SQL analytics on 350TB of csv data In short, what would be the best method/tricks/techniques/tools for performing ad hoc sql (style) queries on 350TB of csv data? Would there be other options, tool wise that would do it faster if we dropped the "sql" requirement?

Is my best option Hive and as many servers I can't get my hands on? Would spark be of any benefit since this size of dataset wouldn't come close to fitting into memory? I have other related questions /ideas but don't want to bog this down.

Background

The data is originally stored in a binary format that gets converted into an ASCII. The ASCII file is getting turned into CSV (actually tab separated but whatever). 1 binary file = 1 ASCII file = 1 CSV file. The data format is pretty simple, 200 to 500 header columns, each row is a sample of an attribute in 1-second intervals.

Example (first row is header):
t,attribute1,attribute2...<attributeX>
1,val1,val2...<valx>
2,val1,val2...<valx>
3,val1,val2...<valx>
...
X,val1,val2...<valx>

Example queries that could be ran.

1) Take t value (t is non-unique, as each file starts at 1 and ends at some random time) and compare different attribute values between t value of 100-1000 for all other attributes that were sampled at 100-1000.

2)There is UTC value as an attribute value (one of the columns), and same idea as #1 we might want to compare all attributes between X dates and compare those attributes to Y data that is from a different UTC range.

3)We might want to figure out a single particular attribute and trend that from the earliest time we saw that attribute for a different particular attribute.

These are my best examples as provided to me but other types of queries could exist. If you do have a question of why don't we cut out the middle man of the conversions of the binary data to CSV data, we could theoretically. However, the program that converts the files is quite old and is very involved.

There are many tools that are able to support such queries (as you mentioned Hive or Spark), and it is really up to your requirements in terms of number of queries, number of people who are going to query the data, what kind of BI or reporting tools you might want to use with it, etc.

More than that, your requirements are probably going to change in the future. If you provide easier ways to query the data, the more people around you are going to use it.

For example, if you choose to load that data into Amazon Redshift (https://aws.amazon.com/redshift/), it will give you a full SQL flexibility, very fast performance and the ability to connect to several BI, visualization and reporting tools. When you COPY the data into Redshift it is being compressed (column based encoding) 5-8 times, and you can put all your data in a cluster of 6 nodes of DS2.8XL of Redshift, for a couple of weeks for your POC.

The concept of compression can be applied in different solutions as well. For example, instead of using CSV format, you can convert it to Parquet (https://parquet.apache.org/) format, and save on your storage and increase the performance as you need to move around and scan less bytes of data. Once you have your data in Parquet, you can query it for example with Presto, like Netflix are doing - http://techblog.netflix.com/2014/10/using-presto-in-our-big-data-platform.html

Regarding Spark, it can help even if you don't fit all the data into memory. Spark can also help you with the usage of Zeppelin (https://zeppelin.incubator.apache.org/) as an interactive notebook, with nice visualization features, as well as write code with Scala, Python or even R with SparkR.


Model Dump Parser (like XGBFI) for LightGBM and CatBoost Currently my employer has multiple GLM in a live environment. I am interested in identifying new features and interactions to enhance the accuracy of these GLM; for now I am limited to the GLM structure so simply deploying a solution which automatically accounts for interactions is not possible.
I have in the past used XGBoost to identify powerful feature interactions through the use of XGBFI / XGBFIR. I am now looking in to using LightGBM and CatBoost to do the same but cannot seem to find a tool like XGBFI which works on those models.
Question: Does anything similar to XGBFI exist for LightGBM and CatBoost?

Both LightGBM and CatBoost have functions that give you the models' feature importances. They are described in the respective docs here:

LightGBM: plot_importance and lightgbm.importance
Catboost: has get_feature_importance which is described in more detail on its concepts page


What kind of machine learning algorithm can I use? I have a data set of tweets regarding vaccines.  They have been collected from an API because they have keywords like "flu, measles, MMR, vaccine" etc.

I need to find tweets specifically about measles and the outbreak that occurred in California this past February.  It isn't enough to search the data set for words like "California" and "Measles" because tweets like "MMR vaccination rates in Palo Alto on the rise" are about measles and California, but wont be captured by a naive search.

Are there any unsupervised algorithms that could help me out?

A couple ideas:

If you have a large data set of tweets, you could try using latent semantic indexing to find out which terms are semantically related based on their usage and co-occurrence. Then after transformation, you can apply some document similarity metric, e.g. cosine similarity, to find tweets most relevant to your queries, e.g. "California measles".
Use knowledge or lexical databases like DBPedia and WordNet to calculate semantic relatedness between your query and the text of your tweets, or to identify tweets with related concepts.


Scikit Learn: KMeans Clustering 3D data over a time period (dimentionality reduction?) I have a dataset of xyz coordinates with a date component in a pandas dataframe

ex:

date1: $[x_1,y_1,z_1]$,
date2: $[x_2,y_2,z_2]$,
date3: $[x_3,y_3,z_3]$,
..

I would like to classify a sample of object positions over the period of a week
(using indexes to re-map the classification label back to the date), like this:

Week 1: $[x_1,y_1,z_1], [x_2, y_2, z_2], [x_3,y_3,z_3], [x_4,y_4,z_4], [x_5,y_5,z_5], [x_6,y_6,z_6], [x_7,y_7,z_7]$,
Week 2: $[x_8,y_8,z_8],[x_9,y_9,z_9],[x_{10},y_{10},z_{10}],[x_{11},y_{11},z_{11}],[x_{12},y_{12},z_{12}],[x_{13},y_{13},z_{13}],[x_{14},y_{14},z_{14}]$,

When I try to run KMeans it returns

k_means = KMeans(n_clusters=cclasses)
k_means.fit(process_set.hpc)
date_classes = k_means.labels_

ValueError: Found array with dim 3. Expected <= 2

Questions:

Do I have to run it through Principal Component Analysis (PCA) first? if so, how do I maintain date mapping to the classification created?
Are there any other methods I could use?
Am I doing everything completely backwards and should consider a different approach, any thoughts?

Thanks!

I don't have enough reputation to ask questions in comment for clarification before answering it, so I'm going to do both here.
Here are the things that would help answering this question for now:
Can you post part of the process_set.hpc?
What's its format? Is it a numpy array? Is it a Pandas dataframe?
What's the value of cclasses?
And now the answer:
First of all, k-means algorithm is able to find clusters in any n-dimensional data. If n is too big, it is better to use PCA but for n=3 that wouldn't necessarily add any value.
The second thing that looks suspicious to me is that in the documentation for kmeans in scikit-learn, there is no compute_labels option, as seen here. However, that option exists for MiniBatchKMeans as seen here.
Also, if you make your data in the form of a pandas dataframe (if it is not already so), things would be much easier to track and you won't have to reattach the timing information to your data afterwards.
I may be able to give you a more thorough answer if I know a bit more about the format of the data.
Good luck!


Behavioural data required to predict churn I am trying to build a predictive churn model that will identify customers who are likely to churn. I am defining a churned user as someone who hasn't transacted within 60 days. 90% of all transactions occur within 60 days of one another so this feels reasonable.

I have very limited behavioural data; however. I have a record of a user's transactions and I have access to Google Analytics (GA). GA does not, however, allow me to track the specific actions of a user - everything is aggregated across the entire user base.

I have recommended that we track and keep a record of users' navigation across the website, buttons clicked, times logged in etc. but this has been rejected as it has been deemed unnecessary because we have Google Analytics. Am I being unreasonable? And are there any workarounds. I don't think I'm going to be able to make much of a churn forecasting model with basic information like a user's address, age, and transactions.

You are not being unreasonable but it is hard to know if they are being reasonable or not as well. If they feel the existing amount of information is enough, from which one can derive many other things, and what the definition of churn is, then it is also fine to abide by them. Of course, it is better to have the data you are looking for in terms of tracking user behavior on the website but that's not all.

Definition of Churn:

This definition can vary and you need to ask yourself how you are going to define it.

Do you consider someone to have churned once they stopped buying after X days, in your case 60, irrespective of what they are doing on the website? Does a failed transaction or a pending transaction such as added to cart but not yet ordered count as churned?

If you only consider someone to have churned if there has been no successful transaction within 60 days, then you can look into many other factors.

May be you have considered these things already but for future viewers, I will still add how I approached problems like these in the past.

Given that you have data of the customers' orders, you can now derive the following:

1) Average sales value of the customer

2) Count of the number of orders

3) What category of products they have bought

4) Age of the customer since first order or account creation or both

5) When was the last placed order

The above points already show you have more or less tried to get the Recency-Frequency-Monetary aspects of a customer.

Using this, you can now create extra columns of features, if required and helpful, to study how these customers buying products from particular Category Z behave.

1) What is their average

2) How does it fare with an individual customer's buying behavior

3) How often do these group of customers buy products

4) What are the other related products they have been buying

5) The shortest and longest time period for a customer belonging to a particular category to be back again and place an order on the shop

So, there you go, you have just extracted so much information from the most basic order information. You can of course extract more and more out of these data as well. However, whether this is enough or not is a matter of the problem statement, amount of data, and of course opinion.


CNNs: understanding feature visualization Channel Objectives (SOLVED) I'm trying to follow a paper on deep NN feature visualization using beautiful examples from the GoogLeNet/Inception CNN. see: https://distill.pub/2017/feature-visualization/

The authors use backpropagation to optimize an input image to maximizes the activation of a particular (Inception) neuron/feature, or entire channel.

For example, Inception Layer 4a, Unit 11 is feature 12 of 192 from the 1x1 convolution path of Inception Layer 4a before filter concatenation (see: https://distill.pub/2017/feature-visualization/appendix/googlenet/4a.html#4a-11).

For Layer 4a 1x1 convolution the shapes are:

# Layer 4a
input:    [14,14,480]
output:   [14,14,512]

# 1x1 convolution
kernel:   [1,1,480]   # total 192 kernels
output:   [14,14,192] # channels [0..192] of Layer4a output
Layer4a slice:    tf.slice( layer4a_output, (0,0,0), (14,14,192) )

# Layer4a Unit 11
layer4a_unit_11 = tf.slice(layer4a_output, (11,0,0), (1,1,1)) # numpy [11,1,1]

In a related article, the authors state (see: https://distill.pub/2018/building-blocks/) ,

  "We can think of each layer’s learned representation as a three-dimensional cube. Each cell in the cube is an activation, or the amount a neuron fires. The x- and y-axes correspond to positions in the image, and the z-axis is the channel (or detector) being run."

Furthermore, they offer a diagram which super-imposes the cube of Layer4a over the input image with the (x,y) axis overlaying the image itself.

I understand that the Neuron Objective is the input image that produces the highest activation for Layer 4a, Unit 11 which can be found at index=[11,0,0] of Layer 4a output=[14,14,512]. In this case, (x,y)=[11,0]. Each [1,1,480] kernel generates a feature map of shape=[14,14,1] with a total of 196 activations.

kernel => channel or feature map and activation => neuron or feature.

Question

But what is the intuitive concept of the (Positive) Channel Objective? In this example, Unit 11 sits in the same channel as 14x14=196 other neurons, but the channel objectives for all these neurons are different. If the optimized image for the Channel Objective maximizes the sum of neuron activations for channel 0, (e.g. slice=[14,14,0] of 192 1x1 convolutions or 512 total layer 4a channels) wouldn't it be the same for all 192 neurons in the same channel? Obviously, by the examples we see this is not true.

How does the Channel Objective relate to the Neuron Objective for Unit 11?

ANSWER

  I understand that the Neuron Objective is the input image that produces the highest activation for Layer 4a, Unit 11 which can be found at index=[11,0,0] of Layer 4a output=[14,14,512].

This is where my understanding went off the rails. Layer 4a Unit 11 is actually  channel/feature 12 of 192 for the 1x1 convolution. It is NOT the 12 of 196 neuron of channel 1. My fault for confusing 192 channels with 196 neurons/channel.

Instead, as mentioned the in answer, Unit 11 is a single neuron in channel 11, usually located near the center, e.g. Neuron Objective is (x,y,z)=(7,7,11) and Channel Objective is (x,y,z)=(:,:,11)

As you noticed, the activations on mixed4a for a normal sized input is [14,14,512]. Intuitively, these dimensions correspond to [x_position, y_position, channel].

When we talk about "channel" and "neuron" objectives for visualization of channel 11, we mean:

Neuron objective: maximize activations[7, 7, 11]

maximize channel 11 in one position (generally middle)

Channel objective: maximize tf.reduce_sum(activations[:, :, 11])

maximize channel 11 everywhere

This diagram from feature visualization also goes over the difference:


Check Accuracy of Model Provided by Consultant My company has recently engaged a consultant firm to develop a predictive model to detect defective works.

I understand that there are many ways to validate the model, for example, using k-fold cross-validation and I believe that the consultant firm will carry out the validation before submitting the model to us.

However, at the employer's side, how can I check the accuracy of the model developed by the consultant firm ??

Someone suggested that I can give 2000-2015 data to the consultant firm and keep 2016 data for our own checking. However, a model with good accuracy on 2016 data does not imply that it will have good predictive power in the future. In my view, keeping 2016 data for checking seems like adding one more test set for validation, which in my view, is unnecessary since I already hv "k-fold" cross validation.

Could someone advise what the employer can do to check the consultant's model?

Cross validation can be used in parameter tuning or model selection, but it does not evaluate the performance of a model.

When developing a model, you divide your data between train, validation and testing. In the best case scenario, testing is only used once at the end to score the model. You should definitely keep the 2016 data.

If you give all your data, it is easy to have a model learning "by heart" your expected data but it will not generalize well to future years. This is overfiting. The only way to know is by testing it on unknown data, here, 2016

Employed for measuring model performance, cross validation can measure more than just the average accuracy and you can select your features to answer the best accuracy score


What is the difference between CountVectorizer() and Tokenizer() or are they the same? from sklearn.feature_extraction.text import CountVectorizer

from keras.preprocessing.text import Tokenizer

I am going through some NLP tutorials and realised that some tutorials use CountVectrizer and some use Tokenizer. From my understanding, I thought that they both use one-hot encoding but someone please clarify this.
What I don't understand is why CountVectorizer is not used on Deep Learning models such as RNN and Tokenizer() is not used on ML Classifiers such as SVM, Naive Bayes.
Thank you

I thought that they both use one-hot encoding

These are utility to preprocess your text. Like any other utility, it has multiple options to tweak your text. You should explore all the parameters using the official docs.
 I will explain one of these i.e. OHE vs Count
from sklearn.feature_extraction.text import CountVectorizer
corpus = [ 'This movie is bad.Too Bad', 'Awesome Movie. Too Awesome']
vectorizer = CountVectorizer(binary=True) #binary=False will make it Count
x = vectorizer.fit_transform(corpus)

import pandas as pd
df = pd.DataFrame(x.toarray(), columns=vectorizer.get_feature_names())
df

Our end goal is to create Features and each Feature has an indicator for its contextual meaning.

what I don't understand is why CountVectorizer is not used on Deep Learning models such as RNN and Tokenizer() is not used on ML Classifiers such as SVM,

When we are modeling a simple ML algorithm, we generally use scikit-learn.
So there is no point in adding Keras there.
Same is True for Deep learning.
Though, in this case, we have another reason too i.e. Deep Learning generally works on a large dataset. So, we mostly use the idea of embedding on our features. So it's better to use a Framework that provides an end-to-end solution


What is the minimum/suggested sequence length for training an LSTM? My dataset consists of short videos of 4/5 time-steps each (frames), and the problem is classifying this video (multi-label classification). The idea is to use an LSTM but I'm wondering if the sequence length is not enough.
 What are the suggested sequence length? May 4/5 time steps be enough?

P.S. could you please post some link to scientific articles to endorse your claims?

LSTMs would run into problems beyond 500 time steps.
original paper by Hochreiter and Schmidhuber: "LSTM can learn to bridge minimal time lags in excess of 1000 discrete time steps."

Example: Your video is 60 seconds long. I would start by encoding 5 frames per second using a CNN (use resnet (SEnets have better performance but i havent read the paper) till the layer just before final affine layer which feeds into the softmax encode your images. Use these encodings as inputs in a GRU (Hence a 300 time steps for this GRU) and use its final state to make the class prediction (use a separate loss for each label?)


How to get non-normalized feature importances with random forest in scikit-learn I'm using the feature_importances_ attribute in the random forest classifier of scikit-learn to plot the importances of each feature. However, I'd like to plot these importances non-normalized. I have searched around how to do this, but there doesn't seem to be an easy method how to do this. I tried manually:
temp = [t.tree_.compute_feature_importances(normalize=True) for t in clf.estimators_]
arr = np.array(temp)
arr2=[]
for i in range(19):
    arr2.append(sum(arr[:,i]))
arr3 = np.array(arr2)
indices = np.argsort(arr3)[::-1]
indices.reshape(1,-1)

plt.figure()
plt.title("Feature importances")
plt.bar(range(X_train.shape[1]), arr3[indices],
        color="r", align="center")
plt.xticks(range(X_train.shape[1]), indices)
plt.xlim([-1, X_train.shape[1]])
plt.show()

Which gives these results for normalize=True and normalize=False respectively. (the normalize parameter seems to be inverted somehow...?)

For reference, this is the result from using feature_importances_ (the error-bars are not relevant to the question):

It seems you are normalizing the Gini dip for every tree and then summing those values.
 You should do the normalization at the end after the sum.
Result might vary depending upon the overall numbers in the two scenarios.
See this dummy example
import numpy as np
# Three Tree and 3 Features. F1 sum across Trees = 50, F2 = 48, F3 = 20
gini_dip = np.array([[20, 15, 5], [20, 15, 5], [10, 18, 10]])

# Normaliation before sum
gini_dip_n = np.array([((elem-min(elem))/(max(elem)-min(elem))) for elem in gini_dip])
fi_1 = gini_dip_n.sum(axis=0)

# Normaliation After sum
gini_dip_sum = gini_dip.sum(axis=0)
fi_2 = (gini_dip_sum - gini_dip_sum.min())/(gini_dip_sum.max() - gini_dip_sum.min())

fi_1, fi_2

(array([2.        , 2.33333333, 0.        ]),
 array([1.        , 0.93333333, 0.        ]))

F2 is best in scenario #1 and F1 is best in scenario #2