Softmax regression cost function code I really do not understand what does this code do
M = sparse.coo_matrix(([1]*n, (Y, range(n))), shape=(k,n)).toarray()

The code is related to calculating the sparse function in this equation, but I am really confused and I do not know how it iterates through it and what is:
1- sparse.coo_matrix
2- (Y, range(n)))
3-shape=(k,n)).toarray()
??
Also, What exactly does this term means in the equation and how to interpret it into code:
Thank you , and please forgive my poor English.

It's a loss function applied to a regression with l2 penalty on the parameters.
The first square brackets can be interpreted in the following way:

$ - \frac{1}{n} $ has the minus because it wants to minimize.
$\sum_{i=1}^{n}$ means for each data point.
$\sum_{j=0}^{k-1} $ means for each class.
$y_i == j$ means that the fraction after this term is calculated only for true class. This check makes sense because y true is an OHE vector like [0, 0, 1], therefore you want to evaluate only the predicted probability associated with the true class. Take into account softmax function: if you increase the probability of a single output in output in the softmax you are implicitly reducing the probabilities of the other outputs.
the log division is the softmax function.
$ + \frac{\lambda}{2} \dots $ is a l2 regularization term on the model parameters.


Is deduction, genetic programming, PCA, or clustering machine learning according to Tom Mitchells definition? Tom M. Mitchell defines machine learning as

  A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P if its performance at tasks in T, as measured by P, improves with experience E.

For some algorithms that are usually counted towards the field of machine learning, I heavily doubt that they are learning algorithms according to the definition. Interestingly, all of them are part of Tom Mitchell's book "Machine Learning".

Deduction

The "symbolist tribe" uses inverse deduction according to Pedro Domingos (see The 5 tribes of ML). While I'm not too sure what inverse deduction is, the idea of having a set of true statements and inference rules like Modus Ponens to create more/other true statements is clear.

Comming to the definition: What is experience here? How is performance measured? What does a typical task look like?

Genetic programming

From Wikipedia:

  Genetic programming (GP) is a technique whereby computer programs are encoded as a set of genes that are then modified (evolved) using an evolutionary algorithm (often a genetic algorithm, "GA").

What is experience in this case? Maybe "iterations" / "epochs"? (I'm not sure about the terminology)

Tasks: What are the typical use cases of Genetic Programming?

PCA

PCA and some other dimensionality reduction algorithms are in some way "fixed". The algorithm does not change with more data. It does not have parameters. Thinking about it, one might argue that it estimates the projection in a better way with more data.

But then: What is better? What is the performance measure P here?

Clustering

I don't see a clear performance measure for Clustering (e.g. k-means). Also, the only kind of experience I can see is the number of samples.

A while back, when presenting ML basics to experts of other domains, I also looked into the formal definitions; Mitchell's seems to be the most official/prominent is usage. However, I found that the interpretation can be also somewhat subjective. In addition, the lines between Machine Learning, Deep Learning, Artificial Intelligence and Statistics in general, do sometimes become a little hazy in various fields of application - the terms are also often abused and used as buzz-words (just like Big Data). I think the biggest guinine struggle is perhaps the interpretation of each of the terms that Mitchell introduces: Experience (E), Tasks (T) and Performance (P).

Perhaps it would be worth keeping another definition of machine learning in mind (my current favourite); that of Kevin Murphy from his book Machine Learning: a Probabilistic Approach (page 1):

  ... we define machine learning as a set of methods that can automatically detect patterns in data, and then use the uncovered patterns to predict future data, or to perform other kinds of decision making under uncertainty (such as planning how to collect more data!).

This is certainly a lot broader, and may help accept the greater everyday toolbox as machine learning, instead of tight constraints.

For these reasons, I would say my answer below is more of a discussion than a factual answer.

I'd happily be corrected by readers possessing better informed opinions or facts!  :-)

Inverted Deduction

Of the examples you list, I am least familiar with inverted deduction. It is described by Mitchell himself (in Section 10.6 of his book) and   this helpful Quora post, as logically equivalent to induction! The latter link includes one paragraph describing this symbolist method, which makes the idea quite accessible to someone more similar to modern connectionist approaches:

  The most successful form of symbolic AI is expert systems, which use a network of production rules. Production rules connect symbols in a relationship similar to an If-Then statement. The expert system processes the rules to make deductions and to determine what additional information it needs, i.e. what questions to ask, using human-readable symbols.

At this point, it helped me distinguish between mathematical induction and mathematical deduction. The quote above sounds a bit like a expert designed recommender system. Given that deduction implies to refer a universal hypothesis from axioms or known facts (I imagine Sherlock Holmes!), we start with some rules that come from experience (instead of starting with just data). The idea with inverted deduction is that we have some observations and can induce relationships/rules that support the data/observations, and from that we can improve our system. We accept this to be weaker than pure mathematical deduction, and the starting point differs to Mitchell's own definition of machine learning, but if machine learning is an iterative cycle, is our starting point in that cycle critical to the definition?

Putting inverted deduction in the framework of a machine learning  problem (as per our definition with T, E and P), we have the following definition from Mitchell to help:

  ... assume as usual that the training data D is a set of training examples, each of the form $\langle x_i, f(x_i) \rangle$. Here $x_i$ denotes the $i^{th}$ training instance and $f(x_i)$ denotes its target value. Then learning is the problem of discovering a hypothesis $h$, such that the classification $f(x_i)$ of each training instance $x_i$ follows deductively from the hypothesis $h$, the description of $x_i$ and any other background knowledge B known to the system.

He explicitly writes that the Task is to extract the hypothesis (the target function) rules that explain the system as a whole. Experience is the observations plus background knowledge (including the experience of the experts who create/define the rules). Performance remains as the accuracy or strength of the final set of rules. It is especially the interpretation of experience as data as well as expert/background knowledge, which differs slightly from simple observations.

As a facit: induction allows us to make statements about relationships, mapping input to outputs on the known range of our observations, but it is not as watertight as deduction, which by definition must be a universal truth (and therefore becomes a theorem). This makes induction sound a lot like machine learning (and therefore inverted deduction!)- just as neural networks learn relationships very well, but only over their input domain i.e. not universally.

Genetic Programming

Straight off the bat, I would say this does fall into machine learning by the given definition. Models are iteratively assessed on their ability to perform a predefined task, then further training data is used to improve the performance on that task.

Given the usual approach of allowing many models to train, then selecting the best models and allowing them to somehow be fused (equivalent to genetic cross-pollination in biological reproduction). The resulting child-model would thereafter (hopefully) contain the best traits of the two best parent-models. Just as can be the case in biology, or within Reinforcement Learning and exploration step (as opposed to exploitation), the resulting child-model also is sprinkled with a bit of luck/randomness in the form of a mutation. Here is a summary article. The resulting child-model may well be degenerate i.e. less performant than the parents.

  OP: What is experience in this case? Maybe "iterations" / "epochs"?

Given the above rough description, I would call a training case as all the work done between the consolidatory steps (breeding and mutation: producing the child-model). This is also similar to the paradigm used within meta-learning (a great lecture from Ilya Sutskever), whereby a task is defined as the sample dataset, plus information about the task itself. This is inspired by the idea of allowing algorithms to "learn to learn". For me, this is very close to the idea of genetic algorithms and evolutionary models.

PCA

I'm not convinced PCA falls into a machine learning algorithm; however, it is such a commonly used pre-processing method, I would be surprised if it didn't get a mention in any machine learning book.

If I had to force PCA into Mitchell's definition, this would be my best shot: The Task, T, would be simply that of PCA; to reduce the dimensionality of the dataset. The Performance metric, P; I would interpret as the percentage of the variance that is contained with the first $k$ principal components - e.g. just fix $k = 3$, and run the algorithm with more and more data. More data would allow this to be performed more accuractely, i.e. with higher levels of confidence and the variance should be described more densely be the denser set of input points. So, let's say P is given by:

$$
P = \sum_{i=1}^{k} C_i
$$

where $k = 3$, for example and $C_i$ is the $i^{th}$ principal component's normalised coefficient. This sum therefore integrates to 1 over all $k$.

If the value of P actually decreases after adding more data, I would perhaps argue that we have actually learned more about the sample distribution with the extra data, even though our performance metric technically indicates worse results.

Despite this attempt to shoehorn PCA into the definition - I am still uneasy about calling this machine learning (see the second paragraph in my conclusion).

Clustering

I agree with you that the only real experience is the number of samples. To add to any doubts, clustering (like PCA), is an unsupervised method - there are typically no labels on the data.

If experience, E, is then the number of data points, the metric, P, would be the inherent error measured used to iteratively improve the clustering result. For example, in the case if KNN, it could be the sum of squared distances between points one cluster and those of a different clusters. Allowing the model to train with additional data points (assuming the ability to overfit), it would not produce a worse value for P: either the same or better.

In my opinion, clustering is included in many machine learning books, not because it falls nicely into Mitchell's definition, but because it is a widely used and extremely useful tool. That being said, it can also be used in the a non-parametric setting, e.g. using a Dirichlet process  as a prior and allowing the number of clusters to grow as more data is provided. I find that idea to be easier to merge into Mitchell's notion of a model improving a metric given more experience!

The fact that clustering is indeed defined as un-supervised learning, i.e. without any sample labels, just makes the idea here more interesting to me.

Conclusion

I agree with @Anony-Mousse, in that PCA and clustering really don't give the impression of a learning algorithm. I think it is because I conflate statistical learning with human learning. General discussion around machine learning and AI tends to use anthropomorphism: e.g. "AlphaZero learned from scratch how to play a wide range of Atari games"... If we replace AlphaZero with Alice or Bob, it is clear we are talking about a human. This way of describing algorithms leaks into methods like PCA and clustering - people say how PCA learns representations of the variance in the data, even though we struggle to align Mitchell's definition of machine learning to PCA. After the explosion of machine and deep learning since ca. 2012, we see there are still many undisputed terminologies floating around - something that is unavoidable for a rapidly devloping area.

To address the discssion in the comments to Anony-Mousse's answer: Arthur Samuel coined the term machine learning, as a computer being able to learn without being explicitly programmed. Here, in its original context, I believe it meant to learn as a human does, i.e. to exhibit the use of intelligence (paraphrased from quotes in this nice article). Humans learn a great deal from observations, even without knowing the correct answer (the case without labels). Given this, and drawing the direct comparison to unserupervised learning, one could argue that any process by which a machine is able to improve some metric is indeed a learning machine. So clustering as an example, is learning, as it improves a metric and derives new insights into data (without labels). In the case of PCA, the metric is unclear - it must be derived from the usefulness of the results for the task at hand: beauty is the eye of the beholder.

In the end, it doesn't matter too much to me, as long as we know what we are talking about and can make progress in research and applications thereof. My only concern is that this confusion makes teaching the subject more difficult and raises the barrier to entry for newcomers, who have to deal with sometimes contradictory terminology.


Difference between indicator column and categorical identity column in tensorflow I am learning Tensorflow and came across different feature columns used in Tensorflow . Out of these types, two are categorical_identity_column and indicator_column. Both have been defined in the same way. As far as I understand, both convert categorical column to one-hot encoded column.

So my question is what is the difference between the two? When to use one and when to use the other?

indicator_column encodes the input to a multi-hot representation, not one-hot encoding.
The example clarifies more:
name = indicator_column(categorical_column_with_vocabulary_list(
    'name', ['bob', 'george', 'wanda'])
columns = [name, ...]
features = tf.parse_example(..., features=make_parse_example_spec(columns))
dense_tensor = input_layer(features, columns)

dense_tensor == [[1, 0, 0]]  # If "name" bytes_list is ["bob"]
dense_tensor == [[1, 0, 1]]  # If "name" bytes_list is ["bob", "wanda"]
dense_tensor == [[2, 0, 0]]  # If "name" bytes_list is ["bob", "bob"]

The last two examples describe what is meant by multi-hot encoding. for example if the input be ["bob", "wanda"] the encoding will be [[1, 0, 1]].


sklearn: sklearn.linear_model.HuberRegressor vs sklearn.linear_model.ElasticNet I am experimenting different loss functions for my regression model. I noticed that in the sklearn, there are:

sklearn.linear_model.HuberRegressor and sklearn.linear_model.ElasticNet

To me, both use the combination of L1 and L2 loss. What exactly is the difference of this? Thanks!

Yes, many loss functions in regression models are using a combination of L1 and L2 for different purposes. To realize the difference, I start with Ridge regression.

Ridge regression addresses some of the problems of Ordinary Least Squares by imposing a penalty on the size of the coefficients. The ridge coefficients minimize a penalized residual sum of squares.

The HuberRegressor is different to Ridge because it applies a linear loss to samples that are classified as outliers. A sample is classified as an inlier if the absolute error of that sample is lesser than a certain threshold.

ElasticNet is a linear regression model trained with both $\iota_1$ and $\iota_2$-norm regularization of the coefficients. This combination allows for learning a sparse model where few of the weights are non-zero like Lasso, while still maintaining the regularization properties of Ridge. Elastic-net is useful when there are multiple features which are correlated with one another. Lasso is likely to pick one of these at random, while elastic-net is likely to pick both.


What makes columnar databases suitable for data science? What are some of the advantages of columnar data-stores which make them more suitable for data science and analytics?

A column-oriented database (=columnar data-store) stores the data of a table column by column on the disk, while a row-oriented database stores the data of a table row by row.

There are two main advantages of using a column-oriented database in comparison
with a row-oriented database. The first advantage relates to the amount of data one’s
need to read in case we perform an operation on just a few features. Consider a simple
query:

SELECT correlation(feature2, feature5)
FROM records

A traditional executor would read the entire table (i.e. all the features):

Instead, using our column-based approach we just have to read the columns
which are interested in:

The second advantage, which is also very important for large databases, is that column-based storage allows better compression, since the data in
one specific column is indeed homogeneous than across all the columns.

The main drawback of a column-oriented approach is that manipulating (lookup, update or delete) an entire given row is inefficient. However the situation should
occur rarely in databases for analytics (“warehousing”),
which means most operations are read-only, rarely read many attributes in the same
table and writes are only appends.

Some RDMS offer a column-oriented storage engine option. For example, PostgreSQL
has natively no option to store tables in a column-based fashion, but Greenplum has
created a closed-source one (DBMS2, 2009). Interestingly, Greenplum is also behind
the open-source library for scalable in-database analytics, MADlib (Hellerstein et al.,
2012), which is no coincidence. More recently, CitusDB, a startup working on high speed, analytic database, released their own open-source columnar store extension for
PostgreSQL, CSTORE (Miller, 2014). Google’s system for large scale machine learning
Sibyl also uses column-oriented data format (Chandra et al., 2010). This trend
reflects the growing interest around column-oriented storage for large-scale analytics.
Stonebraker et al. (2005) further discuss the advantages of column-oriented DBMS.

Two concrete use cases: How are most datasets for large-scale machine learning stored?

(most of the answer comes from Appendix C of: BeatDB: An end-to-end approach to unveil saliencies from massive signal data sets. Franck Dernoncourt, S.M, thesis, MIT Dept of EECS)


Can LSTM Predict The Next Few Days Of Stock Price? I have searched many websites and forums describing stock price forecast using LSTM.

They shared two things in common: one is that all the sources make predictions with same set of data and none of these websites show a prediction for next 10 days.

So I would like to know whether it's possible to predict stock prices for the next few days (n days) using LSTM.
And why do most sources use the same dataset?

The reason why you find the same dataset everywhere is that financial data is extremely secretive and expensive and nobody is willing to share their financial information because in doing so they will be effectively losing an edge in the market. For that reason public financial data is mostly gathered by academics and not too often.

Second, LSTM can make predictions for the next n-hours, n-days, n-months and even n-years like the article STOCK PRICE PREDICTION USING LSTM,RNN
AND CNN-SLIDING WINDOW MODEL by Sreelekshmy Selvin et. al. and many others show. You just need to build a good training vector and the target can be anything you want. As it's natural in financial markets, accuracy decreases as time increases.


On gradient boosting and types of encodings I am having a look at this material and I have found the following statement:

For this class of models [Gradient Boosting Machine algorithms] [...] it is both safe and significantly
more computationally efficient use an arbitrary integer encoding [also known as Numeric Encoding] for
the categorical variable even if the ordering is arbitrary [instead of
One-Hot encoding].

Do you know some references that support this statement? I get that Numeric Encoding is more computationally efficient than One-Hot Encoding, but I would like to know more about their supposed equivalence to encode unordered categorical variables in Gradient Boosting Methods.
Thanks!

This is actually a feature of tree-based models in general, not just gradient boosting trees.
Not exactly a reference, but this Medium article explains why ordinal encoding is often more efficient.
On the topic of safety, I think the author should have said that the use of ordinal encoding is more safe compared to linear methods, but still not perfectly safe.  It's possible for decision-tree methods to find spurious rules within ordinal encodings, but they don't have the strong assumptions about numeric semantics that linear methods do.

. . . I would like to know more about their supposed equivalence to encode unordered categorical variables . . .

Any rule derived with one-hot encoding can also be represented with ordinal encoding, it just might take more splits.
To illustrate, suppose you have a categorical variable foo with possible values spam, ham, eggs.  A one-hot encoding would create 3 dummy variables, is_spam, is_ham, is_eggs.  Let's say an arbitrary ordinal encoding assigns spam = 1, ham = 2, and eggs = 3.
Suppose the OHE decision tree splits on is_eggs = 1.  This can be represented in the ordinal decision tree by the split foo > 2.  Suppose the OHE tree splits on is_ham = 1.  The ordinal tree will require two splits: foo > 1 then foo < 3


Is Neural Style Transfer considered supervised or unsupervised learning? As far as I understand it, the Neural Style Transfer uses a content image and a style image, and generate a new image based on the two images.

It tries to find a set of pixel values such that the cost function J(C, S) is minimized. It does not have any labels associated in advance, but it has an output (generated image) that should be the target of the learning.

However, I'm not sure if this is considered supervised or unsupervised learning. Which does it belong to?

Neural style transfer is not really machine learning, but an interesting side effect/output of machine learning on image tasks. When performing neural style transfer using a pre-trained model, then a significant amount of supervised machine learning has already occurred to enable it.

The style transfer algorithm is still an example of gradient-based cost function optimisation, which it shares with many supervised and unsupervised learning algorithms.

The output of style transfer is partially a probe into what a network has learned about different levels of structure in the problem domain it has been trained on. However, its main use has been to generate images with altered/mixed aesthetics for art and entertainment.


What's a good Python HMM library? I've looked at hmmlearn but I'm not sure if it's the best one.

SKLearn has an amazing array of HMM implementations, and because the library is very heavily used, odds are you can find tutorials and other StackOverflow comments about it, so definitely a good start.

http://scikit-learn.sourceforge.net/stable/modules/hmm.html

PS: Right now its outdated in June 2019

As of July 2019 you can use hmmlearn (pip3 install hmmlearn)


Classification of multidimensional data to multidimensional clusters with a varying subcluster structure I have a large dataset with mixed (numerical, categorical, textual) data that I need to classify. The clusters are well-defined, but multidimensional (i.e. vector-valued) and have a varying structure as follows:

Cluster Structure 1: $(x_1, * , * , ... , * )$

Cluster Structure 2: $(x_1, x_2, * , ... , * )$

Cluster Structure 3: $(x_1, * , x_3, ..., * )$

...

Cluster Structure N: $(x_1, x_2, x_3, ... , x_m)$

where asterisks ($*$) indicate missing component entries, and each component, when present, (e.g. $x_2$) is a cluster to be identified. The Cluster Structures are disjoint in the sense that each member of the data set is only assigned to one Cluster Structure, but Cluster Structures can share components. For example, Cluster Structure 1 only consists of one "subcluster" ($x_1$) to be identified, whereas Cluster Structures 2 and 3 have two subclusters to be identified, with one of the clustering subproblems ($x_1$) shared and the other ($x_2$ vs. $x_3$) being different. The subcluster $x_1$ (and only that) is present in all Cluster Structures.
For each member of the dataset, I need to classify both the Cluster Structure and the associated tuplet of component clusters $x_i$, but I am not sure if it is better to treat these clustering problems sequentially (first structure then subclusters) or independently of each other (separate classifier for each structure). I could, of course, also define a "metacluster" as $(S, x_1, x_2, x_3, ..., x_m)$ where $S$ is the Cluster Structure id to be classified, but then I'd be treating each cluster structure identically when in fact classification within Cluster Structure 1 (once identified) is potentially much easier than in Cluster Structure N as there is only one subcluster to be identified (namely $x_1$).
I would be interested in any existing solutions (algorithms, research papers etc) that would apply, even remotely, to this problem.

You can pose this as multi-class classification problem (all sub-clusters becomes classes). Since, your input length varies, you should pad your input to get the length equal for all inputs. You can then use neural networks (1D Conv layers followed by Dense and Softmax) to classify this.
An alternate approach to do this would be using tree-based approach where missing values can be handled. Look at CatBoost classifier. This would also require you to pose your problem as multi-class classification.


Getting low accuracy on keras pretrained word embeddings example Original blog post claims that it is possible to get 95% accuracy on the validation set (20 Newsgroup dataset) after only 2 epochs using pretrained word embeddings (glove.6B.100d). All code is located here. I did not changed anything in this example but getting only 40% accuracy on the validation set after 2 epochs and 75% after 10 epochs. I can't get to 95% accuracy even after 20 epochs. Switching from tensorflow to theano backend do not make any significant change. I'm using Keras 2.0.2, tensorflow-gpu 1.0.1, theano 0.9.0, python 2.7.12. What I'm doing wrong?

The code has been changed to remove headers.  See comment on github:

"Newsgroups message contains header like 'Newsgroups: alt.atheism', which inflates the accuracy to 0.95 (2 epochs).
After removing the header, the val accuracy is 0.47 (2 epochs) and 0.71 (10 epochs)."

https://github.com/fchollet/keras/pull/5585

This confused me for days!


How duplicated items can be deleted from dataframe in pandas I have the following data:

userid itemid
  1       1
  1       1
  1       3
  1       4
  2       1
  2       2
  2       3

How can I drop duplicated items (not the index) to make the following DataFrame:

userid itemid
  1       1
  1       3
  1       4
  2       1
  2       2
  2       3

Can someone help me?

import pandas as pd
df = pd.DataFrame({'userid':[1,1,1,1, 2,2,2],
                   'itemid':[1,1,3,4, 1,2,3] })

The best way would be to use drop_duplicates(). If you have a larger DataFrame and only want those two columns checked, set subset equal to the combined columns you want checked.

df = df.drop_duplicates()

or

df = df.drop_duplicates(subset=['userid', 'itemid'])


Processing train and test data I have X numpy array as my features and y numpy array as my target. I split both of it into train and test data. From many QnA i have read they only say to preprocess both train and test separately. I assume i only do it to my feature (X) train and test data and not the target (y). Do we also preprocess the target?

Do we also preprocess the target?

Sometimes. The most common example is scaling the target variable for regression, which is beneficial for some algorithms. Another example is when the distribution of the target variable is skewed and is processed into being normally distributed by log-transformation.


Is there a machine learning model suited well for longitudinal data? I have a fairly large (>100K rows) dataset with multiple (daily) measurements per individual, for a few thousand individuals. The number of measurements per individual vary, and there are many null values (that is, one row may have missing values for certain variables/measurements, but not for all). I also have a daily outcome (extrapolated, but let's assume it's fair to do so, so there is a binary outcome for each day when measurements are taken).

My question goal is to model the outcome, such that I can predict daily outcomes for new individuals.

My background is in research, and I am familiar with some statistics and ML, and overall still fairly new to data science. I am wondering if there are any particular known ML algorithms that can be used to model such data. I am cautious about using logistic regression from something like python's scikit learn because the observations are not independent (they are highly correlated on an individual level). From my knowledge, these kind of data are well-suited for a mixed effects logistic regression or longitudinal logistic regression. However, I haven't been able to find any widely used ML algorithms for it, and I would like to pursue an ML approach rather than fitting a statistical model using something like lme4 package in R.

Could someone recommend an available ML algorithm to model such data?

PS: I did some research and found a few research articles on the topic but nothing widely used or clearly implemented. The structure of the data I am working with strikes me as very common, so I thought I'd ask.

Assuming we are not talking about a time series and also assuming unseen data you want to make a prediction on could include individuals not currently present in your data set, your best bet is to restructure your data first.

What you want to do is predict daily outcome Y from X1...Xn predictors which I understand to be measurements taken. A normal approach here would be to fit a RandomForest or boosting model which, yes would be based on a logistical regressor.

However you point out that simply assuming each case is independent is incorrect because outcomes are highly dependent on the individual measured. If this is the case then we need to add the attributes describing the individual as additional predictors.

So this:

id | day | measurement1 | measurement2 | ... | outcome
A  | Mon | 1            | 0            | 1   | 1
B  | Mon | 0            | 1            | 0   | 0

becomes this:

id | age | gender | day | measurement1 | measurement2 | ... | outcome
A  | 34  | male   | Mon | 1            | 0            | 1   | 1
B  | 28  | female | Mon | 0            | 1            | 0   | 0

By including the attributes of each individual we can use each daily measurement as a single case in training the model because we assume that the correlation between the intraindividual outcomes can be explained by the attributes (i.e. individuals with similar age, gender, other attributes that are domain appropriate should have the same outcome bias).

If you do not have any attributes about the individuals besides their measurements then you can also safely ignore those because your model will have to predict an outcome on unseen data without knowing anything about the individual. That the prediction could be improved because we know individuals bias the outcome does not matter because the data simply isn't there.

You have to understand that prediction tasks are different than other statistical work, the only thing we care about is the properly validated performance of the prediction model. If you can get a model that is good enough by ignoring individuals than you are a-okay and if your model sucks you need more data.

If on the other hand you only want to predict outcomes for individuals ALREADY IN YOUR TRAINING SET the problem becomes even easier to solve. Simply add the individual identifier as a predictor variable.

To sum it up, unless you have a time series, you should be okay to use any ML classification model like RandomForest or boosting models even if they are based on normal logistical regressions. However you might have to restructure your data a bit.


Can we have a dataset with slight difference in target values for same value of feature variable? I am trying to generate a dataset which involves 1 feature variable(X) and 1 target variable(y).

The feature variable represents values on the X-axis on the graph and target variable represents values on Y-axis.

Datatype of X: integer

Datatype of y: floating point

I have N such graphs for same values of X, but a slight variation in y values.

One of the graph is as follows:

I want to fit the data into a regression.

Now, my question is how to generate the dataset for this use case. Should I include values from all graphs into a single dataset? But, in this case, for every unique value of X, I will have N rows with same value of X and a different value of y?

I am doubtful about this approach.

Any help is greatly appreciated!

I'm not sure what's the context of your question but there's no problem with the approach you've outlined. Different values of y for the same unique value of x (over different rows, such that for example you have: x = {1, 1}, y = {1, 2}) are a natural result of the noise usually assumed in the model you fit (e.g. $y = x + \epsilon$).
Hope this helps.


Can I use LSTM models to evaluate multiple, independent time series? Let's say that I would like to predict the temperature tomorrow. I could use the approach whereby I train a model based on a time-series dataset collected from a single location (for example, see this excellent walk-through:https://blogs.rstudio.com/tensorflow/posts/2017-12-20-time-series-forecasting-with-recurrent-neural-networks/).

However, let's say that I want to train a model that incorporated time-series from multiple weather recording sites. In this scenario, let us say that the recording observations from different sites are non-overlapping in time. Ideally, there would be a way to train models using non-consecutive observations (e.g. those collected from different sites), that also enabled us to quantify the influence of 'site' on our prediction.

Is this simply a case where one would train independent LSTM models for each site? Or are there alternative approaches whereby the total training data set can come from multiple observation sets (e.g. sites, non-consecutive observation blocks, etc.)?

If you are looking to predict multiple time series (which would be similar in nature, since each weather station in the area would record similar temperatures, even if they are not identical), using a separate LSTM model for each may prove quite time-consuming.

One approach you could take is one suggested in an excellent answer for another question on Cross Validated. Essentially, the author is describing a means for forecasting sales with LSTM whereby the model is trained on a mini-batch (or subset) of one series, and then a new series is selected. In this case, I would understand this to mean that a subset of data is incorporated from weather station 1, then another batch from weather station 2, etc.

This would have the advantage of essentially creating a unified series that takes the characteristics of all weather stations into account - which allows for maximum utilisation of the data, as well as allowing the network to learn patterns from all stations - not just one or a select few.

One option is to try this method using a subset of data for select weather stations at first, and then compare the prediction accuracy to those that use separate LSTM models for each. If you find that the predictions for the former method are more accurate, then it would make sense to go with that.

If you are using R, then there is also the option of using a package like forecastML to do so - if you are not tied down to using an LSTM model then this could also be another consideration.


Problem to classify multilabel dataset while using random forest algorithm I am using a large dataset with 4 different multilabel classes. I was trying to apply Random forest algorithm on those data set. After preparing data set I separate X (feature columns) and Y(feature class), split those data into train and testing data, fit training data to model, predict test data and was trying to find the accuracy by using testing data then I faced valueError: 'multiclass-multioutput is not supported'. My data set is:

train.head()

And here is my code after pre-processing:

#define X and y
feature_cols=['MFCCs_ 1','MFCCs_ 2','MFCCs_ 3','MFCCs_ 4','MFCCs_ 5','MFCCs_ 6','MFCCs_ 7','MFCCs_ 8','MFCCs_ 9', 'MFCCs_10', 'MFCCs_11', 'MFCCs_12', 'MFCCs_13','MFCCs_14','MFCCs_15','MFCCs_16','MFCCs_17','MFCCs_18','MFCCs_19','MFCCs_20','MFCCs_21','MFCCs_22']
feature_class=['RecordID','Family','Genus','Species']

# X is a matrix, hence we use [] to access the features we want in feature_cols
X = train[feature_cols]

# y is a vector, hence we use dot to access 'label'
y = train[feature_class]

model= RandomForestClassifier(n_estimators=100)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30,random_state=42)
model.fit(X_train, y_train)
preds = model.predict(X_test)
print(accuracy_score(y_test, preds,normalize=False))

and then I faced these value error:

How could I solve these problem?

From the documentation:

  Warning: At present, no metric in sklearn.metrics supports the multioutput-multiclass classification task.

See here and also this issue. Multiclass-multioutput doesn't seem reliably implemented at this point.


Why is the softmax function often used as activation function of output layer in classification neural networks? What special characteristics of the softmax function makes it a favourite choice as activation function in output layer of classification neural networks?

The softmax function is simply a generalisation of the logistic function, which simply squashes values into a given range.

At the final layer of a neural network, the model produces its final activations (a.k.a. logits), which we would like to be able to interpret as probabilities, as that would allow is to e.g. create a classification result.

the reason for using the softmax is to ensure these logits all sum up to 1, thereby fulfilling the constraints of a probability density. If we are try to predict whether a medical image contain cancer or not (simply yes or no), then the probability of a positive result (yes) and a negative result (no) must sum up to one. o the model produces a probability vector for each outcome, in pseudo-code: p = [yes, no].

If the final logits in this binary classification example were p = [0.03, 1.92], we can see that they sum to 1.95. This is not interpretable as a probability, although we can see the value is much higher for no. In other examples where there might be 1000s of categories, not just two), we can no-longer assess so easily, which is the largest logit. The softmax gives us some perspective and (quasi-) interpretable probabilities for the categories.

EDIT

As described by @Neil Slater in his comment below, using the softmax followed by a log loss function does indeed lead to a model that predicts values in the range of true probabilities, thus making them interpretable, as well as providing some nice statistical properties (see the Properties section of Maximum Likelihood Estimation).

Also note: minimising log-loss is equivalent to maximising the Maximium Liklihood Estimation. See a mathematical explanation here.

Implementations of the log-loss include:

the log_loss from Scikit Learn
binary_crossentropy in Keras (for binary problems)
categorical_crossentropy in Keras (for cases with > 2 classes)

These first two method names also highlight that log-loss is also the same as cross-entropy (in the general machine learning context of computing error rates between 0 and 1).


Feature-to-parameter mapping in neural networks For neural networks, can we tell which parameters are responsible for which features?

For example, in an image classification task, each pixel of an image is a feature.
Can I somehow find out which parameters encode the learned information, say from the top-left pixel of my training instances?

Yes, at least you can identify what pixels' are contributing most in the prediction.

Tool like Layerwise Relevance Propagation, used for Explainable AI, serves the similar purpose and evaluate the values(weights) during back propagation and evaluate what pixels are contributing most.

Many opensource implementation are available and on similar track, instead of identifying just relevant pixel you can perform activity for each pixel.

I believe that I'm able to answer your question.


Pretrained models for Propositional logic Are there any pretrained models which understand propositional logic?
For example, the t5 model can do question-answering. Given a context such as "Alice is Bob's mother. Bob is Charlie's father", t5 can answer the question "Who is Charlie's father" correctly, but it cannot say "Who is Charlie's grandmother".
Is there any model that has been/can be trained to do this kind of deduction and answer the question?

As far as I know, automated reasoning with propositional logic can be done with a solver like Prolog (and it's not new). I don't know if it has been done but I don't think it would make sense to train a ML model for propositional logic, since it's entirely symbolic (as opposed to statistical): the right answer can be found deterministically.
The question of logic reasoning from text like in the proposed example is a bit different, because it involves a step of representing the text. I think it could make sense to train a model which converts a text into a formal logic proposition (and back). The logic reasoning should still be done with a tool meant specifically for that imho. Note that question answering doesn't involve any logical reasoning, even if it might look this way to a user. As far as I know a QA system learns patterns about matching a type of question to its corresponding answer, the system is completely oblivious to the meaning.


What's an LSTM-LM formulation? I am reading this paper "Sequence to Sequence Learning with Neural Networks" http://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf

Under "2. The Model" it says:

  The LSTM computes this conditional probability by first obtaining the fixed dimensional representation v of the input sequence (x1, . . . , xT ) given by the last hidden state of the LSTM, and then computing the probability of y1, . . . ,yT′ with a standard LSTM-LM formulation whose initial hidden state is set to the representation v of x1, . . . , xT:

I know what an LSTM is, but what's an LSTM-LM? I've tried Googling it but can't find any good leads.

The definition of a Language Model (LM) is a probability distribution over sequences of words.

The simple illustration of an LM is predicting the next word given the previous word(s).

For example, if I have a language model and some initial word(s):

I set my initial word to My
My model predicts there is a high probability that name appears after My.
By setting the initial words to  My name, my model predicts there is a high probability that is appears after My name.
So it's like: My -> My name -> My name is -> My name is Tom, and so on.

You can think of the autocompletion on your smartphone keyboard. In fact, LM is the heart of autocompletions.

So, LSTM-LM is simply using an LSTM (and softmax function) to predict the next word given your previous words.

By the way, Language Model is not limited to LSTM, other RNNs (GRU), or other structured models. In fact, you can also use feedforward networks with context/sliding/rolling window to predict the next word given your initial words.


How to match a word from column and compare with other column in pandas dataframe I have the below dataframe

Text                                Keywords       Type
It’s a roll-on tube                 roll-on        ball
It is barrel                        barrel         barr
An unknown shape                                   others
it’s a assembly                     assembly       assembly
it’s a sealing assembly             assembly       factory
its a roll-on double                roll-on        factory

I have first found out the keywords, and based on the keyword and its corresponding type, it should true or false

For example,
when the keyword is roll-on , the type should be "ball" or "others"
when the keyword is barrel, the type should be "barr" or "others"

Output

Text                                Keywords       Type         Result
It’s a roll-on tube                 roll-on        ball         True
It is barrel                        barrel         barr         True
An unknown shape                                   others       False
it’s a assembly                     assembly       assembly     True
it’s a sealing assembly             assembly       factory      True
its a roll-on double                roll-on        factory      False

If you know the rules, then you can code them up as you would usually do.
For e.g.
df['Keywords'] == "roll-on" & (df['Type'] == "ball" | df['Type'] == "others") => df['Result'] = True
If you have too many rules and associations to manually code, you could consider using a look-up table + merge procedure. Here you would store all the rules and associations as a table (i.e. this would include Keywords, Type, and Result), and merge Result back on to your original dataframe (by Keywords and Type).


Communication between Keras and Backend like Tensorflow, Theano Keras is a high level neural network API providing python library which uses tensor flow or theano or cntk as backend. What are the primary roles of backend libraries? Is it implementation? or Is it computational heavylifting using GPU, threading etc?

I couldn't find any good resources online to understand how keras interacts with tensor flow or theano(backend) session. Any such resource or direction for understanding interaction is helpful !

Keras is a high level API but the backend is readily available. You simply access it by doing:

from keras import backend as K

The K will then be the same as tf as if you imported Tensorflow like this:

import tensorflow as tf

So you can use K to perform lower level operations with the backend.

For more information you can read the Keras backend documentation.


Which error metric is good for measuring accuracy I am estimating water depth with satellite data (predicted value) and would like to validate my result using bathymetry  lidar data collected on the field and believed to be more accurate (observed value). I have different observations at each water depth. For example, number of observations at water depth range of 0-10 m are 300, where as values at deeper depth range (10 - 20 m) are less (~50 points). I have been using RMSE (as I would like to penalize larger error) to measure my accuracy but wondering if there is a better error metric out that is not sensitive to number of observations. In other words, for water depth 10 - 20 m with 50 points, I have RMSE of ~6m, and I was thinking the value could be lower if I have more observations. Where as for shallow water depth (0-10 m), my RMSE are much lower, perhaps because I have lots of observation.

Since you have a regression problem, you can use many possible evaluation metrics. Other regression evaluation metrics include mean squared error (MSE), mean absolute value (MAE), and mean absolute percentage error (MAPE).
However, it sounds like the issue is variance across different target values. If you switch to Bayesian Regression, it will better model variance and uncertainty across the range of target values.


High dimensional space is dense or sparse? I read some blog articles recently. One mentions that you could not imagine high dimensional space as 2d or 3d as distance between any 2 points in high dimensional space tends to be similar, which means 'dense'. However in the t-SNE paper, it says high dimensional space tends to be sparse such that you have to employ special dimensionality reduction techniques to visualize in 2d or 3d space in meaningful way. So how to reconciliate these 2 different views?

Data in a high dimensional space tends to be sparser than in lower dimensions.  There are various ways to quantify this, but one way of thinking that may help your intuition is to start by imagining points spread uniformly at random in a three dimensional box.  Now flatten the box into a square, pushing two opposite sides together so all the points lie on a single plane.  Do you see that the average distance between a point and its nearest neighbor is now smaller?  Now flatten the square into a line segment.  Do you see that the average distance between a point and its neighbors is now smaller still?

There is no conflict between this and saying that the average distance between any 2 points in the high dimensional space tends to be similar.  The latter statement doesn't imply density.  The real number line is dense (it has no gaps), and yet the distance between points ranges from 0 to infinity.  The point is that the higher the dimension of your space, the more likely the points are to lie near the edges of the space rather than the center.

Again, consider the dimensions we can actually see.  Consider a circle with radius=1, inscribed in a square with sides of length=2.  The circle occupies $\pi / 4$ of the square's area, about 78.5%.  Now consider a sphere of radius=1 inscribed in a cube with sides of length=2.  The sphere occupies $\pi / 6$ of the cube's volume, about 52.4%.  As you see in this example, the odds of a randomly placed point lying close to the center (with close to the center in this case meaning within the circle or the sphere) is lower as the dimension increases.  Points are more likely to be in the corners.  This is why in high dimensions the distance between the points tends to be similar - because randomly placed points tend to be close to the edges of the region.


Alignment of square nonorientable images/data Another post where I don't know enough terminology to describe things efficiently. For the comments, please suggest some tags and keywords I can add to this post to make it better.

Say I have a 2D data structure where 'orientation' doesn't matter. The examples I ran into:

The state of a 2048 game. In terms of symmetry groups this would be D4 / D8, except that an operation doesn't yield an identical state, it just yields another state that has the same solution.
Images of plankton or galaxies (without background). Somewhat similar to above except that any rotation (not just 90o) yields an equally valid image (and one might take scale into account, but let's forget about that).

In both cases I've wanted to transform all these equivalent states/images to remove all but one of the equivalent images. To illustrate with two that worked:

I can use image moments M10 and M01 to transform horizontally and vertically mirrored equivalent data. E.g. apply horizontal mirroring iff it makes M10 bigger. This would transform a 2048 state and it's horizontal mirror image to the same state.
I can use the eigenvector of the covariance matrix which has the largest eigenvalue as the orientation. Then I can rotate the image to align this eigenvector with some predetermined axis (e.g. horizontally).

That still leaves a lot of operations though (diagonal mirroring, rotations around the center, inversion). And these operations do not commute (D8 is non-Abelian). Is there any comprehensive approach?

The reason I want to do this is to help machine learning methods by removing variance that isn't actually meaningful. Hopefully that makes sure they don't have to learn these equivalences, so possibly need less train data (and time).

Fun with Group Theory!

There are only 8 unique rotation-inversion operations for a square matrix.

The four rotation operators are (0,90,180,270).  Further rotation or rotation in the reverse direction is the same as these four.  Two successive rotations just yields one of the rotation operators, so we will only consider these four rotations applied one time.

The five inversion operators are (0,/,\,|,-).  Two successive inversions just yields a rotation, so we will only allow for a single inversion.

We can thus derive all operators by combining these two vectors, which yields 4*5=20 possible states.

(0,90,180,270,0/,90/,180/,270/,0\,90\,180\,270\,0|,90|,180|,270|,0-,90-,180-,270-)

But there is still symmetry to be exploited in the inversion operators.  You can probably intuit that the 4x4 matrix only has 8 final states: inverted or not and rotated by (0,90,180,270).  It turns out that you can arrive at any possible state involving an inversion using any of the other inversion operators followed by one of the rotations.  So we only need to retain a single inversion operator!  So the final set of 8 orthogonal operations are:

(0,90,180,270,0|,90|,180|,270|)

If there is any symmetry in the matrix's members then some of the resulting states may be degenerate.

In terms of mapping possible states into a ground state, it makes sense to apply a set of successive deterministic rules to determine the ground state orientation.  I suggest finding the largest corner square and locating it in the lower right corner. If there are multiple candidates with equally large values in the corner square then use the next closest square as a tie breaker.  There are 16 squares, so you can eventually break all ties or declare degeneracy. There is one remaining \ operation that you can decide to apply in order to locate the larger of the two squares adjacent to the lower right corner at the bottom.  Again, you can use squares adjacent to these as tie breakers.


Label Encoder encoding the same class as two integers First I have defined classes of a label encoder with the keys of a dictionary. Then I have used that label encoder to encode some strings. But for the same string, its giving different integers. Why is this happening?

grid_dict = {'8861892541fffff': 588,'886189254dfffff': 65, '8861892543fffff': 103, '8861892547fffff': 83, '8861892545fffff': 401,
 '8861892569fffff': 131, '8861892727fffff': 39, '886189255dfffff': 13, '8861893597fffff': 2, '8861892555fffff': 4, '886189209bfffff': 13, '8861892549fffff': 3}

le = LabelEncoder()
le.classes_ = list(grid_dict.keys())

Now

le.transform(['8861892547fffff']) #4th class

gives me array([1])
but le.transform(['8861892545fffff']) #5th class
also gives me the same array([1]) which should not happen. Also when I do le.inverse_transform(0) its giving '8861892541fffff' which is the second class.
Also when I do le.transform(['8861892549fffff']) #last class its giving array([5]) and for le.transform(['8861892569fffff']) #class6 its giving array([5]).

I am not fitting the label encoder anywhere. I have some stored the keys in a dictionary and declaring the label encoder classes with the dict keys. Why is this happening?


k-Nearest Neighbours with time series data - how to obtain whole-time-period estimators I have a large dataset for the activities performed by multiple staff in a factory over a long period of time - 01/01/2017 - present.  The activities performed by different staff are recorded at each point in time (since they interact with software).  I have tabulated these to record the number of each activities performed by each operator for each day.

My table looks something like this:

Date         Name    Activity  UnitsProcessed   Shift      Team
01/10/2017   MMouse  Soldering   1000           Shift A   Team A
01/10/2017   MMouse  Welding     3000           Shift A   Team A
01/10/2017   DDuck   Soldering   4000           Shift A   Team B
01/10/2017   DDuck   Welding     1000           Shift A   Team B
...
02/11/2017   MMouse  Soldering   1100           Shift A   Team A
02/11/2017   MMouse  Welding     2900           Shift A   Team A
02/11/2017   DDuck   Soldering   4010           Shift A   Team B
02/11/2017   DDuck   Welding     900            Shift A   Team B

Each line represents the activities performed by each operator each day they worked, and the number of units processed during each activity, along with a classifier for the shift and team that the operator was a member of (for simplicity, assume that this does not change over time).

For brevity, I have omitted values from Shift B, but essentially, the team members of Shift B will (usually) work different days from Shift A, but the proportion of tasks performed will usually be similar to the corresponding Team on Shift A - Team A or Team B.

The problem I want to solve is:  I have the date, activity and units processed - as well as various other variables - for another operator, "GOofy".  But I don't know which shift/team Mr GOofy works.  So I am using the data for Mr Mouse and Mr Duck (obviously, in reality, scores of people) to see if I can try to train kNN to correctly guess the Shift and Team pattern for each Name, given the Date, Activity and Units Processed fields, among others.

My thinking is:

a) There is no particular value in keeping Team and Shift separate, so I should create a joined column which combines both

B) If I create separate columns for "Soldering", "Welding" and the other activities, I can have the number of units processed for each day for each activity by date.  Given that the kind of activity performed is (I hypothesise) a strong predictor of the shift/team combination, I should be able then to train to recognise shift/team based on both the date of that line (perhaps converted to a numeric date) and the relative values for each line of soldering, welding, etc.

The problem is that, when extended to the test set, this will only give me predictors for each individual day.  But I suspect that the days worked is also a significant predictor, so I would like to include this as well.  But I'm not sure how to approach this.

The two approaches I can think of are:

Option 1:  To use kNN to generate estimators for shift/team for each operator/day combination, with the x variables being the date, units soldered and units welded per day, and then calculate, for each operator, the proportion of times "Shift A, Team A", "Shift A, Team B", etc were assigned to that operator - so if DDuck is assigned "Shift A, Team A", on 90% of days, it seems most likely that DDuck is indeed "Shift A, Team A".

Option 2:  To somehow create a variable which represents all of the days worked for each operator - although I'm not sure how to do this - and thus create a  set which contains one row for each operator.

Are either of these approaches valid?  Is there a widely-used alternative approach for this?

Note that this is not a duplicate of previous questions relating to kNN and time-series data, since those were concerned with forecasting, whereas I am trying to analyse retrospective data.

Do you have more numeric features? KNN works off distances (euclidean, L1, etc.) which requires numeric data. It doesn't take in categorical data. Your activity feature will not work with KNN and even converting the date to say an integer will just create a time series which will not perform well with KNN. I think your approaching this wrong. Try logistic regression. It will take in both categorical and numeric features.

  There is no particular value in keeping Team and Shift separate, so I should create a joined column which combines both.

I would build two separate models for team and shift. You'll get higher accuracy because your using the data to predict less classes and letting the models specialize on each task.

Overall I would use logistic regression not KNN. It's more appropriate for your problem. Hope this helps.


Is there some model-based variation of the Q-Learning algorithm which learns on a 3D SxAxS' table instead of a 2D SxA table? Q-learning works with a 2D SxA table of Q values, where S is the current state and A is the action taken. Is there some model-based variant of Q-learning (or SARSA) that uses a 3D SxAxS' table to store Q values, where S' is the resulting state?

A similar table may be used to learn models in discrete environments, but in this case you don't store Q values, but counts.

I couldn't find anything like this, either because I don't know the correct terms to search for, or because it does not exist. It doesn't matter to me if it is useful or not, I just need to know if it exists, and if it exists, how is it called.

The concept of state-action values $Q$ is to denote how good is to be in a particular state and perform a particular action in terms of expected future reward. From what I understand from your question, you are interested in the problem of model uncertainty (uncertainty on the dynamics of the system). In other words, our artificial agent interacts within an unknown environment (transition dynamics $T(s,a,s')$ and reward dynamics $R(s,a)$ or $R(s,a,s')$ are unknown).

The framework you should take a look at is Bayesian Model-based RL. I outline an approach so you can have an idea:

Modelling Transitions

First assume that we have uncertainty on the transitions of the environment $T(s,a,s')$.
To tackle this we will assume that our agent maintains a distribution over possible transitions. Without getting into the theoretical math, I will illustrate this by using a simple Dirchlet-Multinomial model:

The states are sampled from a Multinomial likelihood $s'\sim Mult(p_{ss'}^{a})$ and we assume a prior over the transitions $p_{ss'}\sim Dir(\alpha)$, where $\alpha$ is set to $1/|\cal{S}|$, where is $\cal{S}$ is the state space.
The posterior over transitions will be also a Dirichlet because of the conjugacy of the likelihood and prior distributions. To update such a posterior you need to perform simple algebraic calculations and maintaining the counts of each transition.

The Algorithm

The agent does two processes:

(Simulation or Planning phase) Samples an MDP from the posterior over transitions (in the first iteration you sample from the prior distribution). Having a 'fixed table of tranistions' solves the sampled MDP (e.g with Value iteration) and selects the optimum action.
(Real-world interaction phase) In the real world, you observe the reward of your action and the new state of the environment. You update the posterior counts.

Eventually, and if you have chosen an appropriate distribution for your domain the agent will adapt to the unknown environment. Of course richer priors with non conjugacy will lead to MCMC sampling methods. I refer you to this paper to get an overview of the problem: Model-based Bayesian Exploration and the quite advanced: Bayes-Adaptive MDPs for further research and exploration.


Hierarchical Clustering: Extract observations from large heatmap I'm currently trying to visualize a large data set as heat map. That in itself works smoothly but I struggle with gaining insights from interestingly looking clusters.

Specifically, I have two questions that are very related:

First, I find clusters of interesting features and am looking for a systematic way to extract the a flat clustering at a specific level (but the fcluster function seems to do something different and cut_tree doesn't work with those trees). I would like to have a slice of the hierarchical clustering at a specified depth of the dendrogram. This is probably encoded in the linkage matrix Z but I struggle to understand how exactly I can extract that information from Z.

Second, with the complicated heatmap pictured below, the row names on the right are gene names for every 100th data point (gene). I would now like to have a look at which genes are in some of the little clusters, for instance the little black square for feature MF: LIHC that is marked.

I know the IDs of the genes that are labelled on the right, so I would want to know something like:

Which genes are in the same cluster as CAPN7 at level 5?

Thank you for your help!

Roman

A similar question was asked at stackoverflow. There it was proposed to use the option criterion='maxclust' for the flcuster function:

from scipy.cluster.hierarchy import fcluster
clust = fcluster(Z, t=k, criterion='maxclust')

The description of this option in the flcuster documentation is a bit confusing, but that's how you get a clustering with t=k clusters.

You should be able to retrieve the answer to your second question with the resulting array.


Loading a Model with weights and optimizers without creating an instance in PyTorch I recently downloaded Camembert Model to fine-tune it for my purpose.
Upon unzipping the file the contents are:

Upon loading the model.pt file using pytorch:
import torch
model = torch.load(model_saved_at)

I saw that model was in OrderedDict format containing the following keys:
args
model
optimizer_history
extra_state
last_optimizer_state

As the name suggests most of them are OrderedKeys themselves with the exception of args which belongs to a class argsparse.Namespace. Using vars() we can see args only contains some hyperparameters and values which are to be passed from the command-line.
model["model"] contains the weights which I want to load and use as my base model.
A small part of it is as shown below:
for ans in model["model"].keys():
    try:
        print(ans, "\t" ,model["model"][ans].size())
    except:
        print(ans, type(ans))

decoder.sentence_encoder.embed_tokens.weight     torch.Size([32005, 768])
decoder.sentence_encoder.embed_positions.weight      torch.Size([514, 768])
decoder.sentence_encoder.layers.0.self_attn.in_proj_weight   torch.Size([2304, 768])
decoder.sentence_encoder.layers.0.self_attn.in_proj_bias     torch.Size([2304])
decoder.sentence_encoder.layers.0.self_attn.out_proj.weight      torch.Size([768, 768])
decoder.sentence_encoder.layers.0.self_attn.out_proj.bias    torch.Size([768])
decoder.sentence_encoder.layers.0.self_attn_layer_norm.weight    torch.Size([768])
decoder.sentence_encoder.layers.0.self_attn_layer_norm.bias      torch.Size([768])
decoder.sentence_encoder.layers.0.fc1.weight     torch.Size([3072, 768])
decoder.sentence_encoder.layers.0.fc1.bias   torch.Size([3072])
decoder.sentence_encoder.layers.0.fc2.weight     torch.Size([768, 3072])
decoder.sentence_encoder.layers.0.fc2.bias   torch.Size([768])

However, I cannot use load_state_dict() since I have no instance of this class. How am I suppose to load the weights and optimization parameters without creating an instance? I thought of using sentence.bpe.model but they are for tokenization purposes.

If you are open to using huggingface transformer for fine tuning which is really popular, here is a code sample:
import transformers
class Model(nn.Module):
    def __init__(self):
        super(Model,self).__init__()
        self.Bert = transformers.CamemBertModel.from_pretrained('camembert-base')
        self.fc0 = nn.Linear(768,1)

        nn.init.normal_(self.fc0.weight,std= 0.1)
        nn.init.normal_(self.fc0.bias ,0.)

    def forward(self,input_ids,attention_mask):
        hid= self.Bert(input_ids,attention_mask = attention_mask)
        hid= hid[0][:,0]
        x = self.fc0(hid)
        return x

You can change your last layer as you need.It is just a sample. It will load the pretrained weights from huggingface.You need not provide it.You can install transformers by the following line in terminal.
pip3 install transformers


Time Series Forecasting for Multiple Customers using one RNN I have a product which has univariate and also multivariate time series data from multiple customers. I have variable amount of data available. Ranging between couple of years to couple of months.
What I want to achieve is a model which takes as input training data from all my customers and forecast for 1 at a time. Thus utilizing patterns from all of them and then forecast for one customer at a time.

E.g.

Train:
Date        Customer  Variable
2018-01-01  A         10
2018-01-01  B         5
2018-01-01  C         13
2018-01-02  A         9
2018-01-02  B         7
2018-01-02  C         15

Predict:
2018-01-03  B         ?

Any help on how to formulate this problem and what would be the best model for this?

Here are a few rapidfire ideas:

Does the customers affect each other in any way? If that is the case, you need to feed the information of multiple customers at the same time to your RNN. If not, then I would advise creating a time series for each of your customers and feeding it separately to your RNN.
What is it you are trying to predict? Is it a percentage chance of something? Is it a number (e.g. volume of customer purchases). You need to create your outputs, and the inputs you will use to predict that output accordingly. You may also use multiple RNNs if some of your outputs have no relation to each other. For example, if you are trying to predict the volume of purchase for a customer and whether a customer will leave or not, you may want to create two seperate RNNs that will take different inputs (e.g. use the average purchase volume as input for the RNN that predicts the customer volume, and use the frequency of customer visits as input for the RNN that predicts whether the customer will leave or not
Is there huge discrepancy in values of your variables? (e.g. some customers only do 10 dollar purchases, but the others do 100,000, and you want to feed the information of both to your RNN) If that is the case, you may want to "normalize" your variables as well as the outputs (labels) you are trying to predict
Do you really need an RNN? If your customers' outputs are independent of the sequence of variables from the past, then drop the RNN and use something simpler like feedforward neural network or logistic regression
Are your customers' outputs affected by variables from a long time ago? If so, you may want to add a memory component, such as an LSTM, to your RNN
Does your customers have any periodic components? (e.g. the customer usually buys something 10 days after adding it to the shopping cart). This tells you the lookback (how many past days' data you need to feed to the RNN) you will need for your RNN

Hope it helps. I must admit however, that without getting a sense of the inputs and outputs it is difficult to answer this question.


Calculate implicit rating from streaming behaviour for Recommendation Engine I have a dataset containing some user streams data for particular videos like below:
u_id|start_stream_time_dt|watch_time_ms|video_category
   1|        2021-02-01  | 3600        |  Live

My goal is to build a recommender system for watch streams.
However, I would like to find the optimal watch_stream threshold (or other approaches) that would allow me to define if a user has indeed watched a video because he/she's interested.
In other words I'd like to fill the 1s in the user_item matrix based on these information I have.
Any good approaches to suggest me? or any resources I can look up?
Thanks.

Create a histogram of the watch_time_ms. If you are lucky - you may see a bi-modal distribution (i.e two peaks). The higher/lower peak could be interpreted as interested / uninterested behavior respectively. Then your threshold could lie somewhere in the valley between your two peaks in the histogram.
If your videos are variable length - you may also want to try normalizing the watch_video_ms by dividing with the total_video_length_ms before reviewing the histogram. You then end up with smooth values between 0 - 1 for each video.
If there is no obvious visible pattern in the histogram - then choose a percentile to divide between interested / uninterested views. For e.g. the top 25 percentile of views represent are the 1's whereas the remainder are 0's. Make sure you choose the percentile appropriately based on the problem domain / objective.


Training Neural Networks with unknown length of input I'm currently going into the world of machine learning and Neural Networks, thanks to synaptic (js) that interests me a lot.

So I read a lot, wikipedia links and synaptic's NN 101, but there's a lot of basics questions that I don't understand (but I'd like to) in the use of machine learning (NN) and the point of these technologies.

Let's say, I wan't my network to (kind of) learn (something like) gravity, so to train it I set in input 10 objects with a mass, and a position x, y (and z) and I set output the new x, y (and z) of each objects.
I guess I should give it several configurations and everything but here is the question; can it, then, be able to compute the interactions between 10000, 100000 objects?

At this stage in my learning, what I don't clearly get is what is the point of teaching/training neurons to compute XOR like it's shown in synaptic's documentation:

var trainingSet = [
    {
        input: [0,0],
        output: [0]
    },
    {
        input: [0,1],
        output: [1]
    },
    {
        input: [1,0],
        output: [1]
    },
    {
        input: [1,1],
        output: [0]
    },
];

var trainer = new Trainer(myNetwork);
trainer.train(trainingSet);

Were we just give it all the possible inputs and outputs to a XOR.

Well, as I'm all new to the technologies I think my questions are full of non-sense and everything but thanks for reading and help you might bring :)

Gravity

Basically you'd like to find a function that maps an input (an object with a mass and a location) to an output (new location). It's not necessary to have one set of input neurons for each different object. It's sufficient to encode the input variables generically for all objects.

Like a function:

$$f( x, y, z, mass ) \to ( x_n, y_n, z_n )$$

XOR

Training the XOR function is special as the early neural networks, the perceptrons, were unable to learn the XOR function (because it is not linear separable). Multi-Layer Perceptrons however are able to learn the XOR function. This is just to demonstrate that the NN implementation of Synaptic is capable of learning problems that are not linear separable


How can I "export" a model from Orange once I have prototyped a solution? Using Orange, I would like to use the underlying python model within the control system of my device.  The device is a simple machine which is not connected to the internet.  My intent is to use the model to the machine operating parameters based upon the application requirements.

Is there a way (or could there be) to export the trained model for use?  This seems to be a missing feature or future enhancement for Orange.

You can export any trained machine learning model in Orange using the Save Model widget.
This is what the Save Model widget does under the Model tab. Create your model, then click save model and your model will be saved to a pickle file. Then just load your pickle  file in Python.
Here is a like to a similar question on Stack Overflow:
https://stackoverflow.com/questions/45598627/orange-save-model-in-python-script


Predicting a timeseries that includes categorical and numeric data Given the following properties in a dataset:

Type Of Work [T1,T2,T3,...Ti]
Measurment Unit [U1,U2,U3,...Ui]
Number <int>
Date <datetime>
Status [S1,S2,S3,...Si]

What can i use to predict the Status in a given Date?

Any suggestion is really appreciated.

What you are looking for are time series models that are called "dynamic regression models." These can include forecast distance models like linear models or tree-based methods.

While it is possible to do classification with time series models, it is rarer still to do multiclass classification for time series. If you have a high number of classes to predict, it might be difficult.


How to speed up passing of images to a GPU I am using Ubuntu 16.04 LTS installed on a 4TB HDD. I am working with a large dataset (more than 30 GB and 150,000 images). I have a single 11GB GTX 1080 Ti GPU card on my system. I am training a mask RCNN written in Keras. I noticed that passing the entire dataset of 150,000 images to a single GPU from the HDD will take few weeks at least to process a single epoch. Hence I want to speed up passing an image to a GPU.

So I bought a 256 GB SSD, mounted it and copied only the images directory to the SSD. Then I began to train the Keras model. All python files including the Anaconda install are on the HDD. I noticed that merely moving the images to SSD did not speed up the training. Does anyone know how to speed up the training process by sending images from the SSD to the GPU? Do I need to move the python files and anaconda install to the SSD as well? I do not wish to reinstall the entire OS on the SSD.

Try to use batch version for training. You have to implement a function to manage your data which exist in the memory. For each batch, you can load the data into your memory as much as possible, high RAM size can help. Then you can pass a batch to your network, the size of graphical memory matters here, using the current data which is in memory now. By doing that you reduce the number of disk calls. When it happens to load data from disk, you fill the memory as much as possible, the disk speed matters here; consequently, high speed SSDs are valuable for facilitating the reading operation.


Converting Text Entries into values for machine learning - KNN Consider the following dataset.

| Area | Job Type | Complete |
-------------------------------
| AAA  | Install  |     N    |
| AAB  | Repair A |     Y    |
| OOC  | Repair C |     Y    |
| LCX  | Cease    |     N    |
-------------------------------

I am using JavaScript (Started getting into Tensorflow.js amongs other ML algorithms and theories.) and I am struggling to find a suitable ML method to process this. There could be over 100 areas and over 15 Job Types, but Complete can only be Y/N.

I was thinking about assigning a number to each case as follows

AAA -> 1    |   Install  -> 1   |    Y -> 1
AAB -> 2    |   Repair A -> 2   |    N -> 0
OOC -> 3    |   Repair C -> 3   |
LCX -> 4    |   Cease    -> 4   |
... -> x    |

Is this viable? would it work?

I want to give it another case and return the % chance of that case happening.
I have tried to use Naive Bayes classifier and has some success.

-------------------------------
| LCX  | Cease    |     10%    |
-------------------------------

This is just a small sample there are other x's that I want to include that are a  mix of floating points and other String Values.
the whole main data-set contains over 40 million entries and over 40 possible columns that could be a factor that effects 'completed' so plenty of training data to go around!

What would the best method be what approach would you recommend?

It is not recommended to use the method you suggested of converting text to integer values (unless you are using decision trees or XGBoost as the predictive model). If you are usin KNN (and many other models) you should use a technique called one-hot encoding. The reason for this is that (using your example above) Install is 1 and Repair A is 2, but Install isn't necessarily quantitively twice as small as Repair A but you are assigning them 1 and 2 which are quantitively related (2 is twice as big as 1). So your model will interpret it that way.

That is why you need to use One Hot encoding.

I'm not privy to ML in JS but in python you could do:
With pandas:

import pandas as pd

df = pd.DataFrame({
      'A':['a','b','a'],
      'B':['b','a','c']
    })
df
Out[]:
   A  B
0  a  b
1  b  a
2  a  c

# Get one hot encoding of columns B
one_hot = pd.get_dummies(df['B'])
# Drop column B as it is now encoded
df = df.drop('B',axis = 1)
# Join the encoded df
df = df.join(one_hot)
df
Out[]:
   A  a  b  c
0  a  0  1  0
1  b  1  0  0
2  a  0  0  1

Maybe someone else can give you specific code in JS.


Task not serializable Error import org.apache.spark.SparkContext
import org.apache.spark.SparkConf
import org.apache.spark.sql.cassandra.CassandraSQLContext
object Test {
  val sparkConf = new SparkConf(true).set("spark.cassandra.connection.host", <Cassandra Server IP>)
  val sc = new SparkContext(sparkConf)
  val cassandraSQLContext = new CassandraSQLContext(sc)
  val numberAsString = cassandraSQLContext.sql("select * from testing.test").first().getAs[Int]("number").toString()
  val testRDD = sc.parallelize(List(0, 0))
  val newRDD = testRDD.map { x => numberAsString }
}

This is the code that I have written in Spark. I was expecting it to work properly because i am evaluating the numberAsString value and then using it in the map function but it gives me task not serializable error. I am running the job in local mode.

Looking at another question on Stack Overflow about serialization exceptions in Spark, it says that anonymous functions serialize their containing class, and if that class contains the SparkContext -- which is not serializable -- then an error is thrown.

Perhaps this is happening to you?


Continuous variable to categorical by quartiles? Let's stay I have a field with a continuous variable, like a count of people waiting in line. I want to take those values and create a categorical value based on quartiles. Let's say my range of values is 1 to 80 and the quartiles tell me that a "very short" line is less than 5 people, a "short" line in 6 to 30, a "long" line is 31 to 50 and a "very long" line is >=51

I can think of different ways to write this in python/pandas/numpy but something tell me that one of you can come up with a method/snippet that is short and elegant. Note that I want the method to also generate the quartile values, something that I haven't really done in Python before.

Panda's Categorical Data Type is designed for that type of analysis,  pandas.cut can divide by user-defined bins and pandas.qcut can create quantile-based discretization. Something like this:

import numpy as np
import pandas as pd

df = pd.DataFrame({'value': np.random.randint(1, 80, 20)})
df['group'] = pd.cut(df.value,
                     bins=[0, 5, 31, 51, 80],
                     labels=["very short", "short", "long", "very long"])


Semantic segmentation with greyscale images I'm trying to reproduce a research with greyscale images instead of colour images.
I have found that there are pre-trained networks, like VGG16, with ImageNet. But that dataset has colour images, and I can't use it because I'm going to use greyscale images.
Is there any pre-trained network with greyscale images?
Failing that, I can also train a network with a greyscale image dataset but I can't find any.

Use ImageNet, convert every image to grayscale and train VGG16.


Faster R-CNN: Labels regarding the positive anchors when there are many classes The authors of the original paper of Faster R-CNN when they refer to the positive anchors, they are labeled as 1. I guess they refer in binary classification. What happens in the case in a task we have more than two classes?

I mean the purpose of the RPN is to generate just region proposals (there is an object or not, regardless the label) or it generates region proposals and the respective label as well? I guess it is the second case, since the positive anchors that participate in the training process need to have a ground truth label as well, i.e. (cat, dog, e.t.c).

However, I might misunderstand things and interpret things in a wrong way.

As well as the algorithm used for region proposals in R-CNN and in Fast R-CNN, RPN task in Faster R-CNN is to propose regions that may or may not be an object, so its
cls head with 2k scores corresponds to the classification of every anchor as an object or as a non object (this way all of them may be non objects):

But it could also be a k scores logistic regression (k sigmoids 0 to 1) as they stated in the fourth note of the paper:

  For simplicity we implement the cls layer as a two-class
  softmax layer. Alternatively, one may use logistic regression to
  produce k scores

After proposal generation, Fast R-CNN is used to classify and improve every selected proposal, i.e. grey area in the following image:

So you may better conceptualize the difference between Fast and Faster R-CNN considering the region proposal generation algorithm as a black box, the one could be filled with a RPN head connected to the output convolutional feature map of the deep net used as backbone, or the standard Selective Search used by R-CNN and Fast R-CNN


Labeling hubs in a network Let's say we did some analysis on a network dataset. We have an adjacency matrix which we can use to construct a graph and we can find the nodes with the highest degree's. How would I go about giving a location of these nodes. Would it be okay to refer to the entries of the adjacency matrix (i,j)?

Im not sure if I understand your question. But, you do not need to create a graph to work out node degree. Node degree can be obtained by taking the sum of the row or column in the adjacency matrix. E.g. in this 4 node graph where each node has degree $2$, and the row and column sums are also 2.

  0|1|2|3|              1----0
0|0|1|0|1|              ¦    ¦
1|1|0|1|0|              2----3
2|0|1|0|1|
3|1|0|1|0|

And so if we were to update the matrix to add a diagonal edge between nodes 1 and 3. (apologies for the rubbish graphic). Adding values to the diagonal creates a self-loop edge.

  0|1|2|3|              1----0
0|0|1|0|1|              ¦ \  ¦
1|1|0|1|1|              2----3
2|0|1|0|1|
3|1|1|1|0|

In relation to node location, yes refer to node locations using the indices of the adjacency matrix $(i,j)$.

In python this is done by:

import numpy as np

a = np.matrix([[0,1,0,1],
               [1,0,1,1],
               [0,1,0,1],
               [1,1,1,0]])

a.sum(axis=0) # row sum. aka Node Degree
a.sum(axis=1) # column sum. aka Node Degree
# form a tuple with the result of above
# i.e. (a.sum(axis=1),a.sum(axis=1)) = (1,1)

a.sum(axis=0).argmax() # index of row with max sum (returns 1st index if multiple nodes have same degree)

d= 2 # return nodes with degree value d
np.where(a.sum(axis=0) > d)[1] # returns indeces of nodes with degree > d


How to display the entire output and not shortened versions I'm trying to display all the columns in my dataset that have NaN values:

print(train.isnull().sum())

The output that I get is this:

Id                0
MSSubClass        0
MSZoning          0
LotFrontage       0
LotArea           0
Street            0
Alley             0
LotShape          0
LandContour       0
Utilities         0
LotConfig         0
LandSlope         0
Neighborhood      0
Condition1        0
Condition2        0
BldgType          0
HouseStyle        0
OverallQual       0
OverallCond       0
YearBuilt         0
YearRemodAdd      0
RoofStyle         0
RoofMatl          0
Exterior1st       0
Exterior2nd       0
MasVnrType        8
MasVnrArea        0
ExterQual         0
ExterCond         0
Foundation        0
                 ..
BedroomAbvGr      0
KitchenAbvGr      0
KitchenQual       0
TotRmsAbvGrd      0
Functional        0
Fireplaces        0
FireplaceQu       0
GarageType        0
GarageYrBlt      81
GarageFinish      0
GarageCars        0
GarageArea        0
GarageQual        0
GarageCond        0
PavedDrive        0
WoodDeckSF        0
OpenPorchSF       0
EnclosedPorch     0
3SsnPorch         0
ScreenPorch       0
PoolArea          0
PoolQC            0
Fence             0
MiscFeature       0
MiscVal           0
MoSold            0
YrSold            0
SaleType          0
SaleCondition     0
SalePrice         0
Length: 81, dtype: int64

I actually want to see the entire output and not a shortened version with ..

Alternatively to Marce's answer, you can display all the rows without finding N by using:

with pd.option_context("display.max_rows", -1):
    display(train.isnull().sum())


When would one use Manhattan distance as opposed to Euclidean distance? I am trying to look for a good argument on why one would use the Manhattan distance over the Euclidean distance in machine learning.

The closest thing I found to a good argument so far is on this MIT lecture.

At 36:15 you can see on the slides the following statement:

"Typically use Euclidean metric; Manhattan may be appropriate if different dimensions are not comparable."

Shortly after, the professor says that, because the number of legs of a reptile varies from 0 to 4 (whereas the other features are binary, only vary from 0 to 1), the "number of legs" feature will end up having a much higher weight if the Euclidean distance is used. Sure enough, that is indeed right. But one would also have that problem if using the Manhattan distance (only that the problem would be slightly mitigated because we don't square the difference like we do on the Euclidean distance).

A better way to solve the above problem would be to normalize the "number of legs" feature so its value will always be between 0 and 1.

Therefore, since there is a better way to solve the problem, it felt like the argument of using the Manhattan distance in this case lacked a stronger point, at least in my opinion.

Does anyone actually know why and when someone would use Manhattan distance over Euclidean? Can anyone give me an example in which using the Manhattan distance would yield better results?

According to this interesting paper, Manhattan distance (L1 norm) may be preferable to Euclidean distance (L2 norm) for the case of high dimensional data.
The authors of the paper even go a step further and suggest to use Lk norm distances, with a fractional value of k, for very high dimensional data in order to improve the results of distance-based algorithms, like clustering.


Why some people add results from PCA and other dimensional reductions techniques as features I often see in a well-known datascience competition platform, that a lot of people apply some dimension reductions techniques, but instead of using it to reduce the number of features (complexity) of their models, they append the resulting features to their datasets. Is'nt that adding complexity to their model rather than simplifying it ?

This is feature engineering. You just give the algorithm another look at the data, from another point of view. It often helps to understand better data when you have different point of views.

For example, let's assume you want to learn the US road directions to a simple ML model. You give him all examples from roads 1 to 100 mapping to 0 (if the road is east-west), or 1 (road is north-south), except for the number 47. Then you ask the model to predict 47 and it will answer 0 because it's between 46 and 48 who are labelled 0.

Now if you give the model another point of view (here, the parity of the road number), then it will obviously learn efficiently the road directions.

You can view the results of the PCA as the parity, just another view of the data to understand it better.


Chunking Sentences with Spacy I have a lot of sentences (500k) which looks like this:

"Penalty missed! Bad penalty by Felipe Brisola  - Riga FC -  shot with right foot is very close to the goal. Felipe Brisola should be disappointed."
"Penalty saved! Damir Kojasevic  - Sutjeska Niksic -  fails to capitalise on this great opportunity,  shot with right foot saved  in the centre of the goal."
"Penalty saved! Stefan Panic  - Riga FC -  fails to capitalise on this great opportunity,  shot with right foot saved  in the centre of the goal."
"Penalty saved! Georgie Kelly  - Dundalk -  fails to capitalise on this great opportunity,  shot with right foot saved  in the centre of the goal."
"Penalty missed! Still  FC København 1, Crvena Zvezda 1. Marko Marin  - Crvena Zvezda -  hits the bar with a shot with right foot."

As you see, they are not really robotic, and after ending up writing 1500 lines of php code (with regex) and still being inconsistent, I decided to see my alternatives with machine learning.

What I am trying to achieve is:

For example this one:

"Penalty saved! Stefan Panic  - Riga FC -  fails to capitalise on this great opportunity,  shot with right foot saved  in the centre of the goal."

type => penalty
action => saved
reason => shot with right foot saved  in the centre of the goal
person => Stefan Panic

I stumbled upon spaCy and saw "Named Entity Recognition" and thought maybe I can use it for this purpose. Especially as I have huge training data.

I wanted to ask: Is spaCy's Named Entity Recognition is right for this task? If not, what should I try to learn for this task?

P.S: I know a little about python but nothing about ML

Named Entity Recognition (NER) would extract names of people, organizations and such. Example:

"Penalty missed! Bad penalty by <person>Felipe Brisola</person>  - <organization>Riga FC</organization> -  shot with right foot is very close to the goal. <person>Felipe Brisola</person> should be disappointed."

So it could be helpful for the "person" field, but probably not for the rest. Note that you could also train a system similar to NER in order to predict other fields, but it would require a good amount of annotated data and it's not sure to work well.


Using extracted feature vector to perform zero shot detection I've developed a deep learning model trained from scratch on fruits and vegetables. However, as the data is limited, I can only cover a few different types of fruits and vegetables with the model alone. However, the accuracy on those categories is very high, 90% training and validation accuracy with around 80% testing accuracy (from a different set of data).
Now I tried to use this trained model on unseen data. So I've removed the softmax layer and only kept the other layers of the model. The idea is to use the feature vector and cosine similarity to compare vegetables.
Let's say bananas were in my original dataset but apples and oranges were not. I use the model to extract features of apples and oranges off of google, then compare the vectors of apples with origins. I would expect the cosine similarity to be very low as they are very different vegetables. However, I get a cosine similarity of 0.9, which is very confusing. To test my sanity, I used a pretrained Mobilenet v2 model and also removed the last layer. Then used the same method to extract the features of apples and oranges. Now they only have a similarity of around 0.5, which is much more reasonable.
I have a few ideas as why this could be the case, perhaps the model could not generalize past the categories given to it during training, or it did not learn similar features that could be translated to apples and oranges so it could not differentiate between the two. However I have no certain explanation. Does anyone have any idea why this is happening, and/or how I can prevent this from happening?

The differentiation of shape starts in the initial layers. Colour information is exploited in further layers. Your network has less layers, try increasing your layers (keep filters low in the initial layers).
Also, if you are looking at similarity models, look at contrastive training or ArcFace loss or siamese type models. If you want to stick with classification networks, look at GeM pooling.
Best way would be to use MobileNet (or better networks such as EffNetB4) to finetune with GeM pooling.


How to adapt the softmax layer for multiple labels? Question: A image has multiple labels. Given a set of image with labels, how to adapt the softmax layers?

My idea:

Encode multiple labels to 0-1 variables. Use logistics regression as the output layer.
Choose the top X labels in softmax output. But I do not know

how to determine the threshold.
how to modify the code in a pretrained net. Let's say

{
      "op": "null",
      "param": {},
      "name": "softmax_label",
      "inputs": [],
      "backward_source_id": -1
    },
    {
      "op": "Softmax",
      "param": {"grad_scale": "1"},
      "name": "softmax",
      "inputs": [[510, 0], [511, 0]],
      "backward_source_id": -1
    }

Any recommendations on tutorials are appreciated. I have some theory background from the book ESL.

I don't know what's going on in your code, but you seem to be close: to get multiple labels, simply replace the softmax output layer with a logistic layer (or something else that maps a real number to a probability), then optimize the cross entropy. That way you will have a probability associated with each label such that their sum across labels no longer need to add to unity.


Multiclass classification of timeseries data using NN I have the following dataset that I was thinking of using RNN/LSTM for classifying the protocol. Data contains features from packet capture that has only two provided fields: OUI of the MAC address of the device and the inter-arrival time (IAT) of the packets for a specific protocol.

For example, the labeled data for http and ntp from the point-of-view of a specific device type is provided below:

# MAC, ConnectivityIAT  -> protocol
ff:f1:f2, 10, -> http
ff:f1:f2, 20, -> http
ff:f1:f2, 30, -> http
ff:f1:f2, 0, -> http

ff:f1:f2, 3, -> ntp
ff:f1:f2, 6, -> ntp
ff:f1:f2, 9, -> ntp

The task is to predict the correct class when something like:

ff:ff1:f2, 10 -> ?

is seen.

Questions:

How do I convert the categorical value of OUI to numerical entities
How to approch the solution using a NN ?

How to approach the solution using a NN ?

Given your data, it doesn't look like an RNN/LSTM is the most sensible choice.

It looks like you have two features and a binary output. RNN/LSTM are useful with variable-length input, otherwise, they are a bit overkill.

Also, make sure that you have a reasonable amount of data before using a deep learning approach. They usually have many parameters to train, and if you don't have enough data points, this might be difficult to do.

If you want to stick to NN, I'd say a regular multilayer perceptron might do the job. If you are open to other options, I would try decision trees for this particular problem.

How do I convert the categorical value of OUI to numerical entities

You could use one-hot encoding. There are plenty of implementations out there (i.e. scikit-learn), or you could do it yourself.