LASSO remaining features for different penalisation I am using the sklearn LASSOCV function and I am changing the penalisation parameter in order to adjust the number of features killed off. For example for $\alpha = 0.01$ I have 55 features remaining and for $\alpha=0.5$ I have 6 remaining features.

I would expect that the 6 features I am getting for the second case are a subset of the 55 features I am getting in the first case. This is not what is happening however. 4 out of 6 features are not in the 55 of the first case.

Could someone explain to me the intuition behind why this is happening?

Thanks,
P

Lambda is a tuning parameter („how much regularisation“, I think called alpha in sklearn) and you would choose lambda so that you optimise fit (e.g. by MSE). You can do this by running cross validation.

This page (for the GLMnet package in R) explains how to apply Lasso in a very instructive way (alpha is the elastic-net mixing parameter here, Lambda is the tuning parameter).

You may also look at „ Introduction to Statistical Learning“, Ch. 6.2, in which Lasso is discussed in a very good way.

There are also Python Labs for the book, which should give you a blueprint for how to use Lasso in Python (see Section 6.6.2).


Why does vanilla transformer has fixed-length input? I know that in the math on which the transformer is based there is no restriction on the length of input. But I still can’t understand why we should fix it in the frameworks (PyTorch). Because of this problem Transformer-XL has been created.

Can you explain to me where this problem is hiding, please?

The restriction in the maximum length of the transformer input is due to the needed amount of memory to compute the self-attention over it.

The amount of memory needed by the self-attention in the Transformer is quadratic on the length of the input. This means that increasing the maximum length of the input, increases drastically the needed memory for self-attention. The maximum length is that which makes the model use up the whole memory of the GPU for at least one sentence (once the other elements of the model are also taken into account, like the embeddings which take a lot of memory).

Transformer-XL is certainly a way to take into account as much context as possible in language modeling (its role is analogous to truncated back-propagation through time in LSTM language models). However, the gradients are not propagated through the attention over the memory segment, only through the current segment.

There have been several architectural attempts to reduce the amount of memory needed by transformers, like using locality-constraints in the attention (Dynamic Convolutions model) or using locality-sensitive hashing (Reformer model).

There have been other implementation attempts, like gradient checkpointing(e.g. this), which is a general technique to run computations that don't fit at once in the GPU memory


I keep getting an error message in R while using twitteR here is what I am doing:
tweets=searchTwitter("walmart",n=3000, lang="en",since="2021-01-08",until="2021-01-10")

And this is my error message:
Warning message:
In doRppAPICall("search/tweets", n, params = params, retryOnRateLimit = retryOnRateLimit,  :
  3000 tweets were requested but the API can only return 0

Any help would be appreciated.

See this stackoverflow answer, it seems you can request tweets older than 7 days from the official Twitter API. See also the documentation for the searchTwitter function:

Note that there are pagination restrictions as well as other limits on
what can be searched, so it is always possible to not retrieve as many
tweets as was requested with the n argument.

Try changing the since and until dates to something within the last 7 days to see if that does work.


Python model persistence without pickling Are there any Python machine learning libraries, which allow model persistence without pickling?

More specifically, I want to avoid problems (and time sinks) that could arise upon sharing models with others. (In my case: academic collaborators in non-computational research fields.)

One option is to export the model to Predictive Model Markup Language (PMML) format.

In scikit-learn, sklearn2pmml can be used.


Recognising made up terms Say I have a tagging system on an electrical circuit:

Name          Description
-------       --------------
BT104         Battery. Power source
SW104         Circuit switch
LBLB-F104     Fluorescent light bulb
LBLB104       Light bulb
...           ...

I have a hundreds of tags created by people who should have followed my naming conventions but sometimes they make mistakes add unnecessary additional characters to tag names (i.e. BTwq104 etc.).

Up until now I used regular expressions, that I built over time whilst observing various inconsistencies that users introduce whilst naming different parts of their circuits, to parse the names and tell me what the different elements are. For example: name 'BT104' would tell me it's a battery on circuit 104.

I would like to investigate or use a machine learning technique to identify what a tag name is (same way I used regular expressions). Any suggestions and approaches are welcome.

So far I tried Named-entity recognition suggested technique "Bag of words". Followed a few tutorials here and here (latter being the most useful in learning). None of them produced wanted results if any. I think that "Bag of words" are mostly used for real word rather than made up words.

You could treat this as a spelling error identification problem. The "Name" column should be a set of unique keys. You can calculate the Levenshtein distance, which finds the minimum number of single-character edits (insertions, deletions or substitutions) required to change one word into the other, between each key. Then set a similarity threshold. Any two keys that are greater than the similarity threshold are merged together.

You can write your own based on Peter Norvig's Spelling Corrector or use Python's FuzzyWuzzy package. The code would be something like this:

from fuzzywuzzy import process

names  = set("BT104 SW104 LBLB-F104 LBLB104".split())
threshold = 85

for name in names:
    leave_current_out = names - {name}
    for match, score in process.extract(name, leave_current_out, limit=1):
        if score >= threshold:
            print(f'Potential misspelling: {name} with {match}')


In K-means, what happens if a centroid is never the closest to any point? I am implementing K-Means from scratch and that exercise raised a question.

To update my centroids, for each centroid, I have to find the points for which that centroid is the closest.

In some cases, especially when the number of centroids is high and the number of instances is low (i.e. k=20 and 100 instances), I find centroids for which no point has them as their closest centroid. In other words, they become "orphans" as no instances are allocated to them.

In the example above, the two centroids (X) at the top clearly both have points for which they are the closest centroid. But the centroid at the bottom is never the closest centroid for any instance.

How do I deal with this?

Should the lone centroid remain unmoved?
Should I move that centroid? If yes, how?
Should I remove it?

Is there a standard way to deal with this?

------------ EDIT: --------------

On Github, I found this:

https://github.com/klebenowm/cs540/blob/master/hw1/KMeans.java

This suggests that if a centroid becomes an "orphan", it should be assigned to the point that is the furthest from its centroid.

This seems like a sound method, is there any paper or theory supporting this?

You should leave the lone centroid unmoved. In the next iteration, it's possible that the cluster centers have moved in such a way that now there are some instances that are closest to the lone centroid, and it can get picked up. At the end of the k-means algorithm, you could remove clusters that have no instances associated with them, but it doesn't really matter.

At the end of the day, it's probably not a good idea to use k-means in the manner that you described, and we should not expect it necessarily to produce good results.


KNN Imputation utilize mean or mode? In my current project, I am doing KNN imputation with K = 5 and I am using sklearn.impute.KNNImputer. I have a mix of continuous and nominal variables(encoded as 0/1 or ordinal ones that have been encoded as 0/0.25/0.5/0.75/1 etc). However, the docs say "Each sample’s missing values are imputed using the mean value from n_neighbors nearest neighbors found in the training set." Because of this, I am getting in-between values like 0.4 for nominal attributes. Is there any way to override this to change from mean to mode for nominal columns?
Also, I looked at missingpy and fancyimpute but they both seem to be using mean as well~

By default scikit-learn's KNNImputer uses Euclidean distance metric for searching neighbors and mean for imputing values.
If you have a combination of continuous and nominal variables, you should pass in a different distance metric.
If you want to use another imputation function than mean, you'll have to implement that yourself.


Whats an explanation of PCA that is intuitive for someone in senior leadership who doesn't have a technical background? I've been pulled onto my first Data Science project at work. Classic problem of predicting sales based on web traffic data, etc. While I don't know about the specific techniques I will be using in my model, I think that PCA will probably be tried at some point, and if I use it successfully as a preprocessing step before I chug the data into some estimator, I'm thinking about how I might go about defending/explaining my work.

My question is, what is an intuitive explanation for PCA that doesn't rely heavily on the linear algebra that makes up the process? Perhaps an analogy that draws a parallel to something that any layman can readily understand?

Try to make your explanation simple and brief. Maybe something like "PCA distils a dataset's features into as few attributes as possible; these attributes explain as much of the observed trend as possible while being as different from each other as possible". If you can do this using visual aids, even better.

As far as analogies go, maybe use an example of a motor accident. There are many predictors of an accident, but PCA may combine the majority of them to form 2 features - perhaps vehicle attributes and driver attributes. These are different but arguably are both highly predictive of a motor accident. [I realise this is not technically how PCA works, but might help a layman understand].


breaking joined words into meaningful ones during text mining I'm performing an aspect-based sentiment on consumer complaints. I'm tokenizing at the sentence level.

tidy_complaints <- tidy_complaints %>%
  unnest_tokens(
    output = sentence,
    input = consumer_complaint_narrative,
    token = 'sentences'
  ) %>%
  mutate(sentence_id = 1:n())

However, some of the the complaints contains words that are joined together such as "trailcertified" or "creditorcompany".

Is there any need to break this words up into meaningful words? if yes, how do I accomplish this and still keep the sentence intact?

The compound split is not trivial, but there are solutions that kind of work (as you can read on the link @Aditya shared).

Another way of dealing with the matter would be to tokenize the strings into character n-grams.


Question on bootstrap sampling I have a corpus of manually annotated (aka "gold standard) documents and a collection of NLP systems annotations on the text from the corpus. I want to do a bootstrap sampling of the system and gold standard to approximate a mean and standard error for various measures so that I can do a series of hypotheses tests using possibly ANOVA.
The issue is how do I do the sampling. I have 40 documents in the corpus with ~44K manual annotation in the gold standard. I was thinking of using each document as a sampling unit, and taking 60% of documents for each sample (or 24 documents per sample). However, the issue is that each manually annotated documents does not have the same number of annotations, so that violates using same sample size for each sample.
Any suggestions on how to achieve this bootstrap?

It simply depends what you count as your object of interest: from your description the unit can be either document or annotation. Your method describes using the document as unit, it's fine as long as the tests you plan to do are compatible with this.
Another option is to use the annotation as unit: in this case you would pick 60% of the 44k annotations every time, so you would have a mix of annotations from multiple documents. Depending what you test exactly, this might be an issue, in particular I don't see how you would count False Negative cases in this way.
Since you have text documents of varying size (I assume), you could also consider different options: sentence, paragraph, block of N sentences, etc.


removing words based on a predefined vector I have the dataset test_stopword and I want to remove some words from the dataset based on a vector. How can I do that in R?

texts <- c("This is the first document.",
       "Is this a text?",
       "This is the second file.",
       "This is the third text.",
       "File is not this.")

test_stopword <- as.data.frame(texts)
ordinal_stopwords  <- c("first","primary","second","secondary","third")

texts <- c("This is the first document.",
       "Is this a text?",
       "This is the second file.",
       "This is the third text.",
       "File is not this.")

test_stopword <- as.data.frame(texts)
ordinal_stopwords  <- c("first","primary","second","secondary","third")

(newdata <- as.data.frame(sapply(texts, function(x)   gsub(paste(ordinal_stopwords, collapse = '|'), '', x))))

The output is getting skewed when added in a code block (maybe a bug in SE). But, you would get the desired output.


Tensor Flow Time Series Tutorial Question In the tutorial, they normalize the data and say "The mean and standard deviation should only be computed using the training data"

What does this refer to? Why should you only use the training data?

When building any Machine Learning model, the only observable data you have is training data. Test data is supposed to be unobserved data, meaning that even though you might have it now, you need to act as if you didn't. When you apply normalisation, you first observe the data to get the parameters you need. As you are only supposed to be able to observe the training data, you can't use the test data to calculate those values. Doing so would be like cheating, as you are accomodating your parameters to new unobserved data (how can you observe unobserved data?).

Imagine you build a model today and you want to make predictions tomorrow. You can't use tomorrow data to build your model since you don't have it yet. You are not supposed to know tomorrow's mean and std, though your hope is that they will be similar enough. That is why when you normalose/standarise you get the parameters with the training data and then use them to transform both train and test data, so you can use them as inputs for your model.


How to deal with a sparse matrix when using a perceptron based recommender system? I'm constrained to use a perceptron based method. I have a user-item matrix filled with rating data on scale of 1 to 5 like this, with around 50% of the matrix with no data:

r<- matrix(c(2,4, NA,5,NA,3, NA,5,NA,1,NA,3,NA,5,NA,4,4,NA,NA,NA,1,1,2,NA,1,1,1,1,NA,NA,NA,NA,2,3,4,2,NA,NA,NA,NA,3,4,5,1,NA,NA,2,3,NA), nrow=7).
#one row represents one user, hile one column represents one item
r
    [,1] [,2] [,3] [,4] [,5] [,6] [,7]
[1,]    2    5   NA    1   NA    2    5
[2,]    4   NA    4    2   NA   NA    1
[3,]   NA    1    4   NA   NA   NA   NA
[4,]    5   NA   NA    1   NA   NA   NA
[5,]   NA    3   NA    1    2   NA    2
[6,]    3   NA   NA    1    3    3    3
[7,]   NA    5    1    1    4    4   NA

I'm recoding the NA's to 0's, which makes the perceptron give out only 2 classes (one containing all the 1's and 0's , and the other containing all the 2,3,4 and 5. Which is understandable I guess.) How do I deal with this? I tried mean imputation but the results are not good (20% accuracy)

(I can't give exact code due to this being a proprietary code of my company, but any perceptron based method should work for this example)

Recoding the NAs to 0's is not what you want. Think of what this means: a missing value on an item for a particular user means that said user hasn't seen or listened to or bought (depending on what kind of data you have) that item, while a zero value means that the user disliked that item; so they have completely different meanings.

The goal of a recommender system is basically to fill those NA values in a way that "makes sense". It is very common to have sparse matrices in recommender systems. I don't know exactly what kind of approach you're using or why you're using a perceptron, but I would recommend you to check how matrix factorization for recommender systems works ("Matrix factorization for recommender systems" by Robert Bell, Chris Volinsky and Yehuda Koren is a good paper) and see if this approach works for your problem.

You can also try first a simple baseline model that takes into account the biases of the users and the biases of the items; Bell, Volinsky and Koren's paper is a good start.


Right Package for Federated Learning Could Someone list the pros and cons with respect to using federated learning with the following packages:

TensorFlow federated
PySyft

Are there certain tasks which are specific to either or is one clearly better than the other?
Is there any other module which is better than these? If so could you please link them below.

After doing some reasearch, I found that OpenMined (a company based on FL and Private AI) Are making strides to develop PySyft. Hence I would suggest continuing with PySyft.


What does an Input layer of shape=(None,) or (None,12) actually mean? Is this telling the model that there are two dimensions (i.e. it’s a matrix) but we don’t yet know the size of that particular dimension? If so, how can the model be compiled? Doesn’t the size of each dimension affect the number of nodes in middle layers?

In tf.keras, a None dimension means that it can be any scalar number, so that you use this model to infer on an arbitrarily long input. This dimension does not affect the size of the network, it just denotes that you are free to select the length (number of samples) of your input during testing.


Decomposing R squared or VIF In the context of multi-regression, I am wondering if there is a way to decompose $$VIF_i = 1/(1-R_i^2)$$ where $R_i^2$ is the r squared obtained from the regression of dependent variable = i and independent variables are all other factors.

I want to decompose $VIF_i$ or $R_i^2$ into individual factors to see how much each individual factor contributes to the $VIF_i$ or $R_i^2$

Someone recommended using the square of partial correlation coefficient and that value is linearly related to $R_i^2$. My undestanding is that partial correlation coefficient measures the correlation between two variables, holding the other variables constant.  Is this a viable option?

I suggest calculating R-squared and VIF for each permutation of variable combinations.

Also consider the fact that variable interactions may influence your R-squared and VIF values. Depending on what programming language you are using there are many ways to build interactions into your regression model, and you can iteratively test for effects from there.


Identify outliers for annotation in text data I read the book "Human-in-the-Loop Machine Learning" by Robert (Munro) Monarch about Active Learning. I don't understand the following approach to get a diverse set of items for humans to label:

Take each item in the unlabeled data and count the average number of word matches it has with items already in the training data
Rank the items by their average match
Sample the item with the lowest average number of matches
Add that item to the ‘labeled’ data and repeat 1-3 until we have sampled enough for one iteration of human review

It's not clear how to calculate the average number of word matches.

The idea is to find the documents which are not well represented in the current labeled data. The first point is indeed a bit vague and can probably be interpreted in different ways. My interpretation would be something like this:

For every document $d_u$ in the unlabeled data, count the number of words in common with every document $d_l$ in the labeled data. This value is the "match score" between $d_u$ and $d_l$.

Note: I think that this value should be normalized, for example using the overlap coefficient. Note that other similarity measures could be used as well, for instance cosine-TFIDF.

As output from the above step, for a single document $d_u$ one obtains a "match score" for every labeled document. The average across the labeled documents gives the "average match" for $d_u$.


Can i use other regression types that arent based in decision trees to use it like a weak learners in gradient boosting? I was thinking if i can use polynomial regression like a weak learners in gradient boosting but i read that decision trees are used for that and i cannot find anything that show me the possibility of other weak learners could be used.

The issue of using any linear model (a polynomial regression is a particular case of a linear model, with polynomial features), is that an ensemble of linear models is still a linear model. So, the family of models to optimize from given by the boosted polynomial regression and the single polynomial regression are the same.
This doesn't happen with trees, as an ensemble of trees cannot be expressed as a tree anymore, so that's why trees are used more in the context of gbms.


How is that possible that a reward function depends both on the next state and an action from current state? There's the concept of "expected value of the next reward", often denoted as $\mathcal{R}^a_{ss'}$, and defined as

$$
\mathcal{R}^a_{ss'} = \mathbb{E}\left(r_{t+1}  \mid s_t = s, a_t = a, s_{t+1} = s' \right)
$$

which is the value we expect for the reward at the next time step, that is at time step $t+1$, given that action $a$ from state $s$ brings us to state $s'$.

But the italicized definition does not seem to be "consistent" with the Bellman equation for the value function, which is

\begin{align}
V^\pi(s) &= \sum_{a \in \mathcal{A}(s)} \pi(s, a) \sum_{s' \in \mathcal{S^+}} \mathcal{P}^a_{ss'} \left( \mathcal{R}_{ss'}^a + \gamma V^\pi(s') \right)
\end{align}

Why do I say this?

$\mathcal{R}_{ss'}^a$ depends on $s$, $a$ and $s'$, but in the Bellman equation above, specifically in the inner sum $\sum_{s' \in \mathcal{S^+}} \mathcal{P}^a_{ss'} \left( \mathcal{R}_{ss'}^a + \gamma V^\pi(s') \right)$, we are essentially iterating over all possible next states, $s'$, of $s$. But, if that's the case, then what's the purpose of making $\mathcal{R}_{ss'}^a$ depend on the action $a$? That is, if we already have the next state $s'$, then the action is useless, given that we are already taking into consideration the possible next states, indeed. In other words, $a$ and $s'$ may not be compatible.

For example, suppose we are in a certain state $s$, and we take action $a$ from there, and we end up in a state, say, $x$, which must be different to all next states $s'$ (in the inner summation), apart at most from one.

I hope you can see my point and my doubts, and I hope you can clarify them.

Furthermore, given this doubt, I am unable to implement a reward function. Also, I have seen an implementation of the reward function, that is

def reward(state, next_state, action):
    if GRID[state] != GOAL and GRID[next_state] == GOAL:
        return 10.0
    else:
        return 0

which actually does not use the action, but I don't understand why. Why should we use next_state and not the next state of state according to action (or vice-versa)? How could (and why would) we use both next_state and action, if they may be "incompatible" in the sense I described above?

The equations you have given are consistent with each other. The outer sum over $a$ provides the value of $a$. The "link" between $s, a$ and $s'$ that gives the correct associations is the transition function $\mathcal{P}^a_{ss'}$ which will be zero for any state/action/next-state triplet that does not make sense for the MDP. By "not make sense", I mean not allowed by the dynamics of the problem - such as an action that would successfully move too many spaces or through a wall for a maze problem.

It is normal for the reward to not actually depend on $a$, but only on the successor state. This might happen in e.g. a dynamic or adversarial environment where the desired goal is to achieve a specific end state. However, it is also possible for the reward to depend on action taken in a given state, such as activating some object in a particular location. It depends on how you have set up the MDP. Another example of where $a$ might be important is if there is a cost associated with each action that varies, for example an energy requirement, so that the total reward might be separate contribution from $s'$ (for reaching a goal) and $a$ (for spending energy to reach any next state).

Using an expected reward $\mathcal{R}_{ss'}^a$ that depends on all three terms $s, a, s'$ gives you a generic version of the Bellman equation that can be applied to the widest range of MDP designs. It doesn't matter if, in a specific case, the only thing you care about is $s'$ which is what the example reward function shows. Once you decide that the expected reward is dependent on $s'$, then the Bellman equation has to have that expected reward term inside the inner sum (the only place where $s'$ is defined), and there is no additional structure in the equation needed to have this term dependent on $s$ and $a$ as well.

You could legitimately use a variant $\mathcal{R}_{s'}$ in a Bellman equation that is otherwise identical to the equation you give in the question, to describe the value function for a MDP where reward only depends on the state transitioned to. However, you will not find that in the literature because it is less generic.

One common variant that you will see is $\mathcal{R}_{s}^{a}$ i.e. the expected reward for taking action $a$ in state $s$. You might use this for cost-based systems, where the goal is to complete a task efficiently and taking actions expends some resource such as time. Using your notation, the Bellman equation for that design looks like this:

\begin{align}
V^\pi(s) &= \sum_{a \in \mathcal{A}(s)} \pi(s, a)( \mathcal{R}_{s}^a + \gamma \sum_{s' \in \mathcal{S^+}} \mathcal{P}^a_{ss'} V^\pi(s'))
\end{align}


Are the image data augmentation generators in Keras randomly applied I am working on an image classification problem and using data augmentation in Keras.

from keras.preprocessing.image import ImageDataGenerator

datagen = ImageDataGenerator(
    rotation_range=2,
       horizontal_flip=True)

I would like to know if the ImageDataGeneratorapplies the transformation randomly to image patches. That is for example, rotation may be applied on one image, whilst flip may be applied to another image. I want to know if the decision to apply rotation or a flip is randomly determined.

I suggest having a look at the relevant documentation.
There, it states:

rotation_range: Int. Degree range for random rotations.
...
horizontal_flip: Boolean. Randomly flip inputs horizontally.

Saying each of the operations is applied randomly, I would say, means your images will be generated sometime with and sometimes without the augmentation steps, independently from one another.
If that doesn't convince you, here is the relevant snippet from the source code:
if self.horizontal_flip:
    if np.random.random() < 0.5:
        x = flip_axis(x, img_col_axis)

The code for the rotation step is a little more involved, but contained within the same class that I liked above.


Change distribution of a vector I have the following vector
my_vector=[0.059; 0.223;0.197;0.176; 0.173; 0.171; 0.0421; 0.209; 0.556;0.252;0.198; 0.255; 0.130; 0.176; 0.110; 0.0845; 0.270; 0.192; 0.199; 0.348]

I want to change the vector in such a way that the output derived vector will have the highest possible distribution change. This means the distribution of the two vectors should be highly different (say have large KLD values) but their distribution should be still from the same family (say the F-test or two-sample K–S test will not reject the hypothesis that they are from the same distribution).

How could I generate the new vector from the base vector while I meet the constraint (having "highest" "possible" change in distribution)? I need a machine learning (data-driven) method to achieve such a goal.

Thanks in advance for any help and recommendation.

I think the answer(comment) of TwinPenguins provides you a good start. You could improve this by using a Metropolis Monte Carlo approach instead of just trying again and again generating new vectors.
What you could do is:

create a new vector with length similar to yours randomly sampled from the distribution of your interest and compute KLD$_{old}$ (cf. TwinPenguins)
modify this vector and compute KLD$_{new}$.
if (KLD$_{new}$ is better than KLD$_{old}$), the new vector becomes your old vector
else discard the changes
go back to 2, unless you reach a given target value of KLD

(Note that there may be issues with getting trapped in local minima, but search the Monte-Carlo literature and  you'll find a solution...you can always run multiple times, and then take the best result.)


Customer Intelligence - How to measure success? we are creating models that aim to filter new leads from our current customer base.

We started to create propensity models that calculate a percentage for each customer for a certain product group. I was wondering what are best practices to benchmark such a model and to show the success of a propensity model in advance.

Thx for your replies!

I'm assuming that the "propensity model" predicts a customer's likelihood to take some desired action.

If it's possible at your organization, then a good way to measure the model's effectiveness is an A/B test.  Select a set of leads to act as a control group using a baseline selection method.  Ideally, the baseline method would be whatever is currently in-place at your organization.  Then allow your model to select a set of leads as the experimental group.  Compare conversion rates (or whatever else you care about) between the control group and the experimental group.

If A/B testing isn't viable, then you can usually evaluate your model using the traditional techniques.  You can split your customer base into train/test sets.  Train your model on the training set, then made propensity predictions for the test set.  Compare the propensity estimates with actual conversion rates.


XGBoost Fit vs Train I am trying to do a grid searching using the methodology that mentioned in this post. However, I found that XGBClassifier().fit() is using much more memory than xgboost.train. Does anyone know why? Is this related to sparse matrix?

XGBRegressor vs. xgboost.train huge speed difference? - see this answer, likely the defaults (num_boost_round=10 vs. n_estimators=100) are the cause of this.


Can BERT be used for predicting words? I have a question regarding the pre-training section (in particular, the Masked Language Model).

In the example Let's stick to improvisation in this skit, by masking the word improvisation, after applying BERT (followed by the FFNN and Softmax), by looking at the probabilities of each of the possible English words,  we are able to correctly predict that the masked word was improvisation.

Is it possible to actually play with this by using my own examples? I was wondering if I can input a sentence in a different language (from the multilingual model) and have a sorted list of the most probable words that were masked in the original sentence. If it's possible, what needs to be tweaked?

Any help would be greatly appreciated.

pip install transformers

Then try this

from transformers import pipeline
nlp = pipeline("fill-mask", model="bert-base")
nlp(f"This is the best thing I've {nlp.tokenizer.mask_token} in my life.")


If deep-learning learns features, aren't we saying it can learn association rules? If deep-learning learns features, aren't we saying it can learn association rules ?

Finding valuable association rules is defined base on their support and confidence. ANN is not proper for these kinds of specifications.
Hence, if we use ANN for learning association rules, we can't guarantee the specified support and confidence are satisfied.

Anyhow, there are other machine learning methods to learn association rules such as decision tree. Although, that methods not such that proper to find rules with specified support and confidence. You can know more about this in this article.


Probabilistic gold standard vs Deterministic gold standard I understand that we say something as a gold standard when it involves human intervention/judgement/review.

But can someone help me understand what's the difference between probabilistic gold standard and deterministic gold standard.

For ex: Patient has cancer or not - binary response - Deterministic gold standard which can be provided by humans.

Whereas Patient has 60% chance of being a cancer and 40% chance of not being a cancer. Am I right to understand that this is called as probabilistic gold standard but this can't be produced by Humans right?

Can any human/doctor for example, say this patient has 60% chance of being a cancer and 40 % chance of not  being a cancer?

No, deterministic probability is when you know for certain. If a person does not have a diagnosis, then he doesn't have the disease/condition. Doctors are not supposed to give a probability but we as human beings always like to know the likelihood.

For example, person A who is 27 years old who has the coronavirus is highly unlikely to die of the virus but it's not impossible.


Predicting unknown data using Knn I am self learning data analysis using (mostly) R. I'm currently on K-nearest neighbour topic, and I went through the process of normalizing- splitting the dataset to train/test and fitting the model and evaluating it, and it all went well. But, there is one fundamental thing I just don't understand: what is the point in classifying a known data? How do I predict the value of NEW data? The outcome of knn() is a factor with labels, but after using it for evaluating my model (namely, the value of K I have chosen), what should come next? For example, in regression model I can use predict() to predict new data, but I do I predict (as opposed to just classify) using knn?
Thank!

when you "predict" something in KNN-classify problems, you are classifying new information.

yah, KNN can be used for regression, but let's ignore that for now.

The root of your question is why bother handling known data, and how can we predict new data.  Let's do KNN in R1, with two training examples.  The first one will be 0 and it will be class A, the next one will be 100 and it will be class B.

So, KNN is what's known as a lazy classifier. You actually aren't training, any hyperparameters, just loading the training data.  I've loaded two points, and now I want to classify a new point.  Let's consider the point 0.  Since we already have the point 0 labelled as class A, we know it's going to be class A.  What about the point 4?  It's closer to 0 than to 100, so it's class A.
Similarly, the point 95 is closer to 100, so it's class B.

KNN is really just doing those comparisons with more points and/or more dimensions.

EDIT: user asked the following: how would I predict if it is yes or no, based on the classification I have done before?

The most basic implementations of KNN check the distance from your new case to all of the points in the training data.  If you want to visualize it, it's possible with one, two, and maybe 3 dimensional data.

In python, you can do as follow (from scikit-learn.org):

X1 = np.random.normal(0,1,100) #values centered around 0
X2 = np.random.normal(6,.1,100) #values centered around 6
X = np.concatenate([X1,X2])
X = np.array(X).reshape(-1,1)
Y = [0 for y in range(100)]+[1 for y in range(100)] #2 classes
Y = np.array(Y).reshape(-1,1)

clf = neighbors.KNeighborsClassifier()
clf.fit(X, Y)

now, you can predict by calling predict

clf.predict(1)
clf.predict(4)
clf.predict(.1)

KNN visualized in 2D:

from: https://www.analyticsvidhya.com/blog/2014/10/introduction-k-neighbours-algorithm-clustering/


Anomaly score calculation for multidimensional data set I'm currently using anomaly score calculation methods which work on single dimensional data sets (Timestamp & Value). I would like to calculate the same single anomaly score for multidimensional data set (Timestamp & Value 1 & Value 2 & ..... & Value N).

What method could compare all the values and calculate anomaly score based upon the relationship between them?

One very simple yet efficient way, is auto-regression, which means, you train a regressor on past data, and predict the future, if the prediction is off by too much - then you call it an anomaly.

to make it more formal, if we denote $\vec{v}_{t0}$  as your data at timestamp $t_0$, we train a regression model $R$ that learns from $\{v_{t_0},\dots,v_{t_{n-1}}\}$.

Your anomaly score is a function of $R({v_{t_0},\dots,v_{t_{n-1}}})-v_{t_n}$


Automating the training and deployment of the model as more data becomes available My question is about automating the training of the model as more data becomes available.
In this scenario, I have 1MM items that I split into training and test datasets to train, validate and ultimately deploy the model.
As more data becomes available on a daily basis, is it common in real-life projects or even advisable to automate the process above, as to leverage the new data and potentially make the model more accurate?

This is a big topic with a lot of possible solutions depending on your context, I'll just provide some of my personal experience from projects I've built.
I would say that it is common in industry to retrain models in an ongoing fashion as new data comes in. Many times this happens at the daily level using either a basic cron-like solution or an enterprise orchestration solution (e.g. Airflow/Kubeflow). Cron is easy to set up and manage for small/personal projects, the others require more engineering set up to manage but they also are more powerful/fully featured.
Once you have a model pipeline which trains in an ongoing basis, you would then need to decide whether to promote the new model to production or keep your existing model running. This could be done automatically given certain conditions (e.g. accuracy of new model > old model) or could require a manual step to evaluate the new model and promote after review. Many times the answer comes down to data quality/stability. For instance, if the input data has intermittent quality issues, you would be less likely to retrain and promote automatically without manual oversight.
Potentially helpful links:
Cron for ML: https://towardsdatascience.com/why-crontab-is-the-ultimate-data-science-back-end-tool-e3f212f2b13d
Airflow for ML: https://towardsdatascience.com/10-minutes-to-building-a-machine-learning-pipeline-with-apache-airflow-53cd09268977
Kubeflow Pipelines: https://www.kubeflow.org/docs/pipelines/overview/pipelines-overview/


find the values of theta in a cost function from andrew ng course I was following Andrew NG ML course on coursera. I was stuck at cost function.

How can I find values for Theta-0 and Theta-1 ?

So I came to know Theta-0 as y intercept so the values must be Theta-0 = 0.5 but how can I find the value for Theta-1?

Any help is appreciated.

Elaborating on the previous answer, now hat you have $\theta_0 = 0.5$, take any value of $x$ and the corresponding $h_\theta$ value to find $\theta_1$. For instance, using $x=1$ will yield:

$h_\theta(1) = \theta_0 + \theta_1 \times 1$

From the graphd, $h_\theta(1) = 1.5$ so that:

$\theta_0 + \theta_1 = 1.5$

$\theta_1 = 1.5 - \theta_0$

$\theta_1 = 1.5 - 0.5 = 1$

The final equation is then $h_\theta = 0.5 + x$

As said in the comment, you can also get the slope visually, noticing that if you increase $x$ by 1, $y$ also increases by 1 when you follow the slope.

Does this help?


Purpose of backpropagation in neural networks I've just finished conceptually studying linear and logistic regression functions and their optimization as preparation for neural networks.

For example, say we are performing binary classification with logistic regression, let's define variables:
$x$ - vector containing all inputs.
$y$ - vector containing all outputs.
$w_{0}$ - bias weight variable.
$W=(w_1,...,w_{2})$ - vector containing all weight variables.
$f(x_i)=w_{0}+\sum_{i=1}x_{i}w_{i}=w_{0}+x^{T}W$ - summation of all weight variables.
$p(x_{i})=\frac{1}{1+e^{-f(x_i)}}$ - logistic activation function (sigmoid), representing conditional probability that $y_i$ will be 1 given observed values in $x_i$.
$L=-\frac{1}{N} \sum^{N}_{i=0} y_i*ln(p(x_i))+(1-y_i)*ln(1-p(x_i))$ - binary cross entropy loss function (Kullback-Leibler divergence of Bernoulli random variables plus entropy of activation function representing probability)
$L$ is multi-dimensional function, so it must be differentiated with partial derivative, being:
$$\frac{\partial{L}}{\partial{w}}$$
Then, the chain rule gives:
$$\frac{\partial{L}}{\partial{w_1}}=\frac{\partial{L}}{\partial{p_i}} \frac{\partial{p_i}}{\partial{w_1}}$$
After doing few calculations, derivative of the loss function is:
$$(y_i-p_i)*x_i$$

So we got derivative of the loss function, and all weights are trained separately with gradient descent.
What does backpropagation have to do with this? To be more precise, what's the point of automatic differentiation when we could simply plug in variables and calculate gradient on every step, correct?
In short
We already have derivative calculated, so what's the point of calculating them on every step when we can just plug in the variables?
Is backpropagation just fancy term for weights being optimized on every iteration?

Is backpropagation just fancy term for weights being optimized on every iteration?

Almost. Backpropagation is a fancy term for using the chain rule.

It becomes more useful to think of it as a separate thing when you have multiple layers, as unlike your example where you apply the chain rule once, you do need to apply it multiple times, and it is most convenient to apply it layer-by-layer in reverse order to the feed forward steps.

For instance, if you have two layers, $l$ and $l-1$ with weight matrix $W^{(l)}$ linking them, non-activated sum for a neuron in each layer $z_i^{(l)}$ and activation function $f()$, then you can link the gradients at the sums (often called logits as they may be passed to  logistic activation function) between layers with a general equation:

$$ \frac{\partial L}{\partial z^{(l-1)}_j} = f'(z^{(l-1)}_j) \sum_{i=1}^{N^{(l)}} W_{ij}^{(l)} \frac{\partial L}{\partial z^{(l)}_i}$$

This is just two steps of the chain rule applied to generic equations of the feed-forward network. It does not provide the gradients of the weights, which is what you eventually need - there is a separate step for that - but it does link together layers, and is a necessary step to eventually obtain the weights. This equation can be turned into an algorithm that progressively works back through layers - that is back propagation.

  To be more precise, what's the point of automatic differentiation when we could simply plug in variables and calculate gradient on every step, correct?

That is exactly what automatic differentiation is doing. Essentially "automatic differentiation" = "the chain rule", applied to function labels in a directed graph of functions.


Are there algorithms for clustering objects with pairwise distances, without computing all pairwise distances? I'm looking for a clustering algorithm that clusters objects, by using their pairwise distances, without needing to calculate all pairwise distances.

Normally pairwise clustering is done like this: (see here)

Compute full distance matrix between all pairwise combination of objects
Assuming that the distances there are non-euclidean, one might use Spectral Clustering or Affinity propagation on the distance matrix and retrieve the clustering results.

Here comes the however:

Computing the full distance matrix for all pairwise combination of objects is computationally very expensive. So my though was, whether there are some clustering algorithms that only do lookups on a subset of the pairwise distances, so it is not necessary to compute the full matrix?

I know Spectral Clustering works also on sparse matrices, but since it is theoretically possible to compute all pairwise distances, which ones should be left out?

Exited to hear your ideas, thanks!

Well, one may argue that DBSCAN is based on all pairwise distances, but it uses data indexing to avoid computing all of them using geometric bounds.

And there are other examples if you browse through literature.

For example, the classic CLARA method is an approximation to PAM that avoids computing all pairwise distances.

And there are many more such techniques.


Poor performance for unbalanced dataset Consider a dataset A which has examples for training in a binary classification problem. I have used SVM and applied the weighted method (in MATLAB) since the dataset is highly imbalanced. I have applied weights as inversely proportional to the frequency of data in each class. This is done on training. I have used 10 folds cross-validation for training. After training, I get the confusion matrix on A:

80025 1
0 140

where the first row is for the majority class and the second row is for the minority class. There is only 1 false positive (FP) and all minority class examples have been correctly classified giving true positive (TP) = 140.

PROBLEM: I train again using more data points. Then, I  run the trained model on the a new unseen test data set B which was never seen during training. This is the confusion matrix for testing on B .

50075 0
100 0

As can be seen, the minority class has not been classified at all, hence the purpose of weights has failed. Although, there is no FP the SVM fails to capture the minority class examples.
I have not applied any weights or balancing method on B. What could be wrong and how to overcome this problem?

Try Resampling Your Dataset

One option is to reduce de occurance of the majority class in your training set.

One other option is to over-sampling the minority class. (you may need to ad a bit of noise )

Other idea:
Try Changing Your Performance Metric

The page helped me a lot with unbalance data:
Combat Imbalanced Classes in Your Machine Learning Dataset


I don't know the framework but it seems that's what happens:

Since this is multi-label classification, it's possible that an instance is assigned no label at all by the model. Apparently at least one of the samples contains only instances which are predicted with no label. This means that there are no instances predicted positive for any label in this sample. This causes an NaN value for the precision of this particular sample, which in turn makes the average precision over all samples undefined (hence the error).

I assume that this is not related to the fact that all the classes have a positive precision and recall, since these values would be calculated over the whole dataset, not sample by sample.


File converter: from CSV to HDF5 Can anyone recommend any command line tool for converting large CSV file into HDF5 format?

1st approach: Use append=True in the call to to_hdf:

import numpy as np
import pandas as pd

#filename = '/tmp/test.hdf5'
filename = 'D:\test.hdf5'

df = pd.DataFrame(np.arange(10).reshape((4,2)), columns=['C1', 'C2'])
print(df)
#    C1  C2
# 0  0   1
# 1  2   3
# 2  4   5
# 3  6   7

# Save to HDF5
df.to_hdf(filename, 'data', mode='w', format='table')
del df    # allow df to be garbage collected

# Append more data
df2 = pd.DataFrame(np.arange(10).reshape((4,2))*10, columns=['C1', 'C2'])
df2.to_hdf(filename, 'data', append=True)

print(pd.read_hdf(filename, 'data'))

2nd approach:  you could append to a HDFStore instead of calling df.to_hdf:

import numpy as np
import pandas as pd

#filename = '/tmp/test.hdf5'
filename = 'D:\test.hdf5'
store = pd.HDFStore(filename)

for i in range(2):
    df = pd.DataFrame(np.arange(10).reshape((4,2)) * 10**i, columns=['C1', 'C2'])
    store.append('data', df)

store.close()

store = pd.HDFStore(filename)
data = store['data']
print(data)
store.close()

3rd approach:  using chunksize parameter and append each chunk to the HDF file which  was answered here.

Personally I like 1st and 2nd approach.


What is this type of plot and how to interpret it? I was presented to this plot and i've seen it before, however i would like to know what does the countour around the main line mean.

Ps: i couldn't find enough information about it to investigate by myself, so i had to create this question.

The shaded area likely shows the dark green line plus or minus some error/uncertainty estimate. Common error estimates may be based on the standard deviation, a confidence interval, or the interquartile range depending on the data and the analysis being done. Without more information, we cannot know what the shaded area represents.


Units of mean decrease accuracy on a variable importance plot obtained from a Random Forests classifier? To know the importance variable in a Random Forest I used The mean decrease accuracy and mean decrease Gini.
I would like to understand what are the x-axis units of the mean decrease accuracy and mean decrease Gini on a variable importance plot obtained from a random forests classifier. For example the mean decrease accuracy and gini values range between 0,000 - 0,012 and 0-600 respectively.

Thanks

Gini decrease is calculated based on the mean decrease in Gini i.e. $p_i(i-p_i)$ each time when the Tree is splitted on that Feature. Value is so high because the r package weight the impurities by the raw counts, not the proportions.
Accuracy decrease is calculated on OOB dataset by randomly shuffling the data for that particular feature in the OOB. Then subtracting the two accuracies i.e. before and after shuffling.
Unit -
Accuracy - Unitless - Ratio of correct classification to total records
Gini impurity - Unitless - Measurement of the cleanliness of split.Formal
These links have got good detail:
Stackoverflow
r-bloggers
Quora


Three Staggered Values - Visualisation? I have 3 process parameters that independently cycle between zero and some peak value, as shown on this filled line time-series plot:

Although not a direct control variable, the 'health' of the system can be gauged by the consistency of relative 'stagger' (or difference) between these three parameters at any single point in time. A-B, B-C, C-A.

I am struggling to find an effective way to represent these relative stagger values - on the same timeline. A simple plot of e.g. A - B cycles positive to negative but isn't very intuitive. Ideally I'd like to represent all three relative differences on one discernible plot.

You could compare the relative values over time quite easily with a "stacked" graph, where each of the processes is a layer in the stack. In Pandas' plotting, it is referred to as an area plot.

Here is an example with some dummy data:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

p1, p2, p3 = np.random.random(100), np.random.random(100), np.random.random(100)

df = pd.DataFrame([p1, p2, p3]).T
df.head(2)
# output
#              0         1         2
#    0  0.542490  0.099974  0.831589
#    1  0.988922  0.035026  0.752813

df.plot(kind="area", stacked=True); plt.show()

I think this might allow you to see the relative swings in each of the processes quite clearly - you could also shift the slightly to be in phase, which will make things easier.


How to use Cosine Distance matrix for Clustering algorithms like mean-shift, DBSCAN, and optics? I am trying to compare different clustering algorithms for my text data. I first calculated the tf-idf matrix and used it for the cosine distance matrix (cosine similarity). Then I used this distance matrix for K-means and Hierarchical clustering (ward and dendrogram). I want to use the distance matrix for mean-shift, DBSCAN, and optics.

Below is the part of the code showing the distance matrix.

from sklearn.feature_extraction.text import TfidfVectorizer

#define vectorizer parameters
tfidf_vectorizer = TfidfVectorizer(max_df=0.8, max_features=200000,
                                 min_df=0.2, stop_words='english',
                                 use_idf=True, tokenizer=tokenize_and_stem, ngram_range=(1,3))

%time tfidf_matrix = tfidf_vectorizer.fit_transform(Strategies) #fit the vectorizer to synopses

terms = tfidf_vectorizer.get_feature_names()

from sklearn.metrics.pairwise import cosine_similarity
dist = 1 - cosine_similarity(tfidf_matrix)
print(dist)

I am new to both python and clustering. I found the code for K-means and hierarchical clustering and tried to understand it but I cannot apply it for other clusterings algorithms.
It would be very helpful if I can get some simple explanation of each clustering algorithm and how this distance matrix can be used to implement (if possible) in different clustering.

Thanks in advance!

Each of those selected clustering algorithms can be fit using cosine distances in scikit-learn:

from sklearn.cluster import DBSCAN, MeanShift, OPTICS
from sklearn.metrics.pairwise import cosine_distances

# Define clustering algorithms
algorithms = [DBSCAN, MeanShift, OPTICS]

# Placeholder for results
results = dict.fromkeys((a.__name__ for a in algorithms))

# Fit each clustering algorithm and store results
for algorithm in algorithms:
    results[algorithm] = algorithm(metric=cosine_distances).fit(X)


What is the best hardware/GPU for deep learning? What is the best GPU for deep learning currently available on the market?

I've heard that Titan X Pascal from NVidia might be the most powerful GPU available at the moment, but would be interesting to learn about other options.

And additionally - what might be considered a good machine for data-intensive computing, in terms of processor parameters, RAM etc. ?

I would recommend you read this article carefully: http://timdettmers.com/2017/04/09/which-gpu-for-deep-learning/

Basically,

The most important feature for judging deep learning performance is memory bandwidth. GTX 1080 Ti's memory bandwidth is 484 GB/s and TITAN X is 480 GB/s. Read this Quora answer to understand why memory bandwidth is the most important feature better: https://www.quora.com/Why-are-GPUs-well-suited-to-deep-learning.
Tim has also given a "theoretical" rough comparison between GPUs based on their tech specs. You can also see that in his comparison 1080 Ti and TITAN X are roughly at the same performance level.
Use 2 1080 Ti instead of a TITAN X since GTX 1080 Ti is definitely cheaper.
I've personally evaluated the performance of 1080 Ti vs. TITAN X and they are roughly the same.


What is the relationship between AI and data science? I think they share a lot (e.g. machine learning is a subset of both, right?), but maybe both have elements the other doesn't have?

Could you name some in that case? Or is one a subset of the other?

What is the relationship between AI and data science?

For example, when it comes to the relationship of AI and ML, I always say AI is a superset of ML. And the distinguishing set is search algorithms, which I would include in AI but not in ML. Would search algorithms be included in data science?

Though neither are well defined, as commonly used they are somewhat orthogonal concepts.

In my opinion, AI has a fairly narrow definition - it is about optimization through actions.  AI is about decision making, either in deterministic or probabilistic environments.  Typically, this is operationalized as action selection to maximize some reward function, or equivalently to minimize some loss function.  Supervised or unsupervised learning (i.e. machine learning) about your environment can be helpful in using your experience to aid in selecting optimal actions.

As it's commonly used, data science has no rigorous definition - businesses use the term to refer to anything from creating charts Excel to deep reinforcement learning models that can win Go.  From the point of view of a practitioner, these have absolutely nothing in common. From the point of view of a business owner, the common thread is extracting meaning, and therefore value, from raw data.  A data scientist is a 'meaning extraction layer'.  How this operation is performed, and the techniques used (again in my experience), make no difference to the employer of a data scientist. The job title may as well be 'data magician'.  But the point is that data + data science = business value, whether that comes in the form of insights into customer trends, causal analysis of marketing campaigns, or an AI bot that is 'rewarded' when it recommends you a movie that you like.  I suppose that means that AI is a subset of data science - but you could also say the same thing about clear communication, so it's a bit of a non-statement.


performances evaluation of image classification with different distribution for train and test set I am implementing a CNN to do image classification of 4 classes representing different weathers : Haze, Sunny, Rainy, Snowy. I have as training set 3200 images, and as test set 3038 images.

The problem is that the test and training set are different set of pictures(so to be clear, I didn't split the dataset, but I have been given a training set and a test set in different folders) and they have so a different distribution one with respect to the other.

After training and evaluating, I have that my accuracy on the test set is 0.475313 and the loss always on the test set is 3.060471.

Also if I plot the hihstory of accuracy and of the loss I have the following:

what I don't understand is if I have the accuracy so low because I am overfitting, or because the distribution is different. And also, how do I interpret the plots I have posted about the histories of loss and accuracy? Is it normal that the gap between test and train accuracy is so large?

Thanks in advance.

[EDIT] By searching online, I have found that to cope with this kind of problems, it is possible to use domain adaptation, but honestly I have no idea how to implement it because I have just read about it.

I'm afraid you have too little data. That's why optimizer can't converge to any reasonable place. I'd suggest you try doing transfer learning first. Few links for you to check out:

https://machinelearningmastery.com/transfer-learning-for-deep-learning/
https://www.pyimagesearch.com/2019/05/20/transfer-learning-with-keras-and-deep-learning/
https://www.pyimagesearch.com/2019/06/24/change-input-shape-dimensions-for-fine-tuning-with-keras/
https://www.pyimagesearch.com/2019/06/03/fine-tuning-with-keras-and-deep-learning/

Consider also heavier augmentation. Albumentations is a nice repo, used in many kaggle competitions.


pytorch __init__() got an unexpected keyword argument 'train' I'm using this code in colab. the function below gives me an error when intanciar.
import os
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable
import torch.utils.data as data
import torchvision
from torchvision import transforms
def load_images(image_size=32, batch_size=64, root=""):
        transform = transforms.Compose([
        transforms.Resize(32),
        transforms.ToTensor(),
        transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

        train_set = torchvision.datasets.ImageFolder(root=root,  train=True,transform=transform)
        train_loader = torch.utils.data.DataLoader(train_set, batch_size=batch_size, shuffle=True, num_workers=2)

        return train_loader

data = load_images (28.64, "/ content / data /")

error
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-85-a809f71788e0> in <module>()
----> 1 datotes=load_images(28,64,"/content/datitos/")

<ipython-input-84-0b4989f589a9> in load_images(image_size, batch_size, root)
     15     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
     16
---> 17     train_set = torchvision.datasets.ImageFolder(root=root,  train=True,transform=transform)
     18     train_loader = torch.utils.data.DataLoader(train_set, batch_size=batch_size, shuffle=True, num_workers=2)
     19

TypeError: __init__() got an unexpected keyword argument 'train'

The error implies that it couldn't recognize the train parameter, in the ImageFolder class.
ImageFolder() does not have a parameter train, remove that it will fix the error. Instead of...
train_set = torchvision.datasets.ImageFolder(root=root,  train=True,transform=transform)

...you should have...
train_set = torchvision.datasets.ImageFolder(root=root, transform=transform).

For more check the documentation.


Shapley contribution when coalition is 0 I am exploring Shapley for channel attribution based on [here][1]
Consider C1, C2, C3, C4 as 4 channels in question.
Some of the coalition does not have value, such as
(C1, C2) -> 20
(C1, C3, C4) -> 10
(C1, C2, C3, C4) -> 0

The reason being there is no conversion that went through all 4 channels.
How does Shapley works in this case ?
According to https://en.wikipedia.org/wiki/Shapley_value
At some point, there will be a marginal contribution of for C2 as
v(C1, C2, C3, C4) - v(C1, C3, C4) < 0

Is this still valid ? To interpret this, can we say C2 has a negative contribution to this coalition. But with the similar logic, since v(all Ci) = 0 , contribution of all C1, C3, C4 will also be negative in this case.

One axiom of the definition is marginal
sum(marginal v(Ci)) = v(C1, C2, C3, C4) = 0

However, when I tried Shapley, v(Ci) > 0 . How can they sum up to be = 0

Continuing Q2, I notice I don't have
sum(marginal v(Ci)) = v(C1, C2, C3, C4) = 0

.
[1]: https://medium.com/analytics-vidhya/the-shapley-value-approach-to-multi-touch-attribution-marketing-model-e345b35f3359

In the blog post, the author defines the characteristic function $v$ to be the sum of conversion across all subsets of the coalition.  (If I'm understanding the raw data correctly, this is the number of conversions that would've occurred if you limited the channels to that coalition, which seems a sensible measure.)  You can check that in the code definition for v_function.  So in your example, while no customers converted after hitting all four channels, still $v(C_1, C_2, C_3, C_4)\geq 30$, since it includes the sum of the two other channel combinations you've listed.


How to build a classifier with a rejection class Let's say I need to build a food classifier, and I want a rejection class for the inputs that are not food.

What is the best way to do that?
Should I just add a new class label that includes pictures of anything but food? Is there a best way to chose which pictures I should use?

What is the best way to do that? Should I just add a new class label that includes pictures of anything but food?

Adding a new class which can be called none is a thing that is usual for such tasks.

  Is there a best way to chose which pictures I should use?

Yes, you should pick the images of the none type from the real distribution that your classifier is going to face while test time. It means all the classes should have a same distribution while training and testing. It should be satisfied even for the none class. For instance, suppose that your none type class can be a human hand or a table or something like that. You should not put an elephant image for none type because you are not going to have an elephant in your kitchen.


Backpropagation with a different sized training set? I'm trying to create a NN whose input is a (length m) array of 3d vectors $$\vec{x}_i = [x_{i,1},x_{i,2},x_{i,3}], \hspace{5mm}i=1:m $$

and whose output is a similarly sized array:

$$\vec{h}_{\theta,i} = [h_{\theta,i1},h_{\theta,i2},h_{\theta,i3}], \hspace{5mm}i=1:m $$

BUT, my only training data is not 3d vectors but rather the magnitude/norm of such vectors (with no knowledge of the vector components ($\lambda's$) themselves):

$$y_i= ||[\lambda_{i,1},\lambda_{i,2},\lambda_{i,3}]||, \hspace{5mm}i=1:m $$

So, my concept is to use the cost function:

$$ J = \frac{1}{2m}\sum (||\vec{h}_{\theta,i}|| - ||y_i||)^2 $$

Note, the difference between this and the more usual quadratic cost function ($ J = \frac{1}{2m}\sum (\vec{h}_{\theta,i} - \vec{y}_i)^2 $) which I would use if $y_i$ was the same shape as the output.

In the typical cost function case I've mentioned above the backpropagation algorithm always starts with computing the output layer error (which based on the simple derivative of that cost function is just $\delta^L = a^L - y_i $). From there you follow to the next layer error which depends on $\delta^L$.

My problem is what should my output layer error be? I've tried just using the derivative of $J$ with respect to $a^L$ using my new cost function (I won't write out that derivative here because it's a bit hairy), but I can't seem to simply slot that in as my output error and continue on with normal back propagation with any kind of sensible results?

If you can't suggest how I should approach my BP algorithm, then perhaps it's because you think my cost function is a non-starter? As a note: I have sort of proven the concept to myself using a linear model rather than an NN and training it with this dataset and linear derivatives rather than BP. The accuracy was low though given the nonlinearity of the data so I would like to use an NN approach.

Your cost function implies that you want the network to predict the magnitude of the output vector based on the magnitude of the input vector. Is that what you want to do? If not, e.g. if you want to accurately predict the output vector components, you need to believe that these are predictable based only on the magnitude of the inputs. If you don’t think this is possible, then you need to find new data. If you do, then a normal error function should be fine (e.g. MSE) and you can use normal backdrop as implemented in any NN library. If you’re getting poor performance, it’s probably because the norm of the input vector does not carry enough information to make a good prediction.


Why the number of neurons or convolutions chosen equal powers of two? In the overwhelming number of works devoted to the neural networks, the authors suggest arhitechure in which each layer is a numbers of neurons is power of 2

what are the theoretical reasons(prerequisite) for this choice?

Deep Neural Networks are usually trained on GPUs to speed up training time. Using power of two for the network topology follows the same logic as using power of two for image textures in computer games.

The GPU can take advantage of optimizations related to efficiencies in working with powers of two. (see https://gamedev.stackexchange.com/questions/26187/why-are-textures-always-square-powers-of-two-what-if-they-arent)


How to get spike values from a value sequence? I have pile of vectors where the values could be plotted like this:

Now I want to extract the "spike values" (over a certain threshold say 15,000). In this case there is fifteen. How could this be done with Python? (There is no predefined number of spikes but the threshold is a reliable filter value.)

This is very simple. Let's say your data in Panda format (named data_df), and extracting peaks/spikes over a certain threshold (e.g. 15000 here) is simply:

data_df[data_df > 15000]

If this data is sitting in a particular column, you can use this instead:

data_df[data_df['column_name'] > 15000]

These will return the peak values.

Updated Answer:

If you want local extreme points (e.g. maximum or minimum ) around each peak, check scipy.signal.argrelextrema in Scipy. A concrete example:

Let's make a artificial random data with random spikes:

import numpy as np
import matplotlib.pyplot as plt

random_number1 =np.random.randint(0,200,20)
random_number2=np.random.randint(0,20,100)
random_number=np.concatenate((random_number1,random_number2))
np.random.shuffle(random_number)
plt.plot(random_number)

Now using argrelextrema function you will find the index of relative extreme values (either minimum or maximum)

c_max_index = argrelextrema(random_number, np.greater, order=5)

Please make sure you understand the "order" option. It basically look around 5 neighbouring points, and it returns the maximum in this case. And you can see how it works by pinpointing the found points on the actual graph like this:

plt.plot(random_number)
plt.scatter(c_max_index[0],random_number[c_max_index[0]],linewidth=0.3, s=50, c='r')

Note that you can retrieve peak points via random_number[c_max_index[0]], and c_max_index are just indexes of the extreme points.


How do I calculate the range of a F1-score from a confusion matrix of 3 class,A,B,C Is there any support function to calculate the average F1-score range?

from sklearn.metrics import f1_score

y_true = [0, 1, 2, 0, 1, 2]
y_pred = [0, 2, 1, 0, 0, 1]

f1_score(y_true, y_pred, average='weighted')

From documentation

  Calculate metrics for each label, and find their average weighted by support (the number of true instances for each label). This alters ‘macro’ to account for label imbalance; it can result in an F-score that is not between precision and recall.

https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html


How to predict whether or not a customer will renew I have a dataset of customer contracts that specify a start date and if applicable an end date. Each month a customer is up for renewal. Below is an example of how the data is organized in excel:

ID   Customer Start Date   Customer Drop Date
1    Jan. 2018             Dec. 2018
2    Feb. 2018             July 2018
3    Mar. 2018

Using the above example, I'm trying to predict whether customer 3 will drop in Jan. 2019, Feb. 2019, Mar. 2019, etc.. Essentially I'm trying to calculate the probability that a customer that's still active will renew their contract for a given month after Dec. 2018. What is the remaining life-time value?

Should I graph the the length of all historic contracts and see what distribution they match? If so how would I apply the distribution to the open contracts?

What you are trying to do is called Churn Prediction. Unfortunately, the dataset you have is not enough to train a model. You need a variety of different features for a proper prediction model.

For example, you need demographic data for the customers, but most important, data related to customer's actions. For example, if you have a mobile network company, you might need features as:

Age
Sex
Country
Daily usage (number of call minutes, SMS or mobile data)
Previous day usage
Mobile package (details about the contract etc)

In general, the more data you have around your problem, the easier would be to build such a model.


Can a machine learning model be used as some kind of compression? I'm trying to understand how machine learning is working. I read a lot and now came into my mind that it could be missuses in a practical way. I also hope that this question is on topic here.
Please correct me if I have some wrong assumptions:

All models require sample data
The learning process is some kind of optimization of its functions aka neutrons
The learning process is iterative to find the best values of the parameters to maximize the expected output of the neurons
To make this process faster you do some kind of reduction of the data, so that in the end the neutrons do not need to look at every part of the input
You split up the training data in two buckets to find out of the result became better for the optimization (I would call this verification, not sure how this is really called)
If your verification data are bad or equal to the input data than the results can just be as good as your input data

If this is true or at least most parts I got this idea:
When I set the input data to the same as the output data, then the model is doing nothing, but for equal data I get always the same data right?
So when I train the model as in the table:

input
output

0
3

1
1

2
4

3
1

4
5

5
9

6
2

7
6

8
5

Then the model would return PI based on the correct count of digits (of cause for non learned indexes that would be random)
I absolutely know that this makes no sense at all, but I'm just curious if I understand it correctly that you could "compress" data in a ML-Model.

You're right I'm not sure if I followed exactly what you said, however for your overall question of can a Machine Learning model be used as a form of compression - yes!
A good example is an Autoencoder, which is a neural network that 'breaks down' the input into basic elements and then reconstructs the input using only the elements required.  This is formally know as Dimensionality Reduction.
Some of the practical applications are removing noise for example from images, which if you think about is removing data you don't need but preserving the overall look.  Much like traditional image compression, it won't look exactly the same.


How do I decide if I need to go for Normalization and not Standardization or vice-versa? While designing a ML model, how do I decide if I need to go for Normalization and not Standardization or vice-versa? On what factor is this decision made?

Before we start keep in mind that in most cases it doesn't play much of a difference which of the two you'll choose.

Now to answer your question, generally speaking the choice should be made based on what model you want to employ:

If you use a distance-based estimator (e.g. k-NN, k-means) it's better to normalize your features so that they occupy the same exact range of values (i.e. $[0,1]$). This forces your estimator to treat each feature with equal importance.
If you're using Neural Networks, it's better to standardize your features, because gradient descent has some useful properties when your data is centered around $0$ with unit variance.
Tree-based algorithms don't require any form of scaling, so its irrelevant if you scale or normalize your features.

As a rule of the thumb, I usually standardize the data (unless I'm going to strictly work with distance-based algorithms).


Re-bucket weekly sales data and calculate descriptive statistics I've got sales data in weekly buckets like this:

weekID product SoldQty
1      1       10
2      1       20
3      1       30
4      1       40
5      1       50
6      1       60
7      1       70

1      2       10
2      2       20

Calculating st.dev. of weekly sales per product is pretty straightforward.

Now I'm asking the question: how to calculate the St.dev for the same data, but bi-weekly bucketed? x-weekly bucketed?

Question 2: is there an efficient algorithm for calculating it on weekly data instead of materializing the x-weekly combinations?

From the business side it means that I have various forecast horizons (1wk, 4 wks, 6wks...) per product. And I want to build the confidence intervals for predictions of SoldQty within the forecast horizon.
It all seems very similar to the Safety Stock calculations from logistics but I want to be sure.

UPDATE:

Please consider there are multiple ways to re-bucket (re-combine) this data. I think that if the new bucket contains W weeks, there are W ways to re-bucket data. For an example of 2-weekly buckets.

Way 1

bucket wkid prod sold
1      1    1    10
1      2    1    20
2      3    1    30
2      4    1    40
3      5    1    50
3      6    1    60
4      7    1    70
...

Way 2

bucket wkid prod sold
1      1    1    10
2      2    1    20
2      3    1    30
3      4    1    40
3      5    1    50
4      6    1    60
4      7    1    70
...

Both options make sense business-wise. Essentially you need to calculate st.deviations between sales during fortnights. And, st.deviation in way 2 should be different from way 1.

Thanks in advance for your suggestions!
Constantine.

If the week ID is given as you state, calculate a bucket variable $w_x=int(weekID/x)$.  Then use a SQL statement to summarize the volume to levels of $w_x$.


make seaborn heatmap bigger I create a corr() df out of an original df. The corr() df came out 70 X 70 and it is impossible to visualize the heatmap... sns.heatmap(df). If I try to display the corr = df.corr(), the table doesn't fit the screen and I can see all the correlations. Is it a way to either print the entire df regardless of its size or to control the size of the heatmap?

I found out how to increase the size of my plot with the following code...

plt.subplots(figsize=(20,15))
sns.heatmap(corr)


What is the memory cost of a CNN? I was recently thinking about the memory cost of (a) training a CNN and (b) inference with a CNN. Please note, that I am not talking about the storage (which is simply the number of parameters).

How much memory does a given CNN (e.g. VGG-16 D) need for

(a) Training (with ADAM)
(b) Inference on a single image?

My thoughts

Basically, I want to make sure that I didn't forget anything with this question. If you have other sources which explain this kind of thought, please share them with me.

(a) Training

For training with ADAM, I will now assume that I have a Mini-batch size of $B \in \mathbb{N}$ and $w \in \mathbb{N}$ is the number of parameters of the CNN. Then the memory footprint (the maximum amount of memory I need at any point while training) for a single training pass is:

$2w$: Keep the weights and the weight updates in memory
$B \cdot $ Size of all generated feature maps (forward pass)
$w$: Gradients for each weight (backpropagation)
$w$: Learning rates for each weight (ADAM)

(b) Inference

In inference, it is not necessary to store a feature map of layer $i-1$ if the feature maps of layer $i$ are already calculated. So the memory footprint while inference is:

$w$: The model
The two most expensive successive layers (one which is already calculated, the net one which gets calculated)

Total RAM would be - Batch size X RAM to train one image (since backpropagation happens after the batch)

RAM for one training image -

A/ 4 Bytes X Number of parm

B/ Size of input for each layer considering downsampling and number of features map

(Suppose input are 200 × 300 pixels, the first layer’s feature maps might be 100 × 150, the second layer’s feature maps can be 50 × 75, and the third layer’s feature maps can be 25 × 38. The first convolutional layer has 100 feature maps, this first layer takes up 4 × 100 × 150 × 100 = 6 million bytes (6 MB). The second layer will take up 4 × 50 × 75 × 200 = 3 million bytes (3 MB).

C/ Size for the input image


Using Generative Adversarial Network to generate Single Image Most generative adversarial networks learn the distribution of the dataset and then generate a sample of 10's to 100's of images with similar distributions. I am curious if there is any research regenerating a single image.

I have looked into Super Resolution GAN's and they somewhat model this. They try to estimate a highly resolved image from its lower resolution counterpart. Could that be used to recreate an image?

Thanks for the help.

A generative adversarial network is comprised of two parts which are necessary for the training, the generator and the discriminator. They are playing a game against each other, the discriminator tried to learn what is real and what is fake, and the generator tries to create fake instances which will fool the discriminator.

At the input of the discriminator you will use real images, for example the MNIST numbers. At the input of the generator you will put just random noise vectors. These vectors will then be shaped by the layers of the generator to create some image, hopefully resembling the MNIST dataset. If it fools the discriminator, it wins that round. Otherwise, it will tune its weights using backpropagation in order to produce better quality results in the next round.

Once training is complete, you will detach the discriminator and use only the generator. Then for every single random noise vector fed to the input of the generator you will receive an associated image.

Here is a GaN implementation. You will need to put a path "./gan/images/" i the directory where you are running this GaN.

import numpy as np
import time
from tensorflow.examples.tutorials.mnist import input_data

from keras.layers import Input
from keras.models import Model
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D
from keras.layers import Dense, Activation, Flatten, Reshape
from keras.layers import Conv2D, Conv2DTranspose, UpSampling2D
from keras.layers import LeakyReLU, Dropout
from keras.layers import BatchNormalization
from keras.optimizers import Adam, RMSprop
from keras.datasets import mnist

import matplotlib.pyplot as plt

class GAN():
    def __init__(self):
        self.img_rows = 28
        self.img_cols = 28
        self.channels = 1
        self.img_shape = (self.img_rows, self.img_cols, self.channels)

        optimizer = Adam(0.0002, 0.5)

        # Build and compile the discriminator
        self.discriminator = self.build_discriminator()
        self.discriminator.compile(loss='binary_crossentropy',
            optimizer=optimizer,
            metrics=['accuracy'])

        # Build and compile the generator
        self.generator = self.build_generator()
        self.generator.compile(loss='binary_crossentropy', optimizer=optimizer)

        # The generator takes noise as input and generated imgs
        z = Input(shape=(28,28,))
        img = self.generator(z)

        # For the combined model we will only train the generator
        self.discriminator.trainable = False

        # The valid takes generated images as input and determines validity
        valid = self.discriminator(img)

        # The combined model  (stacked generator and discriminator) takes
        # noise as input => generates images => determines validity
        self.combined = Model(z, valid)
        self.combined.compile(loss='binary_crossentropy', optimizer=optimizer)

    def build_generator(self):

        noise_shape = (28,28,)

        model = Sequential()

        model.add(Reshape((28*28,), input_shape=noise_shape))
        model.add(Dense(256))
        model.add(LeakyReLU(alpha=0.2))
        model.add(BatchNormalization(momentum=0.8))
        model.add(Dense(512))
        model.add(LeakyReLU(alpha=0.2))
        model.add(BatchNormalization(momentum=0.8))
        model.add(Dense(1024))
        model.add(LeakyReLU(alpha=0.2))
        model.add(BatchNormalization(momentum=0.8))
        model.add(Dense(np.prod(self.img_shape), activation='tanh'))
        model.add(Reshape(self.img_shape))

        model.summary()

        noise = Input(shape=noise_shape)
        img = model(noise)

        return Model(noise, img)

    def build_discriminator(self):

        img_shape = (self.img_rows, self.img_cols, self.channels)

        model = Sequential()

        model.add(Conv2D(32, kernel_size=(3, 3),
                 input_shape=img_shape))
        model.add(MaxPooling2D(pool_size=(2, 2)))
        model.add(LeakyReLU(alpha=0.2))
        model.add(Dropout(0.25))

        model.add(Flatten())
        model.add(Dense(64))
        model.add(LeakyReLU(alpha=0.2))
        model.add(Dense(1, activation='sigmoid'))
        model.summary()

        img = Input(shape=img_shape)
        validity = model(img)

        return Model(img, validity)

    def train(self, epochs, batch_size=128, save_interval=50):

        # Load the dataset
        (X_train, _), (_, _) = mnist.load_data()

        # Rescale -1 to 1
        X_train = (X_train.astype(np.float32) - 127.5) / 127.5
        X_train = np.expand_dims(X_train, axis=3)

        half_batch = int(batch_size / 2)

        for epoch in range(epochs):

            # ---------------------
            #  Train Discriminator
            # ---------------------

            # Select a random half batch of images
            idx = np.random.randint(0, X_train.shape[0], half_batch)
            imgs = X_train[idx]

            noise = np.random.normal(0, 1, (half_batch,28,28,))

            # Generate a half batch of new images
            gen_imgs = self.generator.predict(noise)

            # Train the discriminator
            d_loss_real = self.discriminator.train_on_batch(imgs, np.ones((half_batch, 1)))
            d_loss_fake = self.discriminator.train_on_batch(gen_imgs, np.zeros((half_batch, 1)))
            d_loss = 0.5 * np.add(d_loss_real, d_loss_fake)

            # ---------------------
            #  Train Generator
            # ---------------------

            noise = np.random.normal(0, 1, (batch_size,28,28,))

            # The generator wants the discriminator to label the generated samples
            # as valid (ones)
            valid_y = np.array([1] * batch_size)

            # Train the generator
            g_loss = self.combined.train_on_batch(noise, valid_y)

            # Plot the progress
            print ("%d [D loss: %f, acc.: %.2f%%] [G loss: %f]" % (epoch, d_loss[0], 100*d_loss[1], g_loss))

            # If at save interval => save generated image samples
            if epoch % save_interval == 0:
                self.save_imgs(epoch)

    def save_imgs(self, epoch):
        r, c = 5, 5
        noise = np.random.normal(0, 1, (r * c, 28,28,))
        gen_imgs = self.generator.predict(noise)

        # Rescale images 0 - 1
        gen_imgs = 0.5 * gen_imgs + 0.5

        fig, axs = plt.subplots(r, c)
        cnt = 0
        for i in range(r):
            for j in range(c):
                axs[i,j].imshow(gen_imgs[cnt, :,:,0], cmap='gray')
                axs[i,j].axis('off')
                cnt += 1
        fig.savefig("gan/images/mnist_%d.png" % epoch)
        plt.close()

if __name__ == '__main__':
    gan = GAN()
    gan.train(epochs=30000, batch_size=64, save_interval=200)


Which deep learning network would be suitable for classifying this kind of text data? I have some experience with images and have played around with image classification using CNN's but have limited knowledge when it comes to text data.
The input that I currently want to classify is written as:
hjkhghkgfghjkhghkgfghfefdefdcdefghjkjh-hjhgfe
fdcd-dd-fdc-dad-ad-dfe-cde-dggf-ghd-gg-bcd
hjkhghkgfghjkhghkgfghfefdefdcdefghjkjh-gh-gfed
dh-hg-gf-gh-dh-hg-gf-gh-hkhg-kh-hg-gf-gh-hkhg-kh-hg-gf-ghh-hgfg-dfd-dc-fgf-gh

I have over 2000 rows of this data, that needs to be classified. I know that for regular text data RNN networks and LSTM cells have been known t be very effective. Using RNN+LSTM good results can be achieved by pre-processing the data using the usual approaches such as stemming, lemmitization, stop word filtering, tokenization etc. But the same cannot be applied to the text data I have.
Would RNN and LSTM still work on my data? If not which networks do you guys suggest I explore for such a task?

You need character embeddings. I assume you are already familiar with word2vec technology. Its goal it to make a model "learn" the relative meaning of words, placing them into a highly dimensional space.
The same can be done with single characters, instead of whole words. The preprocessing steps you need will be a little bit different, but the embedding technique is the same. In that way, you can generate representations of characters, feed their sequences into some RNN model, and perform the final classification task.
Therefore, RNNs are perfectly suitable for this task. If you are working with tensorflow.keras you can simply tokenize characters, and feed them through an Embedding() layer that will do the job for you. An alternative to RNNs is 1D conv layers, that can do the job as an alternative to recurrent cells. That's up to your preference.


Is there a way to calculate value from a radar (spider) chart? So I am currently working on a project for "sales people". Essentially, the user is giving us a plethora of information related to deals they are trying to win. We have data related to the price of the deal, margin of the deal, number of people working on the deal, caliber of the people working on the deal, and the risk of the deal. I want to assign a "score" to these deals based on the given metrics and visualize how the score is calculated in a simple way.

It would be very easy to visualize this all as a radar (spider) chart (I'm aware that a traditional radar chart has limited power since the ordering of metrics is arbitrary and the area formed by the spider chart doesn't hold much value) - but why can't I take the results from a correlation matrix or regression output to weight the metrics in some way and/or oder them based on importance?

An example - let's say I've built a radar chart for NBA players based on Points per game, assists per game, player efficiency, etc. If I had a logistic regression output to determine the MVP, I would use the output to order and weight the variables (based on p-values?) to build the radar chart. If I then took statistics from the most recent NBA season and plotted each NBA player, wouldn't the player with the biggest area be one of the most valuable?

I can't seem to find much information on building weighted radar charts - although I have seen that percentiles can be used for the actual metrics. Does anybody have any advice or alternative solutions?

You basically want to display the importance of each feature in the model as it relates to your final score. Rather than starting with the visualization, I would figure out the data first and then see which visualization fits. Here, they use a simple bar chart:

http://scikit-learn.org/stable/auto_examples/ensemble/plot_forest_importances.html


Machine learning for object states I have the objects pool with histories of their states, where each transition from one state to another takes some time, ex:
$$
object\ 1: A \overset{1s}{\rightarrow} B \overset{2s}{\rightarrow} C
\\
object\ 2: D \overset{3s}{\rightarrow} E \overset{4s}{\rightarrow} F
$$
The pool consists of many objects with right histories, where sequence of states and transition duration follows some unknown rules.
Also the pool consists a small amount of object with wrong history. But which objects are right or wrong is unknown.

It needs to found mistakes in the pool. Ex, "the transition from X to Y is not allowed" or "the transition is too long for such type".

What machine learning models are more suitable for this problem?

To me this problem looks similar to language modeling: a model is trained on a large amount of sequences, and then it can predict the probability of any input sequence. In your case a low probability would indicate an abnormal sequence.

My background is in NLP that's why I think language modeling, but I guess the same techniques are used for other problems as well. The fact that you have transitions and states suggests Markov Models, for which there are known methods for inference and estimation. So maybe you could design a more specific kind of model for your case and use something like the Baum–Welch algorithm.


Generated training set on convnet I have a dataset with roughly 800 images that are classified in 18 classes.
The classes are spread unevenly, with some classes having 30 images and others having 5.

In order to increase my dataset,I've decided to use image augmentation modifying each image a little,making 20 new images for every image.

I then decided to use my created images as my training set and my original images as my validation set.

Due to unavailability of a GPU,I couldn't train it a lot,but I ended up with around 50% success rate on a training set and 30% on the validation set.

Was the decision of only using my original dataset as validation a good one? If not,why?

I don't think it is wise. Your intention to do validation on your real data is correct. But the way you have it now your model will be prevented from training on data that is from the same distribution as what you actually want to predict.

It is best to first split the images into training and validation sets, then do data augmentation on the training set.


Can I use Keras to give fitness scores to a genetic algorithm? I know that I can use Keras to solve classification problems but is it possible to get it to output a float between 0 and 1000 for instance? I want train my network to approximate the simulation that gives my GA a fitness score because the simulation takes a long time to run.

Yes, you can get a neural network to predict real numbers instead of for classification. This type of problem is called regression.
You can find dozens of tutorials for doing regression with Keras. The official one is here.


Iteratively construct Linear Classifier sorry if this question is out of place. I'm a begginer to machine learning, and I have use for a technique, and I don't even know where to look. The problem is:

I have 5 features which are real valued (Parameters in a deterministic simulation).
This features determine two aspects of the instance (model solution). Its feasibility (binary) and some measure of likelihood given certain experimental data (only for the instances that achieved feasibility).

Since I want to avoid generating "infeasible" combination of features, what I devised was an algorithm that iteratively does the following:

Generate Nc candidate feature vectors
Evaluate Feasibility and Likelihood for each
Find linear combination of features that involves a compromise between least amount of features / holds largest cluster of feasibility. Add this as constraints to the feature vector generation.

In short, it detects and iteratively refines "simple" constraints that once added to the feature vector generation "guarantee" its feasibility to save computational time evaluating combination of parameters that lead to infeasible models. Afterwards, they could be tested by inverting them and looking for other "regions" (if any) of the feature vector where the model is feasible.

Any name of techniques and references I might look for ?

Thanks!

Your algorithm is very similar to a Genetic Algorithm, which has almost the same 3 iterative steps.

You would be missing a Termination condition to end the search for an answer, once your reach an acceptable one.

You can probably adapt your algorithm to fit under a formal GA search heuristic.


Score Columns in Azure ML Studio So I have a data set I have successfully used to train a model, and have decent results. I am using a Two Class Boosted Decision tree for a Boolean output. So far so good.

I now want to analyze each column of my data set and remove any column that is not a meaningful influence on the outcome. I see statistics on columns in my data set:

But I don't see whether a column has a strong relationship with the output variable. Any clues?

Some very helpful articles walked me through how to use the feature selection modules in Azure ML Studio:

https://docs.microsoft.com/en-us/azure/machine-learning/studio-module-reference/feature-selection-modules
https://docs.microsoft.com/en-us/azure/machine-learning/studio-module-reference/filter-based-feature-selection

I used the "Filter Based Feature Selection" Module:

I added it to my training experiment and added input data.

I chose the feature I was training my model to predict.

After running the experiment I viewed the visualization of the Features:

Now I am using this to find which columns are valuable and which need some work.


How to do webscrapping in R on this webpage? I am quite new to R and I am trying to learn webscraping. I basically need to extract documents from this website.

Ideally, the data needs to be structured in three columns: YEAR, DATE, and
INTRODUCTORYSTATEMENT_CONTENT. Can anyone help with the coding?

This should be possible with  rvest in R. Two things make is possible

URL pattern is predictable, https://www.ecb.europa.eu/press/pressconf/2012/html/index.en.html    (replace 2012 with other year values)
Html page applies predictable CSS for INTRODUCTORYSTATEMENT_CONTENT (E.g.: doc-title and doc-subtitle )

Following articles have examples :

https://towardsdatascience.com/web-scraping-tutorial-in-r-5e71fd107f32
https://www.datacamp.com/community/tutorials/r-web-scraping-rvest
https://www.analyticsvidhya.com/blog/2017/03/beginners-guide-on-web-scraping-in-r-using-rvest-with-hands-on-knowledge/


How to detect blocks of texts in document images I am planning to detect texts from document text images like below:

GOAL:

WORK DONE:
I have tried to solve this with some scene text detection algorithms like EAST Text detector and PixelLink. But it only provides result in such a way it detects each and every word individually as below, which is obvious:

What method can help me detect blocks of texts as mentioned under GOAL.

EDIT :

I don't want extract all texts via OCR. What I want instead is to detect texts based on their visual positional arrangement. See in the image, texts positioned together are detected as blocks. And my result should contain all the bounding box co-ordinates of all the detected text blocks.

I would approach the text block amalgamation as a clustering problem. If you define a suitable distance metric or a neighbour predicate between the individual text boxes, you could group the boxes and then determine their minimum bounding box, which is essentially what you are aiming for.

I guess DBSCAN could be a suitable candidate for the clustering algorithm, but more care would have to go into the design of the neighbor predicate - one idea could be that vertical distance could be treated differently than horizontal distance, etc.