Why does Lasso behave "erratically" when the number of features is greater than the number of training instances? From the book "Hands-on Machine Learning with Scikit-Learn and TensorFlow 2nd edition" chapter 4:

  In general, Elastic Net is preferred over Lasso since Lasso
  may behave erratically when the number of features is greater than the number of
  training instances or when several features are strongly correlated.

How and why exactly does Lasso behave "erratically" when no. of features > no. of training instances or when features are strongly correlated? Does it have to do with redundant features?

When p > n, the LASSO model can only sustain up to n variables (this can be proven using linear algebra, the rank of the data matrix in particular), leaving at least p - n variables out (some that might be predictive, consider a model where you use LASSO and all your variables are predictive). This is not necessairly a bad thing though, depending on your use case.

For the colinearity problem, if two variables are highly correlated (that is, they contain the same "information" in some sense) then LASSO will tend to pick one of the two randomly and drop the other one. Again, might not be an issue but can lead to "erratic" behavior in which the variables selected change quite a bit depending on the training set.

In general, I find LASSO to choose different sets of variables that change quite a bit depending on the training set, especially for small datasets and even worse when p > n. Again, not necessarily a problem but you might look to not using explicit feature selection and instead opting for ridge regression. Alternatively, the elastic net is a balance between the two methods.


How to calculate the mean of a dataframe column and find the top 10% I am very new to Scala and Spark, and am working on some self-made exercises using baseball statistics. I am using a case class create a RDD and assign a schema to the data, and am then turning it into a DataFrame so I can use SparkSQL to select groups of players via their stats that meet certain criteria.

Once I have the subset of players I am interested in looking at further, I would like to find the mean of a column; eg Batting Average or RBIs. From there I would like to break all the players into percentile groups based on their average performance compared to all players; the top 10%, bottom 10%, 40-50%

I've been able to use the DataFrame.describe() function to return a summary of a desired column (mean, stddev, count, min, and max) all as strings though. Is there a better way to get just the mean and stddev as Doubles, and what is the best way of breaking the players into groups of 10-percentiles?

So far my thoughts are to find the values that bookend the percentile ranges and writing a function that groups players via comparators, but that feels like it is bordering on reinventing the wheel.

I have the following imports currently:

 import org.apache.spark.rdd.RDD
 import org.apache.spark.sql.SQLContext
 import org.apache.spark.{SparkConf, SparkContext}
 import org.joda.time.format.DateTimeFormat

This is the import you need, and how to get the mean for a column named "RBIs":

import org.apache.spark.sql.functions._
df.select(avg($"RBIs")).show()

For the standard deviation, see
scala - Calculate the standard deviation of grouped data in a Spark DataFrame - Stack Overflow

For grouping by percentiles, I suggest defining a new column via a user-defined function (UDF), and using groupBy on that column.  See

Spark SQL and DataFrames - Spark 1.5.1 Documentation - udf registration


How to interpret my logistic regression result with statsmodels so I'am doing a logistic regression with statsmodels and sklearn.
My result confuses me a bit. I used a feature selection algorithm in my previous step, which tells me to only use feature1 for my regression.
The results are the following:

So the model predicts everything with a 1 and my P-value is < 0.05 which means its a pretty good indicator to me. But the accuracy score is < 0.6 what means it doesn't say anything basically.
Can you give me a hint how to interpret this?
It's my first data science project with difficult data.
My code:
X = df_n_4["feat1"]
y = df_n_4['Survival']

# use train/test split with different random_state values
# we can change the random_state values that changes the accuracy scores
# the scores change a lot, this is why testing scores is a high-variance estimate
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=2)
print(len(y_train)," Testdaten")

# check classification scores of logistic regression
logit_model = sm.Logit(y_train, X_train).fit()
y_pred = logit_model.predict(X_test)
print('Train/Test split results:')
plt.title('Accuracy Score:{}, Variablen: feat1'.format(round((accuracy_score(y_test, y_pred.round())),3)))
cf_matrix = confusion_matrix(y_test, y_pred.round())
sns.heatmap(cf_matrix, annot=True)
plt.ylabel('Actual Szenario');
plt.xlabel('Predicted Szenario');
plt.show()
print(logit_model.summary2())

Something's wrong with your feature selection tool: p-value is NaN, confidence interval includes $0$. Confusion matrix shows that all observations are predicted as Class 1. How many explanatory variables do you have? Try using all of them instead of just one. Are you sure
logit_model = sm.Logit(y_train, X_train).fit()

is correct? Shouldn't it be the other way around,  logit_model = sm.Logit(X_train, y_train).fit()?


Twitter POS and NER: What is state-of-the-art? What is the current state-of-the-art for pos tagging and named entity recognition for twitter data? Are industrial-strength programs like Spacy and SparkNLP accurate for such texts? How about FlairNLP and Stanford's CoreNLP accuracy measures?

SOTA is changing so rapidly in NLP that even Data Science professionists struggle to cope with it. I have two main sources that I constantly check to gain some insights on SOTA:

NLP Progress from Sebastian Ruder. It contains updates on NLP on a whole lot of subfields, NER and POST included.

Paper with code contains a section on NLP. That's a great website for ML in general.

I know these links do not tackle the problem of Twitter specifically, however I don't think that domain is qualitatively different from others. IMO, of course.

About your other question:

Are industrial-strength programs like Spacy and SparkNLP accurate for such texts? How about FlairNLP and Stanford's CoreNLP accuracy measures?

As I wrote above, it's mostly a matter of personal preference and/or contingent project needs. There's no right or wrong tool. Personally, I found Stanford tools to be the best, for either the quality of their predictions and the amount of models available from a single pipeline. But as I said it's very subjective.


How do we know Kernels are successful in making data linearly Separable? When we have linearly inseparable datasets and we are using machine learning algorithms such as SVMs, we use kernels to implicitly map datapoints into a feature space that makes them linearly separable.

But how do we know if a kernel has indeed, implicitly, been successful in making the datapoints linearly separable in the new feature space? What is the guarantee?

You cannot guarantee this. Some data is not separable by any kernel because of duplicates.

By trying too hard, you will cause overfitting. Essentially, you force the implicit mapping to be so complex it contains a copy of your training data (which is exactly what happens if you choose a too small bandwidth with RBF).

If you want a good generalization performance, you will have to tolerate some errors, and use e.g. soft-margin and such techniques.

Perfect separation is not something to aim for. Such a guarantee is just a guarantee of being able to overfit! Use cross-validation to reduce the risk of overfitting and find the right balance between being optimal on training data and actual performance.


Reducing text input size into word2vec without affecting performance too badly? So I am implementing Word2Vec for the first time, and I have a set of training data that I would like to train a word2vec model on. Predictably, the problem is the dataset is rather large, and I have more limited computational power than I would like. This is a very common problem of course, but are there any ways to minimize the input text without too horribly affecting the performance?

For example, if I had the example sentences:

  Mr. and Mrs. Dursley, of number four, Privet Drive, were proud to say that they were perfectly normal, thank you very much. They were the last people you’d expect to be involved in anything strange or mysterious, because they just didn’t hold with such nonsense. Mr. Dursley was the director of a firm called runnings, which made drills. He was a big, beefy man with hardly any neck, although he did have a very large mustache.

I could do some thing like take out the stop words... but as I googled, that's a bad idea according to one link and a good idea according to another... so which is it? I also looked into text summarization but that seems harder than implementing word2vec. Another idea I naively had was to randomly take out X% of sentences from the text. But that would obviously be a performance hit in the model, and I'm not sure how big.

So, are there any general methods for trying to do this, and is stop word removal a way to do it?

I don't think there really is a right or wrong answer to the "removing stopwords" question.  Some people will argue that throwing away information will reduce model performance, while others argue that it'll increase noise.

I personally follow a simple rule of thumb.  If my model depends on sentence structure, then I keep stopwords.  If i'm modeling topics and more interested in important phrases, then I remove them.  This seems to work well.

If you're looking for ways to reduce you matrix, then yes removing stopwords is a perfectly acceptable idea.  Another thing you can do is apply common feature reduction techniques like LSA or chi2 to find the most important words and reduce your input space to the most meaningful words.  However, doing this may dramatically effect the performance or your word2vec model.  But if it is your only choice, then why not give it a go.


Why does bidirectional LSTM have half the parameter count compared to LSTM in keras? I want to implement a unidirectional and a bidirectional LSTM in tensorflow keras wrapper with the same amount of units. As an example I implement the unidirectional LSTM with 256 units, and the bidirectional LSTM with 128 units (which as I understand gives me 128 for each direction, for a total of 256 units). The implementation details:
import tensorflow as tf
in_ = tf.keras.Input(shape=(28,28))

x = tf.keras.layers.LSTM(256)(in_)

model_unidirectional = tf.keras.Model(in_,x)
print(model_unidirectional.summary())

y = tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(128))(in_)

model_bidirectional = tf.keras.Model(in_,y)
print(model_bidirectional.summary())

However, looking at the models' summary, the unidirectional LSTM has double the parameter count compared to the bidirectional LSTM, even if they have the same output shape in both cases:
model_unidirectional summary:
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_11 (InputLayer)        [(None, 28, 28)]          0
_________________________________________________________________
lstm_11 (LSTM)               (None, 256)               291840
=================================================================
Total params: 291,840
Trainable params: 291,840
Non-trainable params: 0
_________________________________________________________________

model_bidirectional summary:
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_11 (InputLayer)        [(None, 28, 28)]          0
_________________________________________________________________
bidirectional_7 (Bidirection (None, 256)               160768
=================================================================
Total params: 160,768
Trainable params: 160,768
Non-trainable params: 0
_________________________________________________________________

Why does the bidirectional approach have significantly less parameters if their output shape is the same?

For Unidirectional LSTM the number of parameters are
4*[(numHiddenUnit+inputSize)*numHuddenUnits+numHuddenUnits]
where 4 is for 4 LSTM gate equations.
For your case numHuddenUnits = 256, inputSize is 28 gives the result 291840
For birectional LSTM the number of parameters are
2 * 4 * [(numHiddenUnit+inputSize)*numHuddenUnits+numHuddenUnits]
where 2 is due to bi-directional weights and 4 is for four gate equations
For your case numHuddenUnits = 128, inputSize = 28 gives the result 160768


Passing the output of a model as input to an other model , computer vision? i want to try my hands at a project where i have to detect the brands + the models of sportswears worn by persons participating to a marathon , the project is quite complex , so my first approach would be to use an already pre-trained model that will draw the bounding box of the persons , then use the outputed bonding boxes as input to an other model that will draw me the bounding box for the shirt , shoes , shorts ...ect , and follow this procedure until i get to the brand and then the model of the sportswear.
theorically , it's super simple , the problem is that , i'm having a hard time figuring out how to implement it by code , passing the output of one model as an input to an other model , is there any good exemples on the internet can illustrate that approach , by code i mean ?

Any suggestion would be much appreciated , thank you.

I find your question a little hard to understand. If I understand correctly you know how to obtain a bounding box for a person.
Using these coordinates, you can select this rectangle using open-cv or matplotlib or any image processing library.
If your second model can handle multi-resolution or multiple sizes you can just use this selection as input.
If, on the other hand, your second model requires a fixed size you can resize the selection and use padding if necessary.


AUC computation on multilabel classification I'm using Tensorflow for an auto-tagging task on audio clips. The problem is actually a multilabel classification problem meaning that each clip can have multiple tags at the same time.

Regarding the model evaluation, I would like to be able to compute the AUC per clip and per tag. My problem is that AUC is basically a metric for binary classification and I don't know how it can be extended to a multilabel problem.

I have found many references of AUC computation on multilabel classification tasks, but none of these explain how to do it.

(An example)

AUC (Area under the ROC Curve) can not be calculated for multilabel classification directly.
Multilabel classification metrics have to be converted to a binary problem. The two most common conversions are:

OvO - One vs One
OvR - One vs Rest

sklearn.metrics.roc_auc_score discusses this in detail.


Depending samples in ad ranking and click rate prediction I am struggling with the following problem:
Suppose we fit a machine learning model to model advertisers click rates. I used a Logistic Regression approach using a one-hot/dummy encoding.

We have two advertisers A and B with a click rate of 10% and 9% respectively. Hence, we would rank A before B and would show users ads from A more often, say 90% of all times. Hence our overall click rate would be close to 10%.

Now we introduce a new feature which says a user either clicked on the advertiser before or not. For advertiser A, we observe a click rate of 8% for users that have not clicked before and a rate of 30% for users that have at least clicked once on an ad of the advertiser. For advertiser B we see no difference in the click rate for this feature.

So when it now comes to ranking the ads and we get a "fresh" user that has not clicked on both advertisers, we would rank B above A. Since fresh users are obviously more often, we would mostly show users ads from B, say 90%.

As a consequence, our overall click rate would drop from almost 10% to a little over 9%. The machine learning model does everything correct since we did not introduce any information that the two states of this new features are timely dependent.

From an intuitive approach I would still rank advertiser A over B for these fresh users since it's not clear if they're going to click.

Any ideas how to tackle this problem to get a ranking that in the end maximizes the click rate?

Let $X$ represent the event "a user clicks an ad" with advertiser X,
Let event $Y$ represent a new user ($\neg Y$ a returning user),

for any advertiser $X$ the probability of a user clicking an ad is:

$p(X) = p(X\wedge Y) + p(X\wedge \neg Y)$

$p(X) = p(X|Y)p(Y) + p(X| \neg Y)p(\neg Y)$

$p(X) = p(X|Y)p(Y) + p(X| \neg Y)(1 - p(Y))$

If it's possible to obtain the information "new user" (event $Y$) in the system, then obviously there should be two distinct rankings, one for $Y$ and one for $\neg Y$. Phrased in a ML context: $Y$ is a binary feature and the predicted best advertiser depends on it. In this scenario a different advertiser would be selected depending on whether the user is new or not, here A for returning users and B for new users.

If not possible, then the advertisers should be ranked according to their overall probability $p(X)$, which depends on the probability of a new user $Y$. Based only on your initial values 10% vs. 9%, this means selecting advertiser $A$ is optimal overall.

To illustrate how this works we can calculate the detail based on the values provided. First we calculate $p(Y)$:

$p(X) = p(X|Y)p(Y) + p(X| \neg Y) - p(X| \neg Y)p(Y)$

$p(X)-p(X| \neg Y) = (p(X|Y)- p(X| \neg Y))p(Y) $

$(p(X)-p(X| \neg Y)) / (p(X|Y)- p(X| \neg Y)) = p(Y) $

We have $p(A)=0.1$, $p(A|Y) = 0.08$ and $p(A| \neg Y) = 0.3$, so we can derive $p(Y)=0.91$.

What happens in this example is that the high value $p(A| \neg Y) = 0.3$ makes up for
the low $p(A|Y) = 0.08$ despite $p(Y)=0.91$:

A: $0.91*0.08+0.09*0.3 = 0.1$
B: $0.91*0.09+0.09*0.09 = 0.09$

Of course this would be different if $p(A| \neg Y)$ was 0.15 instead of 0.3 or if $p(A|  Y)$ was 0.11 instead of 0.09, etc.

[edit based on comments by OP]

Indeed it seems that a more accurate representation of the problem would take into account the probability that a user clicks again (and again, and again...) on an ad.

This reminds me of classical probability exercises such as the Russian roulette game, the tree of possible outcomes is somewhat similar:

user doesn't click on the ad at all: $n=0$
user clicks on the ad at least once

user never clicks again: $n=1$
user clicks on the ad at least twice

user never clicks again: $n=2$
user clicks on the ad at least 3 times

user never clicks again: $n=3$
.......

So the goal would be to represent as accurately as possible the random variable $n$ which represents how many times a user clicks on an ad, and to use the expected value of $n$ (that is, how many times a user clicks on average) to rank the advertisers.

I'm guessing that this can get pretty sophisticated depending on how we choose to represent the problem. I'm probably not able to do anything sophisticated so I'll stick to a simple representation where I assume that there are two probabilities involved:

initial probability $F=p(n\geq 1)$ for a new user to click
probability for a user to click again after having clicked $m$ times: $G=p(n\geq m+1|n\geq m)$ for $m>0$.

This representation has the advantage to be very close to the original probabilities in the question: new user clicks and returning user clicks.
Notice that the model assumes that the probability to click again is a constant, which is unlikely in reality. Also the model doesn't take into account any other features, such as for how long a user hasn't clicked. I'll  just assume that one estimate these probabilities from a training set for the sake of simplicity.

Now let's calculate the expected value [fair warning, I might get it wrong!]

$p(n=0) = 1 - F$
$p(n=1) = F \times (1-G)$
$p(n=2) = F \times G \times (1-G)$
$p(n=3) = F \times G  \times G \times (1-G)$
...
$p(n=i) = F \times G^{i-1} \times (1-G)$

Therefore the expected value is:

$\sum_{i\geq 0} (p(n=i) \times i) = \sum_{i\geq 1} (F \times G^{i-1} \times (1-G) \times i)$

[Ok so normally now we can plug in the values for advertisers A and B. The problem is that I just calculated the result for the first few values and it doesn't really work... I probably made a mistake somewhere, or maybe the values don't represent the same thing... Anyway I hope this gives you the idea, maybe I'll give it another try later]


Apply function on every four rows Let's say I have a dataset like thisX<-matrix(rnorm(30), nrow=100, ncol=6). I am trying to find a way to apply the sum function to every four rows of column 3. This means I want to sum rows 1-4, rows 4-8 and so on.

Are there any functions such as apply or lapply I could use?


KFold cross validation ambiguity I just studied K-Fold cross validation technique for finding model parameters and something seemed to be very confusing. Every tutorial I follow says that for K-Fold validation, the whole dataset will be split into K portions and K models will be fit with one portion as validation dataset each time. So, my doubt is that this method will generate K models. Which model are we supposed to use during actual inference? Or is it that same model will be trained K times with different portion of data as hold-out set?

What you want to achieve with this validation strategy is a robust estimate of what combination of hyperparameters is good enough for your final model, so:

for each combination of hyperparameters, you carry out k trainings (as you ask with your last question), following this schema:

source of info

once you have this k-trained models (i.e. for one hyperparams combination) you find the mean and standard deviation of the k sub models

you repeat this process for all the hyperparameters combinations you wnat to try out

Once you select the model hyperparams which provided the best mean value (and std) of the desired evaluation metric, you retrain with all the training dataset (the green section in the image), and evaluate it with the never-seen-before blue test data.
Luckily for you, this whole process is automated with helpers like thos one from scikit-learn, which already retrains the final model for you, and is accessible via the best_estimator_ attribute.
If you want a bit more detailed documentation about this cross-validation, you can have a look at this link.


Cost functions penalizing certain types of misclassification more heavily in tensorflow I have 2 classes of data A and C.   I want to create a NN with 3 output nodes, classifying data as either A, B, or C.   Classifying an A or C correctly should have zero cost (very good).  Classifying an A as a C or vice versa should have high cost (very bad).  However, for samples that are "blurry", I'd like to classify them as B, which would have relatively low cost (better than a misclassification, but still not ideal).   I am new to tensorflow, and am not comfortable with the different ways to define cost functions.   Can anyone point me in the right direction on this one?  Thanks in advance

You are talking about having different missclassification costs.

Classification consists of outputing a continuous score which is then thresholded to decide the discrete classification. The threshold (ex. probability>0.5 leads to C) is chosen to satisfy your apetite on sensitivty/specificity.

The training algorithm only cares about your continuous score. This means that you can make it easy for yourself by doing simple binary classification (typically with softmax_cross_entropy_with_logits-loss) and create your vague class B after training is done by thresholding say [0,0.4) is A, [0.4,0.6] is B and (0.6,1] is C.

To achieve exactly what you're saying, penalizing certain errors more, then you want to use the weighted_cross_entropy_with_logits. This throws out the interpretation of your outputted score as a probability, and you will need to work more to get a reasonable threshold afterwards so this is not recommended as a first step as it will probably add more complexity than results.

I also recommend reading the wikipedia article on roc analysis


Why MADDPG rather than taking all cooperating agents as a single meta-agent? Since MADDPG uses a centralized critic for training, why not simply treat all cooperating agents as a single meta-agent with a concatenated observation space and a concatenated action space? In my opinion, MADDPG is centralized enough, so it won't hurt to go one step further.

MADDPG can be used to model agents that have limited observation and communication capabilities after training, which is an interesting and useful real world scenario.

why not simply treat all cooperating agents as a single meta-agent with a concatenated observation space and a concatenated action space?

Any real world implementation will then require resources to provide and manage that overview. This may not be practical or desirable in all cases.
There is no single fix for this, it is an open area of research. Whether to invest in better communication and central processing, or better autonomy for multiple agents is likely to have different answers depending on the problem and current technology limits for either approach.
MADDPG reduces the role of central processing to assessment of global reward signals during training. That means:

Each agent works with local signals only, and simpler observation and action spaces as a result. Only the reward signal processing is handled externally.

Trained agents can theoretically be used in environments where a central processor is not available.

So, for example, agents can be trained in simulation with all the oversight that allows, or with a carefully instrumented environment including high bandwidth connections between agents and central processing. They can then be deployed into matching environments where the central oversight is not available, or too costly.


transition to data science job hunting Transitioning from a maths background, is it possible to find a data science job, by just being able to write my own code (kmeans, kNNs, logistic regression etc) without using any popular commercial software?

The basic answer is yes, what you're describing is possible. But the real question is should you?

I'd like to hear a little bit more about your motivations for doing this. The reality is that data science requires a passion & curiosity that most people (even smart people) just don't have. They just want the money in data science.

I read my fair share of resumes on a daily basis and the most difficult part of a resume with someone strictly with a maths background is that they don't have the business knowledge that is required. For most of us working in the field, data science is not theoretical, these algorithms actually get deployed and people work with them. More importantly, they work to solve core business issues/use cases. Most of the applied data science out there (i.e. projects that pay you) will start with a business use case.

Can you effectively stand in front of a room of executives that haven't taken a math course in decades and explain what you want to do without any formulas or any jargon that will lose the audience? Can you effectively work across multiple teams to create all the infrastructure you will need (data pipelines, storage, processing power, etc)?

From there, you also have to "sell" your solution and implementation because, since it's a use case, you may be asking a large corporation to completely shift their approach from one direction to another. Yes, you know the algorithm works. But to them, you're saying, "turn your business over to me and let this algorithm, that you can't see or touch, run your business". Spoiler alert: you're going to get a lot of executives rolling their eyes at you - it's not an easy thing to do!

So you should be asking yourself if you are an expert in a particular business domain or what are the items that you bring to the table in a business discussion? Those are the parts, and the lack of passion for data science, that will be the biggest roadblocks to making the transition.


Using neural network for "features matching" binary classification We have a dataset of numerical features from two images and we want to check if these images match or not using only these features.
Basically we have have these columns:

fA1, fA2, ..., fA14: 14 features from image A
fB1, fB3, ..., fB14: 14 features from image B

We want to predict if image A match image B (y=1) or not (y=0). So it's a features matching problem.

The main usecase is for face recognition using this framework:
BERND HEISELE

So is there any neural networks architecture known for this situation (we have a 7 million annotated training set) ?

N.B: we don't have any images, we have only numerical features.

From what I understand, your dataset is of pairs of images and a binary classification of their pairing?

There are models using an architecture named Siamese Neural Networks which are used exactly for this task of determining similarity.

You can start by reading the following article: Learning to Compare Image Patches via Convolutional Neural Networks. They are using Siamese architecture to compare between different image patches (similar to what you are describing):


Text classification with multiple documents per labeled datapoint I have a dataset with a label TRUE or FALSE for each person, but each person has multiple documents associated with them (emails and documents).

Right now I use a Random Forest Classifier on a bag of words consisting of all words in all documents put together per person (so that I have one row with all words and a label). It performs reasonably well, but I was wondering if you guys have some suggestions about how I can use the information of separate documents.

When I try to find information about this I only encounter multi-label classification, which is the exact opposite problem: multiple labels per document, instead of multiple documents per label.

Why don't you make a person id and add this to your model?

If I understand you correctly, you do:

$$y=\beta X$$,

where each row in $X$ are combined docs per person and $y$ is a vector of true/false, right?

You could try:

$$ y= \beta X + \gamma z$$,

where each row in $X$ is only one doc now and $z$ is a vector of ids per person (so a factor).

Might be worth a try.


Decision Trees Nodes vs Leaves Definition I am having a little trouble understanding the difference between what a "Node" of a tree and a "Leaf" of a tree.

Suppose I am trying to decide the size of coffee a person may like. There are three categories: small, medium, and large based off the peoples age, height, weight, income.

So I have four predictors and 3 possible outcomes. When looking at many gradient boosting algorithms, there are parameters that can increase the number of leaves.

My understanding of this (correct me if wrong), but I will illustrate with a picture. Assuming each yes/no split is 50/50

Does increasing the number of leaves to lets say 3 leaves change it from yes/no aka 50/50 to 33/33/33? This is a little confusing to me. Thank you for any clarification.

Sam

Leaf nodes are the nodes of the tree that have no additional nodes coming off them. They don't split the data any further; they simply give a classification for examples that end up in that node. In your example tree diagram, the nodes that say 'Large', 'Medium' or 'Small' are leaf nodes. The other nodes in the tree are interchangeably called split nodes, decision nodes or internal nodes.

In gradient boosting algorithms, a number of decision trees are grown. Each tree is grown until some stopping criteria is met. One kind of stopping criteria is the maximum number of leaves in the tree. At each stage of growing a decision tree, a leaf node is turned into a split node by creating a yes/no question (we call this a binary split), and two new leaf nodes are created which correspond to each side of the split. Once the total number of leaf nodes in the tree reaches the limiting value, the tree building algorithm stops and begins building the next tree.


Is it a best practice to exclude retweets from the data set? I am going to build machine learning algorithm to identify fake tweets. The data set has huge retweets which I think might be an issue. Do you think given that the focus is the original tweet, it is better to remove all the retweets?

Thank you,

No. I do not believe so and I can explain a few reasons why.

If an entity wants to create waves in twitter with false tweets retweets are probably apart of the plan.
If you want to detect tweets generated by bots looking at the statistical data on said tweets and retweets like time stamps could be relevant to detecting if the tweet is generated by a bot.
If You have a way of checking retweets by bots then removing all retweets would also remove that data.

You should remove retweets if.

The project is focused on analysis of text to determine if a tweet is bot or not.
There is no labeled human or bot retweet data.


Using packages such as sklearn vs building ML algorithm from scratch I have been using different machine learning algorithms throughout various projects at university, and attended some inspirational lectures where industrial companies show and present how they use machine learning, data mining, etc. in their work. I myself mostly use Python, and have previously used libraries such as sklearn. My problem is, that I have a huge difficulty understanding the role of built in algorithms vs making them completely from scratch with pure coding and math - i.e. using theoretical machine learning tools to actually do the work yourself. I understand doing everything yourself can be constrained by time/money/resources. Also sometimes it doesn't make sense to reinvent stuff that has been vastly optimized by others before you.

I keep feeling that using sklearn's built in random forest classifier or using xgboost in python is kind of cheating. I am only preparing the data, cleaning it to get the right formats, maybe do some feature engineering og initial plottings and statistical analysis. The problem is that when all that is done, we simply feed the data to this pre-made algorithm and it does everything behind the scene, and just spits out predictions. I feel that i am not doing anything, and not using all the knowledge i have learned in the data exploration analysis. Neither am i using any of the patterns that i found in the data. Still i hear from big companies that they use xgboost and sklearn - and I can see it actively being used in Kaggle competitions.

Almost every website i find only provides examples using these built in libraries, and do not go through any deeper math or statistics at all. I really enjoy working with machine learning - but i have this strong feeling that im completely missing the "professional" approach of doing things. I know there is a lot of books on theoretical machine learning - but still almost everyone online seems to just use pre-made algorithms. I have been struggling with this understanding for about a year now. The validity of these pre-made algorithms in serious industrial/business/academic use is still not clear to me.

EDIT: To be more specific. My question is: How are these libraries/tools viewed in professional/industrial/academical context in comparison to actually building a model yourself. Are they just a "quick and easy" alternative way to start learning machine learning and data mining for students and amateurs, or are they in fact more powerful (than i at least know) and should not be seen as an alternative, but a viable solution for professionals?

The motivation for my single question above, can be ellaborated by explaining the questions i ask myself. The very questions that began this confusion for me. Is it cheating to use these models? In which situations would you use a pre-built library, and when to avoid it? How do i merge (or use) the knowledge gained from the scientific data analysis i did before modelling, and these pre-built classifiers.

It depends entirely on your goal.

Student Phase  When you're learning about machine learning algorithms, I think it is a really good idea to implement toy examples.  I find this process helps with finding what you understood well and what you did not understand as well as you thought.  It's doing this work where you'll find deeper understanding of how the algorithm really works and different internal choices you have to make.

Professional Phase  When you have a project to deliver, you don't necessary want to rewrite a random forest implementation from scratch.  Even if you could build one in a reasonable amount of time, there's value in having something like sklearn that is well vetted and robust enough to handle edge cases that you wouldn't even consider.  That's the advantage of using pre-built libraries.

I need more Phase  Eventually, you get the point where you understand the math and know how to use the packages well and you realize that there's a feature lacking.  That's when you rip open a framework like xgboost or sklearn and modify existing code or even create you own implementation.  The reason you would do this is because methods are cutting edge so there just isn't anything out there or the implementation of the framework is actually a handicap in production (as I tend to find with sklearn).

The issue you seem to be facing is lack of accountability for your output.  If all you do is clean data push it through a model and get good results, make a chart showing results, I would say you are forgetting the "science" part in data science.

The challenging part isn't using the model it is knowing what moves your model and the potential hurdles your model may face in the real world.  I've seen this play out often in my career, where a junior member will make an awesome model on training and test dataset, and suddenly production comes along and performance tanks.  Why would that happen?  Well, because test and training were similar (and often from the same source), but the junior member failed to question the lack of variety in the data source and question if the real world behaved the same way.

What I am trying to say is, being a data scientist, a small part of the job is cleaning data, running model and making pretty pictures.  The real challenge is asking the why question.  Why does this work?  Why does the model perform poorly?  Why does the model perform well?  Why is this feature important?


classification of groups of images I have been looking through research and cannot find too much on this topic. I believe that I might be searching for the wrong terms.

What I am trying to find is any literature on classifying a group of correlated and different images into a single class. Such that each image might have its own probability of being in a class but an overall meta probability is calculated for the entire group of images.

Almost as though additional images were used to either increase or lower the probability of belonging to a class.

This paper seems interesting:
"Learning Deep Feature Fusion for Group Images Classification"


No target variable in my data I have a list of transactions of bus route from place to place, I don't have any target variable here. I was asked to give meaningful insights from the data, what can I do here?

I cleaned the data, replaced missing values in distance, price. Doing only descriptive analytics, what predictive analytics can we give here?

Here is a sample of my data,

I didn't have enough room in the comments so I'm responding here.

I don't think you'll provide anything useful if you build models to predict any of the columns in your sample.

However, your descriptive analytics should offer a lot of insight, for example, you can say which type of vehicle is more popular for a certain purpose in a certain city and so much more. So while you don't actually predict anything, you're showing the likelihood that something happens for a specific purpose. This itself is useful for business decisions.


Data Science Methodologies What are the best known Data Science Methodologies today? By methodology I mean a step-by-step phased process that can be used for framing guidance, although I will be grateful for something close too.

To help clarify, there are methodologies in the programming world, like Extreme Programming, Feature Driven Development, Unified Process, and many more. I am looking for their equivalents, if they exist.

A google search did not turn up much, but I find it hard to believe there is nothing out there. Any ideas?

Can you elaborate what you mean by 'methodologies'?

In the meantime, take a look at The Field Guide To Data Science by Booz Allen Hamilton. This guide talks about data science processes and frameworks.

Data Science Design Patterns by Mosaic talks about, you guessed it, data science design patterns. This is quite useful to get a sense of common design patterns. They are also working on releasing a book on the same subject.

Then there are several resources out there that will come up as results to more targeted searches, such as machine learning paradigms, recommender systems paradigms, etc. Data Science is a large and varied field, and you'll find many resources out there for each subsection of it. As far as I know, there isn't one book that covers it all.


Spatial Transformer Networks: how is theta differentiable? In the paper Spatial Transformer Networks, the localization network's output, theta, is differentiable, given the current input feature map. How is this theta differentiable?

In spatial transformer networks, basically, the concept of localisation network is to learn to apply a transformation to find the canonical form of the input. Imagine the output of the network $\theta$ as an activation which is passed to another layer. The point is that the sampling sequence of operations is differentiable. $\theta$ is just an output which specifies how the sampling should be performed. The sampling operation that is usually used is bilinear interpolation which although is not differentiable at all points due to the floor and ceiling functions, it can backpropagate the error and is differentiable in most of its inputs. Consider the $\theta$ just as activation which is passed to the bilinear sampler for changing the input of the next network. bilinear sampling is considered to be differentiable.

To understand it better, consider the following figure which illustrates the process inside a spatial transformer easier than the one in the original paper.

As it is clear, the output of the localisation network which is $\theta$ will be passed to the sampling grid. Sampling gird will be multiplied to the $\theta$ to find appropriate regions in the original image. Consider that you don't multiply $\theta$ to the original image. The reason is that if you multiply by the original image, there will be multiple choices for a single pixel while if you multiply the output of the localisation network by the sampling grid, for each entry there is just a single choice. Next, the sampled grid and the original image will be used in the interpolation to find the transformed image. As it is clear, $\theta$ is like the other activations.


How to improve naive Bayes multiclass classification accuracy? I have around 9 string features which I have indexed using string indexer and used vector assembler to get the feature vector and used a normalizer to normalize across features . These are the transformation I have made and used naive Bayes classifier as the estimator , all the above operations after enclosed in a pipeline model. I trained on around 3 lakh samples and tested on 20k samples  the model predicted same label for all of them. I handpicked the features is that the problem? Or any tuning parameters available ? I found out a smoothing parameter which I set it to 1.0 should I increased or decrease to improve the model accuracy? Please help

If you are hand-coding, and found that you got the same prediction for everything in your test set, it is possible that you are multiplying feature probabilities until you hit the floating point limitations of your environment, ending up with a zero value, which probably matches one of your labels.

e.g., $0.001 * 0.0001 * 0.0002 * 0.0003 ... = 0$

Try adding log values instead.

e.g., $log(0.001) + log(0.0001) + log(0.0002) ... $


Selecting most important features for multilinear regression I have a set of 25 features. I would like to choose the best features for my model. Originally, I was looking at the correlation of features with respect to response, and only taking those which are highly correlated and run a regression model. Then, using that model I would predict the outcome based on test data, and compare to actual (metric RMSE) and this would be how I assess it.
I could then add each feature in order of decreasing correlation with response to the feature set and keep calculating above.
Is there any other way I could select features? Could I e.g. run a random forest and use feature importance report from that to also select most important features? Then run regression?
What is the best way to compare each regression model to the next? There are so many metrics: AIC, BIC, ADJ $R^2$ I am confused as to which one is most simplest way to compare... in fact MSE is not even given in the sm.OLS function (stats models in python) summary:

Be careful choosing features based on correlation! Yes it is true that features that are correlated with the response variable may be good predictors, however if the features are correlated with each other then you are introducing multicollinearity into your model, which is bad.
If you want to avoid this you should choose features which are correlated with your response variable but not with each other. For example...
y = a + b + c
Assume:
y is correlated with all three (a, b, c)
a is correlated with b
c is not correlated with a and b
You should only use one of either a or b and c to predict y.
With regard to choosing which model is best, you should use a combination of metrics. As you increase the number of features your R^2 will tend to increase regardless of model performance. AIC and BIC can be used only to compare similar models and the "best" model according to AIC/BIC will be the one with the lowest score relative to the other models.
In my opinion RMSE is the best indicator of a good model, but you should also evaluate R^2, AIC, and BIC.
If you are unsure of which features to use I suggest you try stepwise regression to evaluate many models quickly and (maybe) find the best one.


R packages: How to access csv files in data subfolder? I have successfully written an R package and want to ship it with a specific csv file. I placed the file in the data and data-raw subfolders.

read.csv("data/foobar.csv")

The above command fails. How can I read the csv file?

data-raw is for storing data alongside a short R script that will do the conversion to R data for the user, and the user will just use the data() function.[source] Alternatively, if you want the raw CSV to be user-accessible, I think you need to use the extdata folder, as documented here. Then the user can get the actual path to the file on their system, after package installation, with system.file("extdata", ..., package = "mypackage"). Then, finally, they can feed that path to read.csv() with whatever options they like.


Generating data that look alike my original data If I have a set of N data, each individual data has 4 features. I do not know a priori the relations that could exist (or don't exist) between the features.

Is it possible to generate new data from my initial set that will respect the implicit relations that could exist between the features without having to find explicitly these relations ?

What you want is a generative model.

A simple generative model from the deep learning family are autoencoders: neural networks that receive as input your data and are trained to output that very same data.

There are different types of autoencoders. One of the most simple are contractive autoencoders, that have a bottle neck layer, that is, a layer with very few units. For your case, with only 4 input features, you may have a 2 unit (or even 1 unit, you may try to tune this hyperparameter) hidden layer as bottleneck.

Once fully trained, you only take the part of the autoencoder from the bottleneck to the output, and feed it with random numbers as input, and expect to get as output data that follows the same distribution as the original inputs.

The idea is that the training has allowed the net to learn representations of the input data distributions in the form of latent variables.

Depending on the distribution of your input data, a simple contractive autoencoder may not be able to properly learn good representations. More advanced variants include denoising autoencoders, sparse autoencoders and variational autoencoders.

Other generative models that are currently very trendy are Generative Adversarial Networks.


Do numerical inaccuracies play any role in training neural networks? Are there publications which mention numerical problems in neural network optimization?

(Blog posts, articles, workshop notes, lecture notes, books - anything?)

Background of the question

I've recently had a strange phenomenon: When I trained a convolutional network on the GTSRB dataset with a given script on my machine, it got state of the art results (99.9% test accuracy). 10 times. No outlier. When I used the same scripts on another machine, I got much worse results (~ 80% test accuracy or so, 10 times, no outliers). I thought that I probably didn't use the same scripts and as it was not important for my publication I just removed all results of that dataset. I thought I probably made a mistake one one of the machines (e.g. using different pre-processed data) and I couldn't find out where the mistake happened.

Now a friend wrote me that he has a network, a training script and a dataset which converges on machine A but does not converge on machine B. Exactly the same setup (a fully connected network trained as an autoencoder).

I have only one guess what might happen: The machines have different hardware. It might be possible that Tensorflow uses different algorithms for matrix multiplication / gradient calculation. Their numeric properties might be different. Those differences might cause one machine to be able to optimize the network while the other can't.

Of course, this needs further investigation. But no matter what is happening in those two cases, I think this question is interesting. Intuitively, I would say that numeric issues should not be important as sharp minima are not desired anyway and differences in one multiplication are less important than the update of the next step.

Are there publications which mention numerical problems in neural network optimization?

Of course, there has been a lot of research on vanishing gradients, which is entirely a numerical problem. There is also a fair amount of research of training with low precision operations, but the result is surprising: reduced floating point precision doesn't seem to affect neural network training. This means that precision loss is pretty unlikely to be the cause of this phenomenon.

Still, the environment can affect the computation (as suggested in the comments):

Most obviously, random-number generator. Use a seed in your script and try to make a reproducible result at least on a single machine. After that you can compute the summary of activations and gradients (e.g. via tf.summary in tensorflow) and compare the tensors across the machines. Clearly, basic operations such as matrix multiplication or piece-wise exponent should give very close if not identical result, no matter what hardware is used. You should be able to see if the tensors diverge immediately (which means there is another source of randomness) or gradually.
python interpreter, cuda, cudnn driver and key libraries versions (numpy, tensorflow, etc). You can go as far the same linux kernel and libc version, but I think you should expect reproducibility even without it. cudnn version is important, because the convolution is likely to be natively optimized. tensorflow is also very important, because Google rewrites the core all the time.
Environment variables, e.g. PATH, LD_LIBRARY_PATH, etc., linux configuration parameters, e.g. limits.conf, permissions.

Extra precautions:

Explicitly specify the type of each variable, don't rely on defaults.
Double check that the training / test data is identical and is read in the same order.
Does your computation use any pre-trained models? Any networking involved? Check that as well.

I would suspect hardware differences the last: it's an extraordinary case if the high-level computation without explicit concurrency leads to different results (except floating point precision differencies) depending on the number of cores or cache size.


How to fill NaN based on the previous row that contains a specific value I would like to fill the missing data using the value from the previous trading day of the same stock. In this example, the AAPL stock should be 100.
I've tried with fillna but I am not able to pick a specific row based on the stock.
My data:
date       stock   price
22/12/20   MSFT    87
22/12/20   AAPL    99
22/12/20   FCA     81
23/12/20   MSFT    90
23/12/20   AAPL    100
23/12/20   FCA     80
24/12/20   MSFT    91
24/12/20   AAPL   *NaN*
24/12/20   FCA     78

You can groupby by 'stock', forward fill missing values with ffill and use its result in fillna. For example:
date       stock   price
22/12/20   MSFT    87
22/12/20   AAPL    99
22/12/20   FCA     81
23/12/20   MSFT    NaN
23/12/20   AAPL    100
23/12/20   FCA     80
24/12/20   MSFT    91
24/12/20   AAPL    NaN
24/12/20   FCA     NaN

df.fillna(df.groupby('stock').ffill())

Result:
       date stock  price
0  22/12/20  MSFT   87.0
1  22/12/20  AAPL   99.0
2  22/12/20   FCA   81.0
3  23/12/20  MSFT   87.0
4  23/12/20  AAPL  100.0
5  23/12/20   FCA   80.0
6  24/12/20  MSFT   91.0
7  24/12/20  AAPL  100.0
8  24/12/20   FCA   80.0


Should the weights for CrossEntropyLoss be exactly the inverse of the propotions of training data? I have a classifier network which chooses one of three classifications, and uses cross entropy loss as the loss function.  If the proportions of training data are 100:10:5 for each classification, should I automatically set the weights to 1/100,1/10,1/5 ?
If not, what other issues are there to consider?

I'll refer to an answer for a similar topic. If you have enough data for classes 2 and 3, there's no reason to change your training scheme if you use standard metrics. The baseline should always be training without changing the weights, and if you see that the model does very bad on classes 2 and 3, you can change the training scheme. However, I have rarely seen it work better.


How to know if a model is overfitting or underfitting by looking at graph Just recently got my hands on tensorboard, but can you tell me what features should I look for in the graph (Accuracy and Validation Accuracy)
And please do enlighten me about the concept of underfitting as well.

Overfitting is a scenario where your model performs well on training data but performs poorly on data not seen during training.  This basically means that your model has memorized the training data instead of learning the relationships between features and labels.

If you are familiar with the bias/variance tradeoff, then you can think of overfitting as a situation where your model has high variance, memorizing the random noise in the training set.

Overfitting is easy to diagnose with the accuracy visualizations you have available.  If "Accuracy" (measured against the training set) is very good and "Validation Accuracy" (measured against a validation set) is not as good, then your model is overfitting.

Underfitting is the opposite counterpart of overfitting wherein your model exhibits high bias.  This situation can occur when your model is not sufficiently complex to capture the relationship between features and labels (or if your model is too strictly regularized).

Underfitting is a bit harder to diagnose.  If Accuracy and Validation Accuracy are similar but are both poor, then you may be underfitting.

Edit 1:

Strategies to avoid overfitting/underfitting

Recall that overfitting is caused by the model memorizing the training data instead of learning the more-general mapping from features to labels.
This commonly occurs when training a model with so many parameters that it can fit nearly any dataset.  As von Neumann so eloquently put it, "With four parameters I can fit an elephant, and with five I can make him wiggle his trunk."

You can combat overfitting by reducing the complexity of your model (i.e. reducing the number of trainable parameters).  The specifics of how this is accomplished vary depending on the learning algorithm and the domain.

For neural networks, you can use fewer layers (shallower networks), fewer neurons per layer, sparser connections between the layers (as in convolutional nets), or regularization techniques like dropout.

In the same vein, you can combat underfitting by increasing the complexity of your model.  This has been the driving force behind the push for ever-deeper neural networks in recent years.  With more layers, the network can learn more sophisticated relationships and perhaps perform well on difficult real-world tasks.

Of course, you can't recklessly add layers to a network and expect great performance.  Training deep neural networks is hard, for a number of statistical and technical reasons (one of which is avoiding overfitting).

So to answer your question directly:  If your network is overfitting, adding more layers will almost certainly make the problem worse, since you're increasing model complexity.  If your network is underfitting, adding more layers can help, but it's rarely so straightforward.  You need to think carefully about how you expect the network to operate and what strategies you can employ to ensure that it doesn't begin to overfit.

Edit 2:

P.S. If you're new to the field of machine learning, then it may be helpful to experiment with more intuitive models than neural nets while learning about over/underfitting and the bias/variance tradeoff.

I would recommend decision trees (or model trees for regression problems).  Tree-based models are easy to interpret, and playing around with parameters like the max depth and minimum impurity decrease of a decision tree might help you gain some intuition about the relationship of model complexity to overfitting.


LSTM querying approach I've just realized my prediction approach for LSTM might not be correct.

I am trying to predict character by character, by reading over the book. The way I've approached the problem is as follows:

   b                                    c                d                e
   ^     carry cell state forward       ^                ^                ^
LSTM_t0  ------------------------->  LSTM_t1  ----->  LSTM_t2  ----->  LSTM_t3
   ^                                    ^                ^                ^
   a                                    b                c                d

This means I have 4 timesteps, at at each one I feed next letter into LSTM, expecting it to immediately predict the next letter.

Should I instead do this:

ignore          ignore           ignore             e
  ^               ^                ^                ^
LSTM_t0  ---->  LSTM_t1  ----->  LSTM_t2  ----->  LSTM_t3
  ^               ^                ^                ^
  a               b                c                d

In the first case, I am able to get 4 loss-values, but in the second example, I only have 1 source of gradient, at _t3

My main concern is in first example, I demand LSTM to make prediction of 'b' and 'c' without supplying it enough previous context.  It's fine for 'd' and 'e', but asking for answer at timestep 0 and 1 is a bit unfair?

What would be best for this particular example?

Your first example is basically not a sequential model. You have an input and an output and that's it. You don't need a recurrent layer for that...

What I would suggest you do is:

    e               f               g
    ^               ^               ^
 LSTM_t1  ---->  LSTM_t2  -----> LSTM_t3  ----->  ...
    ^               ^               ^
  abcd            bcde            cdef

Obviously, you will lose the ability of predict the first characters but hopefully your dataset will include a lot more examples using those specific characters so it will learn how to use them eventually. I would say this is the most general way of producing char to char text generation models. A couple of examples including implementations: here and here.


What is Coarse-to-Fine in the context of neural networks? I read in many paper that mentions coarse-to-fine as a technique in deep learning, but I could never figure what exactly they mean. Is it related to multiscale inference, where they use coarse and fine input images?

"Coarse to Fine" usually refers to the hyperparameter optimization of a neural network during which you would like to try out different combinations of the hyperparameters and evaluate the performance of the network.

However, due to the large number of parameters AND the big range of their values, it is almost impossible to check all the available combinations. For that reason, you usually discretize the available value range of each parameter into a "coarse" grid of values (i.e. val = 5,6,7,8,9) to estimate the effect of increasing or decreasing the value of that parameter. After selecting the value that seems most promising/meaningful (i.e. val = 6), you perform a "finer" search around it (i.e. val = 5.8, 5.9, 6.0, 6.1, 6.2) to optimize even further.


Which machine learning algorithms are hard real time capable? Are there any machine learning algorithms or algorithms categories which have predictable execution times (guaranteed minimal/maximal execution time e.g. when the algorithm is run on a single processor which is not loaded with other processing tasks) for predictions given any possible input making them suitable for the use in hard real time systems?

making them suitable for the use in hard real time systems

Not sure what you mean by "hard" real time systems, but what you describe is basically the context of data streams. That is a whole branch with its own algorithms. Since you don't specify much, I leave the answer here, pointing out what kind of algorithms you should research.

Hope this helps!


Why L2 norm in AdaGrad update equation not L1? The update equation of AdaGrad is as follows:

I understand that sparse features have small updates and this is a problem. I understand that the idea of AdaGrad is to make the update speed (learning rate) of a parameter is inversely proportional to the update history of that parameter ($\eta$ / sum of previous updates). This is independent of the other parameters. This makes the small number of updates done to sparse features have a higher learning rate than dense ones.
My question is about how to implement this in the above equation. Why are we summing the squares of update history and getting its square root? I understand that we need to get rid of the negative sign. So, why not directly summing the absolute values?

The reason is because the article introducing the method, which can be found here, has proved bounds on the regret function using inequalities about the L2 norm.
The square is also explained by the fact that is the diagonal of some positive quadratic form.
However I don't know if anyone has ever considered using AdaGrag with L1 norm instead of L2.


how to iterate over certain rows of a certain column an update their value. by applying condition I want to update the value of opening stock and closing stock by adding 20 when dcsDep2 < 3.028512.
 here is the snapshot of dataset:

I tried with this code:

for(index_label,row_series) in dep1.iterrows():
    while row_series["Closing Stock"]<3.028512:
            dep1.at[index_label,"Closing Stock"]=row_series["Closing Stock"]+22

but no output is shown.

You can use df.loc to make the selection and then set the values. Take a look at this example and modify according to your dataset.

In [1]: import pandas as pd

In [2]: df = pd.DataFrame({'a':[1,2,3], 'b':[0.1,0.3,0.9], 'c':[4,5,6]})

In [3]: df
Out[3]:
   a    b  c
0  1  0.1  4
1  2  0.3  5
2  3  0.9  6

In [4]: df.loc[df['b'] < 0.5, ['a', 'c']] += 20

In [5]: df
Out[5]:
    a    b   c
0  21  0.1  24
1  22  0.3  25
2   3  0.9   6

Also, in future try to provide a sample from your dataset as a table of values and not an image so that it becomes easier to reproduce an example.


Finding Correlations in two Datasets I have an assignment where I am trying to find correlations between Lightning Strikes and Telecommunication damage. The two datasets consist of many columns (especially the human-recorded Telecommunication damage one), but let's assume it was something like this :
Telecommunication_Damage_df.columns = (timestamp, geolocation(lat, lon), type_of_damage) and
Lightning_Strikes.columns           = (timestamp, geolocation(lat, lon))
I have done some EDA and cleaned the data, also assigned each lightning strike and each telecommunication damage row to a particular location(cities/areas), but now I am confused as to how I should proceed. Every other Data Science / Machine Learning project I have been involved in was much more direct and usually had one training/testing dataset whereas with this one I am stuck as to how I should proceed, is there a model that could help? Is there a methodology I am unfamiliar with?
I tried following this crime/geolocation tutorial (https://www.kdnuggets.com/2020/02/introduction-geographical-time-series-crime-r-sql-tableau.html) but it's not exactly the same, because there is one dataset being used, crimes, if a crime occurred then that's it, whereas here if a lightning strike occurred that doesn't necessarily mean a telecommunication problem was found and vice versa.
I know it's a bit vague, but I've been stuck for quite a while, and I was hoping that someone could guide me to any direction, because at the moment I am idle.

About the type of problem: apparently here the goal is not to do any kind of supervised learning, it seems to be more a kind of descriptive task.
The first thing I would try to do is to align the two datasets, i.e. merge them based on events in one occuring at the same time and place as in the other one. In order to do that you need to be able to compute for any two events whether they appear in the same area around the same time. I guess you'll need a window, for example defining two events as related if they occur within 1h of each other and are distant by less than 10km. Be careful about noise in the data: the time of a telecom incident might be the time it's reported, not the time it actually happens.
Once you have linked potential related events, you can merge the two datasets into one. You should probably preserve the events which have no match in the other dataset for statistical purposes, so use an outer join.
At this stage you have an exploitable dataset. I would start by doing plots such as overall probability distribution of related events, probability of related event by location. Probability of related events across time.


How prevalent is `C/C++` in machine learning development? I am currently a data scientist mostly doing NLP, and I do most of my work inPython. Since I didn't get a CS degree in undergrad, I've been limited to very high level languages; Java, Python, and R. I somehow even took Data Structures and Algorithms avoiding C or C++.

I'm intending to go to graduate school to study more Natural Language Processing, and I'm wondering how much C/C++ I need to know. Deep-learning frameworks like PyTorch or Tensorflow are written in C++, and CUDA is only available in C. I'm not going to be writing Cython libraries, but I would like to do research and build new models (i.e. like "inventing" CNN's, seq2seq models, transformers).

I don't know how much C/C++ is used, and I'm unsure if it's worth learning the language-specific complexities that may be channeled into learning something else; hopefully somebody can let me know how prevalent the use of C/C++ is?

As you already understand, the vast majority of the data science work is made with rather high level languages such as Python and R. So it's not a matter of prevalence, it's a matter of which part in the big world of data science you want/can do with your skills and your tools.

Imho inventing new models requires:

strong theoretical background in maths and statistics, in-depth knowledge of existing estimation/inference methods
good understanding of computational complexity and (preferably) algorithmic optimization methods.

If additionally you implement your models yourself (that's not necessarily the case), that's where you probably need to know low level languages such as C/C++, because computational efficiency is crucial when people are going to use the model with massive datasets which require a lot of computation.


Is K-NN applicable for binary variables? I need help because I'm just new to machine learning and I do not know if k-nearest neighbors algorithm can be used to identify the appropriate program(s) for Student 11 in the table below.

The school subjects (Math, English, etc.) are the features, while the 'Program' column has the class labels A, B, and C.
The binary values represent the interest of a student in the subjects.
Can K-NN algorithm find the similar students based on the binary variables in the table?

Yes, you just have to find a suitable distance metric, instead of using the default Euclidean distance. Euclidean distance will work, but it loses a lot of its positive points when used on a non-euclidean space.

For you specific case, the Jaccard distance basically measures how many 1's are equal on both instances, ignoring the dimensions where both are 0's. This gives an interpretation like "if a program has a course, but the other doesn't, then they are dissimilar". Jaccard index is very useful in high-dimensional boolean matrices, such as generated by one-hot encodings.

Other more intuitive choice is the perfect match distance, which would simply measure how many dimensions are different for the two instances and can be easily computed by $\sum_{i=1}^n |X_i - Y_i|$. In this case, the interpretation becomes similar to "if both programs have a course, or both program don't have a course, then they're similar".

However, be careful with the K value of your K-NN. You only have two instances of class B, so you will have to choose 1 or 2 (technically at most 3) for the value of K.


Data normalisation and recommendation based on skillset Given the job title return the skillset required for the job
If a user is lacking some skills required for the job, we have to suggest the courses the user has to take to bridge the gap.

Problem with 1) is:

1) The provider is returning the skillset only if the keyword(tile of the job) is exact. For instance: For "Accountant" it gives the skillset whereas for "Accounting" it didn't give any result. One another example is: It gives result for "Developer" not for "Software Engineer" or "Software Developer"

Does NLP or machine learning solves this issue? If yes, can you give some recommendation on how this problem could be approached.

For 2): What is the way to handle recommendations?

Root words

The problem here is that language has many different ways of distorting words to give slightly more information without changing the root of the word. Your database which contains the needed courses for a specific profession should ideally be tokenized into a numerical value. Or if that is not possible you can get the stem of a word as follows

from nltk.stem.snowball import SnowballStemmer
stemmer = SnowballStemmer("english")
print(stemmer.stem("Accountant"))
print(stemmer.stem("Accounting"))

  account account

Recommendations

There's a lot of resources on how to do this, especially using deep learning in recen tyears. You can look at some of the publications from Netflix, Amazon, SoundCloud. All these groups have very powerful recommendation systems. In essence what you want to do is determine some user metrics such as click through sequence, search history. Then build a database with these features and a target such as the next item that is played. From this you can train your model to predict a set of items the user is likely to click on next given his feature set.


Log analysis dataset with labeled cybersecurity issues I am seeking to find a dataset with log files that have labeled cybersecurity issues. As I am trying to build a cybersecurity log analysis model there is no preference on the type of the log, but there is a preference on existence of known cybersecurity issues in the data.
Currently all I was able to find log datasets(HDFS, BGL) that had anomalies which were not cybersecurity issues but rather execution flow errors. Also I have found numerous amounts of network data such as in https://vizsec.org/data/, but they contain network traffic instead of logs. Also, I have found log datasets that actually had cybersecurity issues but the quantity of them were too little to train a model on.
It would also be helpful to know, how is it possible to generate such a dataset in large quantities.

In reference with your little found data either augment it or apply cross validation on top of it.
else
Look for your expected data in https://datasetsearch.research.google.com/


Choosing input variables for Linear Regression for higher accuracy I have a data set describing water levels of rivers. It has following attributes:

Value                   float64
Variable Name            object
Variable Unit Code       object
Date Pulled              object
Site Code                 int64
Site Latitude           float64
Site Longitude          float64
Site Name                object

I am using Linear Regression in trying to predict the Value using Site Code, Site Longitude and Site Latitude but getting results which are no way near the actual values. I am wondering if it's my choice of independent variables that is creating this problem or should I try other types of Regression? If it's the first case, which approaches are recommended to get higher accuracy rate or more precisely, what should one do if there are not many relevant independent variables?

I am using Linear Regression in trying to predict the Value using Site Code, Site Longitude and Site Latitude

Unfortunately none of these three variables can go directly into linear regression.

Site Code looks like a numerical variable, but it is actually categorical. For example Site Code = 10 is probably orthogonal to Site Code = 5, and should not be interpreted as twice as significant. The correct way to handle this is to create a boolean dummy indicator variable for each possible site code. You can also use this method for some of your other object variables which also appear to be categorical.

Latitude and Longitude will not work well in a linear regression because their relationship is highly non-linear. For example two points can have the same latitude/longitude but be very far apart. One typical method is to convert (lat,lng) pairs into predefined zones, and treat the zone as a categorical variable.

The final type of variable you have is a date. You could convert this directly to a categorical variable, but it might be better to take only the month to reduce the number of categories and generalize seasonal effects better.


Word based perplexity from char-rnn model I'm training a character based RNN model for text prediction and want to compare it to similar models. Since most literature uses word based perplexity as a performance metric, what would be the "proper" way to calculate word based perplexity from a character based model?

Actually, there is a formula which can easily convert character based PPL and word based PPL.

$PPL = 2^{(BPC*Nc/Nw)}$

where $BPC$ is character based $PPL$, $Nc$ and $Nw$ are the number of characters and words in a test set, respectively.

The formula is not completely fair, but it at least offers a way to comparing them. The following are some reference.

[1] Hwang K, Sung W. Character-Level Language Modeling with Hierarchical Recurrent Neural Networks[J]. 2016.

[2] Graves A. Generating Sequences With Recurrent Neural Networks[J]. Computer Science, 2013.

[3] T. Mikolov, I. Sutskever, A. Deoras, H. Le, S. Kombrink, and J. Cernocky.Subword language modeling with neural networks. Technical report, Un-published Manuscript, 2012.


Is it possible to cluster data according to a target? I was wondering if there exists techniques to cluster data according to a target. For example, suppose we want to find groups of customers likely to churn:

Target is churn.
We want to find clusters exhibiting the same behaviour according to the fact that they are likely to churn (or not). Therefore, variables not explaining churn behaviour should not influence how clusters are built.

I have done this analysis the following way:

Predict target (e.g. using a Random Forest) and retrieve "most important features" (from feature importance analysis).
Cluster samples with selected features (e.g. using k-means).

However, I am afraid the clustering technique used in the 2nd step might not catch behaviours found in the 1st step which might explain churn (suppose there is a complex interaction in some trees in the RF, this interaction might not be cought in the k-means algorithm).

I was thinking of another way of doing this by using a neural network:

Predict target using a neural network with several layers, and for each sample retrieve activations from a given layer.
Cluster samples with their activations.

If the performance of the neural network is good and if the layer from which activations are retrieved is carefully chosen (not too close to the input or the output layer), I suppose the clusters could show customers displaying the same behaviour explaining the target.

I did not find any articles having this approach. Did anyone deal with the same issue or have other ideas?

Use constrained clustering.

This allows you to set up "must link" and "cannot link" constraints.

Then you can cluster your data such that no cluster contains both 'churn' and 'non churn' entries bybsettingn"cannot link" constraints.

I'm just not aware of any good implementations.


Does Kernels always map data points to higher dimensions When we use kernels in SVM to linearly sperate non linear data points by mapping it to 'another dimensions', does this suitable 'another dimensions' always be a higher dimension with respect to original dimension of the data points?

And is it true that we can always find a higher dimensions that can linearly seperate data points in a training set?

'Another dimension' can be also a lower dimensional space. You are free to choose your kernel. Example is: $k\big((x_1, y_1), (x_2, y_2)\big) = x_1x_2$
Is it true that we can always find a higher dimensions that can linearly seperate data points? Yes. If you have a finite dataset, you can always find a higher dimension that can linearly seperate data points.  The obvious one is mapping n points to a n-dimensional space. If you do any non-trivial mapping (no d points are on a d-1 dimensional hyperspace), then you can always linearly separate any two class assignments.


Why is the result of CountVectorizer * TfidfVectorizer.idf_ different from TfidfVectorizer.fit_transform()? I have a dataframe:

df = pd.DataFrame({'docs': ['gamma alfa beta beta epsilon', 'beta gamma eta',], 'labels': ['alfa alfa beta', 'gamma fi']})

I use count vectorizer:

import numpy as np
import pandas as pd
from itertools import chain
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer

vocab_docs = set(chain(*[i.split() for i in df['docs'].unique()]))
cv_docs = CountVectorizer(vocabulary=vocab_docs)
cv_docs_s = cv_docs.fit_transform(df['docs'])

I do TFIDF:

tfidf_docs = TfidfVectorizer(vocabulary=vocab_docs)
tfidf_docs_s = tfidf_docs.fit_transform(df['docs'])
# tfidf docs
tfidf_docs_s = tfidf_docs_s.todense()

but I see that the results are different:

test = np.multiply(cv_docs_s.todense(), tfidf_docs.idf_)

test != tfidf_docs_s

Why is the result of CountVectorizer * TfidfVectorizer.idf_ different from TfidfVectorizer.fit_transform()?

TfidfVectorizer will by default normalize each row. From the documentation we can see that:

  norm : ‘l1’, ‘l2’ or None, optional (default=’l2’)
  Each output row will have unit norm, either: * ‘l2’: Sum of squares of vector elements is 1. The cosine similarity between two vectors is their dot product when l2 norm has been applied. * ‘l1’: Sum of absolute values of vector elements is 1. See preprocessing.normalize

Setting norm to None will give the result you expect:

tfidf_docs = TfidfVectorizer(vocabulary=vocab_docs, norm=None)


Data Matching Using Machine Learning I have around 4000 customer records and 6000 user records and about 3000 customer records match leaving 1000 unmatched customers. I have created a fuzzy matching algorithm using Levenshtein and Hamming and added weights to certain properties, but I want to be able to match the remaining records without manually doing this. Ideally I want to implement an algorithm to take a customer and user and output match/no match. However, wouldn't I need to train with true negatives? Is there an algorithm that can train with just 1 label?
Thanks

You can obtain one negative example by taking one of the 3000 customer records and pairing it with any user record that is known not to match.  In this way, you can obtain $3000$ positives and $3000 \times 5999$ negatives.  You could then train a boolean classifier on this entire training set.  This might work better than using one-class classification on just the positives.

Even better might be to use techniques for learning to rank.  If $c$ is a customer record that is known to match a user record $u$, and $u'$ is any other user record (which $c$ doesn't match), then you want your classifier to rank the pair $(c,u)$ higher than $(c,u')$.  In this way you can obtain $3000 \times 5999$ such ranking-pairs, and try to train a classifier to learn to rank, then use that to find the best match for each of the 1000 customer records.


Feature Selection for K Nearest Neighbour and Decision Trees I have 2 digits numbers and 9 features.

I must pick 2 features, so decided to plot the features against each other to see whether I can get any insight on the best features to train my algorithm.

The plot colours indicates the two digits.

Algorithms I considered to use were, K-Nearest Neighbour and Decision Tree.
I am very new to machine learning, I chose these two algorithms simply because I have come across them.

Feature matrix of f1 to f9 against f1 to f9

Decision Tree decision boundary

I have a few questions:

Will choosing the feature x against feature y with the least amount of overlap will help achieve the optimal decision boundary?

When I look at features should I initially consider linear data separation.
Then work my way up to use an algorithm that can deal with non-linear separated feature points?

What important visual properties should I look out for when choosing optimal features for training?

How can I visualise the tree in sklearn python?

Thanks.

Like others said Good visual split is a good starting point. (To me it seems the f8-f1 is a good starting point.
However you could get better results by transforming the feature set via PCA and using top eigen factors(new combination features) to train.
You've not clarified whether it is a supervised(you know the no. of classes and actual class for some data set), assuming you don't and it is unsupervised, I'd also try an algorithm like dbscan. It is generally fast.
If you want to try the neural network approach try either sompy or neural gas


If a time series has random time events, how to detect patterns? My app receives messages with a random number of bits at a random time. But two weeks ago I started to notice some almost regular patterns on the metrics of my app. I suspect they are some bots sending artificially generated data to my app. Specifically, I'm looking for sequential subsets of messages in a time series where messages has almost the same number of bits.

I read about some methods but they use data where time is not a random variable. I appreciate any help you can provide, including books, web pages, tutorials (in Python if possible), etc.

As a first step, to segregate the messages that appear to be a bot, you could first try binning by message size. For example, if messages sent by bots are likely to be around 128 bytes to 140 bytes, assign these to a unique bin.

Next, create a time series based on this bin. Try to decompose the time series using an additive or multiplicative method such as Holt Winters. A strong seasonal component would help you identify regular and repetitive messages which are being generated automatically.


Finding levels in an array of floats Before anybody asks please let me confirm this question is related to trading technical analysis.

The problem is quite simple, but I lack the scientific background required to solve it.

Data comes as an array of float values equally spaced temporally. Some local minimums (same apply for maximums) tend to repeat over time (although not at the very same value, but in a rather tight range), because of human psychology: if some price level was important in the past, stopping the price to go below, it will tend to resurface a few times, blocking again the price from going below.

What I'm interested in is to find the most significant levels over a given past period (say 5000 time intervals). Basically the "importance" of a level is determined by the number of times it was tested (meaning the number of times the price tried go past that level and failed); however, a level tested three times at relatively large time intervals (like once each month for a daily chart) is more significant than a level tested three times in a short period (like within a week for a daily chart).

Again, data is as follows:

data comes as an array of float values
data points are equally spaced temporally (however, time is of little to no importance)
some local extremes are more important than others and tend to repeat over time

Can somebody help me with some Python code to extract important levels? Or at least point me to some theory/frameworks to learn? I know about pandas/numpy but I lack the math background required to a least identify what could be used of these...

Thank you!

You could define a window (distance between red and blue line) and slide it from top to bottom over the y-axis and then count the local minima and maxima in that slice. One way with scipy is argrelextrema(x, np.less) for minima and argrelextrema(x, np.greater) for maxima which gives you the indices of the extrema. With len(argrelextrema(x, np.less)) you would get the number of minima.

Each slice would be a level and the results of counting the minima and maxima the corresponding importance. The size of the window is up to you. If your data is very granular you might want to consider smoothing it first (e.g. with numpy.convolve()


What exactly is a Gini Index I am going through the tutorial at this site. Here, I can see the author is explaining the derivation of Gini Index. I want to understand the following terms

 Group
 Classes : As far as I have understood, it represents the possible values of labels in the data which we are supposed to classify. Please correct me if I am wrong.

The website here states that it is the difference between 1 and the probabilities of the classified values within the dataset while creating the split. But the first link does add some more points to the simple derivation. Can anyone please explain in layman terms the derivation of Gini Index?

A class is simply a label you use to categorize a bunch of objects. For example, if you were trying to create an email filter, you might have a spam class and non-spam class.

A Gini index is used in decision trees. A single decision in a decision tree is called a node, and the Gini index is a way to measure how "impure" a single node is.

Suppose you have a data set that lists several attributes for a bunch of animals and you're trying to predict if each animal is a mammal or not. You would have two classes, mammal, and not-mammal. You start making your decision tree by asking if an animal is warm blooded or not and split your data set into two groups based on this splitting criteria. If an animal is cold blooded, it belongs to the not-mammal class, however, if an animal is warm-blooded, it may or may not belong to the mammal class. This new node (e.g., decision) might contain a mix, or group, of animals that may or may not be mammals (i.e., the group could contain birds and mammals). A 50/50 split between mammals and non-mammals at this node would mean the node is impure (with a Gini index of 0.5). A completely pure node would have a Gini index of 0 and would indicate a node is made up of only 1 class.


Find the most relevant columns for each single class in pandas The following question (this one) did not help me.

I have a big dataset, and I want to know which Columns are the most relevant for the Target Variable. I know that, in my case, for each class in the Target Variable, different Columns have a different impact.

In that question, the suggested answer recomend using LDA. From what I understood, it looks like a normal classification algorithm, so it's not what I need

What I what is something like

In :
    magic_function("name_of_target_variable_1")
Out :
    ["really_important_column_a", "really_important_column_b" ...]
In :
    magic_function("name_of_target_variable_2")
Out :
    ["really_important_column_a", "really_important_column_f" ...]

How can I obtain this result? Is there a way, in the first place?

In the end, this is what I've found around the internet. Really helpful

def best_features(df,target):
    features = list(df.columns)
    features.remove(target)

    y = df[target]
    X=df[features]
    #apply SelectKBest class to extract top 10 best features
    bestfeatures = SelectKBest(score_func=chi2, k=15)
    fit = bestfeatures.fit(X,y)
    dfscores = pd.DataFrame(fit.scores_)
    dfcolumns = pd.DataFrame(X.columns)
    #concat two dataframes for better visualization
    featureScores = pd.concat([dfcolumns,dfscores],axis=1)
    featureScores.columns = ['Specs','Score']  #naming the dataframe columns
    return featureScores.nlargest(15,'Score')  #print 10 best features

def worst_features(df,target):
    features = list(df.columns)
    features.remove(target)

    y = df[target]
    X=df[features]
    #apply SelectKBest class to extract top 10 best features
    bestfeatures = SelectKBest(score_func=chi2, k='all')
    fit = bestfeatures.fit(X,y)
    dfscores = pd.DataFrame(fit.scores_)
    dfcolumns = pd.DataFrame(X.columns)
    #concat two dataframes for better visualization
    featureScores = pd.concat([dfcolumns,dfscores],axis=1)
    featureScores.columns = ['Specs','Score']  #naming the dataframe columns
    return featureScores.nsmallest(15,'Score')  #print 10 best features

def largest(df,target):
    features = list(df.columns)
    features.remove(target)

    y = df[target]
    X=df[features]
    model = ExtraTreesClassifier()
    model.fit(X,y)
    feat_importances = pd.Series(model.feature_importances_, index=X.columns)
    feat_importances.nlargest(15).plot(kind='barh')
    plt.show()
    print(feat_importances.nlargest(15).index)

def smallest(df,target):
    features = list(df.columns)
    features.remove(target)

    y = df[target]
    X=df[features]
    model = ExtraTreesClassifier()
    model.fit(X,y)
    feat_importances = pd.Series(model.feature_importances_, index=X.columns)
    feat_importances.nsmallest(15).plot(kind='barh')
    plt.show()
    return feat_importances.nsmallest(15).index.to_list()


Predicting topics for customer reviews based on topics mapped to n-grams? I have a large number of unlabelled customer review data(text column) and my objective is to classify each review to a particular topic.

Also I have a list of unigrams,bigrams and trigrams(not a part of the customer review data) where each n-gram(word column) is tagged with a particular topic(topic column).

Is there a way to classify the customer reviews to the topics associated with the n-grams?
I am mostly thinking in terms of Word2Vec and Doc2Vec approaches.

You might want to look at the Facebook starspace library - their examples are similar but with suggesting Facebook pages and hashtags.

https://github.com/facebookresearch/StarSpace


Why are autoencoders for dimension reduction symmetrical? I'm not an expert in autoencoders or neural networks by any means, so forgive me if this is a silly question.

For the purpose of dimension reduction or visualizing clusters in high dimensional data, we can use an autoencoder to create a (lossy) 2 dimensional representation by inspecting the output of the network layer with 2 nodes. For example, with the following architecture, we would inspect the output of the third layer

$[X] \rightarrow N_1=100 \rightarrow N_2=25 \rightarrow (N_3=2) \rightarrow N_4=25 \rightarrow N_5=100 \rightarrow [X]$

where $X$ is the input data and $N_l$ is the number of nodes in the $l$th layer.

Now, my question is, why do we want a symmetrical architecture? Doesn't a mirror of the deep 'compression' phase mean we might have a similarly complex 'decompression' phase resulting in a 2 node output which is not forced to be very intuitive? In other words, wouldn't having a simpler decoding phase result in the output of the layer with 2 nodes necessarily being simpler too?

My thinking here is that the less complex the decompression phase, the simpler (more linear?) the 2D representation has to be. A more complex decompression phase would allow a more complex 2D representation.

There is no specific constraint on the symmetry of an autoencoder.

At the beginning, people tended to enforce such symmetry to the maximum: not only the layers were symmetrical, but also the weights of the layers in the encoder and decoder where shared. This is not a requirement, but it allows to use certain loss functions (i.e. RBM score matching) and can act as regularization, as you effectively reduce by half the number of parameters to optimize. Nowadays, however, I think no one imposes encoder-decoder weight sharing.

About architectural symmetry, it is common to find the same number of layers, the same type of layers and the same layer sizes in encoder and decoder, but there is no need for that.

For instance, in convolutional autoencoders, in the past it was very common to find convolutional layers in the encoder and deconvolutional layers in the decoder, but now you normally see upsampling layers in the decoder because they have less artefacts problems.


Using Neural Networks to extract multiple parameters from images I want to extract parameters from an image using a neural network.

Example:

Given an image of a brick wall the NN should extract the width and height of the bricks, the color and the roughness.

I can generate images for given parameters to train the NN and want to use it to extract the parameters from an actual image.

I've looked into CNNs. Can I perform this task with them? Do I need special learning algorithms to extract multiple parameters instead of classification? Are there any NNs that are designed for such tasks?

A CNN could be a good choice for this task if you expect variation in the original image scale, rotation lighting etc, and also have a lot of training data.

The usual CNN architecture is to have convolutional layers close to the input, and fully-connected layers in the output. Those fully-connected layers can have the output arranged for different classification or regression tasks as you see fit. Predicting the values of parameters describing the image is a regression task.

If you want accurate measures of size, you may need to avoid using max pooling layers. Unfortunately, not using pooling will make your network larger and harder to train - you might get away with strided convolution instead if that is a problem for you.

If your input images are very simple and clear (because they are always computer generated), then other approaches may be more reliable. You may be able to reverse-engineer image production and derive simple rules such as identifying lines, corners, circles and other easy-to-filter image components, and make direct measurements. There may also be a middle ground in complexity where extracting this data as features and using it to train a simple NN (or other ML model) will have good performance.