what are the top level subsets/domains of ML? I'm not really happy with the mind maps I've been able to find on Google, most of them are algorithm based.  I want to make a good one that is problem/solution domain based.  Do I have this right for my top level nodes?  Here is the general direction I am headed: https://imgur.com/gallery/CugcS

My questions/doubts about what I have so far are:

Is my starting point below generally correct?  e.g. no high level subclass is missing, and everything presented as a subclass deserves to be here?
is Hybrid learning always just a combination of supervised and unsupervised?  Or, are there real examples of other hybrid models (e.g. 'reinforcement' and 'supervised', etc.).  I know theoretically we can combine any methods...I'm looking for what's real/applied/demonstrable today.
does Reinforcement learning belong at this high level, or is it actually a subset of one of the others (or one I've omitted)?

Machine Learning

1.1 Supervised (uses labelled data to train and validate)

1.2 Unsupervised (uses unlabeled data, or ignores labels if they are present)

1.3 Semi-supervised (uses partially labelled (mostly unlabeled) data)

1.4 Hybrid (combines a supervised method and an unsupervised method)

1.5 Reinforcement Learning (uses data from the environment)

Thank you!

Google (in images!) on 'machine learning cheat sheet', to find an example like this:
https://docs.microsoft.com/en-us/azure/machine-learning/studio/algorithm-cheat-sheet

So you are well on your way in identifying all kinds of techniques, but you may want to turn your plot 'sideways', e.g. two-way / multi-class classification are problem (sub)domains and result in different recommended machine learning techniques.


What is the difference between upsampling and bi-linear upsampling in a CNN? I am trying to understand this paper and am unsure of what bi-linear upsampling is. Can anyone explain this at a high-level?

In the context of image processing, upsampling is a technique for increasing the size of an image.

For example, say you have an image with a height and width of $64$ pixels each (totaling $64 \times 64 = 4096$ pixels). You want to resize this image to a height and width of 256 pixels (totaling $256 \times 256 = 65536$ pixels). In the new, larger image you only know the value of $1$ out of every $16$ pixels. How are you going to calculate the values for the rest?

Well, the methods that do that for you are called upsampling techniques. The most common are:

Nearest-Neighbor: Copies the value from the nearest pixel.
Bilinear: Uses all nearby pixels to calculate the pixel's value, using linear interpolations.
Bicubic: Again uses all nearby pixels to calculate the pixel's values, through polynomial interpolations. Usually produces a smoother surface than the previous techniques, but its harder to compute.
Other more complex resampling algorithms, e.g. Lanczos.

An article explaining the differences among image resampling techniques can be found here.


NN for fuzzy classification What loss-function / optimizer to use for fuzzy classification problems?
E.g: Four categories hot, mild, cold, freezing.

Edit:
I use one-hot encoding and have ~ 60 datapoints.

I'll say it because someone else will: It depends on a lot.

Always good to start off with a regular cross-entropy loss for classification problems, especially because it comes out of the box in many libraries (PyTorch, Keras, TensorFlow).
If you're dealing with binary classification task, then the Binary Cross Entropy might be an interesting alternative to try as well.
Surely someone can recommend more obscure losses, but starting with those simple ones could potentially give a very good return for minimal effort!

Again going for simplicity, Adam optimiser or SGD with momentum is often a low hanging fruit you can try.

I'd say use Adam and cross-entropy, and work your way through optimizing your hyper-parameters as much as you can (validation set!) and only then you can move other parts.

Also for more specific answers you should specify how many features, how many datapoints, and how you dealt with your categoricals (label encode / One-Hot).


Transformer model: Why are word embeddings scaled before adding positional encodings? While going over a Tensorflow tutorial for the Transformer model I realized that their implementation of the Encoder layer (and the Decoder) scales word embeddings by sqrt of embedding dimension before adding positional encodings. Notice that this is different from scaling the dot product attention.
I'm referring to the 3rd line of the call method of the Encoder class here: https://www.tensorflow.org/tutorials/text/transformer#encoder
def call(self, x, training, mask):

  seq_len = tf.shape(x)[1]

  # adding embedding and position encoding.
  x = self.embedding(x) # (batch_size, input_seq_len, d_model)
  x *= tf.math.sqrt(tf.cast(self.d_model, tf.float32))
  x += self.pos_encoding[:, :seq_len, :]

  x = self.dropout(x, training=training)

  for i in range(self.num_layers):
    x = self.enc_layers[i](x, training, mask)

  return x # (batch_size, input_seq_len, d_model)

I could not find any mention of this scaling in the papers I've read so far. People always show the input to the encoder as WE + PE, that is word embedding plus positional encoding. But this implementation seems to use sqrt(d_model) * WE + PE.
My questions:

Have you ever seen this extra scaling step mentioned in a paper? I didn't find it in "Attention is all you need" (Vaswani et. al.).
What is this additional scaling trying to achieve?

This is specified in the original Transformer paper, at the end of section 3.4:

Transcription:

3.4 Embeddings and Softmax
Similarly to other sequence transduction models, we use learned embeddings to convert the input tokens and output tokens to vectors of dimension model. We also use the usual learned linear transformation and softmax function to convert the decoder output to predicted next-token probabilities. In our model, we share the same weight matrix between the two embedding layers and the pre-softmax linear transformation, similar to [24]. In the embedding layers, we multiply those weights by √model

This aspect is not justified by the authors, either on the paper or anywhere else. It was specifically asked as an issue in the original implementation by Google with no response.
Other implementations of the Transformer have also wondered if this was actually needed (see this, this and this).
Some hypothesithed arguments (source) are:

It is for the sharing weight between the decoder embedding and the decoder pre-softmax linear weights.
It is not actually needed.
It is to make the positional encoding relatively smaller. This means the original meaning in the embedding vector won’t be lost when we add them together.

For reference, there are other StackExchange questions discussing this (see this and this).


Cross validation test and train errors I came across this sort of flowchart:

Below the flowchart, this is what appears:

“Given a training set, cross-validation error is computed for each configuration of tuning parameters (λ,d). The configuration of tuning parameters with the lowest overall cross-validation error is chosen to be the best as it leads to the best model performance. Using the best configuration of tuning parameters, we then train the models M2 and M3 on the original training set and use the original test set to compute the corresponding test RMSEs.”

They are only mentioning the cross validation error (validation) and never mention the train cross validation error.
Is  the phrase “The configuration of tuning parameters with the lowest overall cross-validation error is chosen to be the best as it leads to the best model performance” correct? I mean, assuming that by “lowest overall cross-validation error leads to the best model performance”, they are referring themselves to the “validation” error of the cross validation technique, I wonder why are they making such assumption? Should we care about the averaged train cross validation error or just the averaged validation error?

I am using a library to play with recommender systems which has a parameter called return_train_measures = True. Then it throws both, train and test errors:

The cross-validation error is calculated using the training set only. Choosing the model that has the lowest cross-validation error is the most likely to be the best model for data that it hasn't yet seen. But it's not necessarily actually the best, so in that sense, strictly speaking, you can say the phrase is wrong.

Even if your sample is a properly randomly drawn from the data generating process, it's still a finite sample. Imagine if the data is one-dimensional: Each example is just a number. The average of the training set could, by sheer bad luck, deviate a lot from the average of the data generating process. It's a simplistic example, but the best model on this data is not necessarily the best on any sample from this process.


Vector representation of documents for text classification I'm looking for proper method of document embeddings. I know that doc2vec will give me the vector representations for given corpus, but how do I embed new documents? I need to train neural network that will classify text, but I have no idea how new documents should be embedded properly.

Not necessarily. Documents embeddings are constructions from (sub)word embeddings. Nowadays, you should always start from a pre-trained model. Please check tensorflow Hub or Hugging Face. Even without retraining, as long as your text is not too specific, the pre-trained encoder should capture enough relations.


Probabilities predicted by XGBoost I looking at football data and trying to predict whether a goal will occur using xgboost with objective binary: logistic.
My data is 1:10 unbalanced with no goals being more dominant. I have used smote or task.over in mlr package to oversample (with a factor of 4).
I train the model, tune and cross validate but the predictions seems reasonable (low aucpr of 30% but high in other stats).
However when I look at the probabilities predicted by the model, it  much larger than the actual average. Is there anything that could cause this?
What are the probabilities? i.e class 1 probs are the probabilities that they are in class 1, so may not truly represent the probability to score a goal.
Thanks in advance

If the predicted outcome targets(i.e., probabilities in this example), are consistently higher than actual, it implies you have statistical bias model. Statistical bias is when there is a systematically different result from the population parameter being estimated.
There are many ways to reduce statistical bias:

Resample at more extreme ratio
Change the loss function to more heavily penalize mispredictions
Change algorithm


Should a Cluster Validity Index contain the same measure(s) as the Clustering Algorithm? I'm currently trying to use cluster analysis as a tool for time-series aggregation for a project of mine. The dataset is high-dimensional (386-d), so no chance in assessing the cluster validity visually.

I'm using three different clustering algorithms (k-means++, k-medoids PAM, fuzzy c-means) to find representative periods. As I do not know how many periods (thus, how many centers/medoids k) are present in the dataset, I want to use an internal cluster validity index (cvi) for it. (Basic procedure: run the clustering with multiple k's and plot the cvi against the k; choose highest/lowest k dependent on cvi optimum).

Let's stick with k-means as an example. It is non-deterministic, thus I initiate it multiple times with varying starting points. It then tries to reduce the intra cluster variance. The result with the lowest intra cluster variance of the various initiations with the same k is then kept.

My question is: should the cluster validity index that I use consider the intra cluster variance as a cohesion measure? So to speak: should the cvi use the objective function of the clustering algorithm as a measure to assess "goodness" of clustering?

On the one hand I think, that this would be a good idea, as it can assess, where the clustering algorithm was "most successfull" in its objective function. On the other hand I think, that for a good clustering it shouldn't depend on the same (cohesion) measure. Thus, using the same measure would increase the likelihood of identifying a "bad" clustering, that might represent an unnatural cluster, drawn from random points.

What's "the truth" here?

It's standard to use, e.g., Silhouette to assess the quality of clusterings, that were obtained with other methods.

I'm not a big fan of this (people always think they would get the "optimal" result this way, but they don't), but it's all over literature. Logically, you are trying to maximizing Silhouette and use the other algorithms as sampling procedure for possible results.


How can l get 50 % examples in training set and 50% in test set for each class when splitting data? l have a dataset of 200 examples with 10 classes. l would like to split the dataset into training set 50% and test set 50%.

for each class, l have 20 examples. Hence, l would like to get for each class : 10 training examples and 10 test examples.

Here are my classes :

classes=['BenchPress', 'ApplyLipstick', 'BabyCrawling', 'BandMarching', 'Archery', 'Basketball', 'ApplyEyeMakeup', 'BalanceBeam', 'BaseballPitch', 'BasketballDunk']

l tried the following :

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(final_data, true_label, test_size=0.50, random_state=42)

However it returns a 50% training set and 50 % test set, without respecting the proportion for each class (l would like to get 10 examples in test set and 10 examples in training set for each class).
Here is the resulted splitting :

sklearn version 0.17 onwards the train_test_split should give you stratified results by using the stratify parameter.

Example code:

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(final_data, true_label, test_size=0.50, random_state=42, stratify=true_label)

From the documentation about the parameter stratify:

  stratify: array-like or None (default is None) If not None, data is split in a stratified fashion, using this as the labels array. New in version 0.17: stratify splitting

Hope this helps!


PyMC3 do not converge I'm trying to run a simple logistic regression on PyMC3. Here the code:

with pm.Model() as logistic_model:
    alpha=pm.Normal('alpha', mu=0, sd=100)
    beta1=pm.Normal('beta', mu=0, sd=100)
    beta2=pm.Normal('beta2', mu=0, sd=100)

    argtemp=alpha + beta1*data['age'].values +beta2*data['educ'].values
    prob=pm.invlogit(argtemp)
    Y_obs = pm.Bernoulli('Y_obs', p=prob, observed=data['income_more_50K'].values)
    map_estimate = pm.find_MAP()
    sampler=pm.Metropolis()

    trace= pm.sample(40000, step=sampler, start=map_estimate)

The sampler run, but at the end it gives this error:

The gelman-rubin statistic is larger than 1.4 for some parameters. The sampler did not converge.
The estimated number of effective samples is smaller than 200 for some parameters.

Can someone help me in understanding what is wrong, and why the sampler do not converge?

Thank you!

It turns out that the problem was the MAP estimation. Commenting out that line makes the code works.

Details in  this thread in the PyMC Discourse forum.


I have a word2vec embedding - now what? I've always relied on the Keras embedding layer for my NLP work. But for my latest project I want to use a custom embedding layer. I have gone through the steps to create a word2vec file but now what? Can someone provide an example of how I can replace the Keras embedding layer with my own layer when constructing a model?

First, make sure, you have the same vocabulary in your word2file and in your Keras code. If you Keras Tokenizer, you can simply call fit_on_texts with the text actually being the vocabulary.

Load your word embeddings as numpy matrix (just read all the numbers and then call np.array).

When creating the Embedding layer, call set_weights and set the loaded matrix and set trainable to False.

You can find more at Keras blog but I am not sure how up to date it is because it is from 2016.


TensorFlow 2 one-hot encoding of labels I was following this basic TensorFlow Image Classification problem, where images of flowers have to be classified into one of 5 possible classes. The labels in the training set are not one-hot encoded, and are individual numbers: 1,2,3,4 or 5 (corresponding to 5 classes). The final layer of the ConvNet however has num_class number of units.
Wouldn't there be a dimension mismatch while computing the loss, since you are finding the difference between a [num_class, 1] (predicted label) dimensioned vector and a [1, 1] (true label) dimensioned vector?
Does the Keras backend automatically convert the labels into one-hot vectors?
Thank you in advance!

The loss function handles the conversion. TensorFlow has a SparseCategoricalCrossentropy and a CategoricalCrossentropy loss function. The first expect your labels to be provided as integers, where the latter expects one-hot encodings. In the given example, they use the SparseCategoricalCrossentropy loss function, therefore it is ok to supply your labels as integers.


Network Plot - R - Modify Col Having this R statement:

itemsets = apriori(data, parameter=list(support=0.05, confidence=0.5))
plot(itemsets, method="graph",
     control=list(type="items"))

How can I change the colour of my plot? I try to insert "col" but don't allows me to do that because I'm getting the following error:

Error in i.parse.plot.params(graph, list(...)) :
  Unknown plot parameters: col

thanks

Use edgeCol and nodeCol:

library("arulesViz")
data(Groceries)
rules <- apriori(Groceries, parameter=list(support=0.005, confidence=0.5))
plot(rules, method="graph",
     control=list(nodeCol="yellow", edgeCol="blue", type="items"))


Data Snooping, Information Leakage When Performing Feature Normalization Assume that we have a training data set (with both features and labels) and a test data set (with only features).

When we build a machine learning model that requires normalization of the features, the correct way of doing normalization (to prevent information leakage) is to only use the training data set. Namely, a wrong way of doing normalization is to stack the training (exclude the Y column) and test data set together, and perform normalization (i.e., using the mean and variance of the entire training + test data set). The intuition here is very clear: if you want to get an unbiased estimator of your model performance in production, then when you train your model, you shouldn't instill any information in the test data set that will be used to gauge the actual performance of your model.

My question is the following: When we have trained the model correctly and about to use this model to do prediction, how should we normalize the test data set? I believe the correct way to normalize the test data set is: using the mean and variance obtained from the training data set to do normalization on the test data set.

However, why not just normalize the test data using its own mean and variance? Or why not stack train and test data set together and using the overall mean and variance to normalize the test data set? In the prediction stage, the idea and intuition of data snooping and information leakage is not clear to me.

The normalization parameters you fitted in training are now part of your model. You fitted the model weights on the training data: the normalization step is part of your model now, and the "parameters" for that step are the mean and variance of the training data, not the test data.

To help make this more concrete, let's pretend that this model is in production. You fit your model on whatever training data you had available, and now it's running as a scoring service. You can evaluate your model accuracy over time as data you've scored gets labeled to ground truth, but you're just evaluating the model not refitting it. How do you normalize these incoming predictions? You're going to need to use the mean and variance from whatever dataset you used to fit the model. If you'd rather do something like using the last K observations to estimate mean/variance, then that's the procedure you need to be testing when you fit the model.


GridSearchCV Acting Weird I am using GridSearchCV to find the best combination of parameters for SVM. However, the parameters chosen by GridSeasrchCV do not seem to be the best ones. I tried some parameters randomly and they are working much better. I tried both combinations in the test set. Here is the picture:
The second confusion matrix + classification report is the random but improved one. Why isn't GridSearchCV selecting this combination?

99% Accuracy is over fitted model and its not the accurate model. There will always be loss associated. secondly you sample size is too small for reaching the global minima. you should also check your ROC-AUC curve for the model.


What courses / subjects are most important to the field of Data Science? I've taken it upon myself to begin a career change. I have a decent background in mathematics, but lack in programming or data science specific skills (such as data munging). I have been looking through data science curricula and already feel overwhelmed by the amount of subjects a proper data science curriculum contains. Just to name a few: natural language processing, machine learning, R, Python, SQL, NoSQL, probability, statistics, numerical methods, algorithms, and the list goes on.

I can't afford to go back to school, but if my goal is to get into data science, what subjects should I focus my attention on?

I fully understand "data science" is a very big field, but certainly there must be several subjects that are important to all, if not the vast majority, of these sub-fields. I suppose this is essentially what my question is asking.  I have not yet obtained the amount of exposure to determine what I want to specialize in, so at this point, my concern is obtaining a foundation that wouldn't restrict me from entering any particular sub-field, or in other words, studying the subjects that benefit all sub-fields.

Note: Math isn't as much a concern for me compared to the other subjects.

Probably the most useful (and fun) answer is provided by the excellent Swami Chandrasekaran ... in the form of a subway map:

I would add just one comment to that:  school (or learning for the sake of learning) doesn't provide the same kind of experience as solving a real problem does.  Therefore, to learn, find real problems you can tackle.  To stay motivated, make them problems you really care about.  Kaggle competitions are a great place to start.  Even reproducing a winning solution (perhaps with some variations, or on a new dataset) will be a huge learning experience.


Final Model fitting - subset vs entire training data If I used a subset of the entire available training data for model tuning and hyperparamater selection, should I fit the final model to the subset training dataset or the entire available training data?
For example, if I have 1M samples available and I took a 100K random samples as a test holdout and 200K random samples as a training dataset for model tuning, should the tuned hyperparamaters used to fit the final model on the 1) 200K training dataset, or 2) 900K available data (excluding test holdout)? In other words, can the hyperparameters by generalized for the entire population?
I am assuming that both the holdout and training datasets are selected randomly and follow the class distribution in the original data.

The general machine learning process is this:
Split your data into two parts, training and test. So in your example I would take 100k for test and 900k for training (don't know why you say only take 200k in your question but I digress). With the 900k training we perform hyper-parameter tuning. This can be done by splitting training into training and validation say 800k/100k or better yet we could do this using k-fold cross validation.
Once you have chosen the optimal hyper-parameters in this manner you evaluate their performance on the test set. The whole point of this process is simply to evaluate the algorithms performance, and from that select and algorithm. That is the only reason for a train/validation/test split. (As a note, this process can be even further improved by using something called nested cross-validation but I will not go into the details).
After you have selected your algorithm and determined its performance (error rate), you take your whole data set (1 million records) and perform hyper-parameter selection on that, either using a single split or by k-fold cross-validation. You no longer need the test set because you have already determined the models error rate in the previous step.
Once you have selected the best hyper-parameters in the previous step you apply them to the entire data set (the 1 million records) and build the model.


How does training a ConvNet with huge number of parameters on a smaller number of images work? I have two questions:

I am wondering why is that a very deep model such as VGG-16 which has approximately 138 million parameters (Source) can be used as a model to be trained on just 1.3 million images? Even though the authors of VGG-16 used dropout and regularisation to avoid overfitting (source), I just feel that the amount of parameters is still significantly more than the number of images.
Is it also true that when you use $N$ number of epochs, you "effectively" have $K*N$ training datasets, where $K$ is the number of training datasets you have? In the case above, $K$ = 1.3 million. Personally I don't think this is the case, but I am not sure.

I probably have not done research or read thoroughly, so I would like to apologize if the answers are already somewhere out there. Thank you.

First of all, you are ignoring the dimensionality of the problem. Images are very high-dimensional. Let's say an image has a resolution of $256 \times 256$, which means each image has $65,536$ pixels. ImageNet images are RGB, so each image has 3 channels, resulting in $196,608$ pixels per image. Now, the whole dataset ($1.3$m images) has more than $255$ billion pixels associated with its images.

Because filters work at pixel-level, the information it gains from one image is more than what a regular ML algorithm would gain from a single training example.

As you can see, the kernel (whose weights you want to update, and accounts to 9 parameters in this case) is much smaller than the input image. It would be a mistake to consider a single image (of around $200,000$ pixels as we say previously) as only capable of updating one parameter.

Secondly, the input of each layer changes from layer to layer, because of the sequential nature of the network. The second layer sees the output of the first,and so on... This means that the second layer's filters won't see the same image as the first and in fact, as training progresses, their input will also gradually change (because the first layer's filters will be getting more effective).

By the time it reaches let's say the $M^{th}$ layer, the input will have changed a lot from the original image, so it would be a mistake to consider that the original image is updating the totality of the parameters in the network.

Thirdly, data augmentation is also a thing. By making simple random transformations to each image (flips, shifts, scales, rotations, brightness/contast adjustments etc.) the network is tricked into thinking that this is a totally new image. This can exponentially increase the size of the dataset.

Finally, you should take a look at where the parameters in a network are. The VGG19 architecture does have more than $143$ million, but around $124$m of those are from the last 3 FC layers. In fact the first of the three layers has around $103$m parameters on its own! This is a very inefficient network design and that the research community has strayed far from recently. A more representative network you should look at is the ResNet architecture. For example a 50-layer ResNet (much deeper than the 19-layered VGG) consists of just around $25$m parameters, while achieving a similar -if not better- performance.


Decision tree regression: Polynomials unnecessary? I am testing out different models for a regression task.
When using OLS, Ridge and Lasso, I use different polynomial degrees of the explanatory variables.
Example: For two variables x and y, degree 2 would give the explanatory variables x, x^2, xy, y, y^2.

When using decision tree, however, I am not sure whether it makes sense to use any any higher degrees than 1 as explanatory variables. Example: Does it make sense to test for x^2, xy and y^2 when applying a decision tree regressor?

The reason I ask, is that the decision tree regressor is a non-linear regressor. On the one hand this could perhaps be an arguement it not making sense to include higher order polynomials, as the decisoin tree already can deal with non-linearity.

Since CARTs (Classification And Regression Tree) are a non-parametric algorithm, they should be able to find interactions between variables and non-linear behaviors.

Nevertheless, building polynomials can help them have a better performance.


Replace part column value with value from another column of same dataframe I have a dataframe with two columns:
Name          DATE
Name1         20200126
Name2         20200127
Name#DATE#    20200210

I need to replace all the #DATE# with the data from the DATE column, and get something like this:
Name
Name1
Name2
Name20200210

How can I achieve this? I've tried things like this, without any good result..:
df_merged_tables["Name"].str.replace("#DATE#",merged_tables["DATE"])

Thanks!

Your solution is close
Maybe just needed to add an apply.
Try:
df = pd.DataFrame({"Name":["Name1", "Name2", "Name#DATE#"], "Date":[20200126, 20200127, 20200210]})

df["NewColumn"] = df.apply(lambda row: row["Name"].replace("#DATE#", str(row["Date"])), axis = 1)

Outputs:


What Models should i try for this problem? I need some advice for a problem i'm working on with automobile data. The vehicles provide a series of codes at every second which are bieng stored, though it can vary how many.

For example , at time step one the vehicle might send 3 codes:

(1 , 2) , (3 , 5)  , (7 , 9)

and there can be a varying number of time steps.(Before an Error occurs)

Now some of these codes have associated Error values. For example (1 , 2) might be a critical error denoting a problem with your tire. But the vast majority of them don't have this. There are about 15 of these critical errors while 5500 of other codes.

The codes are actually made up of two separate independent codes.

The task is to predict ahead of time if is its likely that any of these codes will appear.

What i tried :

I tried to do this with a single layered GRU RNN Network with embeddings for all the codes. I am predicting the probability of the next code in the sequence at every step. (considering each code to be a step) .

I tried tweaking the parameters like the number of neurons etc, but i'm not getting any results with this (Accuracy less than 5 percent after several epochs).

How can i improve this model ?
Is there any method by which i can consider all of these codes at every step ?
What other algorithms
outside of deep learning can i apply here ?

Thanks in advance

If I understand correctly, you currently try to predict the next code among 5500 possibilities. I think that's too many options to choose from, you can't expect a good performance with that.

Since the task is about predicting the 15 error codes I would represent the problem as a sequence labeling problem: the label at each step is either "nothing" (often represented as "_" in sequence labeling tasks) or one of the 15 error codes. The label would represent the most likely outcome for the next step, not for the current one. This way the system can use the history of the codes (regular features) without having to predict all of them, only the critical ones. Note that if several error codes can happen at the same time, you might have to select a single one as label (maybe they can be ranked by priority?).

As far as I know, the standard method for sequence labeling is Conditional Random Fields (CRF). In the traditional approach you would have to do some feature engineering: for example try to not give only the codes as features but also features such as how many times a code happened in the last N steps. I think the modern approach consists in training a RNN and using the vector representation as features for the CRF, but I might not be up to date with this.


Assistance needed on what machine learning approach to use I'm currently writing my Master's Thesis on Subjective tagging of sounds and I feel that I've been stuck with the same problem for quite a time now and need assistance to progress. I'll, in short, describe my goal and what I've accomplished so far.

I'm writing a program that lets users manually tag audio files with a single label according to their own, i.e. subjective, perception. The tags are always adjectives, like happy, spooky or spacy. The goal is to have the program automatically tag similar sounds with the correct label by finding related sounds using machine learning. The program currently looks like this:

Usage of the program goes like this:

The user adds sounds to be labeled.
The program analyses the files and searches for related sounds using Algorithm 1.
The user tags a sound and the program automatically tags related sounds using Algorithm 2.
The user can verify tags given by the program and choose to accept them or give the sound a new tag.
Repeat from step 1 or step 3.

By looking at the program flow above I've been able to implement step 1 using AudioCommons Timbral Models to convert sounds to a usable data format, and step 2 with Algorithm 1 using Mean shift to find groups of related sounds. The idea is to use the groups found by Algorithm 1 as a starting point when no tag has yet been given to a group. When one of the sounds in the group is tagged, all the sounds receive the same tag.

My problem, however, lies in step 3 with Algorithm 2 and step 4. I've tried a few different approaches, but they all seem a bit off and not quite fit for the job.

My goal is to have the program continuously or recursively learn from user validation, hence improving over-usage. My thought process is to have a verified flag on each sound, which the user can use to say if a tag given to a sound by the program is correct or not. Algorithm 2 should (re)learn from verified sounds in step 4 so that when step 3 is repeated, the automatic tagging of sounds done by the program is more accurate according to the user's choices.

So my question is: What kind of algorithm(s) is suitable for step 3 and 4 in place of Algorithm 2?

I've tried to make myself as clear as possible, but I'm quite new to machine learning, so feel free to point out confusing sections or errors.

In hope of assistance,

Andreas

Notes:

I've used tag and label interchangeably throughout the text; they refer to the same thing.
I want all machine learning to occur in isolation for a user, i.e. I don't want to learn from other user's input (like Collaborative filtering).

Just a few observations from my side:

I think you should clarify your goal for yourself. Writing a program isn't a goal in itself. I'm missing the top level picture.
The second part: 'The goal is to have the program automatically tag similar sounds with the correct label by finding related sounds using machine learning.' is more like a goal, but where is the subjective part in it, and how does this answer questions in your thesis?
I'm not sure about the meaning of the output of Algorithm 1. Is this supervised or unsupervised classification? If it would be supervised, the 'supervision' is not subjective (until Algorithm 2). If is it unsupervised, the classes will find similarities, but the similarities are unlikely to correspond to the labels you intend to use.
However, if you take e.g. $k=9$ classes with k-means (you will have to optimize the number) the similarities in the classes will increase, and you could 'learn' the matching of labels and the classes. The $k$ classes should map to one of 3 labels.
Don't try to fit all these steps into one model. Create a few and connect them properly. Measure, interpret and report intermediate steps in your thesis.


How to retrieve Twitter username with UUID I am trying to analyze FakeNewsNet which is a fake news dataset for Social Network Analysis.

There are UUID(I am not sure about the entity name)s of users in the dataset but I am not sure how to retrieve metadata from it.

Can someone explain to me where I can find the information on how to use user_id of User.txt?

Is it even possible to retrieve the metadata of a user with this id? (may using Twitter Search API)

Reference: https://github.com/KaiDMML/FakeNewsNet/blob/master/Data/BuzzFeed/User.txt

It is not possible with the current version of the dataset, the User.txt ids are hashed. But, the pre-processed features are in UserFeature.mat which you can load with something like

import scipy.io
mat = scipy.io.loadmat('UserFeature.mat')

mat['X'][0] will then be the feature vector for the first user id in the text file.

There are no descriptions of the feature names either (or at least I didn't find any)


Character Level Embeddings I am working on a problem that current depends on word level embeddings created using Word2Vec. I am researching new methods to apply to this model and one was a character level embedding. I have not found much information on it, and I don't imagine Word2Vec but at a character level would be effective. Is there any insight on giving vector representations to characters for an overall classification model?

For the sake of completeness, fastText uses sub-word information (character level n-grams) when creating embeddings and then for classification with some interesting results:

can give prediction for the unknown words that have some similarity with known ones
with interesting results for modelling syntactic relations like 'went' - 'go' + 'give' = 'gave'


No correlation found between dataset features I'm trying to build a classification model that predicts the price of New York taxi trips (year 2018).
Datasource page

Since the original file is very large (112 234 626 rows), I constructed the smaller sample file (1 000 000 rows) to make the first exploratory analysis.
By using this sample file I constructed Heatmap chart to display the correlations between dataset features.
I found out that the model's dependent variable (fare_amount) is only correlated to trip_distance variable. The other independant variables didn't have any considerable correlation.

Does this mean that the dataset I use is not appropriate for the classification task?
Or I haven't solved the task correctly?
How many independent variables should correlate to the dependent variable to count it "good enough" for further analysis?
Which method should I prefer for feature selection and dimensionality detection? Should it be Heatmap chart, PCA or something else?

My code in GitHub

Think about the data generating process: For taxi fares, THE relevant thing is distance/time of the trip. This is by definition the case for most taxi services (Uber might be different). Maybe time of the day plays a role as well, e.g. when fares during day/night are different.

In a simple case you might not even need a statistic solution, viz. if the problem is deterministic, meaning fare is a simple linear function of distance.

You can use (for example) a linear regression to predict fare. In this case you could have a model:

fare = b0 + b1 * dist + error.

In this model, there is nothing wrong with having only one predictor (this is a univariate linear regression).

BTW: If you predict a continuous variable, you do regression. If you predict classes (like yes/no) you do classification.

If you have many variables with only weak correlation to y, you can also use regulation (by L1 norm), in linear regression this is called Lasso, to „shrink“ features which are irrelevant.

P.S. Plot price (y) against distance (x). You will likely find close alignment of both.


On regression to minimize log distance rather than distance Suppose I have a lot of points $ x_i \in \mathbb{R}^N $ with corresponding non-negative labels $ y_i \in \mathbb{R} $ and I want to do regression and make a prediction on some new datapoint $ x^* \in \mathbb{R}^N $ for which I don't have a label.   Is there a name for the procedure of choosing a parametric model $ f_\theta : \mathbb{R}^N \rightarrow \mathbb{R} $ so as to minimize the cost function $ \sum_i {|\log(f_\theta(x_i)) - \log(y_i)|^2 } $ rather than $ \sum_i{|f_\theta(x_i) - y_i|^2} $?  It seems that minimizing the difference between logs has some nice properties, and I'm surprised I didn't see this discussed in Bishop's machine learning book, for example.

I thought of this when I was considering a house pricing problem, where I figured I cared more about the percentage by which I was wrong than the pure difference.  After all, in my application (and I'm sure many others like it), being wrong by \$50,000 is terrible for a \$60,000 home, but it's okay for a $2.5M home.

Any data science veterans reading this who have used a cost function like the one I suggested above with logs, or who can tell me what it's called (if it has a formal name)?

There is a loss called Root Mean Squared Log Error (RMSLE): $\sqrt[]{\frac{1}{n}\sum_{i=1}^n{(\log(y_i + 1) - \log(\hat{y_i} + 1))^2}}$ (do not forget the $+1$ as the $log$ is not defined at $0$)

You will find a brief explanation and discussion here. It has also been used in competitions as for example here.


Error trying to do a GridSearchCV() On the following lines of code I am getting

clf = neural_network.MLPClassifier(hidden_layer_sizes=(5, 12))
parameters =[ {'solver': ['lbfgs'],'max_iter': [500,1000,1500], 'alpha': [1e-1,1e-2,1e-3,1e-4,1e-5,1e-6,1e-7],  'random_state':[0,1,2,3,4,5,6,7,8,9]}]
model = GridSearchCV(clf,param_grid=parameters,n_jobs=-1)

On the last line I am getting the following error

ValueError: Parameter values for parameter (solver) need to be a sequence(but not a string) or np.ndarray.

now I know from reading

Parameters in GridSearchCV in scikit-learn

That this means everything must be in an array but all my params are in an array so what am I doing wrong.  thanks

I don't get that (or any) error (after correcting your paramators to parameters).

That said, if you only want to use one solver, you might as well pass it directly to the estimator and leave it out of the grid search.


Euclidean distance for more than 2 datapoints I need to compare $n$ 3-dimensional vectors with $k$ 3-dimensional vectors using euclidean distance. Is that possible?

The Euclidean distance between 2 vectors

$\mathbf{u}$: $(u_1, u_2, ... , u_n)$ and $\mathbf{v}$: $(v_1, v_2, ... , v_n)$

is simply

$\sqrt{(u_1-v_1)^2+(u_2-v_2)^2+...+(u_n-v_n)^2}$


Tensorflow RNN not learning when output included in training variables I have been attempting to train a RNN on a set of time series data. The goal is to predict one of six categorical outputs. The input is given as 5 time steps of 14 inputs, six of which at one-hot attributes for the output.
There is an output at each time step, but the goal is to use previous recorded time spots and their human-assigned outputs to assign an output to the most recent event.

Confusingly the RNN is unable to learn that one of the inputs is in fact the output of the classification. This is simply a sanity check for me, but it seems that it may indicate a larger underlying problem.

The data is heavily imbalanced, 91%, 4%, 2%, 1%,<1%,<1%, but a cost function is being use to weight mis-classification inversely to it's make-up in the data set. Could the imbalance cause this issue? I'm using 60,000 training examples right now, is this not enough?

I'm working off of this dynamic RNN model: https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/dynamic_rnn.py.
Note the additional matrix multiplication for a hidden layer. Is this done correctly?

def dynamicRNN(x, seqlen, weights, biases):

# Prepare data shape to match `rnn` function requirements
# Current data input shape: (batch_size, n_steps, n_input)
# Required shape: 'n_steps' tensors list of shape (batch_size, n_input)

# Permuting batch_size and n_steps
x = tf.transpose(x, [1, 0, 2])
# Reshaping to (n_steps*batch_size, n_input)
x = tf.reshape(x, [-1,n_input])
x  = tf.matmul(x, weights['hidden'])+ biases['hidden']
# Split to get a list of 'n_steps' tensors of shape (batch_size, n_input)
x = tf.split(0, n_steps, x)

# Define a lstm cell with tensorflow
lstm_cell = rnn_cell.BasicLSTMCell(n_hidden, forget_bias=1.0)

# Get lstm cell output, providing 'sequence_length' will perform dynamic
# calculation.
outputs, states = tf.nn.rnn(lstm_cell, x, dtype=tf.float32,
                            sequence_length=seqlen)

# When performing dynamic calculation, we must retrieve the last
# dynamically computed output, i.e, if a sequence length is 10, we need
# to retrieve the 10th output.
# However TensorFlow doesn't support advanced indexing yet, so we build
# a custom op that for each sample in batch size, get its length and
# get the corresponding relevant output.

# 'outputs' is a list of output at every timestep, we pack them in a Tensor
# and change back dimension to [batch_size, n_step, n_input]
outputs = tf.pack(outputs)
outputs = tf.transpose(outputs, [1, 0, 2])

# Hack to build the indexing and retrieve the right output.
batch_size = tf.shape(outputs)[0]
# Start indices for each sample
index = tf.range(0, batch_size) * n_steps + (seqlen - 1)
# Indexing
outputs = tf.gather(tf.reshape(outputs, [-1, n_hidden]), index)

# Linear activation, using outputs computed above
return tf.matmul(outputs, weights['out']) + biases['out']

Strangely, once the columns of each block of 5 data entries were normalized, with null rows included, the accuracy jumped to >99%. Again the outcome is included in the data set but this seemed to correct the issue with this sanity check.


Measuring distance preservation in dimensionality reduction I am looking to compare the distance preserved during dimension reductions for several techniques. I have read some papers on similar topics here and here.

For example, I would like to use the Euclidean Distance to measure the distance preserved during PCA's dimension reduction. However, my point of confusion what are $X$ and $Y$ in
$$d(X, Y) = \sqrt{\sum^n_{i=1}\left(x_i - y_i\right)^2}$$
I understand how to calculate $d\left(X, Y\right)$ given two vectors/matrices, but I don't understand with context to PCA. Let me try to explain.

Let $W_{d \times k}$ be the matrix of $k$ leading eigenvectors, $X_{d\times n}$ be the original data, and $Z_{k\times n}$ be the projection of $X$ onto the reduced subspace.

$Z = W^{T}X$

Back to calculating $d\left(X, Y\right)$. My guess is that the PCA's $X$  correspond to $X$ and $Y$ can correspond to $Z$. But how does this work since $X$ and $Y$ have different dimensions?  I have to be oblivious to something here.

Also, I am not concerned if a Euclidean Distance measure is not a good choice for measuring PCA's distance preservation (unless they are incompatible). This is simply exploration.

Edit:
For example, if I have
$$X_{d\times n} =
    \begin{bmatrix}
    x_{11} & \dots & x_{1n} \\
    x_{21} & \dots & x_{2n} \\
    x_{31} & \dots & x_{3n} \\
    \end{bmatrix}
$$
and say I choose to retain $k = 2$ principal components which are then projected onto
$$Z_{k \times n} =
    \begin{bmatrix}
    z_{11} & \dots & z_{1n} \\
    z_{21} & \dots & z_{2n} \\
    \end{bmatrix}
$$

The confusion

The point of confusion is probably the use of variable $n$ to denote the dimension in Euclidean distance, which is later re-used for the number of points. Euclidean distance is better to be written as
$$d_X(\boldsymbol{x}_1, \boldsymbol{x}_2)=\sqrt{\sum_{i=1}^{\color{red}{d}}(x_1^i - x_2^i)^2}$$
in the original space, and
$$d_Z(\boldsymbol{z}_1, \boldsymbol{z}_2)=\sqrt{\sum_{i=1}^{\color{red}{k}}(z_1^i - z_2^i)^2}$$
in the reduced space. Where $\boldsymbol{z}_a=(z_a^1,\cdots,z_a^k)$ is the transformation of point $\boldsymbol{x}_a=(x_a^1,\cdots,x_a^d)$ by $W$. EDIT: in your notation, $Z[:, a]$ is the transformation of $X[:, a]$.

Distance preservation measure

A measure of distance preservation for transformation $W$ should be defined in such a way that compares the distance of two points in the original space with the distance of the same two points in the reduced space.

Accordingly, a measure that compares all $\binom{n}{2}$ pairwise distances could be defined as:
$$DP(W) = \frac{2}{n(n-1)}\sum_{a=1}^{n} \sum_{b=a+1}^{n} |d_X(\boldsymbol{x}_a, \boldsymbol{x}_b) - d_Z(\boldsymbol{z}_a, \boldsymbol{z}_b)|$$

where for a constant $k$, smaller $DP(W)$ implies better distance preservation by $W$.

Note that various measures of distance preservation can be defined based on this foundation. For example,

Restricting the comparison to only $K$ nearest neighbors of a point in the original space, leading to $nK$ comparisons instead of $\binom{n}{2}$, or
Using the Pearson correlation of distances rather than their difference, since we generally want close (distant) points in the original space to be also close (distant) in the reduced space, or
Measuring the distance to point's tangent space instead of the point itself, etc.


In Affinity Propagation do the cluster centres minimize mean distance to all other points in the cluster? For Affinity Propagation, do the cluster centers minimize the mean distance to all other points in the cluster?

No.

AP maximizes responsibility, which is a combination of availability and affinity. Distances only serve as an indirect source to input affinities.

If you want to minimize average distance to the cluster center, use PAM It minimizes the sum of distances to the representatives, and that is equivalent because the number of points is constant.


What is the advantage of keeping batch size a power of 2? While training models in machine learning, why is it sometimes advantageous to keep the batch size to a power of 2? I thought it would be best to use a size that is the largest fit in your GPU memory / RAM.

This answer claims that for some packages, a power of 2 is better as a batch size. Can someone provide a detailed explanation / link to a detailed explanation for this? Is this true for all optimisation algorithms (gradient descent, backpropagation, etc) or only some of them?

This is a problem of alignment of the virtual processors (VP) onto the physical processors (PP) of the GPU. Since the number of PP is often a power of 2, using a number of VP different from a power of 2 leads to poor performance.
You can see the mapping of the VP onto the PP as a pile of slices of size the number of PP.
Say you've got 16 PP.
You can map 16 VP on them : 1 VP is mapped onto 1 PP.
You can map 32 VP on them : 2 slices of 16 VP, 1 PP will be responsible for 2 VP.
Etc.
During execution, each PP will execute the job of the 1st VP he is responsible for, then the job of the 2nd VP etc.
If you use 17 VP, each PP will execute the job of their 1st PP, then 1 PP will execute the job of the 17th AND the other ones will do nothing (precised below).
This is due to the SIMD paradigm (called vector in the 70s) used by GPUs. This is often called Data Parallelism : all the PP do the same thing at the same time but on different data. See here.
More precisely, in the example with 17 VP, once the job of the 1st slice done (by all the PPs doing the job of their 1st VP), all the PP will do the same job (2nd VP), but only one has some data to work on.
Nothing to do with learning. This is only programming stuff.


How do I use this depth first search code to obtain a topological sort? Problem: I need to implement a topological search using the following depth first search code.

Note: The original code comes from here, and this is a problem given in at the end of the chapter.

I'll be honest. On a coding level, I feel pretty lost. I've added line comments to help show what I think each line is doing. While I'm fairly new to the depth first search, I understand how they work on a functional level (as opposed to my limited knowledge of their implementation). I've attempted to add the start vertices and finish times to a list so that I could then sort them later on, but I've had trouble returning that list. So that I could then sort the vertices in decreasing order of finishing time.

Note that while some solutions seem to use a stack, this code does not state the use of a stack explicitly. Rather, the stack is implicit in the recursive call to dfsvisit.

from pythonds.graphs import Graph
# Definitions
# discovery time == iterations it took for the program to find the vertex and turn it gray
# finish time == iterations it took for the program to turn the vertex black
# pred == predecessor indicator

class DFSGraph(Graph):
    def __init__(self):
        super().__init__()
        self.time = 0                       # self has an attribute 'time' (counter) that initiates at 0

    def dfs(self):
        for aVertex in self:
            aVertex.setColor('white')
            aVertex.setPred(-1)
        for aVertex in self:
            if aVertex.getColor() == 'white':
                self.dfsvisit(aVertex)

    def print_graph(self):
        for key in sorted(list(self.vertices.keys())):
            print(key + str(self.vertices[key].neighbors + " " + str(self.vertices[key].dis)))

    def dfsvisit(self,startVertex):                     # Initiate the visit vunction of the current object (self) at the starting vertex (startVertex)
        finish_times = []                                       # Instantiate the a list to keep finish times for each node
        startVertex.setColor('gray')                    # Set the color of the starting vertex to 'gray' (discovered)
        self.time += 1                                  # Increment the timer
        startVertex.setDiscovery(self.time)             # Assign the discovery time to the current vertex (startVertex)
        for nextVertex in startVertex.getConnections(): # Begin cycling through the connected vertices (nextVertex) of startVertex
            if nextVertex.getColor() == 'white':        # If a vertex (nextVertex) with the attribute color of white is found, then do the following:
                nextVertex.setPred(startVertex)         # Set the predecessor indicator of the next vertex as the current vertex (ie if B is white and connected to A, set B's predecessor as A)
                self.dfsvisit(nextVertex)               # Recursively calls itself with the next vertex until the color of the next vertex is no longer white (i.e. all have been explored)
        startVertex.setColor('black')                   # Set the current vertice's color to black (explored)
        self.time += 1                                  # Increment the timec counter
        startVertex.setFinish(self.time)                # Assign the finish time to the current vertex (startVertex)
        finish_time = [startVertex, startVertex.setFinish(self.time)]  # append vertex and finish times to finish_time list
        return finish_time                                             # return the finish_time list

Again, I'm hoping to understand how to implement the topological search either within this code.

It appears to me that you are doing too much with the code. For DFS, visit every node following adjacent nodes. As you do that, track the topological ordering.

Here is a straightforward implementation of recursive topological sorting a DAG in Python:

from collections import defaultdict

class Graph:

    def __init__(self):
        self.graph = defaultdict(list) # Node: [adjacency nodes]
        self.nodes = set()

    def add_edge(self, u, v):
        "Directed edge from vertex u to vertex v"
        self.graph[u].append(v)
        self.nodes.add(u)
        self.nodes.add(v)

    def mark_as_visited(self, v, visited, topological_ordering):
        visited[v] = True

        # Recur for all the vertices adjacent to this vertex
        for current_vertex in self.graph[v]:
            if visited[current_vertex] == False:
                self.mark_as_visited(current_vertex, visited, topological_ordering)

        topological_ordering.insert(0, v)

    def topological_sort(self):

        visited = [False]*len(self.nodes) # Mark all the vertices as not visited
        topological_ordering = []

        # Sort starting from all vertices one by one
        for i in range(len(self.nodes)):
            if visited[i] == False:
                self.mark_as_visited(i, visited, topological_ordering)

        return topological_ordering

# Create a sample DAG
g = Graph()
g.add_edge(5, 2)
g.add_edge(5, 0)
g.add_edge(4, 0)
g.add_edge(4, 1)
g.add_edge(2, 3)
g.add_edge(3, 1)

# Double check that the algorithm is correct for that DAG
assert g.topological_sort() == [5, 4, 2, 3, 1, 0]


Extract details from bibliometrics data I have set of bibliometrics data (references). I want to extract the author names, title and the name of the conference/journal from it. Since the referencing style used by different papers vary, I am interested in knowing if there are any per-existing tools to do it?

I am happy to provide examples if needed :)

To learn from traditional software in that domain you could start at https://en.wikipedia.org/wiki/Comparison_of_reference_management_software


Do modern R and/or Python libraries make SQL obsolete? I work in an office where SQL Server is the backbone of everything we do, from data processing to cleaning to munging. My colleague specializes in writing complex functions and stored procedures to methodically process incoming data so that it can be standardized and put to work in reports, visualizations, and analytics projects. Prior to starting here, I had very little experience with SQL, aside from writing the most basic of queries. The vast majority of my analysis prep work was all done in R. My boss insists that I improve my SQL skills, even though there seem to be very few assignments that can't be done more efficiently and with far fewer lines of code using R packages like dplyr, data.table, and tidyr (to name a few). My question is--does this make sense?

A couple weeks ago, I found myself faced with the task of getting a list of column names for each row in a table that met certain criteria and concatenating them into a vector of strings. There was a tight deadline and at the time, I was experiencing some blockage and couldn't quite wrap my head around the problem. I asked my boss, who in turn asked my colleague to write a script TSQL to solve the problem. While he was working on it, I figured out a way to do it in R writing a fairly simple function and applying it over the data frame. My colleague came back with his script about two hours later. It was at least 75 lines comprising two nested for loops. I asked him to tell notify when it was finished running and he said it would take several hours. Meanwhile my R script was able to loop over the ~45,000 records in about 30 seconds.

Am I right to assume that R is a much better choice for cleaning and munging data?  Maybe the SQL developer in my office is just inept?  I'm curious if anyone who has worked with both R and SQL (or Python and SQL for that matter) have any thoughts on this.

R and SQL are two completely different beasts. SQL is a language that you can use to query data that is stored in databases as you already experienced. The benefits of SQL versus R lays mostly in the fact of the database server (MS SQL, Oracle, PostgreSQL, MySQL, etc.).

Most, if not all, modern database servers permit multiple users to query data from the same data source and insert, update and delete data in the same tables all while ensuring that the data remains consistent. This is essential for say recording a bank transaction. Can you imagine running a bank on R? That's where database servers come in. They ensure ACID properties of procedures run on the database. ACID stands for Atomicity, concurrency, isolation and durability (see ACID description on wikipedia). R is a single user platform where everything happens in memory. So, if your computer stops working halfway in a big operation, your data will not be stored. You are also the only person who can access the data. To be clear, R is not considered an alternative for database servers and/or SQL.

Another main advantage of database servers is that a good database design will ensure that you can query your database fast by performing query optimization. To achieve this database servers keep track of the design of a table. See for a full discussion of this topic the wiki page. R cannot perform query optimization. Poor database design,  can lead to slow execution of your queries. Database servers can also perform optimization over queries that query multiple tables if foreign keys are properly used in the database design.

The SQL language has a very different syntax and I share your experience that it is shorter to write data munging steps using the data table or dplyr syntax. However, sometimes your data is too big for R or you need to store the results in the database as part of a periodic batch job, which will require to code your logic in SQL.

In my experience there are particular use cases for SQL and R/Python. SQL is great for storing business critical data and for allowing multiple people to access, modify, insert and delete data in a centralized environment. For any one-off data munging R and Python are great. If your data munging needs to be periodically executed, you will need to port your R/Python script to SQL.


Grouping domain specific words/phrases with same meaning I am looking at NLP methods to group together words/phrases which could have the same meaning. For example, in the sentence 'the table is broken' broken could be replaced by the following words/phrases and the sentence would still have the same meaning.

Broken: damaged, ruined, busted, unfit for purpose, missing a leg

I want to do this for texts that contain domain specific and colloquial jargon so existing NLP solutions may not be suitable?

My intention is to do this as a bridging step between named entity extraction and named entity linking.

I suggest you use word2vec for that task. Word2vec is an unsupervised algorithm that calculates N-dimension embeddings for the words in the corpus used for learning. Basically, it gives you a numerical representation in the form of an n-dimension array for each of the words you use in your inputs.

Once the model is built and the emeddings are available, to find similar words is as easy as calculate the similarity between arrays. This is done in word2vec either by looking for the most similar:

>>>result = word_vectors.most_similar(positive=['woman', 'king'], negative=['man'])
>>>print("{}: {:.4f}".format(*result[0]))
queen: 0.7699

or directly comparing terms:

>>> result = word_vectors.similar_by_word("cat")
>>> print("{}: {:.4f}".format(*result[0]))
dog: 0.8798

Have a look here for more examples.

To answer this :

  I want to do this for texts that contain domain specific and
  colloquial jargon so existing NLP solutions may not be suitable?

If you train the model with your own domain specific corpus, you should't have any problems; provided of course that it is long and rich enough.


word2vec -storing a word and its vector yet be able to efficiently run k-nearest neighbour After training a model using word2vec I'd now like to store the trained model with the word serving as a key and the vector as its value. However I'm not sure how I'll be able to implement in this way a k-nearest neighbour search. What would be the correct way to get around this?

A most popular way of obtaining the approximate nearest neighbors is the locality sensitive hash. And here are some practical results. Then once you have the neighboring keys, it's straightforward to use a key-value store to retrieve the corresponding words.


What is the difference between continual learning and active learning? As per my understanding, active learning is a kind of continual learning. Is there any difference between them?

Active Learning:

The main hypothesis in active learning is that if a learning algorithm can choose the data it wants to learn from, it can perform better than traditional methods with substantially less data for training.

Active learning (AL) augments the learning process with a model that guides and extends the learning process. Using AL, the agent can ‘actively’ gather labels from multiple sources, or curate its labels better (including rejecting/refining incorrect or incomplete labels), or embed supplementary information on causality or complications among the data that require added effort in resolving difficulties within the process of labeling or in applying existing learning to similar problems.

Continual learning

Continual learning is the ability of a model to learn continually from a stream of data. In practice, this means supporting the ability of a model to autonomously learn and adapt in production as new data comes in.


GTX 1660 Ti vs. RTX 2060 for a deep learning pc I'm planning to build a PC for Deep Learning, after the launch of AMD Ryzen 3rd gen processors. My budget will allow me to choose between a GTX 1660 Ti and RTX 2060. I'm tabulating the specs of both the cards below.

╔════════════════╦════════════╦══════════╗
║      GPU       ║ GTX-1660Ti ║ RTX-2060 ║
╠════════════════╬════════════╬══════════╣
║ GPU-cores      ║ 1536       ║ 1920     ║
║ Tensor-cores   ║ N/A        ║ 240      ║
║ RT-cores       ║ N/A        ║ 30       ║
║ Texture-units  ║ 120        ║ 120      ║
║ Render-outputs ║ 48         ║ 48       ║
║ Base-clock     ║ 1500       ║ 1365     ║
║ Boost-clock    ║ 1770       ║ 1680     ║
║ Mem-bus-width  ║ 192-bit    ║ 192-bit  ║
║ Mem-bandwidth  ║ 288GB/s    ║ 336GB/s  ║
║ Mem-capacity   ║ 6GB        ║ 6GB      ║
║ TDP            ║ 120W       ║ 160W     ║
╚════════════════╩════════════╩══════════╝

Given this comparison, I'd like to know which one would be a better choice for me. Would the RTX-2060 be worth the extra 70$ for the supposedly better performance? I had an initial look at the usefulness of tensor cores in Deep Learning applications, and found out the following article snippet from Nvidia website.

  Note About Tensor Cores

  Designed specifically for deep learning, Tensor Cores on newer GPUs
  such as Tesla V100 and Titan V, deliver significantly higher training
  and inference performance compared to full precision (FP32) training.
  Each Tensor Core provides matrix multiply in half precision (FP16),
  and accumulating results in full precision (FP32). This key capability
  enables Volta to deliver 3X performance speedups in training and
  inference over the previous generation. All samples are optimized to
  take advantage of Tensor Cores and have been tested for accuracy and
  convergence.

I would have gone with the RTX.
If I was to design deep learning models,I would be prefer more number of cores in the GPU , where RTX is the winner.

Lets not forget the Tensor cores which are present in RTX , which makes it stand out for Deep Learning processing. (Faster Tensor operations).

Go with RTX if you can afford it.


Train and Test Data Split for a Tweet Classification Task I am trying to train a few Machine Learning (ML) algorithms such as SVM, NB and Random Forest to do binary classification on disaster tweets. During this project, I want to train ML algorithms for a combined disaster dataset. However, I want to apply it to individual disasters separately (e.g, train on multiple disasters, apply to earthquake; train on multiple disasters, apply to flood). I have the combined dataset in one pandas DataFrame and individual disasters in separate DataFrames.
In this case, how should I split my train and test datasets? At the moment, I split my combined dataset and apply ML algorithms.
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(all_disaster['cleaned'], all_disaster['target'], test_size=0.33, random_state=42)

But, I feel that is not the proper way, as I want to apply it into individual disasters.

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(all_disaster['cleaned'],specific_disaster['target'], test_size=0.33, random_state=42)

Like this, you train your model on all the disasters data, and you test it on a specific disaster.


Proof subtracting baseline doesn't influence gradient can be used to show no gradient exist at all? I am using David Silver's course in RL to help me write my thesis. However, I am baffled by the proof given in lecture 7 slide 29:
slideshow

\begin{align}
\mathbb{E}_{\pi_\theta}[\nabla_\theta \log_\theta (s,a)B(s)] &= \sum_{s \in S}d^{\pi_\theta} (s) \sum_a \nabla_\theta \pi_\theta(s,a)B(s)\\
&=\sum_{s \in S} d^{\pi_\theta} B(s) \nabla_\theta\sum_{a \in A} \pi_\theta(s,a)\\
&=0
\end{align}

Consider in this proof replacing $b(s)$ with the critic's quality estimate $Q_w(s,a)$ (see previous slide(s)). How does this proof not also show that the gradient of the objective function $\nabla_\theta J(\theta)$ should also be $0$? Does this have to do with the second summation term changing from being over $a$ to over $a \in \mathcal{A}$?

Thank you.

After thinking about this, I've realized that $Q(s,a)$ relies on the action and thus cannot be pulled out of the sum in the same way $B(s)$ can. I'm leaving this up for anyone interested in the same thing.


How to use Statistical Learning theory in real analysis I begin the post trying to say that i don't know if this post is in compliance to community rules, so pardon me for any abuse.

I studied back at the university statistical learning theory. I studied PAC learning, VC Dimension, Uniform convergence etc etc. Recently i watched this talk with Vapnik https://www.youtube.com/watch?v=STFcvzoxVw4&t=1346s in which he claims that deep learning is essentially a "bla bla interpretation" and also claims that 'every problem can be solved with statistical learning theory'.

I'm very confused about this. I can't see how I can apply statistical learning theory on real problems.
Let's suppose I'm facing a new dataset with a clear task of binary classification, with many features and lots of training data. How am I supposed to check for example, if a hypothesis class H is PAC learnable, or in other words if it has a finite VC dimension? Don't take my example too literally, I just want to know if someone can point me out to an article,blog or some kind of answer that clearly shows how we can use this theorems and results in a real analysis.

Thank u

the development of the basic calculus , necessary for both formulating problems and numerical techniques for finding the minimum of a function
the theoretical development of theory of learning, such as the VC theory, in the same way it is used in statistics to do things like prove the central limit theorem


How do I deal with the fact that I have images which are not consistent with the class they belong in an image classification problem with CNN? I am really new to Neural Networks and to Machine Learning in general, and I have been given a dataset composed by images for performing multi-class image classification with a CNN.
The images were already divided into classes, and looking at the images I have noticed that some of them are complitely different from the class they belong, for example If I have a class Fruits, with images of fruits, in the folder of this class I have some pictures of cars, people,..., which of course are not fruits and neither belong to any other class in the classification problem.
The problem is that this creates some problems when I train my CNN, and this results in a low accuracy, infact I cannot go above 0.5.
How do I deal with the fact that I have images which are not consistent with the class they belong?

There are different ways of doing this, but the final idea in regard to this is that you need to clean your dataset.

You could go through it manually and then separate the images. This is extremely slow if you're dealing with a large dataset.

A faster less robust way would be to do a Principle component analysis

step1 :  Do PCA to reduce the dimensionality to a 2D or 3D space
step2 : Plot and see if there are any clusters. Usually, images which are different fall into different clusters
step3 : Cluster it via a convex clustering algorithm like K-means
step4 : Store the images belonging to a cluster into a folder.
step5 : Go through the folders and make the necessary cleaning.


what is the one hot encoding for cancer data classification I am working on a project to classify lung CT dataset using CNN and tensorflow, I know that the order for the category is cancer/no-cancer (only 2 classes), in more than one Github repository I see that they did one hot encoding like the code below:

if label == 1:
    label = np.array([0, 1])
elif label == 0:
    label = np.array([1, 0])

what makes me confused is: 1 means cancer and 0 means no-cancer, as I understand it should be:

if label == 1:
    label = np.array([1, 0])
elif label == 0:
    label = np.array([0, 1])

but why they did one hot encoding like this, I don't know I am wrong or there is another thing that I did not understand, can anyone explain it for? or give me a better way to do encoding for my data, but with code?

Both ways would work equally, but the way you see in the github repo is more standard.

The standard way of converting a integer label $y_i$ (from 0 to K-1) into a one-hot vector encoding is by creating a all-zero vector of length K, and set the element indexed by $y_i$ to be 1, i.e.

label_one_hot = np.zeros(k)
label_one_hot[label] = 1


Linear Regression Loss function for Logistic regression I was attending Andrew Ng Machine learning course on youtube Lecture 6.4
He says what a cost function will look like if we used Linear Regression loss function (least squares) for logistic regression

I wanted to see such a graph my self and so I tried to plot cost function J with least square loss for a losgistic regression task.
Here is my code
import matplotlib.pyplot as plt
import numpy as np
import math

x = np.random.rand(10000)
# x = np.array([0.1, 0.2, 0.7, 0.4])
y = np.round(np.random.rand(10000))
b = 1

J_list = []
w_list = []
for w in np.arange(-500.5, 500.5, 0.05):
    J = (1/10000)*np.sum((1/2)*np.square(((1/(1 + np.exp(-1*(w*x + b)))) - y)))
    J_list.append(J)
    w_list.append(w)

df = pd.DataFrame()
df['w'] = w_list
df['J'] = J_list

import seaborn as sns
sns.lineplot(x='w', y='J', data=df)

The output of lineplot is

Note: w in my code in theta in Andrew Ng's lecture
If anyone can help me spot my mistake, would be really appreciated.

You have taken x,y from random space. So, it should not follow any rule/logic.
Get a real data e.g. Iris(just 2 class) or using sklearn make_classification module
 Figure out the approx value of theta for a good model
 Then loop before and after this value
 Have y_true in probability(not the Class)


What should you do with attributes that predictive in an interaction? I am trying to predict results of football games. Some of our attributes only give meaning for a prediction only when they are considered in interaction with another attribute. To illustrate, a team with an average possession of 48.6 has much higher possession than some teams and much lower than others. Should we modify the features in such a way that we would use the difference between the two teams, instead of two values? On the other hand some algorithms might be good at finding this interactions?

Should we modify the features in such a way that we would use the difference between the two teams, instead of two values?

Yes, you should definitely design your features "to help" the ML algorithm, this is called feature engineering. Be careful though that there can be many different options for representing the contrast in "average possession": you can think of the difference indeed, but also a ratio or many other arithmetic combinations.

  On the other hand some algorithms might be good at finding this interactions?

Most ML handle take into account interactions between features to some extent, but it depends what kind of interaction and which ML algorithm.


How do I approach grouped anomaly detection? I currently have a large stream of data, with points such as HTTP request/response codes (200, 404, 500, etc.). Essentially, I want to perform anomaly detection for when too many signals that are NOT 200 are received. This means that the signal to be analyzed is dependent on groupings of data points (i.e., a single data point indicating that a 404 was sent is not good enough, only a cluster of 404's close together with respect to time means something).

Is there a good algorithm/methodology to approach this? I was previously thinking of having a moving window counter of the previous codes and basing it off of this, but I feel like the window size for example, is very subjective, and I am not sure how to tune that.

It seems that your solution does not have to include an "anomaly detection" in context of data analysis, but can more easily be solved with an iteration through the vector assuming a chronological order of the data (you did not mention any other feature so the problem is 1D).

I would suggest the following approaches:

1) Iterate through the vector and if you find N > threshold non-2xx responses in a row, mark starting index of a "cluster".

    vector<int> status, indices;
    int threshold = 10, counter = 0, flag = 0;
    status = read_data("data.csv");
    for(int i = 0; i < status.size(); i++) {
            if (status[i] < 200 && status[i] > 299) {
                    if (++counter >= threshold && !flag) {
                            indices.push_back(i - counter);
                            flag = 1;
                    }
            } else {
                    counter = 0;
                    flag = 0;
            }
    }
    return indices;

2) Once you cross over some non-2xx example, initialize the weight to, let's say 0.0 (zero). For every new negative example, subtract, let's say 0.2 and for a positive example add 0.05 or something and if you end up bellow -2.0 or above 2.0, reset the procedure and in case that it's < -2.0, return the starting index. You can play around with given numbers.

3) Some more complex strategy?

But yes, your solution does not have to include a machinery, try to start from something more simple and it will lead you in the right direction.


More weightage to a categorical feature for an Autoencoder model I am using autoencoder for anomaly detection. I don't have any labels already and so its unsupervised. If I have categorical variables, I usually one hot encode before giving it to the model. I would like to know if we there is a possibility to give more weights to one particular categorical feature before giving it to the model.

Any help is much appreciated!!

Do you mean giving more weights to a selected part of the input? If so, then, I believe this is possible with neural networks. Each neuron is assigned a random weights at the beginning of training. You can just initialize the first layer of the neural network model with some values representing the desired weights and then freeze those neurons.

An automated way of doing this is by using attention mechanism which automatically provides more weights (attention) to a portion of the input data.


How are new neural network architectures 'discovered' This might not be on topic here, but fingers crossed.

There are all manner of neural network architectures out there, everything from convolutional networks to deep recurrent networks (and even deep recurrent convnets). Some network arcitecture development I can kind of see the intuition behind; recurrent networks allow for feedback loop type things to be taken into account. Other ideas seem to come out of left field and change the field. Yolo is a good example of this - I understand how it works, I've implimented it on my own dataset. But how did someone come to the set up that allowed yolo to work.

Is it just creativity, trying random architectures and new ideas to see what sticks? Is there some methodology that I could apply to take some steps towards a new ground-breaking architecture? (I understand that I won't actually make the next new architecture, but it's interesting to understand the process)

Think of neural networks as math, and ask yourself how is new math discovered? New math comes from a combination of theory, experimentation and feedback from community.

Instead of thinking about it terms of creativity vs randomness to see what new ideas stick; I prefer to think in terms of theory and experimentation.

The current methodology for new neural net discovery is no different than any other applied field of math/engineering.


TF-IDF vectorizer doesn't work better than countvectorizer I am working on a multilabel text classification problem with 10 labels. The dataset is small, +- 7000 items and +-7500 labels in total. I am using python sci-kit learn and something strange came up in the results. As a baseline, I started out with using the countvectorizer and was actually planning on using the tfidf vectorizer which I thought would work better. But it doesn't.. with the countvectorizer I get a performance of a 0.1 higher f1score. (0.76 vs 0.65)

I cannot wrap my head around why this could be the case? There are 10 categories and one is called miscellaneous. Especially this one gets a much lower performance with tfidf.

Does anyone know when or why tfidf could perform worse than count?
I need to formalize this for my thesis.

There are a few possibilities. First, there is some variability in performance. It could have been only by chance that countvectorizer performed better than tf-idf. Did you use cross validation (with how many folds)? Is the superior performance of the countvectorizer reliable? I would compare performance across folds to make sure countvectorizer consistently performs better.

Second, if you find that countvectorizer reliably outperforms tf-idf on your dataset, then I would dig deeper into the words that are driving this effect. It may be that common words (words which will appear in multiple documents) are helpful in distinguishing between classes. There is substantial research that shows that use of some function words (e.g. first person singular pronouns, “I”) change depending on someone’s psychological state. Function words like pronouns are very common and would be down weighted in tf-idf, but given equal weight to rare words in countvectorizer. I’m not suggesting that first person singular pronouns in particular are driving your results, but it’s worth looking at what words are driving the effect. I would examine which words are important in both types of models, countvectorizer and tf-idf, and then think about whether the words that are most important for the countvectorizer make sense in the context of your text documents and labels. Also, are you removing stop words? You could also see how the models perform with and without stop words, which would be another way to test whether frequent words are actually helping you to distinguish between classes.


loss/val_loss decrease but acc/val_acc are consistent I don't know why I am getting such good results.

Epoch 3/10 2937/2937 [==============================] - 12s 4ms/step -
loss: 0.2836 - acc: 0.4679 - val_loss: 0.1937 - val_acc: 0.1980

Epoch 4/10 2937/2937 [==============================] - 12s 4ms/step -
loss: 0.1355 - acc: 0.4679 - val_loss: 0.0866 - val_acc: 0.1980

>Epoch
5/10 2937/2937 [==============================] - 13s 4ms/step - loss:
0.0580 - acc: 0.4679 - val_loss: 0.0342 - val_acc: 0.1980

Epoch 6/10 2937/2937 [==============================] - 13s 4ms/step -
loss: 0.0223 - acc: 0.4679 - val_loss: 0.0120 - val_acc: 0.1980

Epoch 7/10 2937/2937 [==============================] - 14s 5ms/step -
loss: 0.0082 - acc: 0.4679 - val_loss: 0.0040 - val_acc: 0.1980

My training and label sets are float number arrays in range [-0.05, 0.05] and I am using Keras.sequential.model.lstm. Why might this be happening? Previously, I had the opposite problem here: loss/val_loss are decreasing but accuracies are the same in LSTM!, but I couldn't understand the problem.

EDIT: I changed my code from:

model.compile(optimizer = 'adam', loss = 'mean_square_error', metrics=['accuracy'])

to:

model.compile(optimizer = 'adam', loss = 'mean_absolute_error', metrics=['accuracy'])

But the result is same.

I then changed the above line of the code to:

model.compile(optimizer = 'adam', loss = 'mean_squared_error', metrics=['mean_squared_error'])

But it didn't work and the result is as follow:

Train on 2937 samples, validate on 735 samples Epoch 1/10 2937/2937
[==============================] - 90s 31ms/step - loss: 1.6645 -
mean_squared_error: 0.0019 - val_loss: 0.7620 -
val_mean_squared_error: 0.0010

Epoch 2/10 2937/2937 [==============================] - 13s 4ms/step -
loss: 0.5503 - mean_squared_error: 0.0019 - val_loss: 0.3890 -
val_mean_squared_error: 0.0010

Epoch 3/10 2937/2937 [==============================] - 13s 4ms/step -
loss: 0.2837 - mean_squared_error: 0.0019 - val_loss: 0.1938 -
val_mean_squared_error: 0.0010

Epoch 4/10 2937/2937 [==============================] - 13s 4ms/step -
loss: 0.1355 - mean_squared_error: 0.0019 - val_loss: 0.0866 -
val_mean_squared_error: 0.0010

Epoch 5/10 2937/2937 [==============================] - 13s 4ms/step -
loss: 0.0580 - mean_squared_error: 0.0019 - val_loss: 0.0342 -
val_mean_squared_error: 0.0010

Epoch 6/10 2937/2937 [==============================] - 13s 4ms/step -
loss: 0.0223 - mean_squared_error: 0.0019 - val_loss: 0.0120 -
val_mean_squared_error: 0.0010

Epoch 7/10 2937/2937 [==============================] - 13s 5ms/step -
loss: 0.0082 - mean_squared_error: 0.0019 - val_loss: 0.0040 -
val_mean_squared_error: 0.0010

Epoch 8/10 2937/2937 [==============================] - 14s 5ms/step -
loss: 0.0035 - mean_squared_error: 0.0019 - val_loss: 0.0017 -
val_mean_squared_error: 0.0010

Epoch 9/10 2937/2937 [==============================] - 13s 5ms/step -
loss: 0.0022 - mean_squared_error: 0.0019 - val_loss: 0.0011 -
val_mean_squared_error: 0.0010

Epoch 10/10 2937/2937 [==============================] - 13s 5ms/step
- loss: 0.0019 - mean_squared_error: 0.0019 - val_loss: 0.0010 - val_mean_squared_error: 0.0010

In your code:

model.compile(optimizer = 'adam', loss = 'mean_absolute_error', metrics=['accuracy'])

You are using accuracy as a metric, metrics=['accuracy']. This does not work when you are doing regression (which you are doing). Regression is when you have continuous (floats) labels.

So instead your code should look like:

model.compile(optimizer = 'adam', loss = 'mean_squared_error', metrics=['mean_absolute_error'])

Then when you train you should look for both loss and your metric to be decreasing during training. If they are, then your network is behaving normally.


RCNN - recurrent convolutional neural networks - what is a time step? I am currently studying this paper for implementing it into a speech recognition application, but seem to have some problems understanding the concepts of time step.  What is a time step?

I haven't worked with RNN before but would guess that in a continuous stream of data time step would be the data present at time t.

but how do one even implement/train something like that in an iteration based system?...

I mean something like keras, one is not able to involve output received from t-1 to t... or if so, how?, I seem to be unfamiliar with this...

In another paper i found this

  To understand the essen ce ofthe RCL, it is useful to c1arifY the
  concept oftime step in RCL. It is not identical to the time associated
  with the sequential data, and instead it refers to an iteration during
  processing the data.

from RECURRENT CONVOLUTIONAL NEURAL NETWORK FOR SPEECH PROCESSING

which then further conforms my theory of it being being about the data that was involved/processed as iteration t-1.

A couple of vocab words:

layer: An entire set of receptive units with shared weights that creates a convolutional feature map

Unit: a single nxn neighborhood filter operator in a given layer of a CNN (say 3x3 or 5x5, whatever given filter size you are using)

stride length: The distance, or padding, in between the center of every unit of a layer

In a convolutional layer, the convolution operation can be thought of like a sliding window, in which the filter from a given layer is applied to the top left corner patch of pixels, then slid to the left 1 horizontal stride length and repeated until the end of the row, then it hops down 1 vertical stride length and starts again, and so on and so forth. This is implemented in a CNN as a 2D grid of units, all sharing the same weights for convolution (this produces the same result as a sliding window).

The RCL takes the sliding window concept of convolution and adds memory to it, so the filter within each unit will be modified based on the response of the previous unit. The time step means how many strides back you want to directly include in the calculation of the current filter. So, if I have an RCL with a filter size of 5x5, and a stride of 1 for an image size of 100x100, a single convolutional layer (assuming it is padded for the filter radius) will have a grid of 100x100 units. Utilizing a time step of N means that you will include response results from the previous N units on the grid to calculate the new unit response (or, more accurately, the unit weights in backprop).


How should multiclass classifier performance be measured when one type of error is preferred over another? Sorry if this question has been asked before--I am having trouble searching this topic since I'm not sure of my wording.

Say you have a classification problem where there are more than two labels which are discrete but roughly correspond to an increase in some quality--call these labels A, B, and C. Also say in this problem it would be preferrable to over-estimate that quality, rather than to underestimate. Is there a type of metric that captures this skew and penalizes a predicted A on an actual B more than it penalizes a predicted C on an actual B? Or is this preference better handled in a different part of data science methodology?

Define a scoring table like this (You will need to tweak this table to satisfy your particular use case. I am only using it as an example).

Pred   | True Label
Label  | A    B   C
-------+--------------
A      |  0  -1  -2
B      | -3   0  -1
C      | -4  -3   0

Notice this scoring table has the property of favours over-estimate than under-estimates.

Multiply (element-wise) the scoring table with the number of predictions fall into each of the 9 possible scenarios then sum the scores together will give you a metric with the desired properties.


Is the multilayer perceptron only able to accept 1d vector of input data? If yes, why is this so? I am going through the tutorial at the link below which uses MNIST handwritten digit database.

https://machinelearningmastery.com/handwritten-digit-recognition-using-convolutional-neural-networks-python-keras/

The 28x28 sized image data has to be reshaped into a 1D vector of 784 pixels. 28x28=784. Why does the multilayer perceptron insist on only a 1D vector of input data? There are no problems with convolutional and recurrent neural networks accepting input shape of higher dimensions.

You could say every type of neural network gets 1d input data. It's just more convenient to think about 2d-CNNs taking 2d data because the convolution operation is best illustrated by moving squares across a grid, and similarly for max-pooling.

But you could easily write out all the multiplications, additions, and max operations you performed in one line of algebra. And in this line you could easily flatten the 2d input indices into 1d indices. The information passed in and the calculations and output of the network would be exactly the same, but this representation would lose its real-world interpretation so there's no reason to think of it this way.

The point is that there are different ways of representing exactly the same information, and we choose a 1d representation for FNN inputs and sometimes higher dimensions for CNN inputs because it corresponds to the physical structure of the real-world problem and it is easiest think about. This is why most libraries have you input the data in these shapes. But if you think about what's happening at the lowest level of computation on your computer you won't necessarily find the same structure.


Make column as dictionary key and row as value in pandas dataframe I have Dataframe as below.

name    value     id
A       123         1
B       345         5
C       567         4
D       789         2

I need to create a dictionary of the form below

{
  {"name":"A",
   "value":123,
   "id":1
  },
  {"name":"B",
   "value":345,
   "id":5
  },
  {"name":"C",
   "value":567,
   "id":4
  },
  {"name":"D",
   "value":789,
   "id":2
  }
}

I tried below code, but it makes 'name' as the key.

data_set.set_index('name').to_dict(orient='list')

How to get the dictionary of the form i have mention? Do i need to iterate through rows?

Check this:

data_set.to_dict(orient = 'records')


Using the curse of dimensionality for encoding non-ordered (nominal) categorical variables of high cardinality When the dimension is high, all data are approximately at the same distance away from each other. This makes distance-based methods such as k-nearest neighbors less useful if the data are more or less uniformly distributed. This is also referred to as the curse of dimensionality.
However, why not to make lemonade out of this dimensionality lemon?
Why not to encode nominal categorical variables of high cardinality with randomly distributed points in some high-dimensional space but such that the number of dimensions is much less than we would get with the one-hot encoding?
As the distance between any two points is almost the same, no artificial ordering will be introduced. The encoding is very fast. No need to calculate embedding with some neural network.
Will this encoding work? Will it be meaningless or introduce any bias? Is anybody using it? Do you know any references?

When the dimension is high, all data are approximately at the same distance away from each other.

Yes, but this is the worst case scenario of the curse of dimensionality:

This happens only if the data is extremely sparse. In real cases it's not really "all data", because some data points are less sparse than others, and it's not really "the same distance", because there can be small but meaningful differences in the distance between points.
There are known mitigation measures which can prevent this worst case scenario to happen, in particular various feature selection/extraction methods. So in real cases it's simply a mistake to work with some very high dimensionality data left "untreated".

Why not to encode nominal categorical variables of high cardinality with randomly distributed points in some high-dimensional space but such that the number of dimensions is much less than we would get with the one-hot encoding?

This would correspond to a random feature extraction method: instead of trying to group similar features together when reducing the dimensions, features are just randomly projected into a low-dimension space. The disadvantage is clear: instead of simplifying the data while preserving the patterns, the patterns in the data are just ignored. It's likely that the data would become pretty much useless after that.
The goal is not just to reduce dimensionality for the sake of it, it's to reduce dimensionality in a way which leaves the most important characteristics of the data as intact as possible, or even emphasize them by removing the noise. This method would be equivalent to randomly selecting features instead of selecting the most informative ones.
For example, in the case of text data there is often high dimensionality due to the high number of distinct words. A basic but very effective method consists in discarding words which appear rarely: this reduces the dimensions a lot (due to the Zipf distribution of words in a text), prevents noise due to words which happen by chance, and preserves most of the patterns which happen with moderately frequent words.


Non-Real Time Data Augmentation for CNN Classification. What are the drawbacks? When people talk about and use data augmentation, are they mostly referring to real-time data augmentation? In the case of image classification, that would involve augmenting the data right before fitting the model, and a new augmented image is used every epoch. In this case only augmented images are used to train the model and the raw image is never used, so the size of the input doesn’t actually change.

But what about non-real-time data augmentation? By this, I mean augmenting the data in the preprocessing stage, so that you literally expand the sample size of your input. Then you feed all those augmented images along with the original into the cnn, so it’s the same images every epoch. Is this a valid idea? Has it been done and what are the drawbacks? Any logical fallacies or objections by data science and machine learning experts?

Thanks!

Yes, they are mostly referring to real-time data augmentation.

It is not said that raw images are never used. Actually, it depends on what kind of data augmentation function you are using. Original images can either be kept intact or distorted.

Data augmentation si done real time pretty much 100% of the times for the following reason: data would be too big to be stored on your hard drive. Even the smallest image datasets could take hundreds of GBs after augmentation.