How to specify important attributes? Assume a set of loosely structured data (e.g. Web tables/Linked Open Data), composed of many data sources. There is no common schema followed by the data and each source can use synonym attributes to describe the values (e.g. "nationality" vs "bornIn").

My goal is to find some "important" attributes that somehow "define" the entities that they describe. So, when I find the same value for such an attribute, I will know that the two descriptions are most likely about the same entity (e.g. the same person).

For example, the attribute "lastName" is more discriminative than the attribute "nationality".

How could I (statistically) find such attributes that are more important than others?

A naive solution would be to take the average IDF of the values of each attribute and make this the "importance" factor of the attribute. A similar approach would be to count how many distinct values appear for each attribute.

I have seen the term feature, or attribute selection in machine learning, but I don't want to discard the remaining attributes, I just want to put higher weights to the most important ones.

A possible solution is to calculate the information gain associated to each attribute:
$$I_{E}(f) = - \sum \limits_{i = 1}^m f_ilog_2f_i$$
Initially you have the whole dataset, and compute the information gain of each item. The item with the best information gain is the one you should use to partition the dataset (considering the item's values). Then, perform the same computations for each item (but the ones selected), and always choose the one which best describes/differentiates the entries from your dataset.
There are implementations available for such computations. Decision trees usually base their feature selection on the features with best information gain. You may use the resulting tree structure to find these important items.


Curse of Dimensionality : How many dimensions is too many dimensions? Say I have a dataset with 1000 columns and 3M rows.
I know that this is will definitely suffer from Curse of Dimensionality and that I need to reduce the number of dimensions. But to what extent am I supposed to reduce the dimensions by?

With each additional dimension the number of datapoints required such that the data is not too sparse increases exponentially according to my understanding.

So how do I know know for different number of columns what the golden number of data points is? Assuming that I have the capability to collect infinite amount of data but would still have a small cost associated for each datapoint, how much should I collect?

I am using these slides to understand the concept:
http://www.dataminingbook.info/pmwiki.php/Main/BookPathUploads?action=download&upname=slides-chap6.pdf

As karthikeyan mg mention in his answer, you could use the explained variance score to get an idea of how many columns you can drop. Unfortunately, there isn't a magic number to know in advance. If you write code in Python, you should read this blog post in towardsdatascience.com

An Approach to Choosing the Number of Components in a Principal Component Analysis

After you rescale your data to 0-1, you can run this snippet of code and get a plot of variance loss for each components number you choose.

#Fitting the PCA algorithm with our Data
pca = PCA().fit(data_rescaled)
#Plotting the Cumulative Summation of the Explained Variance
plt.figure()
plt.plot(np.cumsum(pca.explained_variance_ratio_))
plt.xlabel('Number of Components')
plt.ylabel('Variance (%)') #for each component
plt.show()

This is the result from the blog post. As you can see, the variance starts to drop a lot after the 5 components. So, this number might be the one you could use.


Why does this paper say that 0-1 loss is insensitive to scaling of weights in a neural network? When discussing capacity control using norms of weights in a neural network,this paper says the following(see P4):

  Capacity control in terms of norm, when using a zero/one loss (i.e. counting errors) requires us in
  addition to account for scaling of the output of the neural networks, as the loss is insensitive to this
  scaling but the norm only makes sense in the context of such scaling. For example, dividing all the
  weights by the same number will scale down the output of the network but does not change the $0 / 1$ loss, and hence it is possible to get a network with arbitrary small norm and the same $0 / 1$ loss. Using
  a scale sensitive losses, such as the cross entropy loss, does address this issue (if the outputs are scaled
  down toward zero, the loss becomes trivially bad), and one can obtain generalization guarantees in
  terms of norm and the cross entropy loss.

I'm a little bit confused about this paragraph, since it seems that the 0-1 loss is not insensitive to weight scaling. For example, if we scale down the weights, the output would also be scaled down, so the 0-1 loss would definitely be changed(since the true labels are not changed).Can anyone explain to me which part of my understanding goes wrong? Also, I would like to know how the cross-entropy loss settles this problem.

Zero-one loss only really makes sense for (hard) classification, so the output of the network needs to already include a discretization.  Presumably the authors have a particular system in mind, and scaling weights doesn't affect the discretized predictions.

Two possibilities (though very closely related) come to mind, though I can't tell whether the authors had either of them in mind.  If the final output of the model is on a linear scale, and the decision is based on the sign of the output, then scaling weights will not change the sign and hence not the decision.  Similarly, if the final output of the model is on a probability scale via a sigmoid activation, then scaling the weights will not change the log-odds's sign, hence will preserve whether the output is above or below 0.5, hence not change the final decision (assuming a 0.5 cutoff).


Can distortion be derived from inertia rather than recalculating it from scratch in case of kmeans? I got this definitional difference between distortion and inertia from here:

Two values are of importance here — distortion and inertia. Distortion
is the average of the euclidean squared distance from the centroid of
the respective clusters. Inertia is the sum of squared distances of
samples to their closest cluster centre

However, when I searched for an example from here:

distortions.append(sum(np.min(cdist(X, kmeanModel.cluster_centers_,
                                        'euclidean'), axis=1)) / X.shape[0])

I took one inertia value from:

and 1 distortion value from :

however the values 217.64 and 3.45 aren't simply a division of 217.64 by count of variables in x (17).
Can anyone detail on what exactly is the formula for distortion?

The reason is that
distortions.append(sum(np.min(cdist(X, kmeanModel.cluster_centers_,
                                        'euclidean'), axis=1)) / X.shape[0])

does not square the distance as stated in the definition

Distortion is the average of the euclidean squared distance from the centroid of the respective clusters

If you replace
    mapping1[k] = sum(np.min(cdist(X, kmeanModel.cluster_centers_,
                                   'euclidean'), axis=1)) / X.shape[0]

with
    mapping_squared[k] = sum(np.square(np.min(cdist(X, kmeanModel.cluster_centers_,
                                   'euclidean'), axis=1))) / X.shape[0]

you will get the desired result:
IN: for key, val in mapping_squared.items():
    print(f'{key} : {val}')

OUT:
1 : 12.802768166089965
2 : 4.025210084033612
3 : 0.954621848739496
4 : 0.7467787114845938
5 : 0.5647058823529413
6 : 0.42156862745098034
7 : 0.29901960784313725
8 : 0.24019607843137258
9 : 0.16666666666666669

That is, the values in mapping_squared can be multiplied by the number of samples to match inertia:
IN: np.array(distortions_square) * X.shape[0]

OUT:
array([217.64705882,  68.42857143,  16.22857143,  12.6952381 ,
         9.66666667,   7.51666667,   5.66666667,   4.08333333,
         3.        ])

On a side note: Distortion and SSE are usually used interchangeably. See, for example, the paper Scaling Clustering Algorithms to Large Databases:

Distortion is the sum of the L2 distances squared between
the data items and the mean of their assigned cluster

This question touches the subject too.


What is the difference between model hyperparameters and model parameters? I have noticed that such terms as model hyperparameter and model parameter have been used interchangeably on the web without prior clarification. I think this is incorrect and needs explanation. Consider a machine learning model, an SVM/NN/NB based classificator or image recognizer, just anything that first springs to mind.

What are the hyperparameters and parameters of the model?
Give your examples please.

Hyperparameters and parameters are often used interchangeably but there is a difference between them. You call something a 'hyperparameter' if it cannot be learned within the estimator directly. However, 'parameters' is  more general term. When you say 'passing the parameters to the model', it generally means a combination of hyperparameters along with some other parameters that are not directly related to your estimator but are required for your model.

For example, suppose your are building an SVM classifier in sklearn:

from sklearn import svm
X = [[0, 0], [1, 1]]
y = [0, 1]
clf = svm.SVC(C =0.01, kernel ='rbf', random_state=33)
clf.fit(X, y)

In the above code an instance of SVM is your estimator for your model for which the hyperparameters, in this case, are C and kernel. But your model has another parameter which is not a hyperparameter and that is random_state.


AttributeError: 'DataFrame' object has no attribute 'as_matrix' is the error AttributeError: 'DataFrame' object has no attribute 'as_matrix' is the error

It could be that your code used to work and now it doesn't because you updated your pandas package.

If you check pandas.DataFrame.as_matrix's documentation you will see that this method has been deprecated.

Instead, you should use:

X_mat = X[['height', 'width']].values()


k-means in R, usage of nstart parameter? I try to use k-means clusters (using SQLserver + R), and it seems that my model is not stable : each time I run the k-means algorithm, it finds different clusters. But if I set nstart (in R k-means function) high enough (10 or more) it becomes stable.

The default value for this parameter is 1 but it seems that setting it to a higher value (25) is recommended (I think I saw somewhere in the documentation).

So I'm a bit confused...
Any advice ?

nstart option attempts multiple initial configurations and reports on the best one. For example, adding nstart=25 will generate 25 initial random centroids and choose the best one for the algorithm. Hope this helps!

You can read more here...


Interpret results from a lightFM factorization machines I built a recommendation model on a user-item transactional dataset where each transaction is represented by 1:

model = LightFM(learning_rate=0.05, loss='warp')

Here are the results of this recommendation model:

Train precision at k=3:  0.115301
Test precision at k=3:  0.0209936

Train auc score:  0.978294
Test auc score : 0.810757

Train recall at k=3:  0.238312330233
Test recall at k=3:  0.0621618086561

Can anyone help me interpret this result? How is it that I am getting such good AUC score and such bad precision/recall? The precision/recall gets even worse for 'bpr' Bayesian personalized ranking.

Prediction task:

users = [0]
items = np.array([13433, 13434, 13435, 13436, 13437, 13438, 13439, 13440])
model.predict(users, item)

Result from the prediction task:

array([-1.45337546, -1.39952552, -1.44265926, -0.83335167, -0.52803332,
   -1.06252205, -1.45194077, -0.68543684])

How do I interpret the prediction scores?

Thanks

The prediction scores are only used for ranking. The scores themselves do not provide more insight than that.


ValueError: Found input variables with inconsistent numbers of samples I am trying to do svm model training and it gives this error:

 ValueError: Found input variables with inconsistent numbers of samples: [91, 212]

Code:

target = data.target
target.head()

##splitiing data into training and test data
X_train, X_test, y_train, y_test = train_test_split(train, target, test_size=0.3, random_state=101)
print("X_train size ===>", X_train.shape)
print("y_train size ===>", y_train.shape)
print("X_test size ===>", X_test.shape)
print("y_test size ===>", y_test.shape)

#Create a svm Classifier

clf = svm.SVC(kernel='linear')

#training the model using the training sets

clf.fit(X_train, y_train)

#predict the response for test dataset

y_pred = clf.predict(X_train)

This error usually indicates your train and test in the train_test_split() function call are different sizes. You may need to reshape one to get them to match. Look at the shapes of train and test to see what is the problem.


Linear Regression with Category variables I'm currently learning and exploring machine learning and understand the basics of linear regression based on two numerical variables, but now I wish to go a little further and need some guidance understanding how to go about it.
Specifically, I'm now learning about linear regression with categorical variables, and I understand the gist of it: We just encode the categorical variable to some sort of numerical representation (like one-hot encoding) and put it in the model. Great.
While there are many guides on how to do various encoding methods etc. online, I haven't really found a resource that explains the use-cases of such a method: under what kind of circumstances would using categorical data to predict a numeric value be useful?
And what type of data format should I have my data in before doing the encoding? (does having two columns with one numeric results and the other the corresponding category work?)
I would also like to know the different ways we can visualize and analyse the results of our model (and its predictions), especially if we have a sizeable amount of categorical variables.
Sorry if this is too many questions in one post, I need some guidance on the concept. All the online resources are telling me how to implement the model, but not when and why to use it.

We just encode the categorical variable to some sort of numerical representation (like one-hot encoding)

The choice of representation matters, because it has to preserve the properties of a categorical variable: one-hot-encoding is a standard option, but directly encoding categorical values as integers is a mistake because it introduces order where there is none.

the use-cases of such a method: under what kind of circumstances would using categorical data to predict a numeric value be useful?

There are many applications for which regression from categorical data is useful. For example text data is often represented as "bag of words" (BoW), where each word is a categorical variable. Some tasks involve predicting a numerical target from some text input, for example:

Predicting the grade of an essay
Predicting a score representing the likelihood of plagiarism (e.g.Turnitin)
Predicting sentiment of a text on a scale from 1 to 5

There are many other examples, and not only from text data.

And what type of data format should I have my data in before doing the encoding? (does having two columns with one numeric results and the other the corresponding category work?)

The data can have as many independent variables (features) as needed. Since the categorical variables are typically one-hot-encoded, what matters is the number of features after encoding. It's often necessary to preprocess the data in order to avoid overfitting (typically rare values should be discarded).

I would also like to know the different ways we can visualize and analyse the results of our model (and its predictions), especially if we have a sizeable amount of categorical variables.

As far as I know there's no specific method for analyzing the result of the model, it's the same principle as with numerical features. If one wants to analyze the impact of an individual feature on the performance of the model, a simple technique is to train the model with/without this feature and compare the results.


Multivariate Regression Error “AttributeError: 'numpy.ndarray' object has no attribute 'columns'” I'm trying to run a multivariate linear regression but I'm getting an error when trying to get the coefficients of the regression model.
The error I'm getting is this: AttributeError: 'numpy.ndarray' object has no attribute 'columns'
Here's the code I'm using:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as seabornInstance
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn import metrics
%matplotlib inline

# Main files
dataset = pd.read_csv('namaste_econ_model.csv')
dataset.shape
dataset.describe()
dataset.isnull().any()

#Dividing data into "attributes" and "labels". X variable contains all the attributes and y variable contains labels.

X = dataset[['Read?', 'x1', 'x2', 'x3', 'x4', 'x5', 'x6' , 'x7','x8','x9','x10','x11','x12','x13','x14','x15','x16','x17','x18','x19','x20','x21','x22','x23','x24','x25','x26','x27','x28','x29','x30','x31','x32','x33','x34','x35','x36','x37','x38','x39','x40','x41','x42','x43','x44','x45','x46','x47']].values
y = dataset['Change in Profit (BP)'].values
plt.figure(figsize=(15,10))
plt.tight_layout()
seabornInstance.distplot(dataset['Change in Profit (BP)'])
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
regressor = LinearRegression()
regressor.fit(X_train, y_train)
coeff_df = pd.DataFrame(regressor.coef_, X.columns, columns=['Coefficient'])
coeff_df

Full error:

Traceback (most recent call last):
File "", line 14, in coeff_df = pd.DataFrame(regressor.coef_, X.columns, columns=['Coefficient'])
AttributeError: 'numpy.ndarray' object has no attribute 'columns'

Any help on this will be highly appreciated!

Using .values on a pandas dataframe gives you a numpy array. This will not contain column names and such. You do this when setting X like this:

X = dataset[['Read?', 'x1', .. ,'x47']].values

But then you try to get the column names from X (which it does not have) by writing X.columns here:

coeff_df = pd.DataFrame(regressor.coef_, X.columns, columns=['Coefficient'])

So store your column names in a variable or input them again, like this:

coeff_df = pd.DataFrame(regressor.coef_, ['Read?', 'x1', .. ,'x47'], columns=['Coefficient'])


How to reduce time R takes for model building I am building machine learning algorithms in my laptop. It has i3 procesor and 16 GB RAM. Despite using multiple cores(3 out of 4), it takes 2 days to run all the techniques that i am trying to run an obviously data is huge (close to 1.3 million rows and 20 variables).

Is there any way to reduce this time required for running algorithms into fraction of the time it takes currently? Lets say some hours instead of days? I have heard from a friend who has computer science background that spark takes less time than stand alone R. Not sure if Spark can reduce the analysis time from multiple days to some hours. I am open to suggestions and solutions(preferably open source). Thoughts?

I am sure a solution for this must exist as R is pretty old and some genius would have found a way to solve this painful problem.

Depends on the models you are trying to run. Your data isn't that big. For example using a support vector model from the kernlab package, you run into problems. Not every model is fast or has a fast implementation.

Without more information on what you are doing it is difficult to say what causes the bottleneck. But if you just want a speed boost in running models, have a look at the xgboost package, the h2o package (GLM, GBM, rf, deeplearning), ranger for a faster implementation of a randomforest model.


validation after hyperparameter tuning I tuned my hyperparameter with random search and i used cv=5. Is it important to validated the hyperparameter with model and testdata or is it okay to use the given back accuracy from random search?

In machine learning there is a fundamental difference between what is known as hyperparameters and parameters.
Parameters
They are the variables that define your model, or in other words the relationship between your inputs and the output you are trying to predict.
For example, in simple linear regression, the goal is to be able to make predictions for a target variable $y$ given some new input value $x$ of the input variable on the basis of a set of training data comprising $N$ input values $\mathbf{x} = (x_1, \dots, x_n)^{\mathbf{T}}$. Relationship between inputs and output is given by
$$
y = w_0 x + w_1
$$
where $w_0$ and $w_1$ are the parameters our algorithm tries to "learn" during training to produce the most accurate possible predictions for $y$. FYI the term $w_0$ is called weight and the term $w_1$ is called bias.
Hyperparameters
These are different from the parameters above in the sense that they act on the training process itself. Hyperparameters can be seen as knobs you would tweak to adjust the learning of your model in the best possible way.
An example of hyperparameter is that of the learning rate $\eta$ used to adjust how far apart from each other values are computed when looking for best possible parameters of a model using Gradient Descent. Another example is the number of decision trees to have in the forest when using Random Forest algorithm.
Hyperparameter tuning
Unlike model parameters, hyperparameters can be set by the user before training a machine learning model. To be fair the Random Search Cross-Validation method you mentioned is pretty efficient for finding most suitable values for hyperparameters given a machine learning problem. You can reasonably use results from your cross validation for hyperparameters.
Next steps should include testing your model on a held-off test data to check that your model parameters determined during training are indeed correctly setting your model up.


Student Churn prediction I am working on an ML model for student churn prediction. It is a classification problem if some student will churn or not. I have a lot of data like the student data and the activities of the student. There are two problems which I would like to ask about:

The churn of the student in the first 6 weeks
The overall churn of the student after the 6 weeks

Would you split your work between 2 models: in 6 weeks and after 6 weeks?
How would you start such a project?

Churn models often simply predict a binary output: will the student churn? Yes or No, 1 or 0. In your case there is an added component, namely the time factor of 6 weeks, so the question is more "when is the student likely to churn?".

Does your dataset include how long students stayed on the course before churning (i.e. leaving the course)?

At the highest level, you could model this problem as either a classification problem or a regression problem (with some post processing).

Classification approach

If you choose classification, you should form your target variable (the actual churn information) for each student into several discrete classes. For example, you could create 3 classes:

0 -> the student didn't churn
1 -> the student churned later than 6 weeks
2 -> the student churned within 6 weeks

Then you can select any model that can consume your data (input features) and classify each case as one of those three cases. Something like a decision tree might work well as a base line model. You could then perhaps try an SVM model.

Regression approach

In this case, you would be predicting the exact time each student will churn. For this, your dataset must contain e.g. the number of days or weeks into the course that churn students left the course.

In this case, you target variable is then simply this information, and your model will predict numbers, like 3.7 weeks or 26 weeks; you will then need to simply post-process these results into your 3 categories (as listed above).

In this case, again, you could try decision tree (regression variant) or an SVM model.


Math of Logistic regression cost function In the current scikit-learn documentation for binary Logistic regression there is the minimization of the following cost function:
$$\min_{w, c} \frac{1}{2}w^T w + C \sum_{i=1}^n \log(\exp(- y_i (X_i^T w + c)) + 1)$$
Questions:

what is the $c$ term? It is not explained in the documentation
What is the cost function minimized when LogisticRegression(multiclass=multinomial) is used instead?

The c (small one) term is bias or intercept added to the model. This is similar to intercept we add in case of linear regression. The library allows you to set bias term to zero too.
When set to multinomial model, the cost function will try to minimize cross-entropy loss. Hard for me to write down the equation here, but I always go back to this useful explanation of how cross entropy loss works and how it is minimized. In other words, a multinomial regression will work like a single layer neural network with logistic activation function.


Fewer observations & larger documents vs More observations & smaller documents Let's suppose that I have a dataset of 1000 documents.

Each document is a restaurant review (so relatively short text) and it has labels {Negative, Indifferent, Positive}.

Let's suppose that the dataset has 600 positive reviews, 200 indifferent reviews and 200 negative reviews.

I want to train a classifier to classify a review as Negative or Indifferent or Positive based on the text of the review.

I am not thinking about using any word embeddings for now so I will probably use a TF or TF-IDF model (even though this may be a bit off topic for current question).

Let's suppose that in my case I split (in a stratified way) my dataset into a training set of 800 observations and into a test set of 200 observations.

My question is the following:
Is it better to have 800 separate documents in my training set or to merge these documents based on its categories/labels and create 3 very big documents?

There 800 separate documents of any of the 3 labels or 3 big documents of each of the labels is the best way to go and why?

My question stems from the fact that in the latter case if for example I do TF-IDF then this will be applied based on different categories/labels since each document will be about a category/label.

On the other hand, if I do (as we usually do actually) like in the former case then the TF-IDF will be categories/labels-agnostic and I do not know this helps things.

Is the answer simply that this an interesting but pretty bad idea because in this way you simply massively decrease the number of the observations with which the model/algorithm is trained and so you make much harder for him to figure out how to successfully classify things?

There 800 separate documents of any of the 3 labels or 3 big documents of each of the labels is the best way to go and why?

The first thing you need to think about in any ML problem is: what is an instance for the problem? In other words, what is going to be the input for which you want a prediction at the end of the process?

Imagine you train your model with 3 big documents, one of each label. Then the input for such a model is a big set of documents with the same label. So it can only predict a label for a set of documents sharing the same label. This means that somehow you need to have the labels before applying your model... difficult isn't it? :)

This is why in this case an instance must be a single document. It's the job of the learning algorithm to learn to discover the label based on the instances, and for that it needs many instances of each possible label.

  On the other hand, if I do (as we usually do actually) like in the former case then the TF-IDF will be categories/labels-agnostic and I do not know this helps things.

This is where there is a confusion: the TF-IDF weights are not supposed to encode the label in any way, they represent the importance of a particular word in a document. The learning algorithm will use this information for all the words, that is it's going to learn the difference between when the word delicious has a high TF-IDF and when the word disgusting has a high TF-IDF (for instance).

  Is the answer simply that this an interesting but pretty bad idea because in this way you simply massively decrease the number of the observations with which the model/algorithm is trained and so you make much harder for him to figure out how to successfully classify things?

That would be true as well, but the main issue is the one I mentioned above: you won't be able to provide the same kind of input when you apply your model on your unlabeled data.


Connection between the first pooling layer and the second convolutional layer This is a follow-up question regarding this question. I managed to verify in python that the output of the first pooling layer will be a $14 \times 14\times 32$ tensor.

I interpret this as having 32 $14 \times 14$ images. Maybe my interpretation is mistaken. In the second convolutional layer we apply $64$ filters.

When I type in python conv2.shape I get

TensorShape([Dimension(55000), Dimension(14), Dimension(14), Dimension(64)]).

How that comes? I mean I am in interested in the outcome of the depth, which is $64.$

The first dimension is the batch size. From the tutorial you're linking to:

  We store the training feature data (the raw pixel values for 55,000 images of hand-drawn digits)

So it looks like your code is performing each gradient update step using all 55,000 training examples. That's unorthodox, but it's fine as long as your computer can handle it. See this video for an explanation of batching in gradient descent.

Your interpretation of the shape as being 32 14x14 images is fine for now, but it is more proper to refer to these 32 matrices as "feature maps". A grayscale image is/contains a single feature map, while a color image would consist of three of them, one for each red, green, and blue (assuming RGB color space). Also, note that at this point in the network the feature maps are no longer "images" as any human would understand the word.


Multivariate and multi-series LSTM I am trying to create a pollution prediction LSTM. I've seen an example on the web to cater for a Multivariate LSTM to predict the pollution levels for one city (Beijing), but what about more than one city?  I don't really want a separate network for every city, I'd like a single generalised model/network for all x cities. But how do I feed that data into the LSTM?

Say I have the same data for each city, do I...

1) Train on all data for one city, then the next city, and so on until all cities are done.

2) Train data for all cities on date t, then data for all cities on t+1, then t+2 etc.

3) Something completely different.

Any thoughts?

I would say that option 1 will not work out too well: in my experience, the model will either only be good for the first or last model you train, depending on how much freedom you give the algorithm to change weights as time goes on (e.g. with the learning rate).

You really need to decide what you are going to be predicting. Is it the pollution level for a single city: Which features do you have for each city?

It could can make sense to train all cities at the same time if the features you have are also general ones that really can explain the target variable. So if you have temperature, humidity, some transport statistics for that city etc. then training everything together could make sense.

I would think about each sample leading to one target pollution level, and if that sample has enough information (based on the features) to distinguish itself from samples of the other cities, the model should pick up on and leverage those subtleties in the data.


linear regression with "partitioned" data I am not a stats/math expert by any stretch of the imagination, but have been trying some linear regression with census data and think I have run across a fundamental problem/obstacle.

The response variable is a percent. The independent variables are percents by state. Here is an example of what I only know to refer to as  "partitioned" data. One of the groups of independent variables is "lighting" and has the following variables and values for one city (for example):

electricity - 40%
kerosene - 10%
candle - 25%
generator - 15%
solar - 5%
other - 5%

When added together, those six variables = 100%. There are only six choices and a household can only have one type of "lighting". Using all six of these variables, causes my model to blow up producing coefficients that are off the charts and clearly something is wrong.

Any insight on how to handle or at least still make some use of partitioned data with linear regression? Is there another name for these types of variables?

It is due to the fact that the 6th predictor is a linear combination of the other 5. You can write:

electricity = 100%-kerosene-candle-generator-solar-other

Linear regression models do not handle linear combinations at all. However the solution is easy, since the 6th variable is embedded in the first 5, you can just drop a column since it is implied by the other 5.


What are the main types of NLP annotators? I'm new to the world of text mining and have been reading up on annotators at places like the UIMA website. I'm encountering many new terms like named entity recognition, tokenizer, lemmatizer, gazetteer, etc. Coming from a layman background, this is all very confusing so can anyone tell me or link to resources that can explain what the main categories of annotators are and what they do?

Here are the basic Natural Language Processing capabilities (or annotators) that are usually necessary to extract language units from textual data for sake of search and other applications:

Sentence breaker - to split text (usually, text paragraphs) to sentences. Even in English it can be hard for some cases like "Mr. and Mrs. Brown stay in room no. 20."

Tokenizer - to split text or sentences to words or word-level units, including punctuation. This task is not trivial for languages with no spaces and no stable understanding of word boundaries (e.g. Chinese, Japanese)

Part-of-speech Tagger - to guess part of speech of each word in the context of sentence; usually each word is assigned a so-called POS-tag from a tagset developed in advance to serve your final task (for example, parsing).

Lemmatizer - to convert a given word into its canonical form (lemma). Usually you need to know the word's POS-tag. For example, word "heating" as gerund must be converted to "heat", but as noun it must be left unchanged.

Parser - to perform syntactic analysis of the sentence and build a syntactic tree or graph. There're two main ways to represent syntactic structure of sentence: via constituency or dependency.

Summarizer - to generate a short summary of the text by selecting a set of top informative sentences of the document, representing its main idea. However can be done in more intelligent manner than just selecting the sentences from existing ones.

Named Entity Recognition - to extract so-called named entities from the text. Named entities are the chunks of words from text, which refer to an entity of certain type. The types may include: geographic locations (countries, cities, rivers, ...), person names, organization names etc. Before going into NER task you must understand what do you want to get and, possible, predefine a taxonomy of named entity types to resolve.

Coreference Resolution - to group named entities (or, depending on your task, any other text units) into clusters corresponding to a single real object/meaning. For example, "B. Gates", "William Gates", "Founder of Microsoft" etc. in one text may mean the same person, referenced by using different expressions.

There're many other interesting NLP applications/annotators (see NLP tasks category), sentiment analysis, machine translation etc.). There're many books on this, the classical book: "Speech and Language Processing" by Daniel Jurafsky and James H. Martin., but it can be too detailed for you.


Skipgram - multiple formulations? I've been reading about the Skipgram model and I have found what I interpreted as multiple definitions.

1 - Taking a look at this blog post and Andrew Ng's Deep Learning Specialization, I understood that, for each word, we generate one training sample for each context word.

So, if we have the sentence "cat sat on the mat" we will have samples:

(cat, sat)
(sat, cat)
(sat, on)

And so on.

Then you train your network which will have the dimensions.

Input: $(V,1)$
Weights 1: $(d,V)$
Hidden layer: $(d,1)$
Weights 2: $(V,d)$
Output layer: $(V,1)$

Ok, so these dimensions match and we are good. For one given input, given that the weights are the same, we always have the same estimation in the output layer.

In this definition we have symmetric samples (for example, (cat, sat) and (sat, cat)) and saying that we use center words as inputs and context words as outputs is meaningless since they are interchangeable?

2 - Watching Stanford's NLP with Deep Learning class (38:54), it seems like for the same center word we can get different outputs:

The numbers in the red circles that I drew should be the same from what I understand. I don't really understand how it is possible to get different outputs if you multiple Weights 2 by the Hidden Layer.

3- I saw in other places (couldn't find reference now) another formulation: for each input word, we represent the context as a vector with ones for the context words and zeros elsewhere. So, in this case, instead of one training example for each context word, we would have one training example for each center word.

Andrew Ng word2vec @1:29. In the video, his choice of window size is more than one. In his example, if "orange" is the center word, the target word could be "juice", "glass", "my", which are all within the window size of 5.

In your example, "cat sat on the mat", if you choose a window size of one, you can get 7 training samples:

(cat, sat),
(sat, on),(sat, cat),
(on, sat), (on, the),
(the, on), (the, mat)

If this is the entire corpus, p(cat|sat), probability of sat occurs as the center word, cat being the context, is 0.5, p(sat|cat), cat being the center word, sat being the context is 1. The distributed word representation of "cat" and "sat" being the center word should be different.

"it seems like for the same center word we can get different outputs:" That's because they are predicting more than one context words.

@31:35 224 2017 lec2, the slide says "predict surrounding words of radius m for every word". That is, the problem is to find a distribution of context words given the center word, so for one center word, the model will have more than one output words (unless you set m to 1 and specify the direction as well).

"I don't really understand how it is possible to get different outputs if you multiple Weights 2 by the Hidden Layer."

@39:37 224 2017 lec2 The professor did a dot product between the matrix output and the word representation. It's also shown on the top of the note: "$u_o ^T v_c$"

Multiply the hidden layer (d * 1) by Weights 2 (V * d) will get a V*1 vector. That's $u_o$, output vector representation. Then we do $u_o ^T v_c$ to get the output for each context word. After that, we apply softmax and compare that value with the context word's one-hot representation ($y$ vs $\hat{y}$), three comparisons in the prof's example.

That sounds like the idea of the Continuous Bag of Words model.


Is it acceptable to make new independent variables out of old ones? I've recently been tasked with an Data Science interview assignment and looking over the variables, I wondered if it is professionally acceptable to make a new independent variable out of old ones (or at least modify the independent variables in a way to potentially get more worthwhile data).

For example, one of the variables has data on a person's birth state/territory. So instead of trying to model using 50+ categories, creating a new variable that would categorize each state by region (North East, Midwest, South, Etc.) might lead to more significant and easier to interpret analysis.

I know the objective, or what we're trying to answer, is important, but I wonder if making brand new variables or significant changes like this is appropriate for an interview assignment?

Thanks!

It is completely acceptable if you make out new categories out of old ones or add a different representation of them like in the case of (North East, Midwest, South, Etc.) from state this process is called feature engineering. All you need to pay attention to is that you are not loosing any information in any way when you are doing so because you want to feed your model with as much information as possible.

Sometimes it is useful to keep both the new variable you made out of other variables like for say you have a dataset containing infect_people_from_covid19 in region and population_of_region you can create a variable as ration of people infected but when building a model and feeding data you have to decide weather adding all 3 of them will be a good idea or you just need the summary variable you made.


Backpropagation of a transformer when a transformer model is trained there is linear layer in the end of decoder which i understand is a fully connected neural network. During training of a transformer model when a loss is obtained it will backpropagate to adjust the weights.
My question is how deep the backpropagation is?

does it happen only till linear layer weights(fully connected neural net) ?
OR does it extend to all the decoder layer weight matrices(Q,K,V) and Feed forward layers weights?
OR does it extend to the even the encoder + decoder weights ?

Please help me with the answer.

Backpropagation extends to the full model, through all decoder and encoder layers up to the embedding tables.


What is the best way to cluster this kind of data? I have data that looks like this:

The chart on the left is the trend and the smaller chart on the right is the box plot showing the distribution of means. Each color is the output of a particular tool. I dont need to do an ANOVA and Tukey tests on it because it can easily be seen that there are statistically significant differences between the tools.
However, I would like to group/cluster the tools based on their mean.
For example, visually speaking, one can separate all the tools into 4 groups. 4 tools with a mean around 32, 12 tools with a mean around 30, 5 tools whose variance is quite high, and whose mean don't really fall into a particular group and the rest whose mean is around 30.
Update:
I have used the k-means clustering technique and the Hierarchical Agglomeration techniques. However, I have to specify the number of clusters in advance. Is there any clustering technique where I don't have to specify the number of clusters a-priori?

In hierarchical clustering, both agglomerative and divisive, you do not have to pre-specify the number of clusters. You can create all possible clusters and then select the number cluster to use at the end.


Need input on which features to drop in classification model This is the correlation of features with my target variables. I have done all the features engineering but I am left with these features.

Any input on what columns to keep for model training and what to drop. Is there any criteria for dropping features that I don't need. It seems credit history is the only feature that has high correlation.

Loan_ID              0.011610
Gender               0.017987
Married              0.091478
Education           -0.085884
Self_Employed       -0.003700
ApplicantIncome     -0.004710
CoapplicantIncome   -0.059187
LoanAmount          -0.037318
Loan_Amount_Term    -0.022549
Credit_History       0.561678
Total_Income        -0.031271
Total_Income_Log     0.007240
LoanAmt_Log         -0.037536
CH__0               -0.540556
CH__1                0.432616
EMI                 -0.011552
EMI_Log             -0.028496
Dependents_1        -0.038740
Dependents_2         0.062384
Dependents_3        -0.026123
Property_Area_1      0.136540
Property_Area_2     -0.043621
Loan_Status          1.000000
Name: Loan_Status, dtype: float64

Welcome to the community Sai!

Let's assume your problem is a regression problem (i.e. you have continues target).

There are some points:

First of all there is no written rule for this. Feature Engineering is an EDA kind of thing. There is no final solution. Your model selection strategy will choose some.
Just as a reminder, if these are Pearson correlations, be careful that nonlinear correlations might exist which is not captured by linear correlation analysis. Plus the fact that linear correlation analysis always comes with visual inspection.
Negative correlations inhibits information. If whenever a variable increases the other decreases, then knowing it tells you about the other one! So take them into account. You better use Mutual Information for checking dependencies.
Sparse Linear Models seem to be fruitful here. I suggest letting LASSO or Ridge Regression choose the final set of features.
In case you really insist on your current way (e.g. if your supervisor asked for it), use the threshold as the hyperparameter of your model selection and find the optimal. It means, you train and validate your model using different threshold (which results in different sets of features) and according to empirical error (validation error) you choose the best threshold.

Hope it helped. Good Luck!


The right Machine Learning solution for this problem Throughout my understanding about using Machine Learning, there was always a column whose name Target or Label, that was causing us to call methods Supervised learning. Now I am dealing with not a supervised or an unsupervised procedure. I do not know what exactly is. I have some data-frames that each of which has 3 columns. The columns have a relation with each other but it does not matter. I am looking for a way to show the effect of samples on each other. In the sense that, in my topic, theoretically, it has been proved that in a specific situation, some samples are more important than the others. Now I need to show it using Machine Learning. I have generated the required data for having that situation but I do not know how to find the effect of samples on each other.? I could have shown some differences on samples but couldn't find any meaningful scenario to be used in machine learning. To clarify, I have defined a value which a computation between each column value and I would like to know in the computation of this value in a specific index, which sample are playing a more important role.  Now The question is that,
**HOW TO FIND THE EFFECT OF SAMPLES ON EACH OTHER USING SIMPLE OR ADVANCED TECHNIQUES?**

To understand better let's see some sort of data here.
an example of data frame is below:
S   C     E
0   0.2   1e-2
1  -0.15  2e-2
2   0.24  2e-3
3  -0.1   3e-1
4   0.3   2e-1

the third column is a combination of the first 2 columns(There is the equation of combination) Now I need to know for example for the E[2], what is the effect of c[0],c[1],c[2],c[4].

First things first, columns in your dataframe are called as Feature,
So, The right question for you will be
How to find Co-Relation between features?
As mentioned in the question, You want to find co-relation among them,
SO,
What is Co-relation ?
Answer :Correlation is a statistic that measures the degree to which two variables move in relation to each other.
Example: Ice Cream Sales
The local ice cream shop keeps track of how much ice cream they sell versus the temperature on that day, here are their figures for the last 12 days:

And here is the same data as a Scatter Plot:

We can easily see that warmer weather and higher sales go together. This kind of relation is called as a positive co-relation which means if one variable increase the other gets increased too hence both of the Variables are providing the same information.
In general case, it is better to drop one of them.


How to "count" certain events in a time series What techniques are suitable and what do I need to learn in order to
detect and count the number of "events" (pic) that consist of:

a) shape 1 3 and 4 and not of (2 and 5) or

b) of all shapes above the blue line with a certain minimum "volume" like shapes 1 and 4?

My first idea is to use a window function and somehow integrate the area but i want to be sure to use an appropriate procedure. I havent worked with time series before and I dont know were to get startet. I have to use python for this project.

Try generating a dictionary of patterns you want to identify.  You can then use convolutions/ cross-correlations to identify where these patterns appear in your data.
https://en.wikipedia.org/wiki/Convolution
https://en.wikipedia.org/wiki/Cross-correlation
http://paulbourke.net/miscellaneous/correlate/

This method is also called 'matched filter'.


Unsynchronized time series visualization I would like to visualize a large amount of events composed of time serie windows. A typical event would be:

Problem is, my events are not synchronized, and so if I plot them all, it would look like:

Question
Is there any way to visualize all my events so I can see their original/"typical" shape (preferably in the time domain) despite their unsynchronization ?
What I have tried so far:

Visualize features: approach is good but I have to guess what I am looking for.
Synchronized events: in the example above it might be possible, but for some other cases it won't be. (such as multiple peaks spaced with different time lapses)
Use statistical values to visualize: that was my first shot, to plot not all events but only their mean, median, p95,... Problem is, using this approach gives a deformed rendering of the data because of the unsynchronization:

What I have thought about
The main problem of applying the mean, p95 or whatever statistical function is that it is applied along an axis that is not synchronized between events. I don't know if it exists or if it is even feasible, but I was thinking about an approach that would display those statistical values checking their temporal neighbors ? I know Dynamic Time Warping deals with this unsynchronization but I am not sure if and how I could use it to plot my events in the way I'd like.
An idea of a possible output could be (any other output idea is welcomed!):

Any help, idea, would be greatly appreciated ! Thanks

As far as estimating similarity between the time series series there is a variety of methods you may want to investigate. Some of those are:

cross correlation: this will be affected by the amplitude and will not be able to estimate lagged correlations, prone to noise.
coherence: normalised frequency based correlation (cross-spectrum), not prone to amplitude or noise.
wavelet coherence: similar to above but based on wavelet transformations instead of STFFT.
dynamic time warping: measuring similarity between two temporal sequences, which may vary in speed.


What makes the posterior intractable? In the setting of Variational AutoEncoders, i.e. when we want to find the posterior distribution

over the data generating, latent variable z, given some observations x, what exactly (which part of the equation) makes this posterior distribution intractable and why?
Cheers

It's usually the denominator $p(x)$ (the "evidence") which is intractable. You could attempt to compute it by marginalizing over the latent variable $p(x) = \int p(x|z)p(z)dz$. However, you would need to evaluate all possible values of $z$ which would require exponential time. (That's why in maximium-likelihood estimation you have no intractability problem because you can treat the evidence as a constant.)


How to extract insights from the given data? Ok, I have this data with 3 columns, unique id, raw text, and review text. My task is play with the dataset and find meaningful insights from it. Raw text is in plain English but review text is in another language. I have no idea how to proceed with the dataset. Even after I clean the data from the raw text, what should I do with review one because it is in another language. Which text analysis should I do and how can I implement it on the dataset?

The language that are used on review text just happened to be my native language. I can confirm that the review_text at least from what you showed above is a direct translation of the raw text (although I would say the translation is not perfect).

Maybe you can consider making unsupervised model and probably compare between these two, see how much they match (theoretically since they are the same text they should have many overlaps).


Saving LSTM hidden states while training and predicting for multi-class time series classification I am trying to use an LSTM for multi-class classification of time series data.

The training set has dimensions (390, 179), i.e. 390 objects with 179 time steps each.

There are 37 possible classes.

I would like to use a Keras model with just an LSTM and activation layer to classify input data.

I also need the hidden states for all the training data and test data passed through the model, at every step of the LSTM (not just the final state).

I know return_sequences=True is needed, but I'm having trouble getting dimensions to match.

Below is some code I've tried, but I've tried a ton of other combinations of calls from a motley of stackexchange and git issues. In all of them I get some dimension mismatch or another.

I don't know how to extract the hidden state representations from the model.

We have X_train.shape = (390, 1, 179), Y_train.shape = (390, 37) (one-shot binary vectors)/.

n_units = 8
n_sequence = 179
n_class = 37

x = Input(shape=(1, n_sequence))
y = LSTM(n_units, return_sequences=True)(x)
z = Dense(n_class, activation='softmax')(y)

model = Model(inputs=[x], outputs=[y])
model.compile(loss='categorical_crossentropy', optimizer='adam')

model.fit(X_train, Y_train, epochs=100, batch_size=128)
Y_test_predict = model.predict(X_test, batch_size=128)

This is what the above gives me:

ValueError: A target array with shape (390, 37) was passed for an output of shape (None, 1, 37) while using as loss 'categorical_crossentropy'. This loss expects targets to have the same shape as the output.

There didn't seem to be any way to build a working trainable model while also returning the hidden states with return_sequences=True.

The fix I found was to build a predictor model and train it, and save the weights. Then I built a new model which ended with my LSTM layer, and fed it the trained weights. So, using return_sequences=True, I was able to predict on new data and get the data's representations at each hidden state.


Parameter tuning for machine learning algorithms When it comes to the topic of tuning parameters, most of the time you read grid search. But if you have 6 parameters, for which you want to test 10 variants, you get to 10^6 = 1000000 runs. Which in my case would be several months of processing time.

That's why I was looking for an alternative. On the Kaggle website, I found a tutorial that uses a different approach. It almost works like this:

1) Test all 6 parameter individually with the other 5 parameters as default value and plot the results

2) Change the default values for alle 6 parameters to the best value of the associated plot

3) Test all 6 parameter individually with the other 5 parameters as last best value and plot the results

4) Repeat step 2 and 3 until the results does not change anymore

This approach has the advantage of requiring much fewer runs. Is this a scientifically accepted procedure? And does this approach have a name in the literature?

In general, your approach will get stuck in local minima. This is why it is not scientifically accepted.
(Notice that this may be different in very special cases, in particular if the performance of the algorithm is a strictly convex function of all input parameters).

To see how the approach fails, suppose your machine learning algorithm has two parameters, $x$ and $y$, which can be either $0$ or $1$. The default values are $x=1$ and $y=1.$

The performance of your machine learning algorithm is $f$ and should be as high as possible.
Assume the following performance levels $f(x,y)$:

    | x=0 | x=1
----|-----|-----
y=0 | 0.9 | 0.2
y=1 | 0.1 | 0.3

Your approach would do the following:

First, choose the default value $x=1$ and compute $f(x=1, y=0) = 0.2$ and $f(x=1, y=1) = 0.3$. Second, choose the default value $y=1$ and compute $f(x=0,y=1)=0.1$ and $f(x=1, y=1) = 0.3$.
Change the default values to the best value. In this case, this requires no change since $x=1$ and $y=1$ are the best values, respectively.
The result did not change. Report $(x=1, y=1)$ as the best parameter combination.

But the global performance maximum occurs at $(x=0, y=0).$


Why are pandas/R/etc so common in a data scientist or analyst's workflow when one can resort to UDFs in an RDBMS? It seems common for an analyst to have this workflow when using an RDBMS: use SQL to get a subset of data from the database, export it or use a connector, and then apply a data mining / model algorithm on it (e.g. kNN, regression models, etc) using something like Pandas, R or Matlab. In other words one operates within the DBMS and then operates outside of it.

One can use User-defined Functions (UDF) to implement the algorithm of choice in many languages (build a UDF in C using e.g. PostgreSQL) and remain operational within the DBMS environment, i.e. one does not give up the benefits obtained of operating inside of a DBMS. Furthermore, one can seamlessly integrate the UDF with the SQL construct, which is very powerful if one requires to perform subsequent SQL operations on the data which passed through a model.

I was wondering what is the reason behind operating outside of the DBMS given that such functionality is available and that the DBMS is, ironically, good at managing data (supported by many years of research).

There are other discussions which seem to conclude that SQL is for preprocessing and Pandas is for data analysis; to clarify I am building on this conclusion: why did things end up this way, when the DBMS is very well-suited for data analysis as well?

I think the answer might be this:

The main difference between Python -and R- and SQL is that the first are Turing complete.

A given programming language is said to be Turing-complete if it can be shown that it is computationally equivalent to a Turing machine.

That is, any problem that can be solved on a Turing machine using a finite amount of resources (i.e., time and tape), can be solved with the other language using a finite amount of its resources.

Typically, one proves a given language is Turing-complete by providing a recipe for translating any given Turing machine program into an equivalent program in the language in question. Alternately, one can provide a translation scheme from another language, one that has already been proven to be Turing-complete.

Nearly every existing computer language is Turing-complete. About the only computer languages that aren't Turing-complete are a handful of special languages that are capable of LESS than a Turing machine - usually because some limitation is "hard-wired" into the language's structure or definition. (For example, Hofstadter's designed a language called BlooP so that it was impossible for it to have a iteration structure with an arbitrarily high upper bound.)

Turing Completeness definition

The question of yours is: Why we don't use SQL (or another DBMS) to do all the Data Science process? The answer is that because SQL lenguage is limited in certain aspects that Python and R are not. And one of those aspects is key to Data Science: Recursion.

Just imagine Decision Trees and all their variants without recursion! They are impossible!

Is SQL Turing Complete?


Predicting Age of Birth I have a pet project to figure out the birth year of a significant person in history.

I'm collecting a lot of data on other people with similar status during that time period. I have data such as education length, year married, year of child bearing, information about siblings and age difference between each child, marriages, etc...

The age of this person is disputed between two years, one making the person very old, and the other making the person young. I want to regress the persons age. My first idea was to draw gaussians over each variable and see if one is more likely to be an outlier than the other.

What way would you tackle this problem?

It seems like you have a classical bayesian problem. You have some sort of prior distribution, a distribution over years of birth, your prior distribution is bimodal with peaks at the two years,  you can probably use a convolution of two normal distributions to model this variable. Then have it spit out a posterior distribution after you feed in some data.

The real problem that I have with this analysis is it seems your features aren't particularly good. It is true these vars might have information about birth year, for example for the 20th century the average age of first marriage has steadily been increasing. But I suspect that the signal is going to be fairly weak. Essentially, if I tell you that I got married at age 24, had my first child at 26, and that my older brother is 3 years older than me and my younger sister is 2 years younger than me, can you tell me in what year was I born, 1956 or 1989?

I suspect that without additional data this information that I provided would be completely useless, mostly because it is a very noisy signal. That information could apply equally to someone born in 1956 or 1989. It isn't very helpful.

Essentially, what I am saying is that when you update your prior, it isn't going to change very much. (Your posterior distribution would look very similar to the prior distribution.) Instead of doing some mustache twirling over what is the right algorithm to crack this problem, I think a much more fruitful exercise would be to think up some better features.


Using Google Translate API to create a Translation Dataset Is it a good idea? ;-)
Is it legal to do so? Is it legal to release such a dataset to public?

Say I have a language X for which I want to create a dataset for translation to/from English, for which I have no online sources to scrape out the data from. But our giant Google Translate somehow has the support for language X.

Can we just dump the translations from English datasets for my language X using Google Translate API and create a rough dataset to start with?

It depends. Google Translate works very good with some pairs of languages, and not good on others. Based on my personal experience, translating from North-European languages (Dutch, Danish, Swedish) to English worked almost perfectly, while English-Italian translation lead to bad results.

You can find a Spanish-English dataset here, this is a very official source.

Additionally, it seems DeepL is a great German <-> English online translator around.


Print the prediction of convolutional neural network I have designed a convolutional neural network using tensorflow which looks as follows

#Define a convolutional neural network function
def conv_net(x, weights, biases):
    conv1 = conv2d(x, weights['wc1'], biases['bc1'])
    conv1 = maxpool2d(conv1, k=2)

    conv2 = conv2d(conv1, weights['wc2'], biases['bc2'])
    conv2 = maxpool2d(conv2, k=2)

    conv3 = conv2d(conv2, weights['wc3'], biases['bc3'])
    conv3 = maxpool2d(conv3, k=2)

    conv4 = conv2d(conv3, weights['wc4'], biases['bc4'])
    conv4 = maxpool2d(conv4, k=2)

    # Fully connected layer
    # Reshape conv2 output to fit fully connected layer input
    fc1 = tf.reshape(conv4, [-1, weights['wd1'].get_shape().as_list()[0]])
    fc1 = tf.add(tf.matmul(fc1, weights['wd1']), biases['bd1'])
    fc1 = tf.nn.relu(fc1)
    # Output, class prediction
    out = tf.add(tf.matmul(fc1, weights['out']), biases['out'])
    return out

The code I'm using to train and test the model is as follows :

#Train and Test the Model
with tf.Session() as sess:
    sess.run(init)
    train_loss = []
    test_loss = []
    train_accuracy = []
    test_accuracy = []
    summary_writer = tf.summary.FileWriter('./Output', sess.graph)
    for i in range(training_iters):
        for batch in range(len(X_train)//batch_size):
            batch_x = X_train[batch*batch_size:min((batch+1)*batch_size,len(X_train))]
            batch_y = y_train[batch*batch_size:min((batch+1)*batch_size,len(y_train))]
            # Run optimization op and Calculate batch loss and accuracy
            opt = sess.run(optimizer, feed_dict={x: batch_x,
                                                 y: batch_y})
            loss, acc = sess.run([cost, accuracy], feed_dict={x: batch_x,
                                                              y: batch_y})
        print("Iter " + str(i) + ", Loss= " + \
                      "{:.6f}".format(loss) + ", Training Accuracy= " + \
                      "{:.5f}".format(acc))
        print("Optimization Completed")

        # Calculate accuracy for all test data
        test_acc,valid_loss = sess.run([accuracy,cost], feed_dict={x: X_test,y : y_test})
        train_loss.append(loss)
        test_loss.append(valid_loss)
        train_accuracy.append(acc)
        test_accuracy.append(test_acc)
        print("Testing Accuracy:","{:.5f}".format(test_acc))
        summary_writer.close()

This prints the accuracy but not the predictions. How can I print the predictions?

p.s new to neural network

You can replace these lines:

opt = sess.run(optimizer, feed_dict={x: batch_x,
                                     y: batch_y})
loss, acc = sess.run([cost, accuracy], feed_dict={x: batch_x,
                                                  y: batch_y})

by:

opt_, loss_, acc_, out_ = sess.run([optimizer, cost, accuracy, out], feed_dict={x:batch_x,
                                                  y: batch_y})

In this way, you can evaluate all the parameters given in the list (optimizer, cost, accuracy, out) in a single sess.run(). After, if it is classification task, you will probably need to apply softmax and argmin to out_


Can we specify the number of data generated(minority class) using SMOTE? I am trying to improve classification of imbalanced dataset creditcard fraud using SMOTE imbalanced_learn. But, in this it generates the data to 50%, can we give a specific number for the data to be generated? I want to track the classifier performance with the increase in generated data. Any help will be highly appreciated.

SMOTE is an algorithm for generating as many minority samples as you like. Thus, you can generate as many samples as you want.


Ordering scrambled 1D data sets by continuity This is a cute little clustering problem that was probably solved a million times over, but I couldn't find a good reference for it.

I have 20 1D datasets with 400 entries each. In the picture they are denoted by different colors.

As you can see, they are also pretty continuous. However, for each index i the datasets have been re-ordered by magnitude, i.e. instead of nice continuous lines, the color now jumps at every intersection of each two datasets.

Is there a way to bring the the datasets to their original order? I.e. cluster the data into 20 continuous lines? This could be easily done by eye.

Thank you very much!

Ofri

P.S. Here's what I tried so far. For each index i I assumed the datasets are already ordered up to i-1. I now prepare 20 bins by extrapolating the ordered datasets from the first i-1 indices to the i index. Now I have 20 values to put in 20 bins, with exactly one value in each bin. I can try 20! combinations and find the one with the minimal error but there must be a more clever/efficient way.

I ended up soling this by cutting the datasets into segments and then gluing them together. Here are some of the disentangled datasets:

First I cut all the datasetes into continuous segments by identifying the discontinuities (I looked at peaks of the second derivative). I then sowed the segments together based on which segment is the closest to the 2nd order interpolation from the neighboring segment.

There were some special cases that I had to deal with - for example actual divergences in the signal. I dealt with these by identifying segments that cannot be continuously sowed into any other segments (these are the ones going to +-infinity) and gluing them to other segment returning from +- infinity.

Ofri


What is the logic of the epoch? What is the logic of the epoch? For example, 1 time, 2 times, etc. I just do not know what else is working to give better results than I know.

There are two terms which relate to the number of examples while learning. Epoch and iteration. During each epoch, you feed all the examples in your training set and update the network. You can feed the data simultaneously, not for large-scale tasks, or batch by batch. Each turn you pass each batch is called iteration. Consequently, each epoch may have multiple iterations. The reason we have to pass the data multiple times is that we don't know the height of each local optimum along each axis, feature. Consequently, we use a learning rate to limit our steps to small ones and take multiple steps to get close to the desired point.

Due to the request of one of our friends I add the explicit definitions:

Epoch: It means how many times the entire dataset has been passed through the network.
Iteration: It means for each epoch, how many times you have passed the chunks of the dataset. These chunks of data are called batch and the number of times you pass them through the network is called the number of iterations.


What is the difference between "expected return" and "expected reward" in the context of RL? The value of a state $s$ under a certain policy $\pi$, $V^\pi(s)$, is defined as the "expected return" starting from state $s$. More precisely, it is defined as

$$
V^\pi(s) = \mathbb{E}\left(R_t \mid s_t = s \right)
$$

where $R_t$ can be defined as

$$
\sum_{k=0}^\infty \gamma^k r_{t+k+1}
$$

which is a sum of "discounted" rewards after time $t$, i.e. starting from time $t+1$.

$V^\pi(s)$ can also be interpreted, even more precisely, as the expected cumulative future discounted reward. This denotation contains all the words which refer to specific parts of the formula above, where

"Expected" refers to the "expected value"
"Cumulative" refers to the summation
"Future" refers to the fact that it's an expected value of a future quantity with respect to the present quantity, i.e. $s_t = s$.
"Discounted" refers to the "gamma" factor, which is a way to adjust the importance of how much we value rewards at future time steps, i.e. starting from $t + 1$.
"Reward" refers to the main quantity of interested, i.e. the reward received from the environment.

Meanwhile, I've heard the term "expected reward", but I am not sure if it refers to the same concept or not, that is if "expected reward" or "expected return" are the same thing or not.

I know there's also the concept of "expected value of the next reward", often denoted as $\mathcal{R}^a_{ss'}$, and defined as

$$
\mathcal{R}^a_{ss'} = \mathbb{E}\left(r_{t+1}  \mid s_t = s, a_t = a, s_{t+1} = s' \right)
$$

which, again, is the value we expect for the reward at the next time step, that is at time step $t+1$, given that action $a$ from state $s$ brings us to state $s'$.

Is the "expected reward" actually $\mathcal{R}^a_{ss'}$ instead of $V^\pi(s)$?

Is the "expected reward" actually $\mathcal{R}^a_{ss'}$ instead of $V^\pi(s)$?

In short, yes.

Although there is some context associated - $\mathcal{R}^a_{ss'}$ is in the context of specific action and state transition. You will also find $\mathcal{R}^a_{s}$ used for expected reward given only current state and action (which works fine, but moves around some terms in the Bellman equations).

"Return" may also be called "Utility".

RL suffers a bit from naming differences, however the meaning of reward is not one of them.

Notation differences also abound, and in Sutton & Barto Reinforcement Learning: An Introduction (2nd edition), you will find:

$R_t$ is a placeholder for reward received at time $t$, a random variable.
$G_t$ is a placeholder for return received after time $t$, and you can express the value equation as $v_{\pi}(s) = \mathbb{E}[G_t|S_t=s] = \mathbb{E}[\sum_{k=0}^{\infty}\gamma^kR_{t+k+1}|S_t=s]$
$r$ is a specific reward value
You won't see "expected reward" used directly in an equation from the book, as the notation in the revised book relies on summing over distribution of reward values.

In some RL contexts, such as control in continuous problems with function approximation, it is more convenient to work with maximising average reward, than maximising expected return. But this is not quite the same as "expected reward", due to differences in context (average reward includes averaging over the expected state distribution when following the policy)


What is the efficient way to generate a similarity score when comparing two face images? I am working on a face recognition application using deep learning. To plot the ROC curves and do performance evaluation, I extracted the features from the last layer of the deep neural network and I have tried Euclidian and Manhattan distances between high dimensional feature representation, but this won't give me accurate measure in terms of comparison as faces have intra-personal and interpersonal differences and other challenges such as pose, lighting condition, and expression. I came across techniques such as joint bayesian, sparse projection, KL-divergence based regularizer/prior(EM algorithm) etc. Are there any other approaches that I'm missing here? What would be the efficient approach to generate similarity scores?

A common method is defining "landmarks", a set of specific points that exist on every face . For example, the outside edge of each eye or the bottom of the chin are landmarks on most faces. Train the deep learning network to learn the relationship between these landmarks which will form the embedding space. Then define a distance metric in this embedding space. Finally, create a threshold for similarity between sets of points in this embedding space. That threshold can be learned through "one-class" classification techniques.

One gotcha is pose. Faces turned different directions look totally different to a computer. Assuming you have isolated the faces in an image, it is best practice to warp each face so that the eyes and lips are always in the place in the image. This (and other normalization steps) should be done early in the processing pipeline to make subsequent analysis much easier.


Getting wrong ch2 values from sklearn chi2 I have a feature vector table which looks like this-

This a table with 156 columns or features.I want to apply feature selection algorithm to this before applyinh my classification model.

This is what I am using-

dataset = pd.read_csv('.csv')
X = dataset.iloc[:, 1:157].values
y = dataset.iloc[:,0].values

##normalize

scaler = MinMaxScaler()
scaler.fit(X)
MinMaxScaler(copy=True, feature_range=(0, 1))

X_normalized = scaler.transform(X)

##feature selection

sel = SelectKBest(chi2, k='all')

sel.fit_transform(X_normalized, y)

print(sel.scores_)

this is the result of print(sel.scores_) I am getting-

As can be seen they are not all between 0 and 1.

I a referring this research paper as my source-

source-http://courses.ischool.berkeley.edu/i256/f06/papers/yang97comparative.pdf

I don't know what is your source that said that value of chi-square should be between 0 and 1. Imagine as in that equation in the picture say two terms t and c always co-occur, meaning D(t|~c)=0 (number of documents with no c but t in them) and D(c|~t)=0 (number of documents with c in them but no t). This means the equation in the snap reduces down to

$\chi^2(t, c) = N*(AD)^2/A^2D^2 = N$.

Edit 1:

After reading your comment, I understood you are confused regarding the 'normalised' part. A normalised value does not always mean a scaled value between 0 and 1. Normalised value does mean a value that is scaled appropriately for comparing and in my opinion, the denominator gives that comparable effect. The type of normalisation that you are associating this situation to is feature scaling. Check out normalization.


predict numeric value from images I have two questions:

1) I have images and I must predict a continuous value.

What is the approach I must follow?

Use for example a pretrained network like this?

 last_layer = pretrained net
 x = Flatten()(last_layer)
 x = Dense(512, activation='relu')(x)
 x = Dropout(0.4)(x)
 predictions = Dense(1)(x)

So, just use a Dense layer at the end with 1 node?

and then procees to prediction as usual?

2) If I have again a numerical values to predict but it is ordinal.

If I want to go with regression in this case, can I work like in previous example?

And just assign some probabilities to the result?

If result >0 and <=0.5 then classify as 0
If result >0.5 and <=1 then classify as 1

and so on?

It seems you are doing it right. The difference between case 1) and 2) should be mainly the loss function (and maybe the activation of the predictions layer).

For 1) you would use e.g. mean_squared_error, while for 2) you would use the binary cross entropy from logits (logits are what you would feed into a sigmoid or softmax layer). In the keras-documentation it seems like keras does not provide cross entropy from logits as a loss, so you have to set the activation of predictions to sigmoid and use binary_crossentropy. I guess then you could do the If result >0 and <= 0.5 then classify as 0 ..., the result >0 would be taken care ouf by definition of the sigmoid, however you might want optimize the threshold w.r.t. the precision and recall you need.


any reason for this project to use hadoop/spark? I am setting up for a couple self study projects to explore machine learning techniques.

1st project has 10,000 time series with 24 float data points each day for 10 years (876 million points). I will be creating a bunch of calendar and weather features for the data, then trying to forecast using a variety of machine learning techniques.

2nd is some 13 million rows of text data (several paragraphs for each row)  for classification. (currently in solr database)

My compute rig is 6-core, 32g ram, gforce GPU. I plan to install Ubuntu 14.2.

I expect to be using python for file processing, scilearn, pylearn2 and word2vec for general exploration and training. R for getting a taste of the language.

Clearly data set 1 will require joining weather and calendar data to date/time and aggregation across time and location. I know how to stuff it all into a MySQL database and do the aggregations and joins there, but I have been reading about spark and wondering.

......

If I take the time to simulate a cluster using virtual box/hadoop/spark (for my learning experience, not performance), can/should I do the aggregations there and write the results to the distributed data store?

Since deep learning can not be run on spark, does that mean I would need to copy the aggregated data back out to the local file system to use some of those techniques?

For data set 2, I want to run the word2vec algorithm as found in the kaggle tutorial https://www.kaggle.com/c/word2vec-nlp-tutorial/details/part-2-word-vectors. In that example, that is a deeplearning method, so I should just leave the data in the solr.. right?

In general I am looking for appropriate applications and insight into data flow from app to app to help me get to the part where I start trying various ML techniques.

Thanks for helping me along

Especially when your main goal is learning, I would break it into several phases:

Get familiar with pandas dataframes and visualization using matplotlib. Try loading subsets of your datasets using pandas and visualize them using matplotlib, for example plot the time-series, or word count histograms. This will come in handy lateron to be able to understand the predictions/clusters you will create using machine learning methods. Pandas also provides functions to clean, aggregate and resample the data, which will be useful to align multiple time-series (e.g. weather and time-series) to the same sampling rate.
Familiarize with scikit-learn using the articles and examples in the excellent online documentation. Play with the feature extraction, dimensionality reduction, classification and regression methods relevant to your data, apply them on smaller subsets and visualize the results. Learn about cross-validation and find a good algorithm and parameters that work well on a subset of your data.
One you have a pipeline setup and you want to run it on the full dataset, Spark can come in handy for feature extraction and cross-validation, as those tasks can be often run very well independently and in parallel. Install the Spark libraries on your machine, and reproduce the Pyspark examples (e.g. calculating Pi) in local mode. No need to setup a Hadoop cluster for that, Spark can make full use of your machine in local mode, reading from local files on disk. Once you have that running, try to express the expensive steps of your code from step 2 as Spark RDD operations. As Spark can make use of all cores of your machine, you should already see a speedup - debugging becomes much harder though compared to working with pure Python scripts.
Play with advanced stuff like deep learning, using the word2vec feature extraction. Compare with traditional feature extraction (bag of words).
(optional) If you want to experience the speedup you can get from Spark in a distributed setting, get an account in AWS, put your data on S3 and fire your Spark scripts against a multi-node cluster instantiated using Amazon EMR.

Regarding data storage, from my experience CSV files or similar work best to load the data into both Pandas and Spark. There are Spark connectors for databases (e.g. Cassandra), but I am not sure about MySQL and Solr. For processed data and intermediate results (e.g. models) I would always use Python's serialization framework pickle, so you can directly get the object instances back into Python without the need to parse file formats and reinstantiate objects.


Reconize a card from a video stream I want to make an mobile app where you scan using the camera and it gives you the card you just scanned. (from a board game)
I got a PNG of each and every single existing cards, but I don't really know what to use.

For now, I've been using the text in the card using OCR to find the card, but it's not 100% accurate as the typography is pretty special.

I've been thinking about using image classification, but I'm not sure it's the right way to do it, as I would be training for hundreds of classes. (got over 400+ different cards)

Do you have any idea what I should be using ? (I'm sorry if this seems like a basic question, but I just started learning about CV and ANN)
Thanks for your help !

First Approach:

You can probably train a CNN ( Convolutional Neural Network ) on the images you have. If the number of images is less, I prefer you to use Transfer Learning.

  In Transfer learning, we have a highly trained model ( like Inception, ResNet ) and we modify it so that it can classify images which we need.

Second Approach:

Use object detection. You need to localize an object ( card ) from the image and then classify that cropped image. This could be tedious if you're a beginner.

You can refer to these links:

https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/training.html
https://towardsdatascience.com/creating-your-own-object-detector-ad69dda69c85
https://pythonprogramming.net/training-custom-objects-tensorflow-object-detection-api-tutorial/

  You need to train a localization model which can track and crop your images ( of cards ).


Cause of periodic jumps in loss function I might be missing something obvious as I am new to machine learning. I am training an SSD Inception V2 for detecting buildings from satellite images. I use the Tensorflow Object Detection API. I am having troubles interpreting why the value of loss seems to change periodically:

Please let me know if I need to add more information

Looks like you may be feeding training data in a very specific way or weights and biases reset after a specific period of time, e.g. end of epoch?
I would start with checking that

training and validation sets consist of a desired class ratio
datapoints within mini batches are being shuffled
model parameters are being correctly updated during training


How many images per class are sufficient for training a CNN I'm starting a project where the task is to identify sneaker types from images.  I'm currently reading into TensorFlow and Torch implementations. My question is: how many images per class are required to reach a reasonable classification performance?

From  How few training examples is too few when training a neural network? on CV:

It really depends on your dataset, and network architecture. One rule of thumb I have read (2) was a few thousand samples per class for the neural network to start to perform very well. In practice, people try and see.

A good way to roughly assess to what extent it could be beneficial to have more training samples is to plot the performance of the neural network based against the size of the training set, e.g. from (1):

(1) Dernoncourt, Franck, Ji Young Lee, Ozlem Uzuner, and Peter Szolovits. "De-identification of Patient Notes with Recurrent Neural Networks" arXiv preprint arXiv:1606.03475 (2016).
(2) Cireşan, Dan C., Ueli Meier, and Jürgen Schmidhuber. "Transfer learning for Latin and Chinese characters with deep neural networks." In The 2012 International Joint Conference on Neural Networks (IJCNN), pp. 1-6. IEEE, 2012. https://scholar.google.com/scholar?cluster=7452424507909578812&hl=en&as_sdt=0,22 ; http://people.idsia.ch/~ciresan/data/ijcnn2012_v9.pdf:

  For classification tasks with a few thousand samples per
  class, the benefit of (unsupervised or supervised) pretraining is not easy to demonstrate.


Find average sequence from a set of sequences I have a set of user sessions. Session consists of an ordered list of types of actions that user made (for example, bought a gun, played a mission, etc).
I want to create/calculate session that have most possible similarity to all provided sessions (most common types of actions users make in order they make them)

Unfortunately, I know nothing about data science but I tried to google a way to do that.
I've found this document: https://cran.r-project.org/web/packages/TraMineR/vignettes/TraMineR-state-sequence.pdf
And it looks like 9.1 and 9.2 describe things similar to what I want. But I dont know this for certain and even if it's true I stil don't know how to use it for my scenario.

One way would be not to approach this as a calculation per session. Most data science solutions like to end up with a number, probability or classification. I suggest you structure your data differently so that you try to answer the question - what next action is likely given the last action.

In order to do this you would have to restructure your session data and use information from across all your sessions. For example, if you compare in how many sessions a player 'buys a gun', and if so record over all those sessions what their next action is, e.g. in 60% they 'play a mission' next. You will then have a probability of their next action based on the number of choices players made in all those sessions.

Once you have those probabilities, you will be able to answer the question, 'What comes next?'. This will in turn enable you to build that most average session that you are after by stepping through a session and building it by the most probable next step.


Machine learning for predicting HTML Elements on a web page? My goal is to implement an assistant for crawling web data for users that don't understand anything about HTML or DOM. I will show a web page to the user and the user has to select, what data he is interested on the page (or what data he is not interested in).

Example: If the user clicks on the cell inside a table, it is very likely he wants to extract all elements inside that column. He might only be interested in the table row, or he might be interested in only the one cell. So the algorithm proposes him three selectors for each possibility. The user might choose one of the proposed selector or he might click on another element to get a new proposition.

So far the use case. The component I want to create:

has as input data a list of elements inside a DOM (hierarchy of HTML elements), that the user wants to crawl
shall output a (probability) ranked list of selectors that will fit the users selection to 100% and possibly include other elements that the user is probably interested in.

The problem here is the high variability of websites, and the possibility to create similar looking results with very differently structured elements. Manually creating rules for prediction can therefore probably cover the most basic uses (like getting data from a table column), but will fail when it comes to finding article parts on a news page.

So finally my questions. I am thinking about the following processing:

first generate selectors that will fit the users selection
sort out the selectors that do not fit the users choice of elements by 100%
use a (machine learned?) model to predict which selector the user might be interested in

Do you think the above processing will be productive for this problem?

What algorithm could be most suitable for ranking the selectors in 3.?

Is there an idea on how to auto-magically come up with a list of selectors for the selected elements?

I hope I am making sense - I do not have a very strong background in data analysis and machine learning so I am hoping for some directions where to look into.

There are at least 2 distinct problems:

Understanding the non-programmers intention for the webpage's content
Understanding the webpage's content

The 1st problem is a problem not best solved with data science or machine learning, hard-coded rules will work better. The 2nd problem is mostly a software engineering problem. I would look into existing solutions for inspiration, for example scraper or import.io.

If you still want to sort and present the "selectors" to users, that is most frequently called a "learning to rank" problem. Learning to rank problems are common in information retrieval systems / search engines.


Approximate Text compression by training model? I have a key-value form of dataset. Basically two sets of integers in many-to-one mapping. Like f(123) corresponds to 4652, f(24) corresponds to 12 and so on. Size of this dataset is say around upto 10^7.

This is too large. So my Idea is to train this dataset on some model and store trained model's parameters only, so when I have to answer say f(24), I can pass 24 as input to my model and print model's output. My thinking was overfitting some model on the dataset could possibly result in high accuracy of say greater than 99%. I strictly want size of parameters to be less than 50,000 bytes.

I tried making vanilla neural network but error percentage is way too high.
I found an interesting blog trying same approach as mine. Kindly let me know if something is unclear to you in comments.

Concrete Example Problem
To give you a concrete example of what I am trying. Here is the DataSet, ith line stores f(i) in this file. I am trying to make a code which is of 50k bytes and can answer queries f(i). Ofcourse while training there are no limits, but after training it should be a file of 50k length at max.

Your problem can be solved with a hash map. If the dataset is very large, you can use an approximate hashmap. Examples of approximate hashmaps are Bloom Filters and Cuckoo Filters.


Scoring metric for recommendation system I'm working on a project that involves building a news recommendation system. I've come as far as quantifying user interaction with different articles  on the site into user's affinity towards atopic using a bayesian function. I also have quantified the recent articles using LDA into the proportion an article talk about each topic.
my users topic-affinity for a user x looks like this(target-x):
 user_id  interest-topic-0  interest-topic-1  interest-topic-2  interest-topic-3  interest-topic-4  interest-topic-5  interest-topic-6  interest-topic-7  interest-topic-8  interest-topic-9
       0            0.0257            0.2956            0.0386            0.0643            0.1285            0.0000               0.0            0.0257            0.0386            0.1671

My quantified articles looks something like this(vectors-v):
post_id   topic-0   topic-1   topic-2   topic-3   topic-4   topic-5   topic-6  topic-7  topic-8   topic-9
      x  0.055048  0.000000  0.742544  0.032286  0.059630  0.000000  0.000000  0.01173      0.0  0.095441
      y  0.000000  0.051172  0.000000  0.000000  0.158314  0.042632  0.022281  0.00000      0.0  0.720676
      z  0.028615  0.000000  0.020919  0.000000  0.000000  0.018940  0.882862  0.00000      0.0  0.046078

The shape of target will always be (10,)
The shape of vectors will always be (num_articles, 10)
Both vectors do not follow the same distribution.
Now I'm trying to figure out the best way to recommend articles from vectors v, for a target user x, given x and v. I've tried distance similarity functions like cosine similarity to find the distance between vectors. The results are satisfactory but I'm looking for a better function/metric to pick out top n recommendations for a user.

Other ways to measure real-valued vector distances:

Euclidean distance
Manhattan distance
Higher order Minkowski distance
Chebyshev distance

Beyond just trying different distance metrics, you can try re-framing the problem or using A/B testing to see which recommendations are most empirically useful for users.


Machine learning in electrical circuit simulation In the field of VLSI design, and specifically in digital circuits it is common to use electrical simulation software to find signal integrity issues.
I would like to use machine learning instead of running a true electrical simulation to do the same, in order to boost run time (simulations can be very heavy). For example, a NAND logic gate with 2 inputs and 1 output, and on each input there is a noise glitch, and I want to predict the glitch which may occur on the output. A glitch is a waveform represented as a series of voltage and time points. That is, for a given electrical circuit I would like to build a model which can be used to predict an output waveform given input waveforms. Instead of running a (spice) simulation code which calculates results analytically, the output waveform will be predicted or estimated by the model.
Training data can be generated plentifully by running real simulations as many as needed.
I need a direction for types of machine learning algorithms and architectures.
From my research I think using a graph-neural-network (GNN) could be a good direction since an electrical circuit is actually a graph, and it is important to capture the relationships between its elements and other graph properties.

Thanks!

For waves one usually uses a Sequence Model but for this purpose I think it will probably be the simplest to use a CNN. I am not very familiar with waveforms, but they seem to be similar to standing waves (time independent) or maybe even better as they also have scale invariance. Assuming the number of input waveforms are the same, I would simply stack them side by side and treat the result as any other image and do image classification using a CNN. You can probably also do the analysis by building a multi input CNN, but you probably have to build it from scratch, I don't know any samples of this kind of architecture available online.


reprocessing steps for images before training classification models I have a data set of images for classification task.
I read some articles about image reprocessing (before training CNN models) which summarize in those steps:

scale image values (img / 255.0)

remove noise (using Gaussian blur)

morphology

I'm not sure when to use each step and what is the right order of those steps:

Do we need to remove noise before scaling images ? (or it dosn't matter) ?

I didn't found many articals about morphology step:
2.1 When will we use morphology ? (Is it always right to use this reprocessing step ?
2.2 What is the right order ? use morphology after scaling and removing noise ?

I assume you talking about preprocessing instead reprocessing.

Divided the image with 255.0 value is a normalization technic called min/max normalization. Like the other normalization methods, min/max normalization used to improve performance of CNN's.

Image process methods like gaussian blur, average filter etc. are using to remove the noise as you said. You might want to denoise your input image to avoid performance losses.

Using purpose of morphology methods are depends on what is your data and what do you want to do your dataset with these methods. You can refer this link to see the methods that cv2 shared with code snippets.

Greetings


The Meaning Behind the Cross Validation Score in Factor Analysis In order to choose the best number of underlying factors for my data using factor analysis, I decided to use the tutorial outlined in scikit-learn's documentation.

Running cross_val_score(fa, X) outputs a score (usually a negative number). What is this score actually measuring? Any references with your answer would be much appreciated!

The score you get is determined by the estimator you use. In your case this is Factor Analysis. Looking at the documentation of the algorithm, you can find the explanation:

  FactorAnalysis performs a maximum likelihood estimate of the so-called loading matrix, the transformation of the latent variables to the observed ones, using expectation-maximization (EM).

and the reference:

score(X[, y])   Compute the average log-likelihood of the samples

So your number is the logarithm of the maximum likelihood estimate of the factor loading matrix.