What is the best Data Mining algorithm for prediction based on a single variable? I have a variable whose value I would like to predict, and I would like to use only one variable as predictor. For instance, predict traffic density based on weather.

Initially, I thought about using Self-Organizing Maps (SOM), which performs unsupervised clustering + regression. However, since it has an important component of dimensionality reduction, I see it as more appropriated for a large number of variables.

Does it make sense to use it for a single variable as predictor? Maybe there are more adequate techniques for this simple case: I used "Data Mining" instead of "machine learning" in the title of my question, because I think maybe a linear regression could do the job...

Common rule in machine learning is to try simple things first. For predicting continuous variables there's nothing more basic than simple linear regression. "Simple" in the name means that there's only one predictor variable used (+ intercept, of course):

y = b0 + x*b1

where b0 is an intercept and b1 is a slope. For example, you may want to predict lemonade consumption in a park based on temperature:

cons = b0 + temp * b1

Temperature is in well-defined continuous variable. But if we talk about something more abstract like "weather", then it's harder to understand how we measure and encode it. It's ok if we say that the weather takes values {terrible, bad, normal, good, excellent} and assign values numbers from -2 to +2 (implying that "excellent" weather is twice as good as "good"). But what if the weather is given by words {shiny, rainy, cool, ...}? We can't give an order to these variables. We call such variables categorical. Since there's no natural order between different categories, we can't encode them as a single numerical variable (and linear regression expects numbers only), but we can use so-called dummy encoding:  instead of a single variable weather we use 3 variables - [weather_shiny, weather_rainy, weather_cool], only one of which can take value 1, and others should take value 0. In fact, we will have to drop one variable because of collinearity. So model for predicting traffic from weather may look like this:

traffic = b0 + weather_shiny * b1 + weather_rainy * b2  # weather_cool dropped

where either b1 or b2 is 1, or both are 0.

Note that you can also encounter non-linear dependency between predictor and predicted variables (you can easily check it by plotting (x,y) pairs). Simplest way to deal with it without refusing linear model is to use polynomial features - simply add polynomials of your feature as new features. E.g. for temperature example (for dummy variables it doesn't make sense, cause 1^n and 0^n are still 1 and 0 for any n):

traffic = b0 + temp * b1 + temp^2 * b2 [+ temp^3 * b3 + ...]


How to know when to treat a problem as a classification task or a regression task? Facing this dataset: https://archive.ics.uci.edu/ml/datasets/wine+quality I treated it both as a classification task and as a regression task.

Due to the fact that classifying the quality of a wine as 8 while it is a 3 is a huge difference while classifying a 5 as 6 is not that bad

This has two subquestions:

Is there an easy way to change the weight of the classification in a (scikit-learn) classifier?
Following the regression approach and having the output being rounded to the nearest valid wine quality value (converting the outputs to a classifier-like outputs) hides any pitfalls?

Many thanks

In effect it is ordinal regression/classification. I suggest you to go for Mean Absolute Error to take into account the missclassification that play a bigger role when far from the ground truth :

$MAE = \frac{\sum_{i=1}^n|h(x_i) - y_i|}{n}$, creating your own scoring function in python.

Given the number of different outputs you have, I recommend you to go for regression. Anyway, rounding any hypothesis output would only play on the score, it's up to you to decide if it makes more sense to see wine quality scores as classes or not.


Can I scrape data from government websites if there is no mention about commercial usage? I am trying to be sure that can I scrape government data from several websites if there is no mention of any commercial usage? I am willing to scrape US navy data(Link) and Canada Industrial Data (Link) and not sure if I should do. I personally don't hesitate to scrape govt. data but sometimes it may take the whole company in trouble specially in US. I need some valuable suggestions how to classify any website whether we should scrape or not.

Short answer: It all depends on how you plan to use the data you scraped. If you plan to upload it to your website, that is generally not OK since content can be copy write protected and you could get sued. If you plan to just use the data for personal use offline you should generally not have a problem.

I suggest you take a look at these links:

https://www.archives.gov/global-pages/privacy.html
http://www.ic.gc.ca/eic/site/icgc.nsf/eng/h_07033.html

Those are the privacy pages found at the bottom of the links you provided.

You will find all the information you need there, including email addresses to which you can write to if you have any questions.

Though to help you out a bit, both of the sites you provided state that most of the content or all of it is copy write protected.

In conclusion, ask permission if you are going to re post the data somewhere on the internet. Websites that allow you to do so clearly state it in their privacy policies or terms of use.


Decision tree with multiple outputs I have a sample with 10 independent variables (X1, X2, X3 ....), and multiple output labels (y1, y2, y3).

Here y1 will depend on X1, X2

y2 will depend on X3, X4 and so on.

y1, y2, y3 might or might not be correlated.

Can you please suggest pros and cons of clubbing this into single model, or should i go with multiple models with single output

X1,X2,y1

X3,X4,y2

.....

Thanks

You basically answered your question: if y1, y2, y3 are independent, there is no point to use a single model. If they are dependent (say, y1=1 implies that y2 is not 2), then the single model helps to exclude the non-existent case.

However, if y1 is a function of x1 and x2, and y2 is a function of x3 and x4, and x1,x2,x3, and x4 are independent, y1 and y2 are also independent.

I would start from whatever seems simpler and more logical. Based on your description of weak dependence between variables, multiple models seem more promising to me.


Where does the "deep learning needs big data" rule come from When reading about deep learning I often come across the rule that deep learning is only effective when you have large amounts of data at your disposal. These statements are generally accompanied by a figure such as this:

The example (taken from https://hackernoon.com/%EF%B8%8F-big-challenge-in-deep-learning-training-data-31a88b97b282 ) is attributed to a 'famous slide from Andrew Ng'. Does anyone know what this figure is actually based upon? Is there any research that backs up this claim?

The original slide in question "Scale drives deep learning progress" is possibly what you currently can find at https://cs230.stanford.edu/files/C1M1.pdf (page 13). It may be roughly interpreted with "low bias learners [in that plot, larger neural networks] tend to benefit from more training examples".


What does "Norm" term mean? In this paper (page 1 abstract) which considers regularization technique, the author used the word "norm" - what does it stand for?
Is it related to Batch Normalization / L1 or L2 Normalization?

  "We stabilize the activations of Recurrent Neural Networks (RNNs) by penalizing
  the squared distance between successive hidden states’ norms"

Googling "what is a norm Neural Net" doesn't provide any links

A norm is a concept in linear algebra which assigns a size to a vector. Many different norms exist you can read up on their many uses here.

In this paper, they are training a recurrent neural network. These are used for time series data. The assumption is a hidden state should be similar for successive time inputs $t$ and $t-1$. Thus, the author proposes an additional constraint based on the relative size of the successive hidden state vectors as

$||h_{t}||_2 - ||h_{t-1}||_2$.

The more different the norm of these successive states, the greater the cost will be.


Bagging with Neural Networks Best practices I am trying to build a majority vote system for 3 Neural Networks, and I came across the concept of Bagging method. Actually, I want to use neural networks as weak learners (I know it's debatable, but some papers have tried it and I want to try it too).
For more information about the voting system I tried to construct/constructed, please read the following thread (The softmax Layer is better the the function in the thread, because the majority function only gives equivalent accuracies of the 3 NNs, but doesn't improve the overall accuracy).
I read that bagging can improve the overall accuracy of the weak learners, but as you can see I only have 3 learners and there aren't any clear information about bagging with Neural Networks. I only found after some reading that I can use ensemble learning with Neural networks by using the output of the trained NNs in a linear fashion. If I want to detail this into steps, I would write:

Divide Dataset into training and validation sets
From the Training sat, construct 3 bootstraps samples ?, Or I need more?
Train/Develop the 3 neural Networks on the 3 bootstraps samples?
Test the Neural Networks on the validation/test set
How can I join decisions of the NNs by bagging?

If possible, I need your insight on these implementation steps and to know if they follow the best practices of bagging with NNs.
PS: I just started reading about bagging and boosting, so I apologize for any conceptual mistakes and contradictions I might have said.
Best regards,

I think you have the right general idea.

Divide the dataset into training and test.
Divide the training dataset into k folds, call this M.

For each iteration of hyperparameter tuning (whatever you choose to tune);

For each fold i in M;

Set the validation set to be equal to fold i, and the (inner) training set to be equal to all folds that are not i.
Using the inner training set, generate N bootstrap resamples. Fit a single neural network model to each bootstrap resample, generating N models.
Predict the validation set using each of the N neural network models.
Count the votes of each neural network model. Set the final prediction for an observation in the validation set to whatever has the most votes as predicted by your N models. You should have a matrix of m observations by N predictions. Since you are actually classifying objects, try to keep N odd so that there are no ties. Alternatively, don't classify objects, and instead generate probabilities from your N models for each observation in the validation set. Then, calculate the mean predicted probability for each observation in your validation set, and choose a threshold for classification based off the costs of false positives/false negatives (probably superior, but I won't discuss this).
Calculate your error using your final predictions from step 6 and the actual ground truth labels from the validation set. Save these error scores.

end

Calculate the average error from all of the folds. If this average error is lower than a previous set of hyperparameters error, then save these hyperparameters from this iteration of tuning.

end

Generate N bootstrap resamples of the entire training set (the training set created in step 1) and fit a single neural network model to each of the N bootstrap resamples, using the optimal set of hyperparameters that you found above for each neural network model.
Generate predictions on the test set (created in step 1) using each of your N models. Set final predictions for the test set using the exact same methodology you used in step 6.
Calculate the error using your final predictions from step 10 and the ground truth labels found in the test set. This is your final, unbiased estimate of model performance.

In general, the more bootstrap resamples you can generate, the better. However, since you are fitting neural networks you might need to be conservative with how many models you decide to bag due to computational costs. I recommend setting the number of bootstrap samples, N, to be as large as possible while being mindful of your own computational/time constraints.

EDIT: In response to the comments in which we require a faster method, one can replace step 2 (and delete step 3) and instead divide the training dataset into an inner training set and a validation one. We then validate our model to a single validation set rather than k sets, which has the potential to increase the likelihood of overfitting to a specific validation set (though since we are presumably dealing with a large dataset, it probably will be fine).

All other steps should remain the same, though there is clearly no loop over folds anymore. In step 8, there will also be no need to calculate the average error since we will only have one estimate of model error from the single validation set.


How to control number of branches for each split in sklearn DT classifier? I am trying to code a two class classification DT problem that I used SAS EM before. But trying to do it in Sklearn. The target variable is a two class categorical variable. But there are a few continuous independent variables. In SAS I could specify the "Maximum Number of Branches" for each split. So when it is set to 4, some leaf will split into 2 and some in 4 (especially for continuous variables). I could not find an equivalent parameter in sklearn. Looked at "max_leaf-nodes". But that controls the total number of "leaf" nodes of the entire tree. I am sure some of you probably has faced the same situation and already found a solution. Please help/share. I will really appreciate it.

Scikit-learn uses an optimized version of the CART algorithm, which follows binary splits. What you are looking for is a multi-way split on nodes for information gain.

You should look for a C4.5 or C5.0 implementation of Decision tress in python.


How to find what events cluster together? I have a data set that looks at a five-year timespan of peoples' lives and indicates if specific events have occurred (Divorce, Birth of Child, Health Shock, etc.).

# Sample data in R
n <- 100
set.seed(1337)
exampleFrame <- as.data.frame(cbind(1:n,replicate(7, sample(0:1,n, TRUE))))
colnames(exampleFrame) <- c('Person',paste(rep("Event",7),1:7, sep = ''))

First thing I did was simply show a correlation matrix.

cor(exampleFrame)

And it gives an idea of what items follow each other, so next I ran a logistic regression to see how events are likely to improve odds of occurring together.

glm(Event1 ~ . - Person,family = binomial(link = 'logit'), data = exampleFrame)

And this is interesting, but I feel like I'm missing something more compelling. Am I chasing a phantom? Does anyone know a better way to demonstrate which life events occur together in this five-year span?

It's just a simple idea but you could calculate the conditional probabilities of an event given another:

library(plyr)

condProbEvents <- function(d) {
  ldply(colnames(d), function(col1) {
    t(ldply(colnames(d), function(col2) {
      nrow(d[d[,col1] & d[,col2],]) / nrow(d[d[,col2],]) # p(col1|col2)
    }))
  })
}

This gives you values which are easy to interpret, e.g. for p(divorce|marriage): "X% of the people who had a marriage also had a divorce"


nltk.corpus for data science related words? from job description I scraped from the internet, I've went through all nlp processes and I've got to place where I found:

freq = nltk.FreqDist(lemmatized_list)
most_freq_words = freq.most_common(100)

which outputs:

[('data', 179),
 ('experience', 86),
 ('work', 78),
 ('business', 71),
 ('team', 59),
 ('learn', 56),
 ('model', 49),
 ('skills', 47),
 ('science', 41),
 ('use', 41),
 ('build', 39),
 ('machine', 37),
 ('ability', 36),.....

and so on. My problem is I do not want to consider words like "experience", "work", and only consider keywords related to data science. I'm guessing there is a corpus for data science terms which I can use like how I use stop word corpus to not select them. Let me know if there is a way, Thanks!

I have a way through which you can solve your problem. For it you will require a,

A pretrained embedding generator. It can be Word2Vec or GloVe. Any of them could work.

  Next, we have a corpus of words which have higher frequencies. Suppose
  we have a set of 100 such words where the 1st word has the highest
  frequency.

Now, we convert every word in this set to a vector using our pretrained word embedding.
Hence you will have a set of vectors for the words from the corpus. Let's call it $z_i$

We have the word "data science". Get a vector for this too. Let's call it $x$

Measure the euclidean distance between the vector $x$ and $z_i$.
Or, you can measure cosine similarity betwee $x$ and $z_i$.
Both of above methods will produce a set of values which would show proximity of $x$ with values of $z_i$.
From these 100 values, we get the least 10 values and convert them to words again.

These 10 words would have the highest similarity with the word "data science".


Best way to store scraped web pages? I'm going to scrape the HTML code from a large number of ULRs and store them on my computer, for machine learning purposes (basically, I'm going to use Python and PyTorch to train a neural network on this data). What is the best way to store the HTML code for all the web pages?
I want to be able to see which URLs I have already scraped, so that I don't have to scrape them again, and for each piece of HTML code (1 piece of HTML code = all extracted HTML code from one URL), I may also want to see which URL it came from (but maybe this turns out to be an unnecessary requirement). I also want to be able to see the timestamp when these pages were created so I can read them in chronological order (I will be able to extract timestamp when I download the web pages), and possibly other metadata. I imagine that the total file size of the HTML code can reach many GB, if not TB, and speed (both for reading and scraping) is a high priority.
In an ideal world, I would be able to just use the URLs as file names, have one file for each piece of HTML code, and store all files in one folder. But I'm not sure this is such a good idea, or even possible, for several reasons, for example:

URLs contain characters that are not allowed or at least not suitable in file names (e.g. '/'). It may be possible to hash each URL and let that be the name of the file, but then I don't know what URL the HTML code came from by looking at the file name.
I'm not sure the operating system likes having millions of files in one folder because of of the file I/O overhead. Maybe grouping a number of pieces of HTML code in each file would be a better, e.g.
I'm not sure whether storing each piece of HTML code in a separate file is optimal when it comes to file reading.
It's not obvious how to store the metadata for each piece of HTML code.

Simply put you can create an "index data base" in the following format:
ID | URL | Timestamp | link_to_file | other_metadata

You could even store the actual file, instead of just the link with most databases.
However the simplest way might be the best here. This "index db" would be created automatically in the scraping process and serve as navigation and check to avoid double scraping, etc.


If you pass header=None, pandas.read_csv assumes that the first row contains data and names the columns '0' to '12'.

Instead you should pass header=0 to specify that the column names are in the first row or equivalently skip the header argument.

You can then still continue with X = balance_data.values[:, 1:12], because calling values returns a numpy array without the column names.

Alternatively, you could also select your feature columns like so:

feature_names = ['A','AAAA',....]
X = balance_data[feature_names].values

You can then pass the same list of feature_names to graphviz.

Also note that you don't have to pass a numpy array to scikit-learn's functions. It can handle pandas DataFrames as well, so values is optional.


Why non-differentiable regularization lead to setting coefficients to 0? The L2 regularization lead to minimize the values in the vector parameter.
The L1 regularization lead to setting some coefficients to 0 in the vector parameter.

More generally, I've seen that non-differentiable regularization function lead to setting coefficients to 0 in the parameter vector. Why is that the case?

Look at the penalty terms in linear Ridge and Lasso regression:

Ridge (L2):

Lasso (L1):

Note the absolute value (L1 norm) in the Lasso penalty compared to the squared value (L2 norm) in the Ridge penalty.

In Introduction to Statistical Learning (Ch. 6.2.2) it reads:
"As with ridge regression, the lasso shrinks the coefficient estimates towards zero. However, in the case of the lasso, the L1 penalty has the effect of forcing some of the coefficient estimates to be exactly equal to zero when the tuning parameter λ is sufficiently large. Hence, much like best subset selection, the lasso performs variable selection."

http://www-bcf.usc.edu/~gareth/ISL/


Uses of NoSQL database in data science How can NoSQL databases like MongoDB be used for data analysis? What are the features in them that can make data analysis faster and powerful?

To be perfectly honest, most NoSQL databases are not very well suited to applications in big data. For the vast majority of all big data applications, the performance of MongoDB compared to a relational database like MySQL is significantly is poor enough to warrant staying away from something like MongoDB entirely.

With that said, there are a couple of really useful properties of NoSQL databases that certainly work in your favor when you're working with large data sets, though the chance of those benefits outweighing the generally poor performance of NoSQL compared to SQL for read-intensive operations (most similar to typical big data use cases) is low.

No Schema - If you're working with a lot of unstructured data, it might be hard to actually decide on and rigidly apply a schema. NoSQL databases in general are very supporting of this, and will allow you to insert schema-less documents on the fly, which is certainly not something an SQL database will support.
JSON - If you happen to be working with JSON-style documents instead of with CSV files, then you'll see a lot of advantage in using something like MongoDB for a database-layer. Generally the workflow savings don't outweigh the increased query-times though.
Ease of Use - I'm not saying that SQL databases are always hard to use, or that Cassandra is the easiest thing in the world to set up, but in general NoSQL databases are easier to set up and use than SQL databases. MongoDB is a particularly strong example of this, known for being one of the easiest database layers to use (outside of SQLite). SQL also deals with a lot of normalization and there's a large legacy of SQL best practices that just generally bogs down the development process.

Personally I might suggest you also check out graph databases such as Neo4j that show really good performance for certain types of queries if you're looking into picking out a backend for your data science applications.


Appending the values of multiple columns into one whilst retaining UIDs I have a table structured as below:

UID|ColA|ColB|ColC|ColD

I want to restructure so that all columns are combined (appended) into one, retaining the UID of each as below:

UID|ColA
UID|ColB
UID|ColC
UID|ColD

I'm very new to Pandas and was wondering how this could be done? Thank you!

Assuming you already have a dataframe df, created as below:

In [1] : df = pd.DataFrame(np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]]),
                           columns=['a', 'b', 'c'])
In [2] : df
Out[2] :
   a  b  c
0  1  2  3
1  4  5  6
2  7  8  9

you could try doing this:

In [3] : df.melt("a").sort_values("a")
Out[3] :
   a variable  value
0  1        b      2
3  1        c      3
1  4        b      5
4  4        c      6
2  7        b      8
5  7        c      9

where the sorting might be optional in your case. Also, you should swap "a" for the name of your column containing the UID values.

You can additionally drop the autogenerated variable column too, so the final solution would be:

In [4] : df.melt("a").sort_values("a").drop("variable", axis=1)
Out[4] :
   a  value
0  1      2
3  1      3
1  4      5
4  4      6
2  7      8
5  7      9

From what I remember, the melt functionality is inspired on the melt function in the R programming language, where DataFrames are one of the core built-in data-types. It is really useful for plotting using the ggplot library (in R).


How to choose the features for a neural network? I know that there is no a clear answer for this question, but let's suppose that I have a huge neural network, with a lot of data and I want to add a new feature in input. The "best" way would be to test the network with the new feature and see the results, but is there a method to test if the feature IS UNLIKELY helpful? Like correlation measures etc?

A very strong correlation between the new feature and an existing feature is a fairly good sign that the new feature provides little new information.  A low correlation between the new feature and existing features is likely preferable.

A strong linear correlation between the new feature and the predicted variable is an good sign that a new feature will be valuable, but the absence of a high correlation is not necessary a sign of a poor feature, because neural networks are not restricted to linear combinations of variables.

If the new feature was manually constructed from a combination of existing features, consider leaving it out.  The beauty of neural networks is that little feature engineering and preprocessing is required -- features are instead learned by intermediate layers.  Whenever possible, prefer learning features to engineering them.


Classifying text as emails or ages I am trying to classify small strings into three categories.
Examples:
X(str column)                    y(str column)
abcd@gmail.com                   email_address
32                               age
@sadn                            invalid(since this is not a valid email address)
1234                             invalid(since this is not a valid age)

I am wondering which ML model needs to be used to achieve this case. And also what kind of text processing techniques need to be applied.

You can start with rule-based methods, such as a combination of regular expression and if/then statements.


Platt's Scaling for multi-label classification? How does one do platt's scaling for multi-label classification? For example, if the final layer of my DNN is a soft-max activation with 10 classes, then how does platt's scaling work exactly?

Do I train multiple logistic regressions using one-vs-many classification? Or is there a better way?

There are a few multiclass variants of Platt scaling. The easiest approach is as you have described; simply perform one Platt scaling on each class.

However, there are more sophisticated options--a very simple one to implement is training a standard logistic regression on the logits (the values before the softmax activation is applied). This has called matrix scaling and can overfit pretty easily, so only use this if you have a large calibration set. Alternatively, a fewer-parameter version called vector scaling is relatively simple to implement, where the weights matrix inside the logistic regression is restricted to be a diagonal matrix. Finally, a very simple option that has been shown to work well for neural networks is temperature scaling, where all logits are simply scaled by a single scalar parameter.

You can read more about these and their application to neural networks in Section 4.2 of "On Calibration of Modern Neural Networks" (2017) - available here


Neural Network with Connections to all forward layers In classical neural nets, we have that each layer is connected only with the following layer.  What if we relaxed this constraint and allowed it to be connected to any or all subsequent layers?  Has this architecture been explored?  It seems to be backprop would still work on this.

At the very least, this type of network could be emulated by artificially creating identity neurons at each layer that bring in the value of every earlier node, where the incoming weights to those layers are fixed at 1.

What you describe has been explored in Deep Residual Neural Networks.

A residual block will combine two or more blocks from a standard architecture like a CNN with a skip connection that adds the input to the first block to the output of the last block.

The intuition is that deep networks have a harder and harder time learning the identity function between layers, which has been proven to be useful especially in image recognition tasks. Residual connections also mitigate the problem of vanishing gradients.

Residual connections help solve the "degradation" problem, where deeper architectures lead to reduced accuracy. For example GoogLeNet won ILSVRC in 2014 with a 22-layer CNN, but in 2015 Microsoft ResNet won with a 152-layer Res Net.


What does Make Density Based Clusterer in Weka do? In Weka, there is a clustering algorithm with the name as Make Density Based Clusterer. When going through its properties, it takes a clusterer as base clusterer(I took it as K-means with k=3).
It initially performs k-means and creates three clusters. I see prior probabilities for each cluster and attribute-wise normal distribution means and standard deviation in the result buffer.

What happens after k-means clusters are calculated?
What role mean, standard deviation and prior probabilities play here?
Why is it called density based?

Based on this paper, MakeDensityBasedClusterer is  a  metaclusterer  that  wraps  a  clustering  algorithm  to make  it  return  a  probability  distribution  and  density.   To  each  cluster  and  attribute,  it  fits  a discrete distribution or a symmetric normal distribution (whose minimum standard deviation is a parameter).


xgboost with tree_method = 'hist' in R According to a benchmark of GBM vs. xgboost vs. LightGBM (https://www.kaggle.com/nschneider/gbm-vs-xgboost-vs-lightgbm) it is possible to implenet xgboost with the argument

tree_method = 'hist'

in R.

However doing so gives me always an error:

Error in xgb.iter.update(bst$handle, dtrain, iteration - 1, obj) :
Invalid Input: 'hist', valid values are: {'approx', 'auto', 'exact'}

What am I missing?

The fast hist mode is available in newer versions of XGBoost. You can find infos on https://github.com/dmlc/xgboost/

For small data sets (100k lines), you wont feel much gain in training speed.


Autoencoders for the compression of time series I am trying to use autoencoder (simple, convolutional, LSTM) to compress time series.

Here are the models I tried.

Simple autoencoder:

    from keras.layers import Input, Dense
    from keras.models import Model
    import keras

    # this is the size of our encoded representations
    encoding_dim = 50

    # this is our input placeholder
    input_ts = Input(shape=(2100,))
    # "encoded" is the encoded representation of the input
    encoded = Dense(encoding_dim, activation='relu')(input_ts) #, activity_regularizer=regularizers.l2(10e-5)
    # "decoded" is the lossy reconstruction of the input
    decoded = Dense(2100, activation='tanh')(encoded)

    # this model maps an input to its reconstruction
    autoencoder = Model(input_ts, decoded)

    # this model maps an input to its encoded representation
    encoder = Model(input_ts, encoded)

    # create a placeholder for an encoded (%encoding_dim%-dimensional) input
    encoded_input = Input(shape=(encoding_dim,))
    # retrieve the last layer of the autoencoder model
    decoder_layer = autoencoder.layers[-1]
    # create the decoder model
    decoder = Model(encoded_input, decoder_layer(encoded_input))

    autoencoder.summary()

    adamax = keras.optimizers.Adamax(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.01)
    autoencoder.compile(optimizer=adamax, loss='mean_absolute_percentage_error')

Convolutional autoencoder:

    from keras.layers import Input, Dense, Conv1D, MaxPooling1D, UpSampling1D
    from keras.models import Model

    window_length = 518

    input_ts = Input(shape=(window_length,1))

    x = Conv1D(32, 3, activation="relu", padding="valid")(input_ts)
    x = MaxPooling1D(2, padding="valid")(x)

    x = Conv1D(1, 3, activation="relu", padding="valid")(x)

    encoded = MaxPooling1D(2, padding="valid")(x)

    encoder = Model(input_ts, encoded)

    x = Conv1D(16, 3, activation="relu", padding="valid")(encoded)
    x = UpSampling1D(2)(x)

    x = Conv1D(32, 3, activation='relu', padding="valid")(x)
    x = UpSampling1D(2)(x)

    decoded = Conv1D(1, 1, activation='tanh', padding='valid')(x)

    convolutional_autoencoder = Model(input_ts, decoded)

    convolutional_autoencoder.summary()

    optimizer = "nadam"
    loss = "mean_absolute_error"

    convolutional_autoencoder.compile(optimizer=optimizer, loss=loss)

LSTM autoencoder:

    from keras.layers import Input, LSTM, RepeatVector
    from keras.models import Model

    inputs = Input(shape=(1, 500))
    encoded = LSTM(128)(inputs)

    decoded = RepeatVector(1)(encoded)

    decoded = LSTM(500, return_sequences=True)(decoded)

    sequence_autoencoder = Model(inputs, decoded)
    encoder = Model(inputs, encoded)

    sequence_autoencoder.summary()

    sequence_autoencoder.compile(optimizer='nadam', loss='mean_absolute_error')

To check for compression loss, I use the SMAPE formula.

I got such results. The average loss for simple autoencoder is 14.28%, for convolutional autoencoder is 8.04%, for LSTM-autoencoder is 9.25%.

My question is: is it practical to compress time series with losses using a neural network if the compression time does not matter? Perhaps i should pay attention to other methods? How can I achieve better compression?

Thanks!

Above all, you should take care of the time series.
Despite from that, AEs are thoroughly used for time series, especially LSTM+AE. Did you vary the topology? For the CAE it looks reasonable but the other models lack some layers, or?
Furthermore, some regular advices would be to standardize the input, change the activation functions (tanh worked well for me in the output layer) as well as the number of neurons per layer and the amount of layers in general.
Could you provide the head() of the input data? An AE expects to fit X on X, maybe you missed that?


Numpy element wise comparison for a particular value in two arrays If I have two arrays as shown below:

a = numpy.array([0, 0, 1, 0, 1, 1, 1, 0, 1])
b = numpy.array([1, 1, 1, 0, 0, 1, 1, 0, 0])

Is there an easy way using numpy to count the number of occurrences where elements at the same index in each of the two arrays have a value equal to one. In the above two arrays, the elements in position(zero-indexed) 2, 5 and 6 are equal to 1 in both the arrays. Thus I want to get a count of 3 here.

Thank you for any help that you may be able to provide.

There are two ways I'll show you (there are probably a lot more using NumPy):

First method: chaining operations

You can use "masking" followed by the comparison and finally a sum operation:

We want all values in a from the indices where b is equal to 1:

part1 = a[b == 1]

Now we want all places where part1 is equal to 1

part2 = part1[part1 == 1]

now we are left with all the places where a and b are equal to 1, so we can simply sum them up:

result = part2.sum()

Method 2: built in numpy.where

This is much shorted and probably faster to compute. NumPy has a nice function that returns the indices where your criteria are met in some arrays:

condition_1 = (a == 1)
condition_2 = (b == 1)

Now we can combine the operation by saying "and" - the binary operator version: &.

part1 = numpy.where(condition_1 & condition_2)

To get your desired output, we can take the length of the resulting set of indices:

result = len(part1)

Read the documentation about numpy.where to see the other things it can do for you!


Is there a quick way to check for multicollinearity between categorical variables in R? I have a large amount of categorical and dummy variables (36) and I would like to remove a number of them based on their multicollinearity (or just collinearity). Instead of using Chi Square tests over and over again, are there any functions that can check for (multi)collinearity in my variables and return variables with multicollinearity (or collinearity)?

In my work I usually use Normalized Mutual Information (NMI) to get an understanding of how "correlated" two categorical variables are.

Normalized Mutual Information is an information-theoretic measure that tells you how much information is shared by two variables. If NMI is close to 1, the two variables are very "correlated", while if NMI is close to 0 the two variables are "uncorrelated".

I wrote this function that computes the NMI between the first two variables in a data.table. It is quite fast thanks to data.table. Feel free to use it!

compute_mutualinfo <- function(df){
  require(data.table)
  require(entropy)
  var1 = names(df)[1]
  var2 = names(df)[2]
  tmp_tab <- df[,.N,by=c(var1,var2)]
  names(tmp_tab)[1:2] <- c('V1','V2')
  cross_tab <- tapply(tmp_tab$N,list(addNA(as.factor(tmp_tab$V2),ifany = T), addNA(as.factor(tmp_tab$V1),ifany = T)), sum)
  cross_tab[is.na(cross_tab)] <- 0
  mi.plugin(cross_tab)/sqrt(entropy(df[,.N,by=var1,with=T][,N])*entropy(df[,.N,by=var2,with=T][,N]))
}


Which algorithms should I use for identifying similar characteristics between data points (the intersections)? I am working with a dataset that has been coded and categorized, so that each datapoint has a set of coded characteristics. An example data point would be something like the following:

Example Data Point:

Quality
Service & Support
Price

Each data point can have multiple codes associated with it.

What I'm looking to do is identify the "intersections" between the data points so that I can answer questions like the following:

When a data point has "Quality" as a characteristic, 25% of the time it also has "Price" as a characteristic

I've been struggling with the right way to ask this question in my Google searching and realized I should just come to the experts on topics like this and get your help and guidance.

To do this type of work, what algorithms should I be investigating?

Thank you for your help!

You could achieve this by creating and analyzing a confusion matrix of characteristics that appear together. Here is an example:

From this image, you can see (from the price-price section) that price appears 4 times. Then, you can also see that (from the price-quality section) that price and quality appear 3 times together. So, you can conclude that 75% of the time that price is a characteristic, quality is also a characteristic.

Other information you can extract:

price appears the most
price and quality is the most common pair
service_and_support is only 50% of the time paired with quality

Below is the code to generate this plot:

import seaborn as sn
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

characteristics = {'quality':0, 'service_and_support':1, 'price':2, 'customer_relations':3}

data_points = [
    ['quality', 'service_and_support', 'price'],
    ['quality', 'price'],
    ['quality', 'customer_relations', 'price'],
    ['service_and_support', 'customer_relations', 'price'],
]

counts_matrix = np.zeros((len(characteristics), len(characteristics)))

for data_point in data_points:
    for characteristic1 in data_point:
        for characteristic2 in data_point:
            counts_matrix[characteristics[characteristic1]][characteristics[characteristic2]] += 1

keys = list(sorted(characteristics.keys(), key=lambda x: characteristics[x]))

df_cm = pd.DataFrame(counts_matrix, index = keys, columns = keys)
plt.figure(figsize = (10,7))
sn.heatmap(df_cm, annot=True)

plt.show()


Negative R2_score Bad predictions for my Sales prediction problem using LightGBM My project involves trying to predict the sales quantity for a specific item across a whole year. I've used the LightGBM package for making the predictions. The params I've set for it are as follows:
params = {
'nthread': 10,
'max_depth': 5, #DONE
'task': 'train',
'boosting_type': 'gbdt',
'objective': 'regression_l1',
'metric': 'mape', # this is abs(a-e)/max(1,a)
'num_leaves': 2, #DONE
'learning_rate': 0.2180, #DONE
'feature_fraction': 0.9, #DONE
'bagging_fraction': 0.990, #DONE
'bagging_freq': 1, #DONE
'lambda_l1': 3.097758978478437, #DONE
'lambda_l2': 2.9482537987198496, #DONE
'verbose': 1,
'min_child_weight': 6.996211413900573,
'min_split_gain': 0.037310344962162616,
'min_data_in_bin': 1, #DONE
'min_data_in_leaf':2, #DONE
'num_boost_round': 1, #DONE
'max_bin': 7, #DONE
'extra_trees': True, #DONE
'early_stopping_rounds':-1
}

My dataset consists of daily sales data (columns= date, quantity) for the years 2017, 2018, 2019 and 3 months of 2020. I've been trying to use the 2017 and 2018 data for training and cross-validation and trying to test it for 2019 data. However my predictions for the year is way off the mark while considering the quantities on a weekly, monthly, quarterly or yearly basis (error ~ 40-50%)(I've tuned the params to bring the error down to this values). Moreover while considering the predictions, my r2_score is giving me a negative value of around -2.9148426301633803. Any suggestions on what can be done to make it better?
Script for lightgbm:
lgb_train = lgb.Dataset(train_x, train_y)
lgb_valid = lgb.Dataset(test_x, test_y)
model = lgb.train(params, lgb_train, \
                  valid_sets=[lgb_train, lgb_valid],\
                  verbose_eval=50)
test_df_pred = df[(df.date >= '2019-01-01') & (df.date < '2020-01-01')]
#test_df_pred = df[(df.date >= '2019-01-01') & (df.date < '2019-02-01')]
#test_df_pred = df[(df.date >= '2019-01-15') & (df.date < '2019-01-22')]
test_df_pred['month'] = test_df_pred['date'].dt.month
test_df_pred['day'] = test_df_pred['date'].dt.dayofweek
test_df_pred['year'] = test_df_pred['date'].dt.year
col = [i for i in test_df_pred.columns if i not in ['date','id', 'qty']]
y_test_pred = model.predict(test_df_pred[col])
test_df_pred['qty_pred'] = y_test_pred
mse = mean_squared_error(y_true=test_df_pred['qty'], y_pred=test_df_pred['qty_pred'])
mae = mean_absolute_error(y_true=test_df_pred['qty'], y_pred=test_df_pred['qty_pred'])
mape = mean_absolute_percentage_error(y_true=test_df_pred['qty'], y_pred=test_df_pred['qty_pred'])
qty = test_df_pred.qty.sum()
qty_pred = test_df_pred.qty_pred.sum()
diff = qty_pred - qty

I assume you are new to the field, thus, I would suggest using tutorials to achieve your goal. Because what you did is completely wrong and your approach is incorrect. I guess you want to model the sales as time series without using any predictor instead you want to model future values by looking at the past values. To achieve that, you need to use algorithms like ARIMA, exponential smoothing, etc. Here what you have done is trying to correlate the year, month, and day with the sales, which does not possess any information about the sale as expected (also you decoded it wrongly). Thus, your performance metric shows you a negative result. As a reference, check these which are similar to your problem. Source1, Source2, Source3. These will solve your issue.


Overfitting results with Random Forest Regression I have one image that contains for each pixel 4 different values.
I have used RF in order to see if I can predict the 4th value based on the other 3 values of each pixel. for that I have used python and scikit learn. first I have fit the model, and after validate it I used it to predict this image.
I was very happy and scared to see that I got very high accuracy for my model : 99.95%!
but then when I saw the resulted image it absolutly wasn't 99.95% of accuracy:
original image:

result image:

(I have makrd the biggest and most visible difference).
My question is- why would I get this high accuracy when the visualization shows very well that there is much less accuracy? I understand it might come from overfitting but then how this different is not detected?
edit:
Mean Absolute Error: 0.048246606512422616
Mean Squared Error: 0.00670919112477127
Root Mean Squared Error: 0.0819096522076078
Accuracy: 99.95175339348758

Where are you evaluating the performance of your algorithm?
Are you making a train test split and evaluating in the test split? It might be that you overfitted your train and you are just measuring the accuracy there.
If you have made correctly the train/test split and the evaluation it could be that the images that you are predicting do not have the same properties/configuration/topology than the with you are trainning


K-modes implementation in pyspark I'm looking for an implementation of k-modes in pyspark. I found this and this as implementations.

First, I tried implementing k-modes using the first link and faced issues. So I went ahead and tried the second implementation on github. This one worked (as in ran without errors), but I'm still unable to use it in the proper way because of no guide/usage techniques. According to the github page:

n_modes=36
partitions=10
max_iter=10
fraction = 50000 * partitions / (data.count() * 1.0)
data = data.rdd.sample(False,fraction).toDF()

method=IncrementalPartitionedKMetaModes(n_partitions = partitions, n_clusters = n_modes,max_dist_iter = max_iter,local_kmodes_iter = max_iter,
 similarity = "frequency", metamodessimilarity = "hamming")

cluster_metamodes = method.calculate_metamodes(data)

I'm not able to understand how to use this cluster_metamodes, or what it even means.

  In short, k-modes will be performed for each partition in order to
  identify a set of modes (of clusters) for each partition. Next,
  k-modes will be repeated to identify modes of a set of all modes from
  all partitions. These modes of modes are called metamodes here.

Can someone explain the concept and tell how to use this method of clustering on a pyspark dataframe of categorical values?

Modes are just the analog of centroids in k-means. A distributed implementation assigns points to clusters in parallel on subsets of the data, then computes new centroids in parallel, and finally collects those results to perform a (weighted) average of those centroids to get the real centroids. Similarly here, after cluster assignment, modes are computed on subsets of the data in parallel, but then they need to be collected and combined into the real new modes. For whatever reason they're calling the 'real' modes 'metamodes'.


Do Convolution Layers in a CNN Treat the Previous Layer Outputs as Channels? Lets say you have a max pooling layer that gives 10 downsampled feature maps. Do you stack those feature maps, treat them as channels and convolve that 'single image' of depth 10 with a 3d kernel of depth 10? That is how I have generally thought about it. Is that correct?

This visualization confused me:
http://scs.ryerson.ca/~aharley/vis/conv/flat.html

On the second convolution layer in the above visualization most of the feature maps only connect to 3 or 4 of the previous layers maps. Can anyone help me understand this better?

Related side question: If our input is a color image our first convolution kernel will be 3D. This means we learn different weights for each color channel (I assume we aren't learning a single 2D kernel that is duplicated on each channel, correct)?

Lets say you have a max pooling layer that gives 10 downsampled feature maps. Do you stack those feature maps, treat them as channels and convolve that 'single image' of depth 10 with a 3d kernel of depth 10? That is how I have generally thought about it. Is that correct?

Yes. The usual convention in a CNN is that each kernel is always the same depth as the input, so you can also think of this as a "stack" of 2D kernels that are associated with the input channels and summed to make one output channel - because under the convention that $N_{in\_channels} = N_{kernel\_depth}$ this is mathematically the same. Expressing as a 3D convolution allows for simpler notation and code.

  On the second convolution layer in the above visualization most of the feature maps only connect to 3 or 4 of the previous layers maps. Can anyone help me understand this better?

The diagram is non-standard in that respect, although it seems to show pooling and fully-connected layers as normal. It might be a mistake in the diagram, or something unconventional about that specific CNN.

  If our input is a color image our first convolution kernel will be 3D. This means we learn different weights for each color channel (I assume we aren't learning a single 2D kernel that is duplicated on each channel, correct)?

Correct. You can see this in the visualised filters for AlexNet (do note that for computational reasons, AlexNet specialised one half of its filters to work in greyscale, and had other clever optimisations that we don't use nowadays because available GPU power is high enough to not need them). Most implementations will also treat a greyscale image as a 1-channel 3D shape for consistency.


How to add a new category to a deep learning model? Say I have done transfer learning on a pre-trained network to recognize 10 objects. How can I add a $11^{th}$ item that the network can classify without losing all the 10 categories I already trained and the information from the original pre-trained model? A friend told me that active research is going on in this field, but I cannot find any relevant papers or a names  to search for?

If this is just a one-time case, you can simply re-train the neural network. If you frequently have to add new classes, then this is a bad idea. What you want to do in such cases is called content-based image retrieval (CBIR), or simply image retrieval or visual search. I will explain both cases in my answer below.

One-time case

If this just happens once - you forgot the 11th class, or your customer changed his/her mind - but it won't happen again, then then you can simply an 11th output node to the last layer. Initialize the weights to this node randomly, but use the weights you already have for the other outputs. Then, just train it as usual. It might be helpful to fix some weights, i.e. don't train these.

An extreme case would be to only train the new weights, and leave all others fixed. But I am not sure whether this will work that well - might be worth a try.

Content-based image retrieval

Consider the following example: you are working for a CD store, who wants their customers to be able to take a picture of an album cover, and the application shows them the CD they scanned in their online store.
In that case, you would have to re-train the network for every new CD they have in the store. That might be 5 new CDs each day, so re-training the network that way is not suitable.

The solution is to train a network, which maps the image into a feature space. Each image will be represented by a descriptor, which is e.g. a 256-dimensional vector. You can "classify" an image by calculating this descriptor, and comparing it to your database of descriptors (i.e. the descriptors of all CDs you have in your store). The closest descriptor in the database wins.

How do you train a neural network to learn such a descriptor vector? That is an active field of research. You can find recent work by searching for keywords like "image retrieval" or "metric learning".

Right now, people usually take a pre-trained network, e.g. VGG-16, cut off the FC layers, and use the final convolutional as your descriptor vector. You can further train this network e.g. by using a siamese network with triplet loss.


How match output (pred value) to input value I'm working with data(with 4 columns which are p(product), M(name of the store)), I want predict the demand of store for that I sued SVR on the data by theses formulation:
dfn = pd.get_dummies(df)
x = dfn.drop(["demand"],axis=1)
y = dfn.demand
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler(feature_range=(0,1))
dfn = scaler.fit_transform(dfn)
.
.
.
from sklearn.metrics import r2_score
pred = regressor.predict(testX)
SVM_R2 = print('r2= ' +str(r2_score(testY,pred)))
print(pred)
# array example is between 0 and 1
array = np.array(pred)
#scaled from 200 to 800
minimo = 200
maximo = 800
output=array * minimo + (maximo - minimo)
print(output)
df2=pd.DataFrame(output)
df2.to_excel(r'/content/Book1.xlsx', index = False)

and now I get the output of this prediction. My question is, how can I match these outputs to inputs, or how can I found which demands are related to each market?

The outputs of your model are in the same order as your inputs, so the first row in your output array corresponds to the first row in the testX array. If you want to have both the inputs and the model prediction in one table you can just concatenate them along the column axis.


Importance of normal Distribution I have been reading about probability distributions lately and saw that the Normal Distribution is of great importance. A couple of the articles stated that it is advised for the data to follow normal distribution. Why is that so? What upper hand do I have if my data follows normal distribution and not any other distribution.

This is an interesting question. So sorry for a long winded answer. The tl:dr; is it is a mix of some real applicability, theoretical basis, historical baggage (due to limited compute power) and obsession for analytically tractable models (instead of simulation/computational models). We should be very careful and discerning while using it in real problems.
Details
The importance of normal distribution comes from the following facts/observations,

Many naturally occurring phenomenon seem to follow normal distribution when sample size is large (more on this below).
In Bayesian statistics, if you assume a Normal distribution prior on parameters, then posterior distribution is also normal. This makes computations easier.
Somewhat related, central limit theorem tells us that average of a samples from any distribution (no fat tails) follows normal distribution. So normal distribution is useful and provides theoretical basis for doing population level parameter estimates from samples (think of election predictions). But again, this assumes the underlying data comes from a distribution which is well behaved and extreme values are very unlikely.

In short, normal distribution can be thought of as a good base case, which is analytically tractable, easy to code up and also seems to be applicable to many models of nature. Somewhat broken analogy, but in Physics we consider linear second order differential equations to study many systems. Now not all systems actually are linear second order, but it is a reasonable approximation under some constraints that is easier to analyze and code up.
And over-usage of normal distribution everywhere is actually controversial.

As we have to more computing power and access to Monte Carlo based simulation based method, we are no longer limited by using only analytically tractable distributions. We can use distributions which more accurately fit the reality.
Normal distributions are useful for natural phenomenon (heights of students in a class) but are way inaccurate to model mostly man made systems (income of people in a town, potential swings of stock indices during panic).
For example, many critics of probabilistic financial models observe that the underlying models use normal distribution. But real market swings are mostly fat tailed (distributions where extreme outcomes are more likely than normal distributions). If you want to go down on deeper into this, start with statistical consequences of fat tails by Nassim Nicholas Taleb. Fun fact, if you look at the wild swings in the price of GameStop stock from the r/wallstreeetbets saga, Taleb pointed out that the swings are not actually wild if you consider a fat tailed distribution.


Matrix class not being recognized I have a data frame, and I am only interested in the numerical variables (variable class is double):
Example
A    B    C    D    E
0.13 2.22 3.44 3.30 6.54
3.55 4.23 0.43 5.33 3.55
...

I want to build a linear regression xgboost model, so I converted the data frame to a dense matrix.
I took out all categorical variables from my data, and am left with the five variables of interest. But I can't seem to be able to use the dense matrix, despite it being a type of input that xgboost accepts. See here. I am instead getting this error:
Error
"Error in xgb.DMatrix(train_mat[, -n], label = train_mat[, n]) : [23:57:13] amalgamation/../dmlc-core/src/io/local_filesys.cc:66: LocalFileSystem.GetPathInfo 2007 Error:No such file or directory".
Code
train_mat <- as.matrix(subset(train, select = -c(Genre, Publisher, Rank, Name, Platform)))

lm.boosted <- xgb.train(data = train_mat, booster= "gblinear", max_depth=3, nthread=1, nrounds = 2, eval_metric="error", eval_metric= "rmse", objective= "reg:linear")

I tried to convert to a xgb.DMatrix, but I am not clear on what it's asking for to do the conversion.
train_mat <- xgb.DMatrix(data = train_mat, label= train$Global_Sales)

How do I input data into xgboost?

Following this example:

Suppose the last column in train_mat is the label that needs to be predicted, i.e. Global_Sales, then you construct the xgb.DMatrix as:

n <- ncol(train_mat)    # or if not the last, the position of the label column
dtrain <- xgb.DMatrix(train_mat[,-n], label=train_mat[,n])

To train the model, simply pass dtrain to the data parameter:

lm.boosted <- xgb.train(data = dtrain, booster= "gblinear", max_depth=3, nthread=1, nrounds = 2, eval_metric="error", eval_metric= "rmse", objective= "reg:linear")


How to decide which images to label next? We have a custom dataset of 20 thousand images with two pixel-wise labeled classes.
However we have 1 million more raw images, which we would like to label.

We want to label the most important new images first. Importance is defined as:

Images with the more new information
Images helping our deep learning bounding box classifier to improve

So instead of labeling images, where we already have about thousand similar ones, we first want to label the images which are quite different from the already-labeled ones and help more to improve our classifier.

How can we assign priorities and decide which images to label first?

This type of problem is considered to be part of 'active learning'. There is a lot of research being done on this topic at the moment, but some first approaches are relatively easy, depending on the type of model that you are using. Since you mentioned that you are using deep learning bounding box detectors, I will showcase a few examples of how to approach this problem using Convolutional neural networks.

The core idea is that we want some measure of potential gain of an unlabeled sample. That way we can train our model on our labeled training set, predict the labels for our unlabeled set and measure which examples will be most useful to label.

In case of classification you could use the sigmoid/softmax output and get some kind of uncertainty from there, however deep learning models are usually fairly certain about their predictions and a high probability doesn't automatically mean that it predicts it well.

Another approach is to use dropout in your model during training, and then apply dropout to your predictions on your unlabeled set as well. By sampling multiple dropout masks and comparing all the different predictions, you could measure how different the outputs are. If the outputs are very similar, it's unlikely that your model will learn much more if you label this, but if the outputs vary wildly, maybe this example lives in a part of your feature space that your model doesn't know or understand very well yet.

There are a lot of ways to approach this, what I have written here is just an introduction to the concept of 'active learning'. There are a lot of papers available about this topic! EDIT: I haven't actually read a lot of this research, but here are a few:

https://arxiv.org/pdf/1703.02910.pdf

https://arxiv.org/pdf/1707.05928.pdf

https://arxiv.org/pdf/1701.03551.pdf


What algorithm can help me discover synonyms? In the  context of text mining, I'd like to discover potential synonyms in my dataset. The current dataset is stackexchange's stackoverflow data on archive.org. The result doesn't have to be perfect, I can post-process it by hand. But I need help to have a clue of which term seems to be "similar" to another term.

Here examples of synonyms I am looking for:

postgresql, postgres, pgsql, psql
mobile, phone, android, iphone

Also in best case, it should be possible to also guess multi word synonyms like:

rdbms, relational database management system
obama, barack obama

The algorithm doesn't need to compute whether those are many-way synonyms (like rdbms and relational database management system) and one-way synonyms (like iphone is mobile but not all mobile are iphone).

I read that word2vec can be helpful but I am not sure how to use it.

word2vec is probably the way to go. It maps words to a point in n-dimensional space. You can use Euclidean (or whatever distance) to find the nearest points to a given word. If training went well, the closest points should be a synonym.


Building a time-series demand forecasting model I am forecasting demand for certain types of goods and services, which I expect to be correlated to a sub-set of a basket of macroeconomic indicators (considering 15-20 indicators)

I do not know which indicators influence the demand more, whether they have a simple correlation influence or whether a derivative of change influences the demand (i.e. GDP or GDP change for example) or whether there’s a delayed effect on demand (e.g. increased government spend in last year may better predict this year’s demand?). Some macro indicators may be correlated to each other.

I have some basic hypotheses on likely indicators - that may be right or wrong.

Questions 1. What are good time-series forecasting models? What can be considered, apart from just a multivariate regression? 2. Is there a tool whereby I can input the historical demand, historical macro indicators, which will then output which set of indicators best predict the demand and which model works best?

I know how to do regressions in excel, but that’s just one set of indicators at a time. 20 indicators (plus derivatives, plus lag) throws up so many possibilities I cannot manually simulate.

Any help appreciated.

Time series forecasting with exogenous variables (a.k.a. external regressors, like GDP you mentioned) can be complex. I'd suggest starting with an ARIMA (autoregressive integrated moving average) model that includes exogenous variables, like arima in R or sarimax from statsmodels in Python.


Optimum way to train a keras nn by a dataset I have a training dataset with all the images and no sub -directories , also the images are named as random long ids and the target labels are stored in another file labels.txt with ids and the corresponding  labels.
How should I implement a generator using keras to pass these images to the model without loading the dataset in ram.

edit:

One way I see it as

sorting the image names and labels so that i can use ImageDataGenerator.flow

I would recommend running a script before even using Keras just to clean up your data into a more standard format such as .h5. You can then bin this data into batches with a given index in the key id. Then writing the DataGenerator in Keras becomes extremely easy.

The way I do it is to have the following type of keys:

dataset_identifier_ix

where dataset is either train, validation or test, depending on the set you wish to use. Identifier in your case will be either image and label. Lastly, ix is the batch index. Inside each of these batches you have $n$ instances with which to train your model.

If this is the structure of your .h5, then the DataGenerator code can be like

class DataGenerator(keras.utils.Sequence):

    'Generates data for Keras'
    def __init__(self, dataset, filename, X_identifier, Y_identifier,
                 batch_size, percent_data_use = 1, shuffle=True):
        super(DataGenerator,self).__init__()

        with h5py.File(filename+'.h5', 'r') as hf:
            keys = list(hf.keys())
            # Get dimensions of the input space
            temp = [i for i in keys if dataset+'_'+X_identifier in i]
            x_dims = hf[temp[0]].shape[1::]
            num_files = len(temp)
            # Get dimensions of the output space
            temp = [i for i in keys if dataset+'_'+Y_identifier in i]
            y_dims = hf[temp[0]].shape[1::]

        # The batch size
        self.batch_size = batch_size
        self.num_files = num_files
        # Assumes the file ids are always from 0 to num_files
        self.file_ids = list(range(int(percent_data_use*num_files)))
        # Calculate the number of batches
        self.num_batches = int(percent_data_use*num_files)

        self.filename = filename
        self.dataset = dataset
        self.X_identifier = X_identifier
        self.Y_identifier = Y_identifier

        # Dimensions of the input and the output
        self.x_dims = x_dims
        self.y_dims = y_dims
        self.input_shape = (int(x_dims[0]*sampling_percent),) + x_dims[1:]

        self.sampling_percent = sampling_percent
        self.shuffle = shuffle
        if shuffle is True: self.on_epoch_end()

    def __len__(self):
        """Number of batch in the Sequence.

        Returns
            The number of batches in the Sequence.
        """
        return self.num_batches

    def __getitem__(self, index):
        """Gets batch at position `index`.
        Arguments
            index: position of the batch in the Sequence.
        Returns
            A batch
        """
        X = np.zeros((self.batch_size,) + self.x_dims)
        Y = np.zeros((self.batch_size,) + self.y_dims)

        with h5py.File(self.filename+'.h5', 'r') as hf:

            # Input and output identifiers
            x_id = self.dataset + '_' + \
                    self.X_identifier + '_' + str(self.file_ids[index])
            y_id = self.dataset + '_' + \
                    self.Y_identifier + '_' + str(self.file_ids[index])

            X = np.asarray(hf[x_id])
            Y = np.asarray(hf[y_id])

        if len(X) == 0:
            return None

        ## Restructure X, and Y for use in the Keras network
        return X, Y

    def on_epoch_end(self):
        'Updates indexes after each epoch'
        if self.shuffle == True:
            np.random.shuffle(self.file_ids)


Size Issue when Plotting the Predicted Vaue I wrote a simple Stock Prediction Algorithm and got the predicted value. Then, I wanted to plot the relation between Adjusted close price and predicted value, but got the ValueError: x and y must be the same size. I tried to reshape it, but no luck. I'm having problem with the last 5 lines of the following code.
How can I resize X and y_predict in order to have a same size of them? What is the exact problem with my resize code?

import quandl
import pandas as pd
import numpy as np
import datetime
import matplotlib.pyplot as plt
%matplotlib inline
from sklearn.linear_model import LinearRegression
from sklearn import preprocessing, svm, datasets
from sklearn.model_selection import cross_validate, train_test_split

stock_data = quandl.get("WIKI/AAPL")
stock_data = stock_data[["Adj. Close"]]
forecast_out = int(30)
stock_data["Prediction"] = stock_data[["Adj. Close"]].shift(-forecast_out)
X = np.array(stock_data.drop(['Prediction'],1))
X = preprocessing.scale(X)
X_forecast = X[-forecast_out]
X = X[:-forecast_out]
y = np.array(stock_data["Prediction"])
y = y[:-forecast_out]
LinearRegression().fit(X.reshape(-1,1),y)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state = 0)
lin_reg = LinearRegression().fit(X_train,y_train)
y_predict = lin_reg.predict(X_test)

LinearRegression().fit(X.reshape(-1,1),y_predict)

plt.scatter(X_train, y_predict,  color='blue')
plt.plot(X_train, regr.coef_[0][0]*X_train + lin_reg.intercept_[0], '-r')
plt.xlabel("Adjusted closing price")
plt.ylabel("Predicted price")
```

You are using the full X dataset and want to plot it with the y_predict values, this is not possible since the size of both arrays is not the same. y_predict are the predicted values on X_test while X contains all inputs (i.e. X_train and X_test). You should therefore use X_test instead of X.


What does it mean when your NN model has "multiple" inputs or outputs in Keras? I'm reading the guide on Keras website: https://keras.io/guides/sequential_model/
And it says don't use a sequential NN if you have multiple inputs or outputs.

What is the difference between multiple inputs and just having many input variables?
Does it mean having two np.arrays of features?
Why wouldn't someone just combine the two arrays into a single array?
Also, what does "multiple" outputs mean?
Would this just mean that you want to jointly produce two predictions?
Could someone give an example (non - code) of a real life application so I understand what "multiple" input and output means?

Thank you!

A Sequential model is not appropriate when:

- Your model has multiple inputs or multiple outputs
- Any of your layers has multiple inputs or multiple outputs
- You need to do layer sharing
- You want non-linear topology (e.g. a residual connection, a multi-branch model)

The docs suggest that you shouldn't use the Sequential API of Keras when you have such requirements as they also provide Functional API.

Assume that you want to caption an Image automatically. Now in the training time, you need to have images and texts both as Inputs at the same time, right? That's multiple inputs to the network.

Again, Assume that you want to do some image classification for some random image as well as predict the bbox coordinates of the desired object. Now you will have a regression output as well as a probability based output layer, right?

Another example for multiple outputs, Suppose you want to classify an image as to some class as well as find out which color has maximum coverage in the image or what's the color of the object (e.g. a black tee shirt, jeans etc). That will have multiple outputs at the very end.

The last variant possible is combination of both, having multiple inputs and outputs together.

Try to avoid it when your requirements are different or slightly involved...
Sample Image From link


KShape cluster centers offset? I am using tslearn KShape to cluster time series data. I am generally happy with the results, as upon inspection, the clusters seem to make sense because of the similarity in shape and magnitude. I can see the general shape that emerges as seen in the figures below.

What I find odd is the cluster centers using k_shape.cluster_centers_. There seems to be a significant shift from some of the clusters. I've included two figures illustrating my point. In Cluster 0, the cluster center is plotted as the red line, and there is a significant shift from where that shape actually emerges in the time series. In Cluster 4, the cluster center is much more centered, for lack of a better term.

What could be contributing to the major offset seen in Cluster 0? What I find odd is that not all clusters have such a dramatic offset, and not all are offset to the beginning or the end of the time series. Any thoughts?

It turns out that the red shape in each cluster indicates the cluster centroid, which is the shape that all the other shapes in the cluster are compared with. There is an offset occasionally because it's a centroid, and there may be a some variance within the cluster. For that reason, the standard deviation set in the k-shape algorithm can affect the centroid, with larger standard deviations contributing to less "exact" centroids.


What are the "extra nodes" in XGboost? When training an XGboost model some of the information printed regards "extra nodes". I can't find an explanation of these anywhere in the documentation. What exactly are extra nodes?

[14:13:09] C:\dev\libs\xgboost\src\tree\updater_prune.cc:74: tree pruning end, 1 roots, 54 extra nodes, 0 pruned nodes, max_depth=5
[14:13:09] C:\dev\libs\xgboost\src\tree\updater_prune.cc:74: tree pruning end, 1 roots, 58 extra nodes, 0 pruned nodes, max_depth=5
[14:13:09] C:\dev\libs\xgboost\src\tree\updater_prune.cc:74: tree pruning end, 1 roots, 48 extra nodes, 0 pruned nodes, max_depth=5
[14:13:09] C:\dev\libs\xgboost\src\tree\updater_prune.cc:74: tree pruning end, 1 roots, 46 extra nodes, 0 pruned nodes, max_depth=5
[14:13:10] C:\dev\libs\xgboost\src\tree\updater_prune.cc:74: tree pruning end, 1 roots, 48 extra nodes, 0 pruned nodes, max_depth=5
[14:13:10] C:\dev\libs\xgboost\src\tree\updater_prune.cc:74: tree pruning end, 1 roots, 50 extra nodes, 0 pruned nodes, max_depth=5
[14:13:10] C:\dev\libs\xgboost\src\tree\updater_prune.cc:74: tree pruning end, 1 roots, 60 extra nodes, 0 pruned nodes, max_depth=5
[14:13:10] C:\dev\libs\xgboost\src\tree\updater_prune.cc:74: tree pruning end, 1 roots, 44 extra nodes, 0 pruned nodes, max_depth=5
[14:13:10] C:\dev\libs\xgboost\src\tree\updater_prune.cc:74: tree pruning end, 1 roots, 50 extra nodes, 0 pruned nodes, max_depth=5
[14:13:11] C:\dev\libs\xgboost\src\tree\updater_prune.cc:74: tree pruning end, 1 roots, 46 extra nodes, 0 pruned nodes, max_depth=5

Backtracking the updater source code, it looks like "extra nodes" are calculated this way:

At each boosting stage, looking at the gradient boost tree,

Extra Nodes = (the total number of nodes) - (the number of start roots) - (the number of deleted nodes)

At each boosting stage, there might be different starting roots (sub trees) and different deleted (so far) nodes. The extra nodes can provide some intuition into how much your processing tree is utilized.

updater_prune.cc

tree_model.h

xgboost "train" api


What is the best way to predict multiple outcome from a single entity? Let's say i have three model: Facial recognition, Face landmark detection, Emotion recognition.
Now if i want to predict those three feature from a single image. What should be my approach?

Should i combined those three model? or
Run three model in three different thread?

All three models fit to single GPU

Since you have already trained the models and models are separate (do not share the features), you could construct the computational graph in a way that you have only one input (your image), but that input is pushed to three different branches of the computational graph (each branch is one of your three models). At the output of such constructed graph, you will get three outputs (one from each of three branches).

This way you will run all three models at once.

If you are using TF, it will look like this:

output_1, output_2, output_3 = sess.run(output_op, feed_dict:{input_layer: input_image})

where the output_op holds a list of outputs from three models (hence, we unpack them to three variables output_1, output_2 and output_3); input_layer is the tensor operation which takes the image and pushes it to three branches as already described.

This is only possible if your GPU memory is large enough to fit all three models into the memory.

All three models do not fit to single GPU

In this case, assuming you have access to multiple GPUs, you could modify the computational graph which combines three models to use different GPUs for each branch.

Run one after another

Also, this can be always done.

If you are using TF, this link could be useful.


Guiding principle of Neural Network structure building I study Neural Networks and I pretty much understand the logic of the structure with layers, activation functions and connections. But a fundamental question is not clear: how does one put together the actual structure from the several possible combinations? It seems me that most people just combine these quite randomly without any proper reason (or with brute force "grid search" in better cases).

I've read several tutorials but nobody explained the reason of choosing "sigmoind" activation function over "tanh" for instance, not to mention which one to pick in case of different activation functions in different layers. I stress that I understand the working of these functions by themselves, but the logic of the order of them is quite a puzzle.

You would use a sigmoid if you want your activations to be between zero and 1 and tanh between -1 and 1, this is important for the final layer, but for a hidden layer there is not much difference between them.

Here is an article on how to build a neural network in a more systematic way.

The basic principle is to start with only one layer, and make it bigger and bigger until you see satisfactory results, if you don't, add another layer and start over. Oh and make sure you use batch normalization, and don't use sigmoids or tanh's, use PreLUs or ELUs:
https://www.linkedin.com/pulse/keras-neural-networks-win-nvidia-titan-x-abhishek-thakur?trk=prof-post

In general:

Sigmoids: use for output layer which are probabilities with logloss loss function, and for gates in RNNs.

Tanh: use for updating the state in RNNs (linear-like activations might make it unstable), or for an output layer with variables between -1 and 1.

Rectifiers: use for activations in hidden layers of feed-forward network.


Word2Vec how to choose the embedding size parameter I'm running word2vec over collection of documents. I understand that the size of the model is the number of dimensions of the vector space that the word is embedded into. And that different dimensions are somewhat related to different, independent "concepts" that a word could be grouped into. But beyond this I can't find any decent heuristics for how exactly to pick the number. There's some discussion here about the vocabulary size: https://stackoverflow.com/questions/45444964/python-what-is-the-size-parameter-in-gensim-word2vec-model-class  However, I suspect that vocabulary size is not most important, but more important is how many sample documents you have and how long they are. Surely each "dimension" should have sufficient examples to be learnt?

I have a collection of 200 000 documents, averaging about 20 pages in length each, covering a vocabulary of most of the English language. I'm using the word2vec embedding as a basis for finding distances between sentences and the documents. I'm using Gensim, if it matters. I'm using a size of 240. Is this reasonable? Are there any studies on what heuristics to use to choose the size parameter? Thanks.

I have checked four well-cited papers related to word embedding: 2013 Word2Vec, 2014 GloVe, 2018 BERT, and 2018 ELMo. Only GloVe has experimented on the embedding dimension for the analogy task (answering "a" is to "b" as "c" is to ?).

Other papers did not report an experiment on embedding dimension size. They are all using an arbitrary dimension on the order of hundreds (100 and 300 are used more frequently). The lack of experiments for embedding size implies that the performance is not very sensitive to this parameter and only the order of magnitude matters, and also other aspects of the model architecture are more important to investigate.

You can carry out a similar experiment using values from three orders of magnitude 10, 100, and 1000. The required dimension size will depend on the task too. I speculate that a classification (discriminative) task would require fewer dimensions than a sentence generation task.

Generally, for hyper parameter optimization, methods like Bayesian Optimization can be used to find the best hyper parameter (here, embedding dimension) with as few (costly) training-evaluations as possible. These techniques are more useful in competitive settings where every improvement counts.


How to resolve "invalid number of 'breaks'" Code:

 f_custdata <- gsub(".", "", f_custdata)
 f_custdata <- as.numeric(f_custdata)
 hist(f_custdata)

Error: Error in hist.default(f_custdata) : invalid number of 'breaks'

I've also tried:

  df_custdata <- gsub(".", "", df_custdata)
  df_custdata <- as.numeric(df_custdata$income)
  hist(df_custdata)

It gives me 2 errors:

Error in df_custdata$income : $ operator is invalid for atomic vectors

AND

Error in hist.default(df_custdata) : 'x' must be numeric

It looks like your data isn't actually a 2-dimensional frame, just a 1-dimensional series of strings.  (Hence the error when you try to select out an income column.)  So casting to numeric doesn't make sense.

You'd want to split rather than just replacing periods, but also your data strings aren't period-separated; it's almost space-separated, but you also have data strings that have internal spaces ("Homeowner free and clear"), so you can't just split on spaces.  You'd have to be more clever in splitting up the strings into the columnar data you want.  Or, hopefully, your data is more cleanly set up elsewhere.


Loading a Keyword and Evaluating the Information I am an Ex Service Veteran and need assistance with a Small Program to use with my Rats of Tobruk Project, for the purpose of evaluating Archives Information, which I normally do by Manual Means, but my List has grown to 11,663 records that need Verification.

I have put this together as a Guide to what needs to happen.
I hope someone with experience can assist me with a Program.

Automate Finding Service Records & Evaluation

I produce an Excel List from my National Rats of Tobruk Database, consisting of two (2) Columns – Service Number and NAA.

a) Service Number will contain the List of Service Numbers to be found and evaluated, in sequence.
b) NAA will initially be Blank, but will be changed to (Y) if a Condition is found “True” for the Loaded Service Number.

Open Browser (Firefox or Chrome)

a) Browse to (https://recordsearch.naa.gov.au/SearchNRetrieve/Interface/DetailsReports/SeriesDetail.aspx?series_no=B883)
b) Set Button to “List Report”
c) Click Tab = “Basic Search”

From Excel Service Number List, Enter the first “Service Number” in the Pad = “Keywords” and Press “Enter” or Click “Search”

a) If the Column “Digitised Item” is Blank – Loop Back – load the next Service Number from the Excel Service Number List and evaluate.
b) If the Column “Digitised Item” shows a "Document Icon" or Other Content, the Condition = True – Go To Excel Service Number List, and show “Y” in the NAA Column, for that Loaded Service Number.

On the Web Page, Click “New Search” and Load the next Service Number from the Excel Service Number List and Repeat the Evaluation Process for each Service Number, till the end of the Excel Service Number List is exhausted.
When the Service Number List is completed, Save the Excel File to Desktop/ROTWEB/Service Number NAA Test. (ROTWEB is a Folder on my Desktop)

I hope you are able to follow my thoughts - if not please contact me, either with a Reply Post or by my Email ( ocar23@iinet.net.au )
Best Regards
Owen Carlton
( www.ratsoftobruktribute.com )

You ask for quite a bit. The webpage is dynamic, so you would need to run an automated browser, navigate to the search page, enter the service number, press "search" and see if there is any item indicating a digital copy.

One way to do this is in Python, a free programming language.
Find Python here and a "howto install" tutorial here.

Make sure you install Python with pip, since it lets you install additional free modules very easily. To install additional packages you simply need to type cmd in the search field in windows. This shows you the command line interface (a black window). When you have Python with pip installed, you can simply type pip install XYZ (press enter), to install the package XYZ (where XYZ is the name of the package).

You also need to download the Selenium Chrome driver (the version must match the version number of you current Chrome browser). This free software allows Python to automatically use Chrome to browser pages. The Chrome driver is a single file, which must be copied to the place where you have Python on your computer.

The script below does exactly what you want. It searches a list of "numbers" from an excel file one by one on the NAA page. For each request, the script looks for digital documents and "notes" the number of found digital documents. Finally, the results are saved to an excel file. My dummy result would look like:

     servnr  digicopy
0      1234         2
1      6516        15
2     51651         2
3  51651651         0

Maybe you have a friend or someone in your family who can help you install Python. Usually this is a little daunting for people with no programming experiance. If you elaborate my script below, you could even download the digital documents or extract information (e.g. by using OCR software like tesseract).

Basic code for the webcrawler:

# Import required additional packages (to be installed via pip first)
from bs4 import BeautifulSoup
from urllib.request import urlopen
import re, time, datetime, json, requests
import pandas as pd
# Chrome Webdriver https://chromedriver.chromium.org/downloads
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By

# When you see something like "ModuleNotFoundError: No module named 'openpyxl'"
# You need to install the pip package, e.g. pip install openpyxl
# Install pip packages in the console window (type cmd in the search field in Windows bottom left, open it, type pip...)

# Load Excel containing service numbers
# Change the path to the file is needed
df = pd.read_excel('D:/service_numbers.xlsx')

# Start chrome driver https://chromedriver.chromium.org/downloads
options = webdriver.ChromeOptions()
#options.add_argument('headless') # in background option
options.add_argument('log-level=3')
browser = webdriver.Chrome(chrome_options=options)

# Set up empty data frame to store results
results = pd.DataFrame([])

# Loop over each number (aka row) in the excel file (column name 'snr')
for s in df['snr']:
    browser.get("https://www.naa.gov.au/")
    time.sleep(0.5)
    # Go to record search
    browser.find_element_by_xpath("//*[@id=\"block-naa-mainpagecontent\"]/div/div[4]/div[2]/div/div/div/div/a").click()
    time.sleep(0.5)

    # Find searchfield and send number "s", press enter
    print("Loading number %s" %s)
    searchfield = browser.find_element_by_xpath("//*[@id=\"ContentPlaceHolderSNR_tbxKeyword\"]") # find field
    searchfield.send_keys(s) # enter searchterm (the service number)
    searchfield.send_keys('\n') # press enter
    time.sleep(0.5)

    # Load the content of the current page (I do not iterate over all resultrs but only look at the first page here)
    soup = BeautifulSoup(browser.page_source, "html.parser")
    table = soup.find("table", id="ContentPlaceHolderSNR_tblItemDetails")
    # You can also load/print the entire table, you can even go directly to the digital contant (if present)
    #print(table.text)

    # See how many "images" indicating digital content are there
    imgs = soup.find_all('img', {'class':'digital_copy'})
    try:
        numberofdigiitems = len(imgs)
    except:
        numberofdigiitems = 0
    print("There are %s digital copys" %numberofdigiitems)

    # Append result to data frame
    results = results.append(pd.DataFrame({'servnr': s, 'digicopy': numberofdigiitems}, index=[0]), ignore_index=True)

print(results.head())

# Save to excel
results.to_excel("D:/myresults.xlsx", index=False)


Can features negatively correlated with the target be used? In feature selection (for a regression problem), can features that are negatively correlated with the target variable be chosen to predict the target?
I don't think negative correlation means the predictor does not provide any information about the target.
Some feature selection methods (like Filter method) are based on using only those predictors that have high correlation to the target variable, and dropping those with low correlation.
My question is - shouldn't negative correlated features also be considered? I think the problem of feature selection should be whether a feature is "simply correlate " with the target or not, rather than whether it is a Positive/Negative correlation. Am I right? Can someone please clear my confusion?

Negative correlation is not the same as low correlation.
If variables $x$ and $y$ have a correlation value of $c$, then $-x$ and $y$ will have a correlation of $-c$.
When people talk about "low correlation" they usually mean correlation that is close to 0.
In the context of machine learning, negative correlation is just as good as positive correlation; do your filtering based on the absolute value of the correlation.


Canonical VS Graph Isomorphism I'm having a having a hard time understanding the difference between an isomorphism in graphs and canonical graphs. I have read through the Wikipedia articles, but it still isn't clicking.

Can somebody explain the difference, perhaps with an example?

this might take a few iterations, but perhaps it can be moved to chat if it gets too long.  are you wondering about a specific application or just the general concept?  as far as i know, the canonical form of something is the unique/simplest arrangement/representation of that something.  as for isomorphism (one to one relationship, a function gets you from one set to the other), you can oversimplify it and think about factors of numbers.

for a moment consider an analogy - instead of graphs, using sets of numbers.  this is randomly off the top of my head so hopefully there aren't too many holes in it.  we have

$A = 6, 10, 14$

and

$B = 18, 30, 42$

let's call $A$ and $B$ isomorphic because to get between any value in $A$ to the corresponding value in $B$, you multiply or divide by 3.  now let's introduce another isomorphic set:

$C = 3, 5, 7$

it looks like $A$, $B$, and $C$ are all isomorphic because there is a one to one relationship between any of their initial values.  the same can be said for any of their middle values, or any of their ending values.  while it's great that they're all isomorphic, we'll consider something additional about $C$.  i'm going to call $C$ canonical because if you remove the greatest common factor from $A$, or you remove the greatest common factor from $B$, you get $C$.  there is no common factor we can remove to reduce $C$ itself.

extending this, you can see that a canonical representation of a graph is an isomorphic version of that graph, but not necessarily the other way around.  all isomorphic versions of said graph will reduce to the same canonical form.

EDIT:

that being said, the definition of canonical is more complicated for graphs.  first, for graph isomorphism, this page about a code named bliss presents some small graphs as an example.  they define $G_{1}$ as

and $G_{2}$ as

and offer two transformations for $G_{1}$ that will make $G_{1}$ and $G_{2}$ isomorphic.  first, they consider $\{\langle 1\to 2, 2\to 4, 3\to 1, 4\to 3\rangle\}$, and then $\{\langle 1\to 2, 2\to 4, 3\to 3, 4\to 1\rangle\}$ - both of these transformations swap vertices.  the transformations result in the following isomorphic graphs:

each pair of connected vertices has a corresponding pair in $G_{2}$ ($2\to 1,3,4; 1\to 4; 3\to 4$).

now, to discuss canonical graphs, we must identify a cost function with which we can measure what is the 'most canonical' form.  this paper mentions metrics such as area, minimum angle, and total number of bends.  for simplicity, i'll try to minimize bends and favor angles that are closer to $90^{\circ}$.

if you use a coordinate transformation, you can see that this simple graph

and this graph

are isomorphic.  the same edges connect the same pairs of vertices, and if you move vertices from one around, you can make the other.  this graph

is also isomorphic with the first two, but since it has the optimal amount of bent edges (zero) and optimal angles (four right angles), it is also the canonical representation for these graphs.

for further reading, you might consider this paper, which discusses several algorithms used to find canonical representations for web mining, or this paper, which discusses canonical representations of binary decision diagrams, word-level decision diagrams, binary moment diagrams, taylor expansion diagrams, and finite field decision diagrams.

tl;dr:

just have a look at the minion with bananas on slide 10 of this presentation.