What do the hidden layers of a neural network try to learn? Typically all the layers of an artificial neural network are trainable. But what are the hidden layers trying to learn?

Each hidden layer represents a nonlinear application of a function on the inputs of the previous layer.

The first layers perform a function on our data-inputs and the last layer is responsible for either using softmax for classification purposes or be an affine layer for regression.

The in-between layers are necessary in order to stack complexity and therefore avoid having to create a layer of very high complexity.

In fact, in theory, two level deep neural network can represent any function. But how complex would this function need to be?
A layer of a very high complexity would be able to map our raw inputs into our final classification or regression task.

However what would be simpler to do is to let the intermediate hidden layers learn useful features regarding the given task and then the next hidden layer could exploit these features in its own way. And then the next hidden layer could use these newly transformed features in its own way and so on. So stacking nonlinear functions one on top of the other is more clever than requiring a single nonlinear function to do all the hard work.

There is a limit on how many layers you can stack together for the main reason that these hidden layers get trained by backpropagation which uses the chain rule of derivation in order to pass the deltas of the error back to the weights of the layers and tune them but a long series of these operations could end-up with exploding or vanishing gradients which means really poor performance or a model that cannot be trained.


Finding linear transformation under which distance matrices are similar I have $n$ sets of vectors, where each set $S_i$ contains $k$ vectors in $\mathbb{R}^d$. I know there is some unknown linear transformation $W$ under which the distance matrix $D_i$ (a $k\times k$ matrix) is approximately "the same" (i.e. has a low variance) among all sets $S_i$.

To illustrate, it might be the case that for each vector in each $S_i$, the first $\frac{k}2$ numbers are random noise, and the latter $\frac{k}2$ are the same for each $i$. In this case, $W$ would recover the last $\frac{k}2$ elements from each vector. But in practice the structure of the vectors may be more complex than in this toy example.

Is there a method to learn $W$ - either using direct linear algebra methods, or through learning (e.g. by optimizing a specific loss on the distance matrices), without finding the naive solution of $W$ being the zero mapping, or other mapping that maps all vectors to the same vector?

EDIT:

One possible approach I was considering is to apply a variant of Canonical correlation analysis (CCA) on the distance matrices. CCA projects two "views" of the same data to a shared subspace that maximizes the total correlation between the views; is there a variant of CCA in which the views are not pairs of vectors, but rather sets of vectors that each has some internal structure (as expressed by the distance matrices)? while in CCA we maximize the correlation between the pairs, here I'd like to maximize the correlation between the distance matrices of the different sets.

Thanks

I think what you are searching is related to metric learning. PCA is an unsupervised learning method but there are equivalent method in which you can introduce label information to learn the best linear transformation that maximize the distance between data from different labels.
http://contrib.scikit-learn.org/metric-learn/introduction.html#:~:text=Distance%20metric%20learning%20(or%20simply,%2C%20clustering%2C%20information%20retrieval).


Is there any way to explicitly measure the complexity of a Machine Learning Model in Python I'm interested in model debuggin and one of the points that it recommends is to compare your model with a "less complex" one in order to see if the performance is substantially better on the most complex model.
It raises my questions:
Suppose you have a Ensemble model and a Linear model for a classification task
"It seems natural to think that the ensemble model is more complex than the linear model"

But, what would it be a model agnostic way to numerically measure the model's complexity in order to be able to compare two or more models in such terms?

Is there any python implementation that can help with such a task?

I have not heard of any model agnostic way to measure model complexity. There are several strategies but they are model dependant.
You can tackle the problem using different families of models.

For linear models you can count the number of nonzero parameters that is using. Number of features used for the prediction.

For decision tree you can count the maximum depth that the tree achieves.

For Neural Networks you can count the number of parameters that your NN is optimizing.

For ensemble methods (random forest, gradient boosting) you can use an aggregation of the different weak learners used in the model.

For the python implementation there are several implementations depending on for which model you want to measure it. Some of them if you notice are really easy to measure.
Its intuitively hard to compare complexity between different model families. What is more complex a linear regression with 4 coefficients or a decision tree with max_depth=3?
On the topic of deep learning complexity, Hinton, Oriol, Jeff Dean published a paper Distilling the knowledge of a Neural Network. Where they talk about simplifying the complexity of a Neural Network.


Help with deep learning for motorbike inspection First of all, I am very new in machine learning and data science, so I am really sorry if my question is completely stupid.

I am doing an internship in machine vision, and people of my office want me to implement a Deep Learning model to inspect a motorbike after being fully assembled. Basically, they want to inspect different parts of the motorbike, and detect if there is a defect or not. However, before doing it in real life, they want to use a miniature of a motorbike in order to study the viability of this project.

This being said, I was thinking about creating one model for each region to be inspected, programing a robot with a camera to take pictures of these regions, and letting the DL models evaluate them. In addition, the DL models would be used for One-Class Classification to detect if the region inspected is OK or Not OK only by analyzing images labeled OK.

However, there are several problems that I will list as follows:

1 – We don’t have (any) data (yes, I told them it is stupid to try DL without any data);

2 – In internet, I could not find dataset of motorbike, much less specifically of the regions to be inspected;

3 – They want to inspect a specific model of motorbike, so they are asking me to do something very very specific, and I suppose that even if I find a dataset of the specific regions of different motorbikes, it will be kind of useless.

Finally, with all the conditions and problems that I mentioned above, it seems that it is impossible to do what they want, but I would like to ask you about it before giving up, because as I said, I am very new in this subject and I might be wrong. Could you give me your opinion/advice about everything I mentioned here?

Thank you very much!!

Data

If you are falling short of data, you obviously need to create a custom dataset i.e by labelling each part of a motorcycle. If you can collect some images, we can use a pre-trained classifier along with image augmentation technique. You can Google's Dataset Search.

To annotate your dataset, you can find tools here and here. If you need to download a number of images from Google Image Search, try this tool.

Model Architecture

Now, regarding the model, we can use any object localization algorithms to detect various parts of a motorcycle. Next, these individual parts would be fed to different models which will classify the respective faults. For detection, we can use

Faster RCNN and Mask RCNN ( semantic segmentation )
YOLO ( Lower inference time )


how to determine percentage below which a feature can be removed from a model Let feature $feat$ contain one value $A$ that occurs 5% of the time, while 95% of the time it is empty.

Instead of arbitrary saying features that have less than 5% should not be included into the model, is there a way to quantify mathematically this percentage?

Why do people say that?

when you have that much missing values (95%) than that could be encoded as special value, and it will look quantitatively as a value in itself. Now when you start using this feature as "usable" what will most likely happen is that you get noise, i.e. conflicting information/non-discriminatory one. In other words this feature will be saying the same information for all the classes (since there is 95% of it, its most likely). Hence you can do this encoding, fit a random forest and look at gini coefficients (for example!) and you quantify mathematically how important this feature is relative to the problem and the dataset.

If you are looking for quantifying mathematically this percentage generally, thats impossible since its all relative and problem specific. From experience 50%+ missing values is extremely hard to work with.

CAVEAT you dont even have to encode it as value in itself, just fit lgbm model, there missing values will be assigned with values that minimises the loss, and you may even find out that its not that useless after all.


Naives Bayes Text Classifier Confidence Score I am experimenting with building a text classifier using Naive Bayes which has been pretty successful on my test data. One thing i am looking to incorporate is handling text that does not fit into any predefined category that I trained the model on.
Does anyone have some thoughts on how to do this? I was thinking of trying to calculate the confidence score for each document, and if < 80 % confidence, for example, it should label the data as "N/A"
This is my code so far:
df_train = pd.read_csv(__________)

text_clf = Pipeline([('vect', CountVectorizer()),
                  ('tfidf', TfidfTransformer()),
                  ('clf', MultinomialNB()),])
text_clf = text_clf.fit(df_train.text, df_train.label)

df['predicted'] = predicted

Like I said, it works well for documents that do fit into one of the categories, but if I have something that clearly does not fit into anything, it will still try and assign it a label, my guess is based on some kind of confidence calculation but just not sure how that works

First there is a simple theoretical reason why relying on the probability provided by a NB model is not a good idea: what NB predicts is the posterior probability, i.e. the conditional probability $p(C_i|d)$ for every class $C_i$ for a given document $d$. The class $C$ which is predicted is just the one which obtains the highest probability $p(C_i|d)$. The sum of all the posterior probabilities for a particular $d$ is 1 since it's a conditional, and this means that a class is always predicted relatively to the other classes: a high probability for $C$ doesn't mean that the document is very likely to be $C$ in general, it only means that it is much more likely to be $C$ than any of the other classes. If it actually belongs to another class unknown to the model, there's no way the model can represent this.
There is also a practical reason why this is not a great idea: experimentally it is known that NB tends to systematically assign extreme probabilities (and to overfit easily but that's not the point). It does this whether or not the class is correct, and this can easily be checked with your data: if you plot the predicted probability for every instance with a different colour for correct/incorrect prediction, you're likely to see that even the incorrect instances were predicted with a very high probability. It's tempting to interpret the predicted probability as a confidence measure, but it's usually a mistake.
But there is hope: the proper way to deal with potential unknown "classes" (these are not classes technically) is not with regular classification methods, which always rely on a "closed world" assumption (only the classes seen during training exist). There are also open-set classification methods: these are much less common but they are not restricted to the known classes. I think the most common method is probably one-class classification, in particular with one-class SVM.


Finding an appropriate price using thousands of data points If i have a lot of data points describing the price of a used car. How would I find the market value of the car (assuming that the price points in the data set are the only determinant used, and the basis of determination will be the frequency [higher the frequency the better] price data point occurence for that particular car).

A count of absolute value recurrences will not work, as I want to bucket numbers that are similar (less than 5% difference of total price) into one category.

It is not clear from your question if you are trying to predict value for many used cars as a function of different vehicle characteristics (like age of car, model, mileage) or predicting price of a single car.

Assuming the former (prediction using variables), you can set it up as a regression problem, where price is dependent variable and (age, model, mileage) are independent variables. Google "Boston housing price prediction" which is a similar application in different domain.

If you have only prices data (ignoring vehicle characteristics), you are looking for summary statistics. You can start with the mean (average) or median price. You can then it little more complex by modelling the price as some reasonable statistical distribution (normal or Poisson) and then take typical summary statistics on the distribution as estimate (say inter-quantile range).


Why use the .idx data format? The MNIST handwritten digit dataset uses a file format .idx. What are the advantages of this file format over alternatives such as CSV, TSX and ODS?

Generally, you will find datasets being distributed in CSV format for their simplicity and human readable format that you could ingest in any programming language with just the packages that the language is shipped with. Usually, tabular data is exported in CSV format and that is one of the reasons why MNIST dataset is not provided in CSV format.

Here is the quote from LeCun's website for storing the dataset in idx format.

  The data is stored in a very simple file format designed for storing
  vectors and multidimensional matrices.

In terms of performance, binary file formats fairs better compared to text file formats like CSV or rich text format like ODS.

Following are some of the binary file formats that are widely used.

Avro format
Parquet format
Optimized Row Columnar (ORC)
Protocol Buffers (protobuf)

These file formats support data compression, stores data type metadata to serialize and de-serialize data effeciently.


Train building classifier for imagerial data I am using the https://www.arcgis.com/ api for accessing imagery of aerial data.

I would like to train a model that can caputure on the imagery if the provided street address has a house, pool, garage etc.?

Any suggestions how to start such a project? Any recommendations for similar implementations?

Looking forward to your replies!

This blog article might be a good starting point. From what you described and depending on your data, semantic segmentation might be overkill and classification will suffice.

Either way, the first step will be to get your hands on training data. If you do not have labels already, this might mean that you have to sit down for a while an label a bunch of images. This is much easier for classification than segmentation, but the information you end up with is not as rich.

For segmentation you should then check out U-Net or DeepLabv3. Classification is not my strong site, but VGG or a ResNet should do the trick.


Similar students using Machine Learning I have a student performance data, where I have marks of various subjects for the students and I want to find similar students with good marks in a particular subject using machine learning.

How do I go about it, Collaborative filtering?

The end Goal is to recommend jobs based on a student's top performance in a particular Subject.

Suggestions are open. I have attached the head of my data.

First off, I'm not sure that you even need machine learning for this. You probably know what features you are looking for in every specific job (which grades are most crucial). Won't it be easier to define a simple threshold per job?

For example, for job X, you need to have a grade above 80 in course A, above 90 in course A and so on. This can give you the list of all suitable candidates per job (but it won't tell you which job is the best fit per student).

A different approach would be to define the most important courses for each job (preferably weighted, because some are more important than others), than for every student calculate his score for every potential job (weighted MSE or MAE). This will give you the best fit for every student. You will than need to use some sorting algorithm to place all the students in the best fitted jobs overall (I assume that not all students can get their best fit as jobs don't have infinite positions).

For example, for student A and job X, the the score will be:

You will probably still need to use some threshold per job, since having a job being to best fit for the student, doesn't necessary mean that he meets its criteria.


Solution-

df.loc['05-26-2019']

Borrowed from-

https://stackoverflow.com/questions/36871188/how-to-access-pandas-dataframe-datetime-index-using-strings


Maximum Likelihood estimation Given a sample $ X_1,X_2 \dots X_{100}$ and the density function   $ f(x;\theta) = \frac{1}{\pi \cdot \left(1+\left(x-\theta \right)^2\right)}$ , find an approximate solution for  $\hat{\theta}_{MLE.}$
My attempt:
I have found the joint likelihood $L(\theta;x_1,x_2\dots x_{100}) = \prod _{i=1}^{100}\left(\frac{1}{\pi \cdot \left(1+\left(x_i-\theta \right)^2\right)}\right)\:$
$l$ = $\log(L) = -100*\ln(\pi)-\sum^{100}_{i=1}(\ln(1+(x-\theta)^2)$.
I'm not sure of this step
$\frac{\partial }{\partial \theta}\left(\log(L)\right) = \sum_{i=1}^{100}(\frac{2(x_i-\theta)}{1+(x_i-\theta)^2}$
then I used Newton's method to find the maxima.
this is the script I used to calculate the maxima
#deravitive of log(L).
fun1 <- function(theta){
  y1 <- 0
  for(i in 1:length(x)){
    y1 <- y1 + (2*(theta-x[i]))/(1+(x[i]-theta)^2)
  }
  return(y1)
}

#derivative of fun1.
fun1.tag <- function(theta){
  y <- 0
  for(i in 1:length(x)){
    y <- 2*(theta^2+(x[i]^2)-20*x[i]-1)/((1+(x[i]-theta)^2)^2)
  }
  return(y)
}

# The Newton's method.

guess <- function(theta_guess){
  theta2 <- theta_guess - fun1(theta_guess)/fun1.tag(theta_guess)
  return(theta2)
}
theta1 <- median(data$x)
epsilon <- 1

theta_before <- 0

while(epsilon >0.0001){
  theta1 <- guess(theta1)
  epsilon <- (theta_before- theta1)^2
  theta_before <- theta1
}

What I got was $\hat{\theta}_{MLE} = 5.166$
I'm now trying to plot the data(in my case x) and check if $\hat{\theta}_{MLE} = 5.166$ is actually a maxima.

You have a typo in your formula
#derivative of fun1.
fun1.tag <- function(theta){
  y <- 0
  for(i in 1:length(x)){
    y <- y + 2*(theta^2+(x[i]^2)-20*x[i]-1)/((1+(x[i]-theta)^2)^2)
  }
  return(y)
}

There is y +  missing inside the loop.


What causes the Shape (convex or non-convex) of the cost function? There are different types of cost functions like Cross-entropy, absolute error, mean squared error.

When can we expect the cost function to be non-convex? Does this depend upon the type of cost function we choose? Or does this depend upon the Model we choose like when we have higher order polynomial Model?

The cost function is convex if its Second Order Derivative is positive semidefinite (i.e. $\geq0$ ).

But this definition depends on the function with respect to which you take the derivative. This convexity changes when we are talking about Neural Networks, as in that case, our derivatives are taken with respect to the weights.

Refer: Why Cross Entropy is convex, but not necessarily in neural nets and Why RMSE is convex but not necessarily in neural nets


In which cases should Genetic Programming be preferred over Artificial Neural Network trained with Genetic Algorithm I am trying to understand Genetic Programming (GP) but I cannot think of any context where GP can be chosen over training Artificial Neural Networks with genetic algorithms.

What problems each of them is suitable for?
In what context we choose one over the other (if it is not personal preference)?

One key benefit of genetic programming is that it can generate models that are capable of extrapolation. Neural nets tend to be really good at interpolating, but fall apart when trying to extrapolate.
As an example to highlight this we can think of data fitting.
We can start by using just a very simple example. Let the input data be x={1,2,3} and y={1,4,9}.
In genetic programming the proper choice here would be to use symbolic regression, which evolves an algebraic expression to fit data. Given this training data it would likely evolve the expression $y=x^2$. As you can see this expression is capable of taking new values outside of the training data range and mapping it to a response outside of the training responses. For example, you could supply as input 10 and it would use the evolved expression to give a predicted value of 100.
Alternatively, a neural net is very good at interpolating since it creates a network of connections to connect inputs to responses, but when given new data, it tries to fit the new data into its set of rules built to fit only the training data. So likely, the trained neural net would be able to perfectly map $x\mapsto y$, but if you were to give it the number 10 as input, the closest training value it saw was 3, so it would likely give a predicted response of 9 instead of 100.


Loss given Activation Function and Probability Model Can someone please explain this one?

  For output units a good trick is to obtain the output non-linearity
  and the loss by considering the associated negative log-likelihood and
  choosing an appropriate (conditional) output probability model,
  usually in the exponential family. For example, one can typically take
  squared error and linear outputs to correspond to a Gaussian output
  model, cross-entropy and sigmoids to correspond to a binomial output
  model, and − log output[target class] with softmax outputs to
  correspond to multinomial output variables.

At the output we can have an activation that takes a weighted linear sum from the previous layer and transforms it into a new range. Here are a few activations you could use:

$x = x$, the identity activation which is what is usually used at the end for regression models

$\sigma(x) = \frac{1}{1+e^{-x}}$ which maps to $(0, 1)$

$exp(x) = e^x$ which maps to $(0,\infty)$

These activations are differentiable which means backpropagation will keep working. Now that we can manipulate our output to turn into ranges that we want, we can use them to use them for probability distributions. In the $\sigma$ case we can use them directly as the probability at event $Y$ as opposed to not $Y$, so $P(Y|X)$. We can also use multiple output nodes to parameterize a probability distribution. For example let's try to approximate a Gaussian distribution conditioned on our input features $x$. Instead of regressing directly on $y$ we can regress $\mu$ and $\sigma$ that define the Gaussian distribution. For $\mu$ we can use the identity activation because $\mu$ is not constrained, $\sigma$ needs to be strictly positive so we could use $e^x$ for that.

However, in our training data, we don't have $\mu$ and $\sigma$, we only have a $y$, a sample from this probability distribution. What we do know is that given a specific $\mu$ and $\sigma$ what the likelihood is of that $y$. A value of 100 is extremely unlikely from a normal distribution with $\mu = 20$ and $\sigma=20$, but a value of $20$ has a much higher density. The density of a Gaussian is $f(y|\mu,\sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(y-\mu)^2}{2\sigma^2}}$. This means we can check the likelihood of a $y$ given our current estimate of $\mu$ and $\sigma$ given $x$ and try and make the likelihood higher. Or minimize the negative log-likelihood, which has less numerical issues as opposed to normal likelihood.


What is representation learning? I am reading the Chapter-1 of the Deep Learning book, where the following appears:

  A wheel has a geometric shape, but its image may be complicated by
  shadows falling on the wheel, the sun glaring off the metal parts of
  the wheel, the fender of the car or an object in the foreground
  obscuring part of the wheel, and so on.

  One solution to this problem is to use ML to discover not only the
  mapping from representation to output but also the representation to
  itself. This approach is called representation learning.

Here, I did not understand the exact definition of representation learning. I have referred to the wikipedia page and also Quora, but no one was explaining it clearly. The lack of explanation with a proper example is lacking too.

The authors cite the autoencoder as an example for a representation learning algorithm.

  An autoencoder is the combination of an encoder function, which
  converts the input data into a different representation, and a decoder
  function, which converts the new representation back into the original
  format

Can someone explain in simpler terms what exactly representation learning is?

[Would be much better if explained with an example. The car wheel example would do too.]

If you look at older machine learning algorithms, they rely on the input being a feature and learn a classifier, regressor, etc on top of that. Most of these features were hand crafted, meaning, they were designed by humans. Classical examples of features in computer vision include Harris, SIFT, LBP, etc .

The problem with these is that they were designed by humans based on heuristics. Images can be represented using these features and ML algorithms can be applied on top of that. However, they may not be the most optimal in terms of the objective function, i.e, it may be possible to design better features that can lead to lower objective function values. Instead of hand crafting these image representations, we can learn them. That is known as representation learning. We can have a neural network which takes the image as an input and outputs a vector, which is the feature representation of the image. This is the representation learner. This be followed by another neural network that acts as the classifier, regressor, etc .

So, in the wheel example, you can try to manually describe how a wheel should look like and how it can be represented. Say, it should be circular, be black in colour, have treads, etc. But these are all hand crafted features and may not generalize to all situations. For example, if you look at the wheel from a different angle, it might be oval in shape. Or the lighting may cause it to have lighter and darker patches. These kinds of variations are hard to account for manually. Instead, we can let the representation learning neural network learn them from data by giving it several positive and negative examples of a wheel and training it end to end.

Hope this makes sense. Let me know if I can clarify any point I made.


What is the role of the bias? When talking about artificial neurons, inputs, weights and biases, I understand the role of each but the latter.

In short if we have a neuron such as sigmoid(sum(w*x) + b) I get that the weights basically say which of the inputs is more "important", but what about the bias? I've read it in this other question "is as means how far off our predictions are from real values."

But how can this be true, if we start them "at random"? Also this isn`t supposed be be the job of the loss/cost function?

Bias simply takes care of variables which are unaccounted for i.e you did not include it in X. All Machine Learning and Neural Nets do is approximate a function. All functions can be represented as a series of exponentials using Fourier Transform which basically means a function can be expressed as a sum of many other special functions.

In this series there is a constant term to adjust the elevation of the curve above the x-axis. This constant can be thought of as the bias. Thus it basically takes care of the fact that when all variables are 0, a function need not be 0.


issue with early-stopping on f1 score with imbalanced data I have a highly imbalanced dataset with less than 0.5% of the minor class. Using Keras, I'm training DNN on the training set and evaluate performance on validation set. Loss function is binary_crossentropy.

I'm setting my early-stopping on f1 score, instead of validation loss. What I observe during training is f1 score fluctuate wildly up and down while validation loss is decreasing. I actually end up with a very low f1 score with early-stopping, although f1 score was many epochs back...

I'm confused about it. Am I supposed to do early-stopping on the performance metrics? Should we alway use validation loss for early-stopping criterion? thanks.

F1 is based on hard classification; if the probability scores are hovering near the threshold, then the classifications may be flopping a lot, leading to unstable F1 scores.

A low F1 score is not too surprising in the presence of such imbalance; the default cutoff of 0.5 will often lead to high recall but low precision.


Machine learning learn to work well on future data distribution? This is based on my limited machine learning scope and experience, so correct me if I'm wrong. Many of the currently used machine learning models (SVMs, boosted trees, DNNs) work under the assumption that the training, validation and test data sets share the same distribution. They can work to some extent if the distributions differ but not by a lot. Here "can work" means that they work sub-optimally (i.e. can work better if the distributions are the same), not that their theory behind is supposed to deal w/ the distribution difference and can handle them like "nailing it".

Hence my question: is there work on predicting based on the assumption that the data sets are actually moving through a series of distribution changes? A crazy thought would be to observe the distribution difference between training and validation sets, and assume that the same diff will exist between validation and test sets and learn to predict well on test set. This will work great on time series where the nature of the data might change over time.

It has been studied under various names likes Domain Adaptation,   Sample Selection Bias, Co-variate Shift.

Please go trough this survey paper on Transfer learning. It covers all the possible combination like

1)same distribution for train and test data

2) Gradual change between train and test distribution

3) Different but related distribution for train and test

It'll also give you all the necessary resources required to study further on this topic.


Least Squares Regression $Ax=b$ when $A$ is fixed and $b$ is varied The typical setting for least squares regression (or over-determined linear system) for $Ax=b$ is to solve $x$ given $A$ and $b$.
In other words, $A$ and $b$ are fixed when we solve the problem.

My question is that is there any application that $b$ is changed by users (i.e., $b$ is interpreted as a query) while $A$ is fixed?

In least squares regression, the problem that you solve is:

$X^T X \beta = X^T y$

where $\beta$ are the parameters, $X$ the regressors and $y$ the predicted variable. If you want, in your notation $A$ to be constant and $b$ to change, you need to have constant regressors and change the predicted variable $y$. I don't think there are applications that the regressors are not changed but the predictors do change, but if you want to find one it will probably be in online learning.


Why normalize when all features are on the same scale? So I'm doing the tensorflow tutorial found here:

https://www.tensorflow.org/tutorials/keras/basic_classification

Basically, my input is a [28x28] matrix (image) that I flatten to a [1x784] vector.

The tutorial then says:

  We scale these values to a range of 0 to 1 before feeding to the neural network model. For this, cast the datatype of the image components from an integer to a float, and divide by 255.

My question is why do we need to normalize in this case? My understanding is that when we have features that are on different scales, we need normalization if not the output of the model is distorted. But in this case all pixel ranges go from 0 to 255 (all features are the same scale)

I went ahead and ran it with normalization, and get an accuracy of over 85%, whereas no normalization, my accuracy falls to 10%.

Any ideas?

For neural networks, there is another reason. Sigmoid function provides values between 0 and 1; if the task is binary classification, you'd use a sigmoid function at the output. For another task, you might have used tanh function at some layer, a centered input to that neuron works computationally well.

The reason is as follows; outside the range of some input values, the derivatives of the activation functions will be close to zero. At those points, the gradient descent steps will be extremely slow due to the small weight updates. Directly think of the 2-D function graph of the related activation function and think of the range of input values where the derivatives are close to zero. This is why Rectified Linear Unit (ReLU) or Leaky Rectified Linear Unit (Leaky ReLU) outperforms the others in most of the tasks, with the constant derivative of 1, when the input is greater than 1.

Also refer to: https://stats.stackexchange.com/questions/51012/must-i-normalize-inputs-into-a-perceptron-that-uses-a-sigmoid-activation-functio


How exactly dependent variable is expressed in terms of independent variables using Partial Least Square Regression Method? I understand the working of NIPALS algorithm but while doing the regression using PLS how exactly the relation between known and unknown is established using Principle Component Analysis.
The idea is clear up to the points both dependent and independent are expressed in form of their principle components. After I am not able to understand.

PS: I have read text and some papers about it but they are very difficult to understand. This Question may sound a trivial one, but please bear with me. Correct me if I am wrong.

This lecture helped me get my head around PCA: https://www.youtube.com/watch?v=a9jdQGybYmE

The analogy that I am sharing with you comes directly from this lecture:

Imagine that you have a weight that is on a spring. It can bounce up and down and thus is a one degree of freedom system. Imagine that you have three sensors which record the (x,y) position of the weight as it bounces up and down. Now you have six vectors of data (a set of x,y from each camera). PCA tries to infer the actual (x,y) coordinates of the weight as it bounces based only on the data that you get from the cameras.

This analogy is very rough and I highly recommend that you watch the linked video. There is really no short answer to your question!


Why is data science not yet widely applied to Law? Law (judiciary) contains such a huge corpus to apply NLP to, but yet there are only search engines designed for Law. Why is NLP not yet extensively applied? Is it because of dimensionality?

Welcome to the site and thanks for the great question! I recently led an NLP project that dealt with a lot of laws. While I have to obfuscate my actual work, here's a general view:

The laws themselves may not be the best source data. It would take a massively transformed recordset in order to make most laws actionable for modeling. I'm talking about big rooms, full of lawyers providing an annotated version of laws in order to create a recordset that can actually be useful
The above assumes that the laws have been digitized in some easy to digest format. That may not always be the case. In a lot of instances, you are referring back to classic OCR approaches as part of  your data prep and I don't know anyone that likes working with OCR :-)
The human-in-the-loop requirements are very high. So you have an algorithm, now what? That's not something you can just put out on Mechanical Turk for the layman to verify. You need more lawyers to help with the verification of your approach and correct mistakes that are happening
Finally, you must get very sophisticated with your embedding layers in how you create and apply them. That's not an easy thing to do and very processor intensive - a GPU is highly recommended and not a lot of grassroot efforts are going to have this processing power

Good luck!


Is it possible to use TensorFlow inside a python script in Azure Machine Learning Studio? I'm trying to get TensorFlow running inside a python script in Azure Machine Learning Studio. As TensorFlow is not part of Azure Machine Learning Studio, I needed to import it using a zip file. I followed the instructions here: https://stackoverflow.com/questions/44593469/how-can-certain-python-libraries-be-imported-in-azure-mllike-the-line-import-hu

However, when trying to import TensorFlow, I get:

ImportError: No module named _pywrap_tensorflow_internal
Failed to load the native TensorFlow runtime.

It seems like TensorFlow is much more than just a python library. It seems like it needs a native library to work. Sounds logically, but there's this video on YouTube showing that it should work after all: https://www.youtube.com/watch?v=SEvoZ6EnUpk

My question is: is it possible to run TensorFlow inside a python script in Azure Machine Learning Studio? If so, how?

Let me answer my own question: yes it is possible. You simple open this example in ML Studio and you'll get a working tensorflow_keras.zip into your workspace: https://gallery.azure.ai/Experiment/CNN-on-Keras-with-Tensorflow-backend


Fully-Connected DNN: Compute the numbers of free parameter in a DNN A fully-connected DNN has layer sizes of 3-3-4-2, where the first layer size represents the input layer. We assume that all layers are affine ones (no ReLU). Give the dimensions of all weight matrices and all bias vectors in the network and compute the total number of free parameters in this DNN.
According to this task the first layer size represents the input layer, so it must be 3.
a(0) = x = 3

If I know the input layer, so 3-4-2 are also the sizes of the bias vectors.
I know now the dimension of the input layer and the bias vector. The columns of W must be also 3, because of the size of the input layer. The rows of W must be equal to the size of the bias vector.
So I computed the following dimensions:
W(1) = W33, b3, a(1) = 3

W(2) = W43, b4, a(2) = 4

W(3) = W24, b2, a(3) = 2

But how to compute the total number of free parameters in this DNN?

Weights at first layer will depend on the Dimension of the input too.
Let's assume, This is your Network. Input has 5 Features

$\hspace{6cm}$ Image credit - ML Visuals by dair.ai
Per layer weights and biases
\begin{array} {|r|r|}
\hline
 &Weights &Biases\\
\hline
Layer-1 &5*3 = 15 &3\\
\hline
Layer-2 &3*3 = 9 &3\\
\hline
Layer-3 &3*4 = 12 &4\\
\hline
Layer-4 &4*2 = 8 &2\\
\hline
\end{array}
Check with Code
import tensorflow as tf
from tensorflow import keras
model = keras.Sequential()

model.add(keras.layers.Dense( 3, activation="linear",input_shape=(5,)))
model.add(keras.layers.Dense( 3, activation="linear"))
model.add(keras.layers.Dense( 4, activation="linear"))
model.add(keras.layers.Dense( 2, activation="linear"))

model.summary()
#model.get_weights() #Can check weights matrix with this

Output


Reasons and prevention of trivial (and less trivial) misclassification errors? I was not sure about posting this question with mentioning the name of the company, which I quite respect and admire. However, I've figured that a wider exposure might help the team to fix this and similar problems faster as well as increase the quality of the machine learning (ML) engine of their website.

The problem exposes itself by too many occurrences of a quite trivial misclassification error on Amazon's book categories classification (which I'm a frequent visitor of). In the following example, the underlying reason of such behavior is quite clear, but in other cases the reasons might be different. I am curious about what could be other potential reasons for misclassification and what are the strategies/approaches to avoiding such problems. Without much further ado, here's how the problem appears in real life.

I was reviewing some books, related to transitioning from graduate programs (Ph.D., in particular) to work environment in academia. Among several other books, I ran across the following one:

So far, so good. However, let's scroll down a bit further to see the the books ratings in relevant categories. We should expect Amazon to figure out categories, relevant to the book's discipline, topic and contents. How surprised was I (and that's an understatement!) to see the following result of Amazon.com's sophisticated ML engine and algorithms:

Clearly, the only fuzzy fact that connects this book with the subject "Audiology and Speech Pathology" (!) is IMHO the author's last name (Boice), which, is close to the word "voice". If my guess is correct, Amazon's ML engine, for some reason, decided to take into account the book's lexicographical attribute instead of the book's most important and most relevant attributes, such as title, topic and contents. I've seen multiple occurrences of similar absolutely incorrect ML-based decision making on Amazon.com and some other websites. So, hopefully my question makes sense as well as interesting and important enough to spark a discussion: What could be other potential reasons for misclassification and what are the strategies/approaches to avoiding such problems? (Any related thoughts will also be appreciated.)

By looking at the ratings of the book, it looks like they are doing some sort of hierarchical clustering.
First of all it's a book, then a text book, then in medicine & health science.

This is a pretty difficult problem because:

You don't know how many and which topics you have, and the topics are constantly changing.
Since this is a clustering problem (or a semi-supervised problem in the better case), a closed feedback loop for these errors cannot be applied. Clustering is always more difficult than a supervised learning problem.

Apparently they're doing a pretty good job in general. This book in particular is a tough one...


Clustering uncertain data with independent uncertainty per dimension I have $n$ objects located in a $d$ dimensional space, however I do not know their exact coordinates. For each object and each dimension, I have a set of noisy measurements of the coordinate.

I would like to cluster this data in the

For example, with $n = 3$ and $d = 2$, I could have access to the following data:

object $a$, dim 1: 0.8 0.7 0.6
object $a$, dim 2: 1.0 1.0 1.0 0.9
object $b$, dim 1: 0.4 0.3
object $b$, dim 2: 0.2 0.1
object $c$, dim 1: 0.9 0.6
object $c$, dim 2: $\emptyset$

In my current approach, I take the for each (object, dimension) pair that has data the average value, and I imput the average value of the dimension for the missing data.

So I would get

$a$ [0.7 0.975]
$b$ [0.35 0.15]
$c$ [0.75 0.6] (where $0.6 = \frac{1+1+1+0.9+0.2+0.1}{6}$)

Then I use the scikit-learn python library to run the mean-shift algorithm and obtain clusters.

I am not completely satisfied with this methods for two main reasons (maybe they're the same):

an (object, dimension) pair with a single observation is treated as as reliable as one with numerous observations
there is a discontinuity between how a pair with zero observations and a pair with some observations are treated. In the second case, the value of the other objects does not influence the attribute at all.

My questions are: What would be a more principled approach to this problem? If I need to use another algorithm, are there open source decent quality libraries available that would implement it?

I currently use $d = 256$ and $n = 10$ for my testing, but I aim to use $d = 16000$ and $n = 1000$ (but maybe a smaller $n$ if it's unrealistic) in my target application.

Model the measurement process

For each dimension you could attempt to model the distribution of the measurement process variance. Perhaps every dimension has the same measurement process or perhaps it is distinct. Combine all measurements from all dimensions if the process is the same.

If you have enough data then you can build an empirical model of the distribution of errors using perhaps a kernel based estimator.

Otherwise you might choose to use a Gaussian distribution formed from the mean value of each measurement and an overall measure of the variance of the measurement process. Then you replace observations you have with a sample drawn from a  distribution with the observation mean and overall measurement process variance.


How to handle the target variable being in the features I am trying to predict the winner of a tennis match from the players participating and their respective ranks.

Labels of my data:

Winner
Loser
Winner rank
Loser rank

The problem is that the columns holding the player names in my data are labeled 'Winner' and 'Loser'. What is the best course of action to render this dataset usable for machine learning?

Could I just assign the 'Winner' column/vector as my target variable, and construct two new columns ('Player1', 'Player2') and populate them with random choices from 'Winner' and 'Loser'?

I think you need to formalize your problem a little bit.

If I were to predict the winner of a tennis match, I would do something like this. I will be predicting the probability of Player 1 winning (or Player 2, without loss of generality). Then, this is a simple classification problem.

Then, for the features, I will organize it in a way that uses player 1 or player 2 to represent the features.

For example, if the variables are: players-current-ranking and face-to-face-win-rate. Then the feature vector will be something like:

player1_current_rank: 10
player1_2_face_to_face_win_rate: 0.7
player2_current_rank: 20
player2_1_face_to_face_win_rate: 0.3

You can never have target variable in your feature, otherwise it won't make any sense predicting it.


how to train tensorflow chat application on updated dataset everyday I have created chatbot on Cornell movie dataset and it's working fine. I have trained chatbot application up to global step 330000. I am using tensorflow library. I am using ||source|| chatbot. I am also receiving output fine.  Then I created this new dataset file by modifying original dataset and i wish to train chatbot application with updated files. Now should I delete previously saved checkpoints and saved data and start training from zero or should i train from 330000 onwards without worrying changes in dataset.
           I wish to modify database daily basis and train on modified database.
Please kindly let me know if you have a suggestion for me as I am stuck on this issue. I will really appreciate if you help me with this issue.

The distribution of your data won't change too much over a smaller period of time, retraining from scratch seems like a waste. Just add the new data to your set and do one or more epochs on all your data, not just the new ones. It can readjust the weights a little bit for changes in the distribution which is exactly what you want. Make sure you are not overreliant on early stopping as regularization method against overfitting because you continuously train like this, use some other regularizer. I would suggest periodically to retrain from scratch but this doesn't have to happen very regularly, maybe once every two months.


Xor gate accuracy improvement I am pretty fresh in machine learning so probably I made rookie mistake somewhere but I tried a lot last few days and I can't find a way to further improve my network. Hope you guys can help!

So I was asked to make "perceptron Xor gate" in Tensorflow with data that I generated myself. Since single perceptron can't solve it I used MLP. My generated train and test data looks like this:

# TRAIN SET

number_of_elements = 20000

top_right, top_left, bottom_right, bottom_left = [np.random.rand(number_of_elements, 2) for _ in range(4)]
top_left[:, 0] *= -1
bottom_right[1] *= -1
bottom_left * -1

X = np.concatenate((top_right, top_left, bottom_right, bottom_left))

top_right_y, bottom_left_y = np.zeros((2, number_of_elements))
top_left_y, bottom_right_y = np.ones((2, number_of_elements))

y = np.concatenate((top_right_y, top_left_y, bottom_right_y, bottom_left_y))
y = np.reshape(y, [y.shape[0], 1])

# TEST SET

number_of_test_elements = 40000

first_el_random = random.randint(number_of_test_elements / 2, number_of_elements * 2)
second_el_random = random.randint(number_of_test_elements / 2, number_of_elements * 2)
third_el_random = random.randint(number_of_test_elements / 2, number_of_elements * 2)
forth_el_random = random.randint(number_of_test_elements / 2, number_of_elements * 2)

top_right = np.random.rand(first_el_random, 2)
top_left = np.random.rand(second_el_random, 2)
bottom_right = np.random.rand(third_el_random, 2)
bottom_left = np.random.rand(forth_el_random, 2)
top_left[:, 0] *= -1
bottom_right[1] *= -1
bottom_left * -1
X_test_v1 = np.concatenate((bottom_right, bottom_left, top_left, top_right))

top_right_y = np.zeros(first_el_random)
top_left_y = np.ones(second_el_random)
bottom_right_y = np.ones(third_el_random)
bottom_left_y = np.zeros(forth_el_random)

y_test_v1 = np.concatenate((bottom_right_y, bottom_left_y, top_left_y, top_right_y))
y_test_v1 = np.reshape(y_test_v1, [y_test_v1.shape[0], 1])

Then I made two implementations, one in Keras and one in Tensorflow. They look almost the same so I will just show Keras code:

# KERAS MODEL

model = Sequential()
model.add(Dense(64, input_dim=2, activation='tanh', kernel_initializer='normal'))
model.add(Dropout(0.2))
model.add(Dense(32, activation='tanh', kernel_initializer='normal', kernel_constraint=maxnorm(3)))
model.add(Dropout(0.2))
model.add(Dense(16, activation='tanh', kernel_initializer='normal', kernel_constraint=maxnorm(3)))
model.add(Dense(1, activation='sigmoid', kernel_initializer='normal'))

optimize = Adam()
learning_rate_reduction = ReduceLROnPlateau(monitor='val_binary_accuracy', patience=10, verbose=2, factor=0.5, min_lr=0.00001)

model.compile(loss='binary_crossentropy', optimizer=optimize, metrics=['binary_accuracy'])
model.fit(X, y, batch_size=200, shuffle=True, epochs=100, verbose=2, validation_split=0.33,
          callbacks=[learning_rate_reduction])

preds = model.predict_classes(X_test_v1)

real_accuracy = np.mean(np.equal(y_test_v1, preds))
print('Test classification accuracy: %.4f' % real_accuracy)

Model accuracy is around 75%, I tried to make it higher by:
- increasing train data size
- adding additional dense layers, removing layers, adding dropout, removing dropout, adding neurons, removing neurons etc
- changing activation functions, only one that I didn't change is sigmoid on outer layer, when I mess with that my results don't make sense
- messing with different optimizers, learning rates, callbacks. I didn't touch binary_crossentropy as loss function tho.
- changing bath sizes, epochs, setting shuffle to False

The problem seems pretty simple in general but I just can't pinpoint what am I doing wrong. If someone can tell me what am I doing wrong I would be really grateful. Thanks!

After carefully reviewing my code I found a bug in my train and test set.

Bug was in this line:

bottom_right[1] *= -1

should be:

bottom_right[:, 1] *= -1

After that my tensorflow accuracy went up to 99% and my Keras accuracy to 89%.

Anyway if someone has better solution or see something I am doing wrong please let me know :)


partial fitting, how to ensure one hot captures all features consistently Doing some data science on ~4 million samples, with lots of columns being categorical.

One column has ~1000 categories and my boss insists on including it in the analysis.

My output is also predicting classes (I'll use gnb.predict_proba())

So, I'm taking a random subset of my data for partial fitting, and repeating.

# train = ~3 million rows of data as a dataframe
gnb = naive_bayes.GaussianNB()
for i in range(10):
    dds = train.sample(n=10**4)
    (dfX,dfY) = makeXY(dds) #gets one-hot- encoded X and Y dataframes
    gnb.partial_fit(dfX,[getClass(x) for x in dfY.values],classes=np.unique([getClass(x) for x in dfY.values]))

How can I ensure I get all the possible classes AND that they are in the same order every time?

If you're using sklearn, this is a great use of CountVectorizer as a workaround since you can specify a vocabulary.

To start, get a list of all the 1000 categories and set that as the vocabulary in the transformer. Then convert the column to a string data type and apply this transformer to each batch:

from sklearn.feature_extraction.text import CountVectorizer

cv = CountVectorizer(vocabulary = pre_made_category_list)
endoded_variable_matrix = cv.fit_transform(dfX[categorical_column])

Even though it's technically a count vectorizor, since there is only one word in the string, the counts for each row will be 1 for the category it is and 0 for everything else, so its effectively one hot encoding the variable. The order of the columns will be the order of the vocabulary, so the matrix will be consistent between folds.


Guidelines for selecting an optimizer for training neural networks I have been using neural networks for a while now. However, one thing that I constantly struggle with is the selection of an optimizer for training the network (using backprop). What I usually do is just start with one (e.g. standard SGD) and then try other others pretty much randomly. I was wondering if there's a better (and less random) approach to finding a good optimizer, e.g. from this list:

SGD (with or without momentum)
AdaDelta
AdaGrad
RMSProp
Adam

In particular, I am interested if there's some theoretical justification for picking one over another given the training data has some property, e.g. it being sparse. I would also imagine that some optimizers work better than others in specific domains, e.g. when training convolutional networks vs. feed-forward networks or classification vs. regression.

If any of you have developed some strategy and/or intuition on how you pick optimizers, I'd be greatly interested in hearing it. Furthermore, if there's some work that provides theoretical justification for picking one over another, that would be even better.

AdaGrad penalizes the learning rate too harshly for parameters which are frequently updated and gives more learning rate to sparse parameters, parameters that are not updated as frequently. In several problems often the most critical information is present in the data that is not as frequent but sparse. So if the problem you are working on deals with sparse data such as tf-idf, etc. Adagrad can be useful.

AdaDelta, RMSProp almost works on similar lines with the only difference in Adadelta you don't require an initial learning rate constant to start with.

Adam combines the good properties of Adadelta and RMSprop and hence tend to do better for most of the problems.

Stochastic gradient descent is very basic and is seldom used now. One problem is with the global learning rate associated with the same. Hence it doesn't work well when the parameters are in different scales since a low learning rate will make the learning slow while a large learning rate might lead to oscillations. Also Stochastic gradient descent generally has a hard time escaping the saddle points. Adagrad, Adadelta, RMSprop, and ADAM generally handle saddle points better. SGD with momentum renders some speed to the optimization and also helps escape local minima better.


Sort by average votes/ratings I have a data set that's a dictionary of tuples. Each key represents an ID number and each tuple is (yesvotes, totalvotes). Example:

{17: (6, 10), 18: (1, 1), 21: (0, 2), 26: (1, 1), 27: (3, 4), 13: (2, 2)}

I need to find the max key of the set. I want to assign weights so, for instance, key 17 would be ranked higher than key 18 because even though the ratio is much smaller, it has ten times the total votes.

Is there an optimal way to do this? My best guess is simply calculate new ratios by (yesvotes/totalvotes)*(totalvotes+1) but that doesn't seem right... Is there some kind of standardized field of study concerning fair-voting?

Yes, this is a well-studied problem: rank aggregation. Here is a solution with code.

The problem is that the quantity you are trying to estimate, the "score" of the item, is subject to noise. The fewer votes you have the greater the noise. Therefore you want to consider the variance of your estimates when ranking them.


Orange wont allow me to set target for corpus I would like to do a linear regression on an excel file. I have descriptions in one column and category in the other.
I need to be able to set the target to do a linear regression. However, Orange does not allow me to set target variable the way it would when I use the file module. The problem with opening the data as file is that I will not be able to use bag of words (since thats only for corpus).
When I do use Corpus, it automatically detects all columns as meta.
Am I doing something wrong? Is there a work around?

Can't you do something like this? (or am I missing something?)


chart x-axis spacing terminology question In the following hand made charts I show some value for years. In the first chart I've evenly spaced each year. On the second chart I've spaced them relativelly to their actual year value within time (i.e 2016 is closer to 2017 than 2010).

Is there a terminology for the spacing of the second chart? Imagine building a software which would have a toggle control to switch the view from A to B. How would you call it?

It's usually called "Tick" and as it's for x axis, let's say "xtick". You can label it as xticks and give it two values "evenly spacing" and "actual spacing" (This is just an idea as there is no standard terminology as far as I know).


Classifier on top of LDA topic vectors? I have training data in form of pair of documents with an associated label - {doc1, doc2, label}. Label is defined as function of pair of documents.

Now I want to build a model which can predict the label given two new documents.

I want to try different representation of document (instead of common ones say TF-IDF). Can I use vectors (topic distribution) from LDA as features for a classifier?

Yes, that is a reasonable approach. Also try neural network based representations such as doc2vec. I suppose you know how to do the classification part?


Sklearn StratifiedKFold code explanation While going through the following blog I came across the following code snippet

from sklearn.cross_validation import StratifiedKFold
eval_size = 0.10
kf = StratifiedKFold(y,round(1./eval_size))
train_indices, valid_indices = next(iter(kf))
X_train, y_train = X[train_indices], y[train_indices]
X_valid, y_valid = X[valid_indices], y[valid_indices]

and I am unable to understand how it works. Can anyone please help me with an explanation? Thanks

A KFold split will take the data and split it however many times you designate.   StratifiedKFold is used in order to ensure that your training and validation datasets each contain the same percentage of classes (see sklearn documentation for more). The function StratifiedKFold takes two arguments, the array of labels (for binary classification this would be an array of 1's and 0's) and the number of folds.  They have designated the number of folds as 1./eval_size where eval_size = 0.10.  So this is a 10-fold validation.

train_indices, valid_indices = next(iter(kf))

This line derives the indices in order to split the data into the train/validation data sets.

Now that we have the indices, we use those indices to actually split the data.

It is interesting that they wrote next(iter(kf)) and then input the indices into the dataset when they could use sklearn.cross_validation.train_test_split.  train_test_split is just a wrapper for next(iter(kf)), but it is more readable and it's already a function in sklearn.

Hope this helps!


Is it possible to change pandas column data type within a sklearn pipeline? Sklearn pipeline I am using has multiple transformers but one of the initial transformers returns numerical type and the consecutive one takes object type variables.

Basically I need squeeze in a:

data[col] = data[col].astype(object)

for the required columns within the pipeline.

Is there any way to do it?

Note: I am using Feature-engine transformers.

You sure can.

One solution off the bat is to extend your estimator that
 takes object type variables. So what does that mean.

Library that you said are all estimators in the sklearn form fit, predict methodology. So all you have to do is something as follows:

> class modifiedTraf(oldTraf):
>        def __init__(self):
>           super(modifiedTraf, self).__init__():
>
> bla bla
>
>       def fit(self, X):
>           X[col] = X[col].astype(object)
>           super(modifiedTraf,self).fit(X)
>           return self
>       def transform(self, X):
>           X[col] = X[col].astype(object)
>           X=super(modifiedTraf,self).transform(X)
>           return X

Learn more about overriding sklearn classes and super argument for example here


Orange 3 Heatmap clustering under the hood I have recently used the heatmap widget in Orange 3. All the documentation says is "Clustering (clusters data by similarity)". Is this using hierarchical or k-means or some other type of clustering?

On that note, is there a way to look at the code being run by all the widgets to see whats going on under the hood? It would be nice if after you finish the workflow you would get a file with the script run to perform the analysis.

It appears the widget uses hierarchical clustering. I guess the metric is Euclidean distance by default and there doesn't seem to be a way to specify another one (except by using Distances widget and connecting it into the Distance Map widget).

I don't think it is possible to export the widget's workflow as pure code, but you can look at what the widget does in the source code (seems pretty low-level, though). What you can do, however, is select subsets of data (can be saved with Save Data widget) for further analysis if that's of any help.


Can someone clarify what the linear assumption of PCA is? 0
For the past few hours I've been trying to search what this linear assumption is. Some of the articles states that that your independent variables have to be linear in relationship and need some type of transformation if there is no linearity. Other articles state that your data has to be linearly separable. Which is it? Is it both?
Does it mean that that you first have to check if the independent variables are linear in relationship, then after applying PCA, check if the data is linearly separable?
OR
Check if the data, before applying PCA, is linearly separable with techniques like linear programming.
Then there is KERNEL PCA which after searching states that it is an extension of PCA where it is applied to nonlinear data. Does that mean nonlinear in relationship or linear inseparable?

PCA is the best (in the mean-squared error sense) linear decomposition method.

PCA is defined as an orthogonal linear transformation that transforms
the data to a new coordinate system such that the greatest variance by
some scalar projection of the data comes to lie on the first
coordinate (called the first principal component), the second greatest
variance on the second coordinate, and so on.
Wikipedia

The term "Linear" in PCA means:
a. That any data point is simply a linear combination of the principal components.
b. That the data matrix ($A$) can be decomposed via linear similarity transformations to diagonal matrix ($\Sigma$).
Ie
$$A = U \Sigma U^T$$
or
$$AU = U \Sigma$$
$\Sigma$ is the diagonal matrix of variances for each basis vector.
One can see at once that the linear algebra of the above formula makes clear the meaning of Linearity in PCA.
On the other hand decomposition methods like ICA (Independent Component Analysis) cannot be expressed via linear algebra as PCA above, since they require not only decorrelated components but independent components which is a stronger condition requiring non-linearities.
See also: https://datascience.stackexchange.com/a/80361/100269


Newbie: Objective Function I am reading the book "Data Science for Business" by Foster Provost & Tom Fawcett. Only a fourth of the way through.

I am unclear about the concept of Objective Function. I will nevertheless take a stab and would like to seek feedback.

Say that we decided to use standard linear regression for a given problem (assume we have training data). Is the goal of objective function to determine the best parameters (weights) for this training data?

Thanks in advance!

The objective function is the mathematical equation which combines the parameters in a useful sense. For linear regression this is a simple linear combination of weighted features.

You will need to choose which objective function is best suited for your problem. Then based on the function you selected you will know what are the model parameters that will need to be fit using an appropriate training method.

One of the skills of the data scientist is to identify the objective function which has the best likelihood of success given a problem statement.


Which scaling size is better? [0,1] or [-1,1] for LSTM? I see some scale their data between 0 and 1 and some others do that between -1 and 1. But which one is better? Or better to ask: How to make a true/good decision for that?

I think both scalings can potentially behave well. The important matter is to put all the features in the same range, otherwise some features will add up more in the first linear transformation of your layer (remember that any neural network is a linear transformation and a non-linear one), which is undesirable, and will hinder the learning.

Best option is then to try both, and take the one that gives best performance.


Categorical data into numeric in excel I have a large dataset and I would like to convert these categorical data into numeric in binary form to perform k means clustering in R. However, I get an error in value. This is the formula that I have entered into the excel sheet: =LEFT(B2,FIND("  ",B2,1)-1).

This is my dataset.

Why don't you import the data into R and use some built-in function for encoding? createDummyFeatures from mlr package is a function, which can do one-hot encoding of categorical variables.

https://www.rdocumentation.org/packages/mlr/versions/2.13/topics/createDummyFeatures


How to scale up algorithm development? In working on exploratory data analysis, and developing algorithms, I find that most of my time is spent in a cycle of visualize, write some code, run on small dataset, repeat.   The data I have tends to be computer vision/sensor fusion type stuff, and algorithms are vision-heavy (for example object detection and tracking, etc), and the off the shelf algorithms don't work in this context.  I find that this takes a lot of iterations (for example, to dial in the type of algorithm or tune the parameters in the algorithm, or to get a visualization right) and also the run times even on a small dataset are quite long, so all together it takes a while.

How can the algorithm development itself be sped up and made more scalable?

Some specific challenges:

How can the number of iterations be reduced?  (Esp. when what kind of algorithm, let alone the specifics of it, does not seem to be easily foreseeable without trying different versions and examining their behavior)

How to run on bigger datasets during development?  (Often going from small to large dataset is when a bunch of new behavior and new issues is seen)

How can algorithm parameters be tuned faster?

How to apply machine learning type tools to algorithm development itself?  (For example, instead of writing the algorithm by hand, write some simple building blocks and combine them in a way learned from the problem, etc)

First off, if your data has as many variations (in function of time, context, and others) as to make it hard to apply a single strategy to cope with it, you may be interested in doing a prior temporal/contextual/... characterization of the dataset. Characterizing data, i.e., extracting information about how the volume or specifics of the content varies according to some criteria, usually provides with a better understanding (more consise and precise) than simply inferring algorithms on a brute-force fashion.

So, answering each question:

characterization is definitely a means of reducing the number of iterations while trying to select proper algorithms for specific data;
if you have a discrete set of criterias on which your data varies, it becomes much easier to scale up solutions, as will know what information you'd gain/lose if simpler/specific solutions were applied;
after a characterization, you should be also easier to select parameters, since you'd know what kind of specific data you'd be dealing with;
finally, you may use data mining/machine learning algorithms to support this characterization. This includes using:

clustering algorithms, to reduce the dimensionality of data;
classification algorithms, to help deciding on specific properties the data in function of time/context/... may present;
association rules, to predict particular knowledge from the dataset, while also improving/fine-graining the data used for later analysis;
and other possible strategies and analyses.

And here is a list of some criterias on which to analyse data, which you may find helpful.


Improving model prediction for small single data sets by using multiple data sets to fit Have a project I'm working on and am running into an issue.
Essentially I have lots of data sets with a small amount of data. These data sets represent locations in an area. I have one classification data set (y) with categorical values (number from 1 - 6), the location is in the middle of the rest of the data sets.
The issue with these points is that I don't get a lot of data per point (maybe 100 rows).
My goal was to fit a data set (physically closest to the target classification data set) and generalize it to the rest of data. It doesn't work very well, so now I'm trying to improve the model performance by adding 'k' data sets also close to the class data set.
Initially :
model.fit(X_1,y_class)
model.predict(X_2)

These data sets all share the same features and share the same rows as the classification data set.
With basic sklearn models I'm unable to take multiple inputs to fit, and one input to predict.
Essentially what I'm trying to do is the following :

model.fit([X_1,X_2,...X_k],y)

model.predict(X_5)

Where :
All features are numeric (floats)
Each X matrix is about 100 points long with a continuous index [0:100].
I only have one test point (with 100 observations) for each group of points, so it's imperative I use as much data close to the test point as possible to improve the model with the goal is to generalize the data to future points.
Is there another model or technique I can use for this? I've done a bit more research into NN models (not familiar so would prefer to avoid), and found that Keras has the ability to take multiple inputs to fit using their functions API, but can I predict with only one input after it has been fitted to multiple?
https://stackoverflow.com/questions/55233377/keras-sequential-model-with-multiple-inputs

One option is merge nearby data points, this is call binning or bucketing. Binning increases the amount data for a specific target and sometimes increases the signal-to-noise ratio. Binning works very will location based data.


to extract multiple of n images using open cv in python I am using opencv module in python to extract sequences of images(frames) from video (from a file)
My code for extracting images from video goes like this.

import cv2
import os
os.chdir('D:/Python images/videofolder')
cap = cv2.VideoCapture('D:/Python images/test_videos/testcutvideo1.avi')
cap.isOpened()
success,image = cap.read()
count = 1
success = True
while success:
     success,image = cap.read()
     cv2.imwrite("image%d.jpg" % count, image)     # save frame as JPEG file
count += 1

I want my code to select multiple of 'n' images
eg: to select & save multiple of 5 images(to extract multiple of 5) so that I have 5th, 10th, 15th... images.
Please guide me.

You can use the modulo operator to identify multiples of count. Whenever it is a multiple of 5 save the image, otherwise just disregard it and move to the next iteration of the while loop.

Make sure the counter is outside the if statement. It will count every iteration. That is how we will determine if the current image is a multiple of 5.

import cv2
import os
os.chdir('D:/Python images/videofolder')
cap = cv2.VideoCapture('D:/Python images/test_videos/testcutvideo1.avi')
cap.isOpened()

count = 0
success = True
while success:
    success,image = cap.read()
    if count % 5 == 0:
        cv2.imwrite("image%d.jpg" % count, image) # save frame as JPEG file
    count += 1


Interactive Graphing while logging data I'm looking to graph and interactively explore live/continuously measured data. There are quite a few options out there, with plot.ly being the most user-friendly. Plot.ly has a fantastic and easy to use UI (easily scalable, pannable, easily zoomable/fit to screen), but cannot handle the large sets of data I'm collecting. Does anyone know of any alternatives?

I have MATLAB, but don't have enough licenses to simultaneously run this and do development at the same time. I know that LabVIEW would be a great option, but it is currently cost-prohibitive.

Thanks in advance!

For this answer, I have assumed that you prefer open source solutions to big data visualization. This assumption is based on budgetary details from your question. However, there is one exclusion to this - below I will add a reference to one commercial product, which I believe might be beneficial in your case (provided that you could afford that). I also assume that browser-based solutions are acceptable (I would even prefer them, unless you have specific contradictory requirements).

Naturally, the first candidate as a solution to your problem I would consider D3.js JavaScript library: http://d3js.org. However, despite flexibility and other benefits, I think that this solution is too low-level.

Therefore, I would recommend you to take a look at the following open source projects for big data visualization, which are powerful and flexible enough, but operate at a higher level of abstraction (some of them are based on D3.js foundation and sometimes are referred to as D3.js visualization stack).

Bokeh - Python-based interactive visualization library, which supports big data and streaming data: http://bokeh.pydata.org
Flot - JavaScript-based interactive visualization library, focused on jQuery: http://www.flotcharts.org
NodeBox - unique rapid data visualization system (not browser-based, but multi-language and multi-platform), based on generative design and visual functional programming: https://www.nodebox.net
Processing - complete software development system with its own programming language, libraries, plug-ins, etc., oriented to visual content: https://www.processing.org (allows executing Processing programs in a browser via http://processingjs.org)
Crossfilter - JavaScript-based interactive visualization library for big data by Square (very fast visualization of large multivariate data sets): http://square.github.io/crossfilter
bigvis - an R package for big data exploratory analysis (not a visualization library per se, but could be useful to process large data sets /aggregating, smoothing/ prior to visualization, using various R graphics options): https://github.com/hadley/bigvis
prefuse - Java-based interactive visualization library: http://prefuse.org
Lumify - big data integration, analysis and visualization platform (interesting feature: supports Semantic Web): http://lumify.io

Separately, I'd like to mention two open source big data analysis and visualization projects, focused on graph/network data (with some support for streaming data of that type): Cytoscape and Gephi. If you are interested in some other, more specific (maps support, etc.) or commercial (basic free tiers), projects and products, please see this awesome compilation, which I thoroughly curated to come up with the main list above and analyzed: http://blog.profitbricks.com/39-data-visualization-tools-for-big-data.

Finally, as I promised in the beginning, Zoomdata - a commercial product, which I thought you might want to take a look at: http://www.zoomdata.com. The reason I made an exclusion for it from my open source software compilation is due to its built-in support for big data platforms. In particular, Zoomdata provides data connectors for Cloudera Impala, Amazon Redshift, MongoDB, Spark and Hadoop, plus search engines, major database engines and streaming data.

Disclaimer: I have no affiliation with Zoomdata whatsoever - I was just impressed by their range of connectivity options (which might cost you dearly, but that's another aspect of this topic's analysis).


Classifying dates in sentences Let's say I have a sentences that goes like this:

  Hi, how are the kids. I will be going to Los Angeles next Friday and
  will come back the following Monday.

If the date today is October 16 (Wednesday), then next Friday would be October 25th. The sentence later says that the person will come back the following Monday. We know they mean the Monday after next Friday so the 28th of October. If I use the dateutil library, then it would classify both Friday and Monday, but it won't take into account the next before Friday or that we are saying the following Monday.

I was thinking of creating a solution that will split the sentence up into words and then check what comes before actual dates. However, I'd like to capture not only this type of adverb of time, but others as well. For example, next day, this Thursday, 2 days from today, etc. How do I build a framework that will detect these types of date texts and then print out their respective dates?

If you have enough data to train a machine learning model, I believe LSTMs are what you need. Long short-term memory (LSTM) is an artificial recurrent neural network (RNN) architecture used in the field of deep learning.

LSTM networks are well-suited to classifying, processing and making predictions based on time series data, since there can be lags of unknown duration between important events in a time series.

Initially, you can use Pandas LSTM library, you design it from scratch by using Python Tensorflow library. Look at the following links:

Wikipedia

LSTM in python tutorial


What would be the best way to structure and mine this set of data? http://pastebin.com/K0eq8cyZ

I went through each season of "It's Always Sunny in Philadelphia" and determined the character groupings (D=Dennis, F=Frank, C=Charlie, M=Mac, B=Sweet Dee) for each episode. I also starred "winners" for some episodes. How best could I organize this data, in what type of database, and what data science tools would extract the most information out of it?

I was thinking of making an SQL table like so:

             (1)     (2)      (3)     (4)     (5)
Episode# | Dennis | Frank | Charlie | Mac | Sweet Dee
008      |    5   |  3,4  |  2,4    | 2,3 |    1
010      |    5   |  3,4,6|  2,4,6  |2,3,6|    1

...where all the values are arrays of ints. 6 represents that the character won the episode and each number represents one of the 5 characters. Thoughts?

How best could I organize this data, in what type of database?

A simple relational database should do, but you could also use a "fancy" graph database if you want. One table for the users, and one for the "interactions". Each interaction would have foreign key columns for the two participants, labeled winner and loser, and the number of the episode the interaction it occurred.

  Also any ideas on the best way to visually represent this data?

A graphical representation for social network analysis suggests itself. Here are some papers and a subreddit for inspiration. In your case, there is a concept of competition with clear winners/losers, so you could make your graph directed. Have the characters be the nodes, and add directed edges from the winning party to the losing party for each interaction. Collapse repeated interactions, etc. This approach would let you quickly identify overall winners and losers, as well as simply who interacts with whom.


Why does using a standard scalar on my tf idf matrix make it perform better? I have a tf-idf matrix transformed on a list of tweets from a data set i am using. I have a pipeline where i initiate a StandardScalar and then next have my SVM with a linear kernel and auto gamma as the classifier algorithm.
Pretty much as done here in the examples section. With the pipeline the classifier scores an 87 f1 score. Without the pipe it scores a dismal 53.
Why is this?
I thought tf-idf values were already two-fold normalised so shouldn't the standard scalar have no affect as it just performs normalisation again?

I am not sure what you mean by two-fold normalized. TFIDF values are not normalized per se. They will fall within a somewhat constant range of values, but this completely depends on the dataset.
The StandardScaler performs normalization by removing the mean and scaling to unit variance. Since TFIDF values may vary from dataset to dataset, using the StandardScaler will have an effect on the values, which explains the difference in performance.


Aggregating standard deviations Imagine I have a collection of data, let's say the travel time for a road segment.
On this collection I want to calculate the mean and the standard deviation. Nothing hard so far.

Now imagine that instead of having my collection of values for one road segment, I have multiple collections of values that correspond to the multiple sub segments that compose the road segment.

For each of these collections, I know the average and the standard deviation.
From that, I want to aggregate these multiple average and standard deviation in order to get the average and standard deviation for the whole road segment.

For example, let's suppose I have the following dataset :

           subSegmentA , subSegmentB , subSegmentC , subSegmentD
values              20            45            25            70
                    30            55            10            60
                    10            10            10            80
                    15            50            30            75
                    15            40            15            75
                    20            40            20            80
                    30            45            20            65
                    10            40            25            70

average          18.75        40.625        19.375        71.875
stddev      7.90569415   13.47948176   7.288689869   7.039429766

expected_global_average : 150.625
expected_global_stddev  : 18.40758772

For the average there is no problem, a simple sum do the job, but I have trouble with the global_stddev.

I tried multiple solutions from here, without success.

Edit :
After further research, it seems mathematicaly impossible to calculate the standard deviation of a set based only from the standard deviation and average of subsets.

So I am trying to calculate a new metric, that would approximate this global standard deviation.

To do so, I can use in addition to the avg/stddev per subsegments, the length ratio of the subsegment to the road.

It is not perfect but you can try to re-create synthetic data based on the mean and sd. In R, you can use the rnorm function to create a normal distribution from mean and sd. The following is one of the ways to do it. Hope it helps! P.S. I just choose n = 1000 to illustrate how it can be done; you can try using different numbers.

a <- c(20,30,10,15,15,20,30,10)
b <- c(45,55,10,50,40,40,45,40)
c <- c(25,10,10,30,15,20,20,25)
d <- c(70,60,80,75,80,60,65,70)

n <- 1000

e <- c(rnorm(n, mean(a), sd(a)) +
   rnorm(n, mean(b), sd(b)) +
   rnorm(n, mean(c), sd(c)) +
   rnorm(n, mean(d), sd(d)))

mean(e)
sd(e)

Try it online


Validation loss is lower than the training loss I am using autoencoder for anomaly detection in warranty data.

Architecture 1:

The plot shows the training vs validation loss based on Architecture 1.

As we see in the plot, validation loss is lower than the train loss which is totally weird.

Based on the post Validation loss lower than training loss, I understood that it is because of the dropout layer in my model. So I ran the model with dropout layer.

Architecture 2:

Based on the above architecture, I plotted the training vs validation loss. Now validation loss is a bit higher than the training loss.

Clearly it is because of the dropout layer.

Now my question is, whether the model based on Architecture 1 is correct or not? If it is not correct, what kind of changes I could do to make it work?

Thank you ! Any help is much appreciated!!

It is certainly correct in the sense that it is a legitimate neural network. The dropout layer introduces noise that is not injected during the test period. The goal is to combat overfitting so that the error in your test set will be lower due to better generalization. Applying the dropout layer on top of the input layer however throws away a lot of information, meaning it will be difficult to learn a good strategy. Usually dropout layers are applied after the first transformations, which means the neural network has to rely on multiple different transformations that will mean something similar, reducing complexity. However, if you put it over your input layer you are throwing away information that is not accessible in any other way,


Understanding computations of Perceptron and Multi-Layer Perceptrons on Geometric level I am currently watching amazing Deep Learning lecture series from Carnegie Melllon University, but I am having little bit of trouble understanding how Perceptrons and MLP are making their decisions on a geometrical level.
I would really like to understand how to interpret Neural Networks on geometric level, but sadly I am not able to understand how computations of a single Perceptron relate to simple Boolean functions such as OR, AND, or NOT, which are all shown on picture below (e.g. what would be the required value of weights and input in order to model specific decision boundary).

Hopefully, if I were to understand how these computations relate to geometric view shown on picture above, I would be able to understand how MLPs model more complicated decision boundaries, such as the one shown on picture below.

Any help would be appreciated (concrete answer, reading resources, anything at all!). Thanks in advance!

The two pictures you show illustrate how to interprete one perceptron  and a MLP consisting of 3 layers.
Let us discuss the geometry behind one perceptron first, before explaining the image.
We consider a perceptron with $n$ inputs. Thus let $\mathbf{x} \in \mathbb{R}^{n}$ be the input vector, $\mathbf{w} \in \mathbb{R}^{n}$ be the weights, and let $b \in \mathbb{R}$ be the bias. Let us assume that $\mathbf{w} \neq \mathbf{0}$ in all subsequent parts.
By definition, a perceptron is a function
$f(\mathbf{x}) = \begin{cases}
1 & \mathbf{w}^{T} \mathbf{x}+b >0, \\
0 & \text{otherwise.}
\end{cases}$.
Now lets simplify this for a moment and assume that $b = 0$.
The set  $H = \{\mathbf{x} \in \mathbb{R}^n \mid \mathbf{w}^T \mathbf{x} = 0\}$ is called hyperplane, which is a subspace with $\dim(H) = n-1$.
By definition, $H = \mathbf{w}^\perp$, so $H$ is the orthogonal complement of the space $\mathbb{R}\mathbf{w}$.
In simple terms, this means:
For $n = 2$, $H$ has dimension $1$, which is a line that goes through the origin. The line is orthogonal to $\mathbf{w}$. This explains how to get the line, given $\mathbf{w}$ and vice versa. For example, given $\mathbf{w}$, simply draw a line that goes through the origin and is orthogonal to $\mathbf{w}$.
For $n \in \mathbb{N}$, you proceed the same, just that the dimension of $H$ might be higher (for $n=3$ you would need to draw a plane).
In your picture: you see the line in black color.
Note however that the line does not go through the origin. This is handled in the case of $b \neq 0 $.
So let $b \neq 0 $ and let $\mathbf{x}' \in \mathbb{R}^n$ such that $\langle \mathbf{x}',\mathbf{w} \rangle  = -b$.
For any $\mathbf{x} \in H$ we have $\langle \mathbf{x}'+\mathbf{x},\mathbf{w} \rangle  = \langle \mathbf{x}',\mathbf{w} \rangle  + \langle \mathbf{x},\mathbf{w} \rangle  = -b$.
Therefore, $\{\mathbf{x}'+\mathbf{x} \in \mathbb{R}^n \mid \mathbf{x} \in H\} \subset \{\mathbf{x} \in \mathbb{R}^n \mid \mathbf{w}^T \mathbf{x} = -b\}$
Now let $\mathbf{x} \in \{\mathbf{x} \in \mathbb{R}^n \mid \mathbf{w}^T \mathbf{x} = b\}$, then $\mathbf{x} = (\mathbf{x}-\mathbf{x}')+\mathbf{x}'$. Since $\langle \mathbf{x}-\mathbf{x}',\mathbf{w} \rangle  = \langle \mathbf{x},\mathbf{w} \rangle -\langle \mathbf{x}',\mathbf{w} \rangle  = -b+b= 0$, we have
$\{\mathbf{x}'+\mathbf{x} \in \mathbb{R}^n \mid \mathbf{x} \in H\} = \{\mathbf{x} \in \mathbb{R}^n \mid \mathbf{w}^T \mathbf{x} = -b\}$
In simple terms, this means:
The set $\{\mathbf{x} \in \mathbb{R}^n \mid \mathbf{w}^T \mathbf{x} = -b\}=\{\mathbf{x} \in \mathbb{R}^n \mid \mathbf{w}^T \mathbf{x} +b= 0\}$
is nothing but the set $H$ translated by $\mathbf{x}'$.
In particular for $n=2$, the line is translated by $\mathbf{x}'$.
This explains how to describe the line that is depicted in your image.
From the Hesse normal form of the line, you get $\mathbf{w}$ and $b$.
Given $b$ and $\mathbf{w}$, you get $\mathbf{x}'$ by defining  $\mathbf{x}'$ with $\langle \mathbf{x}',\mathbf{w} \rangle  = -b$. Let $i \in \{1,\ldots,n \}$ with $w_{i} \neq 0$. Then $\mathbf{x}' := \mathbf{e}_{i}\lambda$ with  $\lambda = \frac{-b}{w_{i}}$ satisfies $\langle \mathbf{x}',\mathbf{w} \rangle  = -b$, where $\mathbf{e}_{i} \in \mathbb{R}^{n}$ is the vector that is everywhere $0$ except at position $i$, where it has the value $1$.
In simple terms this means, you know how to draw the line given $\mathbf{w}$ and $b$, and vice versa.
Finally, $H^{+} := \{ \mathbf{x} \in \mathbb{R}^{n} \mid \mathbf{w}^T \mathbf{x} +b > 0 \} = \{ \mathbf{x} \in \mathbb{R}^{n} \mid \mathbf{w}^T \mathbf{x}  > -b \}$ is the upper-half space and $H^{-} := \{ \mathbf{x} \in \mathbb{R}^{n} \mid \mathbf{w}^T \mathbf{x} +b \leq  0 \}$ is the lower half-space given by $H$. The purple area in the image is now exactly the upper half-space $H^{+}$ (the area "above" the line), and of course, $f(x) = \begin{cases}
1 & x \in H^{+} \\
0 & \text{otherwise}
\end{cases}$.
Now lets have a look at the upper picture again. It corresponds to three different "trained" perceptrons. The line $H$ seperates the 2D space into two half-spaces. Everything in the purple area gets the value $1$, everything at the opposite gets the value $0$.  Therefore, the perceptron is completely defined by drawing $H$. It defines what value each vector will be assigned to.
Thus, a perceptron is able to represent for example the OR function (left example), as you can seperate $00$, from $01,10$ and $11$.
Meanwhile, the XOR function cannot be represented by a perceptron, as you cannot seperate the points of each class by a line $H$.
Now the bottom picture is about an MLP consisting of 3 layers.
Each neuron in the hidden layer corresponds again to one hyperplane.
Such an MLP thus maintains multiple hyperplanes.
Let us assume we have $k$ neurons in the hidden layer.
Now instead of asking if a vector is within the upper half-space or lower half-space of one hyperplae, a MLP describes the location of a point $\mathbf{x} \in \mathbb{R}^{n}$ with respect to all $k$ hyperplanes.
The output of a node in the last layer (output layer) is computed as $\phi(\sum_{i = 1}^{k}{w_{i}y_{i}}+b')$, where $y_{i}$ is the output of node $i$ of the hidden layer (either 1 or 0, as described before), $\phi$ is some activation function and $w_{i}$ is the corresponding weight.
Let us assume that $w_{i} = 1$ for all $i$ (as in your example image), and let us consider $F:= \sum_{i = 1}^{k}{y_{i}}$ first.
If $F = u$, this means, there are $u$ many nodes in the hidden layer that output $1$, given the input $\mathbf{x}$. Let $l_{1},\ldots,l_{u} \in \{1,\ldots,k \}$ be the indices of these nodes. For each node $i$ of the hidden layer, let $H^{+}_{i}$ be the corresponding upper half-space and $H^{-}_{i}$ be the corresponding lower half-space.
Then, we know that $\mathbf{x} \in H^{+}_{l_{r}}$ for all $r = 1,\ldots,u$ and $\mathbf{x} \in H^{-}_{j}$, for all $j \in \{1,\ldots, k \} \setminus \{l_{1},\ldots,l_{u}\}$.
In simple terms:
If $F =u$, the input $\mathbf{x}$ must be in exactly $u$-many upper-half spaces (and $k-u$-many lower half-spaces).
Now let $\phi$ be again the heavy-side function, thus $\phi(t)=1$ if $t > 0$ and $\phi(t) = 0$ for $t \leq 0$.
Then $\phi(F+b') = 1 \Longleftrightarrow F+b' > 0 \Longleftrightarrow F > b'$.
Therefore, the network will output $1$, if $\mathbf{x}$ is contained in at least $(b'+1)$-many upper half-spaces.
In the example picture, there are 5 hyperplanes and it will output 1, if the ínput vector $\mathbf{x}$ is in the center region.
In simple terms, the MLP uses a finite  arrangement of hyperplanes, see also Stanley.
Each cell (or region) is assigned either to class $0$ or $1$. So the MLP assigns to all vectors within these regions (which are polyhedrons) the same value (either $0$ or $1$).
Now using a different activation function in the hidden layer correspond to using some sort of distance measurement.  With the  perceptron, all points within on cell are assigned the same value. With functions like sigmoid, it would take into accout, how close the vector $\mathbf{x}$ is to the border (the hyperplanes).
Using weights different than $w_{i}=1$, corresponds in grouping different cells together.
Example: Let $n=2$ with $k=3$ hidden nodes, $w_{1} = 1 = w_{2}$ and $w_{3}=-2$. Then $F \in \{-2,-1,0,1,2\}$.
If $F = 0$, then $y_{1} = y_{2} = y_{3}= 0 $ or $y_{1} = y_{2} = y_{3}$.
If $F = 1$, then $y_{3} = 0$ and (either $y_{1} = 1$ or $y_{2} = 1$).
If $F = 2$, then $y_{3} = 0$ and $y_{1} = 1 = y_{2} $.
if $F = -1$, then $y_{3} = 1$ and (either $y_{1} = 1$ or $ y_{2} = 1$).
If $F = -2$, then $y_{3} = 1$, $y_{1} = y_{2} = 0$.
If you set the weight from input to hidden layer to $1$, you will get a representation of XOR.
If you use $b' = 1.5$ you get $\phi(F+b') = 1 \Longleftrightarrow F \geq 2$. Thus $\mathbf{x} \in H^{+}_{1} \cap H^{+}_{2} \cap H^{-}_{3}$  if and only if the MLP will map $\mathbf{x}$ to $1$.
With constant $1$ weights between hidden and outputs layer however,
the MLP will map $\mathbf{x}$ to $1$, if and only if: (1),(2),(3) or (4) holds:
(1): $\mathbf{x} \in H^{+}_{1} \cap H^{+}_{2} \cap H^{-}_{3}$
(2): $\mathbf{x} \in H^{+}_{1} \cap H^{+}_{3} \cap H^{-}_{2}$
(3): $\mathbf{x} \in H^{+}_{2} \cap H^{+}_{3} \cap H^{-}_{1}$
(4): $\mathbf{x} \in H^{+}_{1} \cap H^{+}_{2} \cap H^{+}_{3}$


Find which type of log using machine learning? I am the beginner of Machine learning.
The process is:
I have different log files (System log, MSSQL Server log, Linux log, MySQL Log, FTP log, IIS log).If any input is given, I will find out which type of log using machine learning technique. Each log has a different format. Some logs don't have structure format(Linux, MySQL log, FTP log).
   In my analysis, these are all implemented using KNN algorithm(machine learning).
  But I don't know how to implement this? please give any suggestion about this.

The log type format:

Linux-syslog:

Jan  5 08:39:01 iei-Virtual-Machine CRON[48622]: (root) CMD (  [ -x /usr/lib/php5/maxlifetime ] && [ -x /usr/lib/php5/sessionclean ] && [ -d /var/lib/php5 ] && /usr/lib/php5/sessionclean /var/lib/php5 $(/usr/lib/php5/maxlifetime))

MySQL-error log:

2018-01-05 10:55:20 18856 [Warning] Unsafe statement is written to the binary log using statement format since BINLOG_FORMAT = STATEMENT. Statements writing to a table with an auto-increment column after selecting from another table are unsafe because the order in which rows are retrieved determines what (if any) rows will be written. This order cannot be predicted and may differ between master and the slave. Statement: CALL SubmitGetChangeDetectionInfo(@_SubmitGetChangeDetectionInfo_0

Event log:

IE038,System log,Error,20/11/2017 12:47:51 PM,TerminalServices-Printers,1111,None,Driver HP Deskjet 3520 series required for printer IEC057(Mahendran) Printer is unknown. Contact the administrator to install the driver before you log in again.

FTP log:

2018-01-04 00:00:01 162.254.209.219 INFOEVOL\EC 192.168.0.13 63346
DataChannelOpened - - 0 0 c1df3130-60e6-4678-9dcd-39177cc60d06 -IISAppslog:
2017-11-06 03:25:16 192.168.0.13 GET /IEIAppsLogin.aspx
param=OwjgKJLT+ikfpFnxYbvZS/QWXTFP4GEXmT+qM7TeXTMqi5D7DKexzYjZc3aJNB0x 90 -
182.73.50.19 Mozilla/5.0+(Windows+NT+6.3;+Win64;+x64)+AppleWebKit/537.36+
(KHTML,+like+Gecko)+Chrome/61.0.3163.100+Safari/537.36
http://182.73.50.19/AppsLogin.aspx 302 0 0 234

MSSQL Server:

01/10/2018 07:07:07,Logon,Unknown,Login failed for user 'ms SQL'. Reason: Could not find a login matching the name provided. [CLIENT: 49.65.2.226]

You first need to preprocess your text to convert it into features that can be consumed by a machine learning algorithm. Then you can try this algorithm (a classifier, in this case) with the known examples you have. Then you can use that trained model to recognise logs it hasn't seen before.

If you don't have a preference of language you'd like to use, python with the scikit-learn library would be a good start.

For your problem, work through this tutorial which is very similar to what you're trying to achieve:
http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html


R programming (Jackknife) Hi I would like to ask how to sample out 50 instances from 150 instances of iris data by using Jackknife.
Is it possible?
Thanks in advance

here you go:
http://www.rexamples.com/14/Sample()

sample(x, size, replace = FALSE, prob = NULL)