How to deal with time series which change in seasonality or other patterns? Background
I'm working on a time series data set of energy meter readings. The length of the series varies by meter - for some I have several years, others only a few months, etc. Many display significant seasonality, and often multiple layers - within the day, week, or year.
One of the things I've been working on is clustering of these time series. My work is academic for the moment, and while I'm doing other analysis of the data as well, I have a specific goal to carry out some clustering.
I did some initial work where I calculated various features (percentage used on weekends vs. weekday, percentage used in different time blocks, etc.). I then moved on to looking at using Dynamic Time Warping (DTW) to obtain the distance between different series, and clustering based on the difference values, and I've found several papers related to this.
Question
Will the seasonality in a specific series changing cause my clustering to be incorrect? And if so, how do I deal with it?
My concern is that the distances obtained by DTW could be misleading in the cases where the pattern in a time series has changed. This could lead to incorrect clustering.
In case the above is unclear, consider these examples:
Example 1
A meter has low readings from midnight until 8AM, the readings then increase sharply for the next hour and stay high from 9AM until 5PM, then decrease sharply over the next hour and then stay low from 6PM until midnight. The meter continues this pattern consistently every day for several months, but then changes to a pattern where readings simply stay at a consistent level throughout the day.
Example 2
A meter shows approximately the same amount of energy being consumed each month. After several years, it changes to a pattern where energy usage is higher during the summer months before returning to the usual amount.
Possible Directions

I've wondered whether I can continue to compare whole time series, but split them and consider them as a separate series if the pattern changes considerably. However, to do this I'd need to be able to detect such changes. Also, I just don't know if this is a suitable way or working with the data.
I've also considered splitting the data and considering it as many separate time series. For instance, I could consider every day/meter combination as a separate series. However, I'd then need to do similarly if I wanted to consider the weekly/monthly/yearly patterns. I think this would work, but it's potentially quite onerous and I'd hate to go down this path if there's a better way that I'm missing.

Further Notes
These are things that have come up in comments, or things I've thought of due to comments, which might be relevant. I'm putting them here so people don't have to read through everything to get relevant information.

I'm working in Python, but have rpy for those places where R is more suitable. I'm not necessarily looking for a Python answer though - if someone has a practical answer of what should be done I'm happy to figure out implementation details myself.
I have a lot of working "rough draft" code - I've done some DTW runs, I've done a couple of different types of clustering, etc. I think I largely understand the direction I'm taking, and what I'm really looking for is related to how I process my data before finding distances, running clustering, etc. Given this, I suspect the answer would be the same whether the distances between series are calculated via DTW or a simpler Euclidean Distance (ED).
I have found these papers especially informative on time series and DTW and they may be helpful if some background is needed to the topic area: http://www.cs.ucr.edu/~eamonn/selected_publications.htm

After reading your question, I became curious about the topic of time series clustering and dynamic time warping (DTW). So, I have performed a limited search and came up with basic understanding (for me) and the following set of IMHO relevant references (for you). I hope that you'll find this useful, but keep in mind that I have intentionally skipped research papers, as I was more interested in practical aspects of the topic.
Resources:

UCR Time Series Classification/Clustering: main page, software page and corresponding paper
Time Series Classification and Clustering with Python: a blog post and an ipython notebook
Capital Bikeshare: Time Series Clustering: another blog post
Dynamic Time Warping using rpy and Python: another blog post
Mining Time-series with Trillions of Points: Dynamic Time Warping at Scale: another blog post
Time Series Analysis and Mining in R (to add R to the mix): yet another blog post
And, finally, two tools implementing/supporting DTW, to top it off: R package and Python module


Why does map reduce have a shuffle step? I'm looking at a diagram of map reduce where there is a map step, a shuffle step and then the reduce step. Why shuffle?

The shuffle step occurs to guarantee that the results from mapper which have the same key (of course, they may or may not be from the same mapper) will be send to the same reducer. So, the reducer can further reduce the result set.

for example, your WordCount mappers emit

(hot, 1)
(cold, 1)
(hot, 1)
(rain, 1)
(cold, 1)

the shuffle step will ensure that

(hot, 1), (hot, 1)

get to the same reducer.
So, you get your final answer

(hot, 2)

and the same applied for (cold,1) and (rain,1)


What is a good non cryptographic Hash for string feature translation? What would be a good non cryptographic Hash function to use for converting string features to a numerical representation for feeding into machine learning algorithms?

To explain the scenario my feature set has both categorical data (e.g.: Country) and non categorical data (e.g.: IP Address, Email address). I have used MurMur3 Hash function so far, is there some better algorithm?

See also: Neural Network parse string data?

I do not see a problem with using MurMur3 per se.

For the categorical labels, you can use one-hot encoding / one-of-k encoding.

For the strings, it's an application-specific question.  Presumably if you use exactly those strings as features, it will be very sparse.  The effect of this will depend on the algorithm that you are using, and how the training data compare to the data you see in practice.  You are running the risk that you will effectively either only create a traditional IP/email whitelist/blacklist OR throw out the feature altogether.

You must decide what you want (eg should a certain email address always get a certain output label?) and have some intuition about the application so as to generate more features from IP address and email address.  For example, from email address you can extract the local part (eg "john1972") and domain, and from each of those you can extract:
- length
- character tri-grams
- count/proportion of numbers to alphachars
- number of hyphens
- dictionary validity
...
(From domain you can also extract TLD and possibly subdomains.)
You can try to tokenise .  You can even hit external services to get information like number of Google hits, detected language, spam score etc.


Understanding PyTorch's BCE Notation According to the PyTorch documentation for the Binary Cross Entropy Loss, we can write it as follows:
$$l_{n} = -w_{n}\cdot \left[y_{n}\cdot \log \left(x_{n}\right) + \left( 1-y_{n}\right)\cdot \log\left( 1-x_{n}\right) \right].$$
What do the $w_{n}$, $x_{n}$ and $y_{n}$ mean?

If you look at the documentation of the BCELoss you will the that they mean the following:

$w$ is the weight to use, by which the loss will be rescaled.
$x$ is the output value, which is the value that is predicted by the model.
$y$ is the target value, which is the ground truth value.


Difference between learning_curve and validation_curve What is the difference between these two curves: learning_curve and validation_curve ?

Both curves show the training and validation scores of an estimator on the y-axis.

A learning curve plots the score over varying numbers of training samples, while a validation curve plots the score over a varying hyper parameter.

The learning curve is a tool for finding out if an estimator would benefit from more data, or if the model is too simple (biased).

Above example shows the training curve for a classifier where training and validation scores converge to a low value. This classifier would hardly benefit from adding more training data; a more expressive model may be more appropriate.

The validation curve is a tool for finding good hyper parameter settings. Some hyper parameters (number of neurons in a neural network, maximum tree depth in a decision tree, amount of regularization, etc.) control the complexity of a model. We want the model to be complex enough to capture relevant information in the training data but not too complex to avoid overfitting.

Above example shows the validation curve over a support vector machine's gamma parameter. A too low value of gamma restricts the model too much; both, training and validation scores are very low. A high value of gamma causes overfitting: very good training score but low validation score. The optimal value is somewhere in the middle, where the curves do not diverge too much.

Image source: scikit-learn documentation


Rephrase Neural Network : Find the Fittest Word for Given Meaning/Explanation I am a sophomore student who's interested in deep learning and its method layering up some linear/non-linear operations and constructs up the complex function through the network.

I'd like to comprise a network that finds the word upon the given explanations :

e.g.1 input: a spherically shaped fruit which tastes sour and sweet and gets red when matured well output: apple

e.g.2, input: to make up the boundary between multiple options for the base of one's behavior or decision output: determine.

So basically it's a classification problem but the class might be more than 20,000.

I'd like to name this task "rephrase neural network".

Any good reference or starting point, or, any network recommendation for comprising this neural network?

In a perfect setting, where you have infinite sentences (definitions).

You could use some kind of attention models by chunking sentences into smaller pieces.

[a spherically shaped] [fruit] which [tastes sour] and [sweet] and gets [red] when matured well.

With a distributed representation framework, every little piece of information is simply like a binary indicator (the feature exists or not).

Maybe each piece is vague by itself and it could not tell you anything about the target yet by combining them together they define useful information, see products of experts

To learn p(Y|X) where x consists of the sentence, y is the embedding of some class, so may similar classes share similar sentences.
If you have a model trained on the sentences and learned to project similar sentences near to each other, and assume you learned the word classes by some word2vec like algorithm, you could generalize to new pair even if the sentence and that class target were never paired (never seen in your training data), their respective feature vectors have been related to each other by the mapping function.

At the end of the day if you have a smooth function and your input is continuous then any small change of the input will introduce small change at the output, yet how could we truly extract the knowledge exists in the data. see Generating Sentences from a Continuous Space


how to make a file name as a column in panda? i have a CSV file which name is data.CSV and i want to make column from file name  for example data is column and value must be data as well  how can i do this?

import pandas as pd
import glob

files = glob.glob('path to dir/*.csv')
df = None

for i, f in enumerate (files):
    if i == 0:
        df = pd.read_csv(f)
        df['fname'] = f
    else:
        tmp = read_csv(f)
        tmp['fname'] = f
        df = df.append(tmp)

df.head()


Pandas, make the bars of a bar graph have equal widths What do I need to change so that ax1 uses the same width for bars as ax2?

import pandas as pd
import matplotlib.pyplot as plt

fig, (ax1, ax2) = plt.subplots(2)

pd.Series([1]).plot(kind='bar', width=0.5, ax=ax1)
pd.Series([1, 2, 3]).plot(kind='bar', width=0.5, ax=ax2)

plt.show()

Looking at the documentation, it seems there is no obvious way.

Looking at the source code of the plt.bar method (held on Axes objects), and searching for uses of the width parameter, it starts getting quite complicated and I don't think it is meant to be used for this purpose explicitly! Lots of internal methods, starting with a _, like self._convert_dx is a clue.

I would just offer a nasty hack for your toy example, hoping it is applicable to your actual data:

In [1] : import pandas as pd
    ...: import matplotlib.pyplot as plt
    ...:
    ...: fig, (ax1, ax2) = plt.subplots(2)
    ...:
    ...: pd.Series([0, 1, 0]).plot(kind='bar', ax=ax1)
    ...: pd.Series([1, 2, 3]).plot(kind='bar', ax=ax2)

Gives

So the hack is to make the length of your datasets the same, adding zeros where needed.


What are useful evaluation metrics used in machine learning I am using CNN in order to predict codes after analyzing text. As an example, I will write "I am crazy" .. the model will predict some code " X321".
 All this based on CNN.

I want to evaluate my model. I used Fscore (recall and precision). Can you advice me more metrics?

For model evaluation there are different metrics based on your model:

Confusion matrix

Classification accuracy:
(TP + TN) / (TP + TN + FP + FN)
Error rate:
(FP + FN) / (TP + TN + FP + FN)
Paired criteria

Precision: (or Positive predictive value)
proportion of predicted positives which are actual positive
TP / (TP + FP)
Recall: proportion of actual positives which are predicted positive
TP / (TP + FN)

Sensitivity: proportion of actual positives which are predicted positive
TP / (TP + FN)
Specificity: proportion of actual negative which are predicted negative
TN / (TN + FP)
True positive rate: proportion of actual positives which are predicted
positive
TP / (TP + FN)
True negative rate: proportion of actual negative which are predicted
negative
TN / (TN + FP)
Positive likelihood: likelihood that a predicted positive is an actual positive
sensitivity / (1 - specificity)
Negative likelihood: likelihood that a predicted negative is an actual
negative
(1 - sensitivity) / specificity
Combined criteria

BCR: Balanced Classification Rate
½ (TP / (TP + FN) + TN / (TN + FP))
BER: Balanced Error Rate, or HTER
Half Total Error Rate: 1 - BCR
F-measure harmonic mean between precision and recall
2 (precision . recall) / (precision + recall)
Fβ-measure weighted harmonic mean between precision and recall
(1+β)2 TP / ((1+β)2 TP + β2 FN + FP)

  The harmonic mean between specificity
  and sensitivity is also often used and
  sometimes referred to as F-measure.

Youden's index: arithmetic mean

between sensitivity and specificity
sensitivity - (1 - specificity)
Matthews correlation: correlation between the actual and predicted
(TP . TN – FP . FN) / ((TP+FP) (TP+FN) (TP + FP) (TN+FN)) ^ (1/2)
comprised between -1 and 1 Discriminant power normalized likelihood index
sqrt(3) / π .
(log (sensitivity / (1 – specificity)) +
log (specificity / (1 - sensitivity)))
<1 = poor, >3 = good, fair otherwise

You can find much more here. Also there are some explanations here and you can find useful code snippet from here which are implemented.


Dataset of Tweets by Same User I was reading the Keras documentation on shared layers, and used the example of a model built to classify whether 2 tweets were from the same user or not.  Is there any dataset that I could use to do this?

You'll find great datasets for this task called author verification from the PAN workshop series. Afaik the last one specifically on this task was in 2015: https://pan.webis.de/clef15/pan15-web/author-identification.html. I recommend exploring the website, there are many other datasets for related tasks.


Is active learning able to detect challenging cases? Let's say we have a set of data points that need to be labelled for a classification task. In the pool-based active learning, if we go with the uncertainty measure, is the AL approach able to detect challenging cases? By challenging cases I mean samples that receive a high prediction score for $\hat{y}$ (e.g. >90%) but, most probably, $\neg\hat{y}$ is the correct prediction.
The rationale behind my question is: does adding more samples to the training set always improve the performance of a classifier?

In general it depends on the exact method used to select instances and of course on the data. Assuming that the selection is based solely on the uncertainty measure of a single classifier, then by definition the method will prioritize instances predicted with a probability around 50%, i.e. where the classifier is "unsure". As a consequence an instance predicted with a high probability is unlikely to be selected for annotation. However the iterative training process will make the classifier re-estimate the probability of all the instances, so it's possible that an instance wrongly classified with 90% probability at a particular iteration will later be assigned a lower probability, or even the true class. But overall there's no guarantee: like with any statistical system there can be instances misclassified with a high probability.

The rationale behind my question is: does adding more samples to the training set always improve the performance of a classifier?

In active learning, the performance depends more on how many instances end up being manually annotated than on the size of the unlabelled sample. But as usual the performance strongly depends on the data itself.


Linear classifier and gradient descent I understood that gradient descent is needed to find the local extremum of any function, but how is it applied to the linear classifier (single matrix, two classes case, for example)? How does it step by step modify the elements of this matrix?

Let us break this down into a few steps for better understanding:

Your linear classifier let us represent it with an equation y :
y = m1x1 + m2x2 + m3x3 + ..... where y takes into account all the features you have provided, multiplies it with random weights m .
Now you need to find the ideal values of m in order for you to get maximum accuracy. You could ofcourse brute force your way but there are optimized ways to do this. This is where gradient descent comes into play.
Gradient descent optimizes the values of m by taking into account the final output and understanding which features are important and which are not.
The loss function determines what is to be compared. In your example you are talking about two classes, in this case, the loss function will be a simple cross entropy loss function, that tells you how far you are from the actual values.
The loss is the difference in the predicted value vs actual value.
The only way you can minimize this loss is by playing around with weights. In order to understand which weights to play around with, you use simple first order derivatives to associate impact of output with each feature multiple by its weight.
Gradient descent is then used to compute the next set of weights that helps you reduce the loss by looking at the direction of weights to be used.
When you run this thru multiple epochs and steps, it stabilizes this weight matrix thereby improving performance of your model.

Hope this clears your question, would be happy to elaborate in case you need it.


Custom loss function with multiple outputs in tensorflow my network has two outputs and single input. I am trying to write a custom loss function
$$
Loss = Loss_1(y^{true}_1, y^{pred}_1) + Loss_2(y^{true}_2, y^{pred}_2)
$$
I was able to write a custom loss function for a single output. But for multiple output, I am struck. Below I wrote a mwe I tried
def model(input_shape=4, output_shape=3, lr=0.0001):
    """
    single input and multi-output
    loss = custom_loss(out_1_true, out_1_pred)+mse(out_2_true, out_2_pred))
    """
    input_layer = Input(input_shape)
    layer_1 = Dense(units=32, activation='elu', kernel_initializer='he_uniform')(input_layer)
    #outputs
    y_1 = Dense(units=output_shape, activation='softmax', kernel_initializer='he_uniform')(layer_1)
    y_2 = Dense(units=1, activation='linear', kernel_initializer='he_uniform')(layer_1)

    def custom_loss(y_true, y_pred):

        # both true value of out_1, out_2 are encoded in y_true
        y_true_1     = y_true[:, :1+output_shape]
        y_true_2     = y_true[:, 1+output_shape:]

        #(this part is wrong...I dont know how)
        y_pred_1, y_pred_2 =  y_pred[:, :1+output_shape], y_pred[:, 1+output_shape:]

        #custorm loss for y_pred_1
        entropy_loss = -y_pred_1 * K.log(y_true_1 + 1e-10)

        #mse for y_pred_2
        mse          = -K.mean(K.square(y_pred_2 - y_true_2))

        #net loss
        loss = entropy_loss + C * mse
        return loss

    Network_model = Model(inputs = input_layer, outputs = [y_1, y_2])
    Network_model.compile(loss = custom_loss, optimizer=RMSprop(lr=lr))

    return Network_model

I think, the main issue lies in spiting the y_pred tensor.
PS: For the purpose of mwe, I have use normal cross entropy loss and mse loss functions in the above code. But however, I have a different cost functions.

def model(input_shape=4, output_shape=3, lr=0.0001):
    """
    single input and multi-output
    loss = custom_loss(out_1_true, out_1_pred)+mse(out_2_true, out_2_pred))
    """
    input_layer = Input(input_shape)
    layer_1 = Dense(units=32, activation='elu', kernel_initializer='he_uniform')(input_layer)

    # STEP-1:   name your outputs
    y_1 = Dense(units=output_shape, activation='softmax', kernel_initializer='he_uniform', name='actions')(layer_1)
    y_2 = Dense(units=1, activation='linear', kernel_initializer='he_uniform', name='values')(layer_1)

    # STEP-2:    Define the custom custom_loss. Note that the args should be y_true, y_pred
    def custom_loss(y_true, y_pred):
        entropy_loss = -y_pred * K.log(y_true + 1e-10)
        return entropy_loss

    # STEP-3:   Define the dicts for loss and lossweights with keys as the name of the output layers
    LossFunc    =     {'actions':custom_loss, 'values':'mse'}
    lossWeights =     {'actions':0.5, 'values':0.5}

    Network_model = Model(inputs = input_layer, outputs = [y_1, y_2])

    # STEP-4:   complie using LossFunc and lossWeights dicts
    Network_model.compile(optimizer=RMSprop(lr=lr), loss=LossFunc, loss_weights=lossWeights, metrics=["accuracy"])

    return Network_model

#training example:

#model
Network_model = model(4,2)
# input
S = np.reshape([1.,2.,3.,4.], (1, -1))
#true outputs
action, value = np.reshape([0.1], (1, -1)), np.reshape([0.1, 0.9] , (1, -1))
Y = [action, value]

Network_model.fit(S, [A_t, V_t])

1/1 [==============================] - 0s 2ms/step - loss: 6.7340 - actions_loss: 1.1512 - values_loss: 12.3169 - actions_accuracy: 0.0000e+00 - values_accuracy: 1.0000


What is the significance of model merging in Keras? I have learned that Keras has a functionality to "merge" two models according to the following:

from keras.layers import Merge

left_branch = Sequential()
left_branch.add(Dense(32, input_dim=784))

right_branch = Sequential()
right_branch.add(Dense(32, input_dim=784))

merged = Merge([left_branch, right_branch], mode='concat')

What is the point in mergint NNs, in which situations is it useful? Is it a kind of ensemble modelling? What is the difference between the several "modes" (concat, avg, dot etc...) in the sense of performance?

It is used for several reasons, basically it's used to join multiple networks together. A good example would be where you have two types of input, for example tags and an image. You could build a network that for example has:

IMAGE -> Conv -> Max Pooling -> Conv -> Max Pooling -> Dense

TAG -> Embedding -> Dense layer

To combine these networks into one prediction and train them together you could merge these Dense layers before the final classification.

Networks where you have multiple inputs are the most 'obvious' use of them, here is a picture that combines words with images inside a RNN, the Multimodal part is where the two inputs are merged:

Another example is Google's Inception layer where you have different convolutions that are added back together before getting to the next layer.

To feed multiple inputs to Keras you can pass a list of arrays. In the word/image example you would have two lists:

x_input_image = [image1, image2, image3]
x_input_word = ['Feline', 'Dog', 'TV']
y_output = [1, 0, 0]

Then you can fit as follows:

model.fit(x=[x_input_image, x_input_word], y=y_output]


Reversing the input and output of an ML algorithm to Optimize My dataset consists of multiple input variables (X) and multiple output variables (Y).

For example:

| X1| X2 | X3 |    | Y1| Y2 | Y3 |
----------------------------------
| 1 |  1 |  0 |    | 2 |  2 |  0 |
| 2 |  2 |  1 |    | 3 |  3 |  1 |
| 3 |  3 |  3 |    | 4 |  5 |  6 |

But, I don't actually want to predict the output variables Y. I want to actually predict the input variables X given Y. I understand that switching the variables X and Y might not be 'optimal' since there could be multiple input values that yield the same output values, but I don't see how else I solve this type of problem. I was planning on using a Random Forest or simple neural net.

From the help I've been trying to get, it seems that I should keep input and output variables as is, but rather use optimization techniques to find the best input(the variable I want to predict) for a given output. I am unsure if my initial intuition on reversing the inputs and outputs would 'get the job done'.

For a more detailed breakdown on the actual problem I am trying to solve, please see this link.

The problem is a bit vague but here a couple ideas:

Just use the Y variables as features and predict the X variables. The most basic option is to consider every Xi as independent of the others, and train an independent model for each of them. A more advanced approach is to train a joint model which predicts all the Xi together (i.e. the class would be for instance "1,1,0"), but this probably requires a lot more data.
In the idea of optimization techniques, there might be some way to use genetic learning in order to obtain the optimal X values given the Y values. However I don't see how to design the population with many different Y combinations, but maybe it's possible.


How to validate a chatbot? Let's say I'm building a medical assistance chatbot.

How do I validate that my model is working well or better than others.

Is there any standard validation metrics for it ?

It is impossible to answer this question because you have not defined what precisely you want your chatbot to do (and not to do!). Once you have defined that, you have your metric.

So the answer to your second question is: no, there is no standard validation metric.


Auto ML vs Manual ML for a project I recently was introduced to a AUTO ML library based on genetic programming called tpot. Thanks to @Noah Weber. I have few questions

1) When we have AUTO ML, why do people usually spend time on Feature selection or preprocessing etc? I mean they do at-least reduce the search space/feature space

2) I mean atleast, they reduce our work to some extent and we can work from the output of AUTO ML solution and tune further if required. We don't really have to do gridsearchCV by manually keying in range of values that we might require. Right?

3) Is there any disadvantage to it? I understand it might be black box but for data analysis, don't they make it easier? Computer scientists,may not prefer it.  Ofcourse, we need to have some sort of knowledge to be able to fine tune the model, interpret the results etc

4) What's the advantage of doing manual ML when compared to AUTO ML

5) Will it be possible for us to improve the results further? I mean once we get the output from Auto ML

Can you help me understand this?

1) Feature Selection should be done by AutoML on the other hand preprocessing is normally done by the user in order to make sense fo the data.

2) AutoML takes care of the hyper-parametrization.

3) The disadvantage that I mostly find is that is extremely computationally expensive. And from what I have seen in Kaggle most of the winning solutions use manual ML, not AutoML.

4) For me, one of the advantages is that sometimes it finds a good algorithm that I have not tried (or thought) and that it avoids me spending some coding time. Also, it happens to do some good ensembles of different models.

5) You can do Manual ML in your side and then doing an ensembling with your personal ML model and the AutoML. This doesn't guarantee improvement but it could boost your performance.

You can have a look at H20 AutoML, I quote the documentation, I believe it can be helpful in this case in order to have an intuition about it:

  Although H2O has made it easy for non-experts to experiment with machine learning, there is still a fair bit of knowledge and background in data science that is required to produce high-performing machine learning models. Deep Neural Networks in particular are notoriously difficult for a non-expert to tune properly. In order for machine learning software to truly be accessible to non-experts, we have designed an easy-to-use interface which automates the process of training a large selection of candidate models. H2O’s AutoML can also be a helpful tool for the advanced user, by providing a simple wrapper function that performs a large number of modeling-related tasks that would typically require many lines of code, and by freeing up their time to focus on other aspects of the data science pipeline tasks such as data-preprocessing, feature engineering and model deployment.

You can also have a look at this blog of Bojan Tunguz where he defines what are the phases of AutoML:

Level 0: No automation. You code your own ML algorithms. From scratch. In C++.
Level 1: Use of high-level algorithm APIs. Sklearn, Keras, Pandas, H2O, XGBoost, etc.
Level 2: Automatic hyperparameter tuning and ensembling. Basic model selection.
Level 3: Automatic (technical) feature engineering and feature selection, technical data augmentation, GUI.
Level 4: Automatic domain and problem specific feature engineering, data augmentation, and data integration.
Level 5: Full ML Automation. Ability to come up with super-human strategies for solving hard ML problems without any input or guidance. Fully conversational interaction with the human user.


How to properly predict data with LSTM using train/test? I see one paradox here. If we use Train/Test split and evaluate our Test data, we might get a good score, but any further prediction will not be credible, because model didn't train the Test data and include it's sequences in memory.

On the other side, we can train the data on Train and Test sequence as train data, but then we can not evaluate our predictions, because we have no testing reference.

How do you properly predict LSTM models?

You seek to augment the external validity of your model.

The most common way of doing so is by applying k-fold cross-validation to verify that your model generalizes well on unseen data.

  In k-fold cross-validation, the original sample is randomly
  partitioned into k equal sized subsamples. Of the k subsamples, a
  single subsample is retained as the validation data for testing the
  model, and the remaining k − 1 subsamples are used as training data.
  The cross-validation process is then repeated k times, with each of
  the k subsamples used exactly once as the validation data. The k
  results can then be averaged to produce a single estimation.

This will reduce the variance of your model and will reduce its error on unseen data.


Determine model hyper-parameter values for grid search I built machine learning model for Ridge,lasso, elastic net and linear regression, for that I used gridsearch for the parameter tuning, i want to know how give value range for **params Ridge ** below code? example consider alpha parameter there i uses for alpha 1,0.1,0.01,0.001,0.0001,0 but i haven't idea how this values determine each models.(ridge/lasso/elastic) can some one explain these things?

 from sklearn.linear_model import Ridge
    ridge_reg = Ridge()
    from sklearn.model_selection import GridSearchCV
    params_Ridge = {'alpha': [1,0.1,0.01,0.001,0.0001,0] , "fit_intercept": [True, False], "solver": ['svd', 'cholesky', 'lsqr', 'sparse_cg', 'sag', 'saga']}
    Ridge_GS = GridSearchCV(ridge_reg, param_grid=params_Ridge, n_jobs=-1)
    Ridge_GS.fit(x_train,y_train)
    Ridge_GS.best_params_

I am not sure if I understand your question correctly, but in your model, you are tuning your "alpha" parameter, you have a range from 1 to 0. (1 -> 0.1 -> 0.01 -> 0.001 -> 0.0001 -> 0).

The grid search will evaluate each algorithm (SVD, CHOLESKY,...) with each possible value of your "alpha" parameter.
It will define the score for each alpha parameter (eg. accuracy  / auc). The score metric depends on your estimator / choice

Documentation:

https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html


How much can bias decrease performance of the network at the beginnng of the training? I am writing a custom framework and in it I'm trying to train a simple network to predict the addition function.

The network:

1 hidden layer of 3 Neurons
1 output layer
cost function used is Squared error, (Not MSE to avoid precision problems)
Identity transfer function to make things simple at first
no specal updaters, just the step size
no learning rate decay
no regularization

The training set:

~500 samples
inputs: [n1][n2]; labels: [n1 + n2]
Every element is between 0 and 1. e.g.:
  [0.5][0.3] => [0.8]

The algorithm I'm using to optimize:

samples 64 elements for an epoch
for each sample: it evaluates the error

then propagates the error back
and then based on the error values calculates the gradients
the gradients for each elements are added up into one vector, then normalized by dividing by the number of samples evaluated

After the gradients are calculated a step size of 1e-2 is used to modify the weights.
The training stops when the sum of the errors for the 500 data elements are below 1e-2

I don't have a test dataset yet, as first I'd like to overfit to a training set, to see if it could even do that. Withouot bias the training converges to an optimum in about ~4k epochs.

When I include the tuning of bias into the training, it seems to have a much worse performance,
the network is not converging to the optimum, instead the biases and the weights oscillate next to one another..

Is this a normal effect of introducing a bias?

Here is a chart abuot the weight values throughout the training:

Bias takes care of variables which are latent for i.e you did not include it in the training set.

So you are trying to overfitt on training set, but now you introduce pertrubance. Ofcourse it will have worse performance.


Why there is Different accuracy for two same trained model? Trained the same model twice with the same dataset, the same parameters (Epochs, Batch Size, Learning rate, etc..). But both trained model shows different train as well as test accuracy on the same dataset.
(code is same for two models)
Why both models show different accuracy?

Train the model
Test accuracy = 87.98%

Again Train the model
Test accuracy = 67.18%

Somewhere in the code there should be some parameter that is initialised randomly, this is usually called the random seed. It could be the different initialisation of your neural network weights that is affecting the results, or maybe the k-fold being different if done at random.
The extent of the difference between the performance suggests that you might not have enough data to reliably learn a good model, or your model is underspecified and gets stuck at different local minima. Consider, changing the learning rate and other hyperparameters and see if the effect stays.


How to deal with exception handling in python? Lets say I have a variable city which accepts any city name , like
city='London'

or
city='New York'

It accepts only city name but if user mistakenly specifies city name some different like
city='Londo'

How does should I deal with it means I want to use try catch in it , But How ?

In an interactive program,
If no matching city name found
then shows the input value again and ask the user to enter another value

In data preparation,
select unmatching city names
and find patterns of mistakes
use regular expression accordingly to replace the mismatching city names with the right ones
for example, new_df = df.replace(to_replace ='Londo.*', value = 'London', regex = True)

That is how I would do.


Regarding imbalanced multilabel dataset I am doing multilabel news classification in python language.The dataset I have has two files. First CSV contains articles at each row. Second CSV contains corresponding labels to each article. Here is the snapshot of label file

Here is the dropbox link....https://www.dropbox.com/s/7huzh41je735oqn/labelset.csv?dl=0

Is the dataset imbalanced one?
How do I distribute this dataset properly into training set,
validation and test set?

Note: By properly I mean, Can this unbalanced dataset be split into proper proportions in training,validation and testing set?

I'll go through your questions one by one:

  Is the dataset imbalanced one?

The distinction between a balanced and an imbalanced dataset is a matter of degree. If you think that the difference in number of observations for each class is too big, I suggest you to train your model using Mini-Batch Gradient Descent, and build any mini batch in a way that observations have the same frequency. This would attribute greater weights to the less frequent observations.

  How do I distribute this dataset properly into training set,
  validation and test set?

Make sure that all the classes are evenly distributed among the sets.


Sensitivity analysis of a machine learning model Let’s say I have a set of input variables (A, B, C and D) and I predict a target (y) using a machine learning model (XGBRegressor in my case) with a reasonable performance (5% relative error on test set).

from sklearn.datasets import make_regression
import pandas as pd
from xgboost import XGBRegressor

X, y = make_regression(n_samples=500, n_features=4, n_informative=2, noise=0.3)
X = pd.DataFrame(X, columns=['A', 'B', 'C', 'D'])

model = XGBRegressor()
model.fit(X, y)

Now, I want to do some kind of sensitivity analysis on this model by answering two questions:

What is the impact of a 5% independent increase in variables A, B and C (not D) on the target variable?
From variables A, B, C and D; which combination of values of A, B and C (without touching D) increases the target y value by 10, minimizing the sum of A, B and C.

I have already answered question one (see this gist).
However, how can question 2 be coded? I imagine that this implies an optimization problem.

Without further knowledge of the domain, there is no simple answer.

Are the predictor variables discrete or continuous? If they are discrete, you could search all combinations that sum to 0, then all combinations that sum to 1, etc. until you find a combination that increases y by 10. If they are continuous, then:
Is the target variable monotonically increasing with each of the predictor variables? In other words, given A, B, C and D, when increasing A, B or C, will y always either increase or stay the same? If the predictors are continuous and the target variable is monotonically increasing, you can do a search similar to the one above by increasing each variable by a small step until y has increased by more than 10, this will give you an idea, within the range of that small constant, of where the answer to your question lies. You can then reduce the size of the step to find a more precise answer within that range.
If the target variable is continuous and not monotonically increasing with the predictor variables, this gets much more complicated. Suppose you've found two points (A1, B1, C1, D) and (A2, B2, C2, D) that do not increase y by 10: nothing guarantees that there is not a point between the two, (A3, B3, C3, D), that does. Your best bet then would probably be to explore the model. Depending on the number of trees in your ensemble, this may or may not be possible (computationally). I think you would have to find all possible combinations of leaves where D is unchanged and y is increased by more than 10, then find the minimum amongst these.


Is it possible to have differential weights as per input importance for a Neural Network I have a Computed Tomography data set where central slices are more important as the scan angle approaches 90 degrees. The information in the initial and last slices(0 degrees and 180 degrees) may assigned lesser weight as they contain lateral information. Could anyone please suggest me a way to assign weights as per importance of the input? Shall I focus on any specific hyper-parameter while training?

Yes! It is totally possible, generally weights are never same, they differ for different inputs.This is the first reason why they are called weights, as they associate weight to every input.

Preparing neural network parameters (weights and bias) using TensorFlow Variables in python:(assuming you have 3 inputs and want to assign them with .3,.1 and .8 weight respectively.)

weights = tensorflow.Variable(initial_value=[[.3],[.1],[.8]],dtype=tensorflow.float32)
bias = tensorflow.Variable(initial_value=[[1]], dtype=tensorflow.float32)

Happy to answer.


Neural network with flexible number of inputs? Is it possible to create a neural network which provides a consistent output given that the input can be in different length vectors?

I am currently in a situation where I have sampled a lot of audio files, which are of different length, and have to train a neural network provides me the desired output given a certain input.  I am trying to create a regression network that can generate MFCC feature, given samples of an audio file, which are of different length, which makes different numbered input.

Yes this is possible by treating the audio as a sequence into a Recurrent Neural Network (RNN). You can train a RNN against a target that is correct at the end of a sequence, or even to predict another sequence offset from the input.
Do note however that there is a bit to learn about options that go into the construction and training of a RNN, that you will not already have studied whilst looking at simpler layered feed-forward networks. Modern RNNs make use of layer designs which include memory gates - the two most popular architectures are LSTM and GRU, and these add more trainable parameters into each layer as the memory gates need to learn weights in addition to the weights between and within the layer.
RNNs are used extensively to predict from audio sequences that have already been processed in MFCC or similar feature sets, because they can handle sequenced data as input and/or output, and this is a desirable feature when dealing with variable length data such as spoken word, music etc.
Some other things worth noting:

RNNs can work well for sequences of data that are variable length, and where there is a well-defined dimension over which the sequences evolve. But they are less well adapted for variable-sized sets of features where there is no clear order or sequence.

RNNs can get state-of-the-art results for signal processing, NLP and related tasks, but only when there is a very large amount of training data. Other, simpler, models can work just as well or better if there is less data.

For the specific problem of generating MFCCs from raw audio samples: Whilst it should be possible to create a RNN that predicts MFCC features from raw audio, this might take some effort and experimentation to get right, and could take a lot of processing power to make an RNN powerful enough to cope with very long sequences at normal audio sample rates. Whilst creating MFCC from raw audio using the standard approach starting with FFT will be a lot simpler, and is guaranteed to be accurate.


Train a RNN with strings using positive and negative samples I'm trying to recognize strings coming from a regular language, using positive and negative samples by Recurrent Neural Networks.

In particular, I tried to use the rnnlib by Alex Graves, but I had some troubles in changing the architecture of the network to adapt the network to my needs.

This is an example of my training set:

str1: w1 w2 ... wn --> label 1 (it is a valid string for the language)

str2: w1 w2 ... wi --> label 0 (it does not belong to the language)

where wi are drawn from the alphabet of the unknown language.

In other words, I'm trying to use RNN as a binary classifier.

Can you suggest a good tutorial or reference for this task?

Thanks in advance.

I think Sequence Classification with LSTM Recurrent Neural Networks in Python with Keras is one of the best tutorial available for this task. It does sentiment analysis on IMDB Dataset and is the simplest to understand. Also, it uses Keras framework which I use right now because of it's simplicity and it's online support and documentation available. The RNN of keras has multiple options which you can explore based on your requirement. Also, keras make it real easy to use an optimizer from a wide variety of options.


How to improve the accuracy of my MLP (Current benchmark 77%) My MLP code:

# In[1]:

import pandas as pd
from keras.models import Sequential #Sequential Models
from keras.layers import Dense #Dense Fully Connected Layer Type
from keras.optimizers import SGD #Stochastic Gradient Descent Optimizer
from keras.callbacks import EarlyStopping
from keras.callbacks import ModelCheckpoint

# In[2]:

data = pd.read_csv("doom.csv", delimiter=",")
data = data.drop('id', 1)
data = data.drop('cid', 1)
data = data.drop('alerts', 1)
data = data.drop('hsalerts', 1)
data = data.drop('capacity', 1)
data = data.drop('duplicate', 1)
data.shape

# In[3]:

data.selectedoption.value_counts()

# In[4]:

feature = data.drop('selectedoption',axis=1)
label = data[['selectedoption']]

# In[5]:

from sklearn.model_selection import train_test_split
train_feature, test_feature, train_label, test_label= train_test_split(
 feature, label, test_size=0.3, random_state=42)

# In[ ]:

# In[ ]:

# In[6]:

from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()

scaler.fit(train_feature)
train_feature_trans = scaler.transform(train_feature)
test_feature_trans = scaler.transform(test_feature)

# In[7]:

from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout

import matplotlib.pyplot as plt
def show_train_history(train_history,train,validation):
    plt.plot(train_history.history[train])
    plt.plot(train_history.history[validation])
    plt.title('Train History')
    plt.ylabel(train)
    plt.xlabel('Epoch')
    plt.legend(['train', 'validation'], loc='best')
    plt.show()

# In[11]:

from keras import optimizers

model = Sequential()

model.add(Dense(units=32,
                input_dim=10,
                kernel_initializer='uniform',
                activation='relu'))
model.add(Dropout(0.5))

model.add(Dense(units=32,
                kernel_initializer='uniform',
                activation='sigmoid'))
model.add(Dropout(0.5))
model.add(Dense(units=16,
                kernel_initializer='uniform',
                activation='softmax'))
model.add(Dropout(0.5))

model.add(Dense(units=1,
                kernel_initializer='uniform',
                activation='sigmoid'))

print(model.summary())

adam = optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0)

model.compile(loss='binary_crossentropy',
              optimizer='adam', metrics=['accuracy'])
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=200)
mc = ModelCheckpoint('best_model.h5', monitor='val_acc', mode='max', verbose=1, save_best_only=True)
train_history = model.fit(x=train_feature_trans, y=train_label,
                          validation_split=0.8, epochs=20000,
                          batch_size=1000, verbose=2,callbacks=[es, mc])

show_train_history(train_history,'acc','val_acc')
show_train_history(train_history,'loss','val_loss')

# In[9]:

scores = model.evaluate(test_feature_trans, test_label)
print('\n')
print('accuracy=',scores[1])

My dataset is this:

doom.csv

I made a few changes like, changing the optimizer, changing the learning rate, changing activation functions, changing the units. Nothing seems to work. The accuracy isn't going above 80%. Is there something that i can do to improve the accuracy over 90%?  I am a newbie to this. Thank you.

1. Parameters optimization

As you have not mentioned the strategy, which you use for parameters choosing, I'd suggest to attend to it. For example in this article https://towardsdatascience.com/hyperparameter-optimization-with-keras-b82e6364ca53 you can find, how parameters can be chosen.

The other powerful approach is Bayesian Optimization Methods https://towardsdatascience.com/automated-machine-learning-hyperparameter-tuning-in-python-dfda59b72f8a It allows to move in parameters values space, taking into account previous results of optimization.

Note: it's crucial to consider parameters as a whole, but not just separately (if value $a$ of the parameter $A$ is not the best choice, it doesn't mean, that it won't be the best option in combination with value $b$ of the parameter $B$)

2. Deal with outliers

It would be reasonable to check outliers in the data (objects which appear to be inconsistent with the remaining instances). For this aim you can use box plot approach. It might be the reason, why it's impossible to get particular model performance.

3. Working with feature space

It also might be useful to provide feature analysis.


How to get different length output using keras? I am working on a time-series kind of problem. I have (timesteps, features) and the length of output will be (timesteps,). All outputs as float numbers. But the problem is I have training samples with different timesteps.
Suppose one example has 1500 timesteps then it will output 1500 dimensional vector. Another example has 1000 timesteps and it will have 1000 dimensional output.
I am looking for a way to train such a model using keras.

Its the number of features that has to remain consistent not the number of timestamps.
Outputs will be batches of one row predictions, so in your case its 1500 and than 1000. Model does not care, features should remain same. And beware of dataleakage with time series.


Are linear regression models with non linear basis functions used in practice? I know that popular linear regression models such as Lasso or Logistic Regression are widely used in practice because they perform reasonably good, are efficient and interpretable.

As far as I know, the only way for these models to learn non-linear relationships between $X$ and $y$ is by applying non-linear transformations to $X$ through basis functions $\phi(X)$ (as it is summarized in the scikit-learn documentation).

I wonder if this approach is actually used in real world problems, where usually we don't have any prior information about how to design $\phi$.

Yep, that's a thing. It's called a "Generalized additive model (GAM)":

https://en.wikipedia.org/wiki/Generalized_additive_model

You may also be interested in "multivariate adaptive regression splines (MARS)":

https://en.wikipedia.org/wiki/Multivariate_adaptive_regression_splines

EDIT: Regarding demonstrating that these are actually used, I'm not sure what you're looking for. I've never seen a survey to try and gauge the popularity of specific models, and I'm not sure how useful such a thing would be. I could speak from my subjective experience, but I don't consider that particularly meaningful either.

If you just want examples:

Hits for GAM models on PubMed - 2114 hits
Hits for MARS models on Pubmed - 190 hits

For context:

Hits for Random Forest models on PubMed - 5604
Hits for Deep Learning on PubMed - 4435 hits
Hits for Neural Network on PubMed - 57170 hits

Take these counts with a solid dose of salt: in my experience random forests are in much wider use in practice than neural networks. But like I said earlier, you shouldn't really trust my subjective experience either since that's just anecdotal.


Using text classification for system calls I'm working on a project in which I should classify System calls sequences, my dataset is represented as sequences of integers (from 1 to 340). To do the classification I have inspired from Text classification projects. I'm trying to use on of them but I found a problem in my dataset shape, the code is:

df = pd.read_csv("data.txt")
#df_test = pd.read_csv("validation.txt")

#split arrays into train and test data (cross validation)
train_text, test_text, train_y, test_y = train_test_split(df,df,test_size =
0.2)

#train_text, train_y = (df,df)
#test_text, test_y = (df_test, df_test)
MAX_NB_WORDS = 5700

texts_train = train_text.astype(str)
texts_test = test_text.astype(str)

tokenizer = Tokenizer(nb_words=MAX_NB_WORDS, char_level=False)
tokenizer.fit_on_texts(texts_train)
sequences = tokenizer.texts_to_sequences(texts_train)
sequences_test = tokenizer.texts_to_sequences(texts_test)

word_index = tokenizer.word_index
#print('Found %s unique tokens.' % len(word_index))
type(tokenizer.word_index), len(tokenizer.word_index)
index_to_word = dict((i, w) for w, i in tokenizer.word_index.items())
" ".join([index_to_word[i] for i in sequences[0]])

seq_lens = [len(s) for s in sequences]
#print("average length: %0.1f" % np.mean(seq_lens))
#print("max length: %d" % max(seq_lens))

MAX_SEQUENCE_LENGTH = 100
# pad sequences with 0s
x_train = pad_sequences(sequences, maxlen=MAX_SEQUENCE_LENGTH)  # former des
sequence de meme taille 150, en ajoutant des 0
x_test = pad_sequences(sequences_test, maxlen=MAX_SEQUENCE_LENGTH)
#print('Shape of data train:', x_train.shape)  #it gives (1,100)
#print('Shape of data test tensor:', x_test.shape)
y_train = train_y
y_test = test_y
#if np.any(y_train):
  #y_train = to_categorical(y_train)
print('Shape of label tensor:', y_train.shape)
EMBEDDING_DIM = 50
N_CLASSES = 2
# input: a sequence of MAX_SEQUENCE_LENGTH integers
sequence_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='float32')

embedding_layer = Embedding(MAX_NB_WORDS, EMBEDDING_DIM,
                        input_length=MAX_SEQUENCE_LENGTH,
                        trainable=True)
embedded_sequences = embedding_layer(sequence_input)

average = GlobalAveragePooling1D()(embedded_sequences)
predictions = Dense(N_CLASSES, activation='softmax')(average)

   model = Model(sequence_input, predictions)
model.compile(loss='categorical_crossentropy',
          optimizer='adam', metrics=['acc'])
model.fit(x_train, y_train, validation_split=0.1,
      nb_epoch=10, batch_size=100)
output_test = model.predict(x_test)
print("test auc:", roc_auc_score(y_test,output_test[:,1]))

I got the error: ValueError Error when checking target expected dense_1 to have shape(2,), but got array with shape(1,)

Any suggestion, cause I don't know how to proceed .
Thank you

The problem seems to arise in this line.

predictions = Dense(N_CLASSES, activation='softmax')(average)

Here, N_CLASSES has the value of 2. Meaning you have 2 classes. But the y_train vector has shape ( 1, ).

You need to convert the y_train vector to a one hot vector. Here's any example.

If y_train = [ [ 0 ] , [ 1 ] ] then its one hot vector would be [ [ 1 , 0 ] , [ 0 , 1 ] ]. The one hot vector has shape ( 2 , ) which is needed in the above code snippet.
So make this edit.

y_train = keras.utils.to_categorical( y_train , N_CLASSES )


Extracting NER from a Spanish language text file I am trying to extract various Named Entities from a Spanish language text file. I tried using nltk but to no success. I am using Python 2 with nltk 3.x.

I suggest you take a look at the Python library Spacy it has a model for spanish language that includes NER.


Classification of a time series data I have 5 different classes in which I want to classify some data points. I'm using RNN with Echo-state networks (Reservoir computing).

Normally, a straightforward method consists of computing the outputs y and deciding the class using the argmax[k](y) with k is the dimension of the right class. Now I found another method that consists of computing the argmax[k](sum(y)), and that is what I didn't understand.

I've attached an image of the mathematical approach. The latter is understandable, but it appears not logical for me, since the sum of y over the data points timescale gives the same result for each time sequence, meaning that all data points will be classified in the same class(which corresponds to the max of the sum(y)).

Can someone what I don't understand?

The method you have here is for time series, so input and output are a set of values over a time period.

The sum method sum(y) here is a short hand (abbreviation) for the average over the last mentioned equation, it is not a conventional sum.

Also do note argmax k over the sums is done in separate dimension for each k representing the individual class.

So in my understanding sum is needed only because it is a series of data and sum is calculated per class for the whole series.

The y target (n) is 1 in only in one dimension when match happen, so argmax k finds with highest value the correct class.


What machine learning algorithm should I use for specific user configuration? I have a data-set that contains thousands of employee data, including their role, department (Applications Developer, IT Support, Network Management etc.), and using one-hot encoding all of the hardware or software they have been given.

I want to use this data so that when I input a new user with the data - role, department etc. their account will be compared to the model accounts and a prediction of what software or hardware they will need will be output.

What machine learning algorithm should I use for this? I was thinking an unsupervised approach would be appropriate but I am new to machine learning and data science so I could be going about this completely the wrong way.

Thanks for your help :)

Unsupervised Approach

Unsupervised learning can be a good starting point. You can do a clustering (k-means/hierarchical) of existing user-base to find patterns in it (example: cluster 1 contains employees with Windows laptop with 4 GB RAM and photoshop pre-installed , cluster 2 is Mac with 16 GB memory). Then for a new employee, predict the nearest matching cluster and use the pre-dominant setup of that cluster.

The problem is there may be lot of lot of variation in the same cluster, especially on softwares installed.

Supervised Approach

Other alternative is to convert it to a supervised learning problem with hand-rolled categories as outcome (example: 'laptops with lot of RAM and graphics software', 'laptops for development') and then predict that category. You will need upfront work and domain knowledge to create the categories. But it will reduce the predicted variable space by a large margin.

But How to Create Categories?

Maths can help us a little bit here. Let us assume that you have an N dimensional one-hot encoded vector representing the hardware and software setup. Let us make the assumption that there are a handful of latent K categories of the setup encoded by a large dimensional vector of size N. To go about finding those, you can take a low dimensional projection of the encoding vector. This is probably a mathematical way of finding the categories instead of what you'd do manually on the output of clustering/unsupervised approach.

Radical Alternative: Recommendations
If you have enough data, you can pose it as 'recommending right setup to a new employee' problem. Take any standard movie recommendation kind of setting with users and movie ratings. Replace users -> employees and movies with a unique hardware + software setup. Use standard recommendation algorithm applied to binary data.

All these approaches are actually variant of low dimensional projection of the encoding vector + similarity calculation with existing users.

Hope this helps.


Relation between convolution in math and CNN I've read explanation of convolution and understand it to some extent. Can somebody help me understand how this operation relates to convolution in Convolutional Neural Nets? Is filter like function g which applies weight?

Using the notation from the wikipedia page, the convolution in a CNN is going to be the kernel $g$ of which we will learn some weights in order to extract the information we need and then perhaps apply an activation function.

Discrete convolutions

From the wikipedia page the convolution is described as

$(f * g)[n] = \sum_{m=-\inf}^{\inf} f[m]g[n-m]$

For example assuming $a$ is the function $f$ and $b$ is the convolution function $g$,

To solve this we can use the equation first we flip the function $b$ vertically, due to the $-m$ that appears in the equation. Then we will calculate the summation for each value of $n$. Whilst changing $n$, the original function does not move, however the convolution function is shifted accordingly. Starting at $n=0$,

$c[0] = \sum_m a[m]b[-m] = 0 * 0.25 + 0 * 0.5 + 1 * 1 + 0.5 * 0 + 1 * 0 + 1 * 0 = 1$

$c[1] = \sum_m a[m]b[-m] = 0 * 0.25 + 1 * 0.5 + 0.5 * 1 + 1 * 0 + 1 * 0 = 1$

$c[2] = \sum_m a[m]b[-m] = 1 * 0.25 + 0.5 * 0.5 + 1 * 1 + 1 * 0 + 1 * 0 = 1.5$

$c[3] = \sum_m a[m]b[-m] = 1 * 0 + 0.5 * 0.25 + 1 * 0.5 + 1 * 1 = 1.625$

$c[4] = \sum_m a[m]b[-m] = 1 * 0 + 0.5 * 0 + 1 * 0.25 + 1 * 0.5 + 0 * 1 = 0.75$

$c[5] = \sum_m a[m]b[-m] = 1 * 0 + 0.5 * 0 + 1 * 0 + 1 * 0.25 + 0 * 0.5 * 0 * 1 = 0.25$

As you can see that is exactly what we get on the plot $c[n]$. So we shifted around the function $b[n]$ over the function $a[n]$.

2D Discrete Convolution

For example, if we have the matrix in green

with the convolution filter

Then the resulting operation is a element-wise multiplication and addition of the terms as shown below. Very much like the wikipedia page shows, this kernel (orange matrix) $g$ is shifted across the entire function (green matrix) $f$.

taken from the link that @Hobbes reference. You will notice that there is no flip of the kernel $g$ like we did for the explicit computation of the convolution above. This is a matter of notation as @Media points out. This should be called cross-correlation. However, computationally this difference does not affect the performance of the algorithm because the kernel is being trained such that its weights are best suited for the operation, thus adding the flip operation would simply make the algorithm learn the weights in different cells of the kernel to accommodate the flip. So we can omit the flip.


Filenotfounderror: [errno 2] no such file or directory: 'nsr001.ecg' I am trying to save contents of physiobank Normal Sinus Rhythm RR Interval Database into a numpy array but I keep getting an error:
Traceback (most recent call last):
  File "AverageRRI.py", line 20, in <module>
    averageArray = np.fromfile(file,dtype=float)
FileNotFoundError: [Errno 2] No such file or directory: 'nsr001.ecg'

but the file does exist in the directory.
import os
import numpy as np

for root, dirs, files in os.walk('normal-sinus-rhythm-rr-interval-database-1.0.0'):
    for file in files:
        if file.endswith(".ecg"):
            print(file)
            averageArray = np.fromfile(file,dtype=float)
            print(averageArray)

When I add the pathname like:
averageArray = np.fromfile('normal-sinus-rhythm-rr-interval-database-1.0.0/nsr001.ecg',dtype=float)
            print(averageArray)

It works.
Thanks so much!

You just need to add the root directory in front of the filename so that the full filepath is correct. For this you can use os.path.join:
import os
import numpy as np

for root, dirs, files in os.walk('normal-sinus-rhythm-rr-interval-database-1.0.0'):
    for file in files:
        if file.endswith(".ecg"):
            print(os.path.join(root, file))
            averageArray = np.fromfile(os.path.join(root, file), dtype=float)


Can we call a CMS like Drupal Machine Learning? Drupal process a page and created run time PHP files, Twig files, sometimes CSS/JS as well, from the fragments we have already created based on it's logic.
Since machine learning does some similar thing to act on the data and use own logic to produce result, can we can CMS to be bit of Machine Learning capabilities?

No: a CMS, like many other software tools, processes data. But this process is predefined, it's not "learnt" from the data.
Machine Learning is supposed to involve at least one step where the system "learns" from the data, i.e. uses some algorithmic process to determine the characteristics (patterns) inside the data. In supervised learning there is a specific training stage where the model "learns", which is followed by a testing stage where the model is applied to some new data. In unsupervised learning, learning and applying happen in a single stage: the model discovers the patterns and directly produces an output based on these pattern.


Implementing PatterNet in Python as it is in MATLAB NOTE: This question was first posted in Cross Validated Stack Exchange but I was instructed to move it off that website as it was not a good fit.
I am new in implementation of machine learning, neural networks in python. I am trying to implement patternet in python as it is in MATLAB; more specifically to keep the network configuration settings. Below is what I get when I use paternet(10):
    Neural Network

              name: 'Pattern Recognition Neural Network'
          userdata: (your custom info)

    dimensions:

         numInputs: 1
         numLayers: 2
        numOutputs: 1
    numInputDelays: 0
    numLayerDelays: 0
 numFeedbackDelays: 0
 numWeightElements: 230
        sampleTime: 1

    connections:

       biasConnect: [1; 1]
      inputConnect: [1; 0]
      layerConnect: [0 0; 1 0]
     outputConnect: [0 1]

    subobjects:

             input: Equivalent to inputs{1}
            output: Equivalent to outputs{2}

            inputs: {1x1 cell array of 1 input}
            layers: {2x1 cell array of 2 layers}
           outputs: {1x2 cell array of 1 output}
            biases: {2x1 cell array of 2 biases}
      inputWeights: {2x1 cell array of 1 weight}
      layerWeights: {2x2 cell array of 1 weight}

    functions:

          adaptFcn: 'adaptwb'
        adaptParam: (none)
          derivFcn: 'defaultderiv'
         divideFcn: 'divideind'
       divideParam: .trainInd, .valInd, .testInd
        divideMode: 'sample'
           initFcn: 'initlay'
        performFcn: 'crossentropy'
      performParam: .regularization, .normalization
          plotFcns: {'plotperform', plottrainstate, ploterrhist,
                    plotconfusion, plotroc}
        plotParams: {1x5 cell array of 5 params}
          trainFcn: 'trainscg'
        trainParam: .showWindow, .showCommandLine, .show, .epochs,
                    .time, .goal, .min_grad, .max_fail, .sigma,
                    .lambda

    weight and bias values:

                IW: {2x1 cell} containing 1 input weight matrix
                LW: {2x2 cell} containing 1 layer weight matrix
                 b: {2x1 cell} containing 2 bias vectors

My intention is not to create the exact net data structure, I would like to mainly implement the activation functions to different layers of my network as it is listed in above. Let's assume I can initialize the weights (network parameters).
I followed this post Getting Started with Deep Learning and Python as well as Multi-layer Perceptron to start. In the latter link, I learned that I can implement MLP by utilizing MLPclassifier from sklearn.neural_network library:
MLPClassifier(activation='relu', alpha=1e-05, batch_size='auto',
       beta_1=0.9, beta_2=0.999, early_stopping=False,
       epsilon=1e-08, hidden_layer_sizes=(5, 2), learning_rate='constant',
       learning_rate_init=0.001, max_iter=200, momentum=0.9,
       nesterovs_momentum=True, power_t=0.5, random_state=1, shuffle=True,
       solver='lbfgs', tol=0.0001, validation_fraction=0.1, verbose=False,
       warm_start=False)

However, its setting is not exactly the one that I am trying to have. Reading MLPclassifier document page did not help me further.
I would appreciate it, if I get to know how to implement it as such? I would be happy to be directed to any post or link which has already discussed this matter.

Consider reusing an existing framework and adding the missing activation functions. For example, here you can see how to do that in sklearn.

However, notice that there might be other differences (consider this example) in how the algorithm is implemented, as with the rise of deep learning and convolution networks, neural networks have hugely grown in variety and complexity.


A question on pipeline I made a pipeline with standard scaler and k means .When I fit the pipeline to the training data, Does the standard scaler just fits or fits and transforms the training data?

It fits the scaler and then transforms the training data to pass it to the k-means. The method is called fit_transform in scikit-learn.


How to Convert a Pandas Column having duration details in string format (ex:1hr 50m) into a integer column with value in minutes Lets say i have the following data like below:

  import pandas as pd
    import numpy as np
    df = pd.DataFrame({'Duration': ['1h 50m', '50m', '3h', '2h 30m', '5h', '60m']
                  })
    df

Generated Output:

Column Created in pandas data frame contains duration details in string format i.e (1h 50m,50m,3h etc). I need to know how to convert this column into an integer with the value displayed in minutes (110,50,180).

One liner:

df['Duration']=  df['Duration'].str.replace("h", '*60').str.replace(' ','+').str.replace('m','*1').apply(eval)

Basically, it converts your string into an equation in string format and evaluates it.

Hope it helps!


Difference between zero-padding and character-padding in Recurrent Neural Networks For RNN's to work efficiently we vectorize the problem which results in an input matrix of shape
    (m, max_seq_len)

where m is the number of examples, e.g. sentences, and max_seq_len is the maximum length that a sentence can have. Some examples have a smaller lengths than this max_seq_len. A solution is to pad these sentences.
One method to pad the sentences is called "zero-padding". This means that each sequence is padded by zeros. For example, given a vocabulary where each word is related to some index number, we can represent a sentence with length 4,
    "I am very confused"

by
    [23, 455, 234, 90]

Padding it to achieve a max_seq_len=7, we obtain a sentence represented by:
    [23, 455, 234, 90, 0, 0, 0]

The index 0 is not part of the vocabulary.
Another method to pad is to add a padding character, e.g. "<<pad>>", in our sentence:
    "I am very confused <<pad>>> <<pad>> <<pad>>"

to achieve the max_seq_len=7. We also add "<<pad>>" in our vocabulary. Let's say it's index is 1000. Then the sentence is represented by
    [23, 455, 234, 90, 1000, 1000, 1000]

I have seen both methods used, but why is one used over the other? Are there any advantages or disadvantages comparing zero-padding with character-padding?

If implemented properly, there should be no difference. The very first thing that happens with the indices is corresponding embeddings are loaded. From this perspective, there is no difference between having the pad embedding at the 0th or at the 1000th position.
When you use padding, you should always do masking on the output and other places where it is relevant (i.e., when computing the attention distribution in attention), which ensures no gradient gets propagated from the "non-existing" padding positions.


How to fairly rank the best items having two set of grades? My problem:

I have a set of items.
Two judges will subjectively rate each item giving a grade from 0 to 10.
I have to select the best 10 items.

How do I rank the items so that the judges are equally important?

Suppose one of them is really rigorous, and grade the items with a lower value, with a median of 5 and a standard deviation of 1.

The other median is 8, but his grades are more spread, with a standard deviation of 2.

Here are examples of the grades distributions:

The second judge has more spread grades, so if I just calculate the mean and rank, a lower grade of Judge Two would eliminate one item. Judge One grades would just order the best items of Judge Two. A favorite of Judge One would hardly get into the favorite rank.

Just to rank the items of each judge would make it more fair, but wouldn't take the grade difference between the items.

Procedure-1:

Convert them into percentage and then take average of outcome of both.

so for example for person A, judge-1 rated 5 out of 7 and judge-2 rated 7 out of 10,

converting judge-1 score to percentage i.e., $5/7 *100$ = 71%(rounded 71.45)

converting judge-2 score to percentage i.e., $7/10 *100$ = 70%

Now take average of both i.e., $71+70 / 2$ =  70.5%, now you can convert it back on to a scale from $1-10$ :  This person A's overall rating is 7(rounded 7.05 ) out of 10.

Procedure-2:

You can normalize the rating of both the judges by using this formula,

$$z_i=\frac{x_i-\min(x)}{\max(x)-\min(x)}$$

where $x=(x_1,...,x_n)$ and $z_i$ is now your $i^{th}$ normalized data.  As a proof of concept (although you did not ask for it)

Procedure-3:

You can convert the 0-7 rating to 0-10 and then take average of both.

Once you get the outcome then you can rank them accordingly.

Do let me know if you have any doubts.


Methods to reduce dimensionality within a feature? Suppose that I am interested in predicting an outcome (say, the arrival delay [in seconds] of a flight) based upon a set of features.
One of these features is a nominal variable - carrier - that specifies the airline carrier of the flight. This feature has 16 different values. After investigation of how arrival delay is distributed across each carrier, it appears that some carriers could be collapsed into one value (e.g., "AS" and "HA" or "WN" and "B6").

install.packages("nycflights13")
library(nycflights)

boxplot(
    formula = arr_delay ~ with(flights, reorder(carrier, -arr_delay, median, na.rm = TRUE)),
    data = flights,
    horizontal = TRUE,
    las = 2,
    plot = TRUE
)

In general, are there well-known methods for reducing the dimensionality within a feature?

You could try clustering the data with the carrier feature removed, add it back in, and look at the distributions of the carrier feature per cluster.


How to make sure that the features learned by a neural network are not correlated? Each layer of a neural network learns features of the input data. The first layer learns low-level features (e.g. edges in images). Each subsequent layer learns more abstract features. Then the features from the last hidden layer are combined for classification or regression.

How to make sure that the features learned by a neural network are not correlated?
I think that I can force a neural network to learn correlated features in the following way. Remove the output layer, so that the last hidden layer produces $n$ outputs. Then train this network on $n$ target variables that are not independent but are linear combinations of say two or three independent targets  (maybe with some noise added to avoid true linear dependence). As neural networks are universal approximators, it should be possible to train such a network. However, in that case, all neurons in the final layer will be redundant except those two or three neurons that correspond to the linearly independent targets.
This redundancy can be present in each layer. How to avoid it? Of course, I can check the correlation of neuron outputs after the training, remove the correlated neurons, and retrain the network but it looks inefficient. How to make the network learn uncorrelated features from the beginning?
What comes to my mind is using different and independent (e.g. orthogonal) activation functions for each neuron within the layer. Are there such networks? Usually, activation functions are taken to be the same.
EDIT: I tried to test if dropout can make learned features less correlated. I did not find that it does (or I am doing it in a wrong way?).
This is a toy problem that I tried: $y = a^2+b^2$ where $a$ and $b$ are uniformly distributed between 0 and 1.
a_train = np.random.uniform(low=0.0, high=1.0, size=10000)
b_train = np.random.uniform(low=0.0, high=1.0, size=10000)
y_train = a_train**2 + b_train**2
X_train = np.hstack([a_train[:,np.newaxis], b_train[:,np.newaxis]])

The input features are not correlated.
np.corrcoef(X_train[:,0], X_train[:,1])[0,1]

0.002136928470247475

Neural network:
model = keras.Sequential(
    [
        layers.Dense(32, activation="tanh", name="layer1", input_dim=2),
        layers.Dense(16, activation="tanh", name="layer2"),
        layers.Dense(2, activation="tanh", name="feature_layer"),
        layers.Dense(1, name="out_layer"),
    ]
)
optimizer = tf.keras.optimizers.RMSprop(learning_rate=0.001)

model.compile(loss='mean_squared_error',
                optimizer=optimizer,
                metrics=['mean_squared_error'])

model.fit(X_train, y_train, epochs=100, batch_size=128, validation_split = 0.2, verbose=1)

feature_extractor = keras.Model(
    inputs=model.inputs,
    outputs=model.get_layer(name="feature_layer").output,
)

features=feature_extractor(X_train).numpy()

The correlation between the outputs of the next-to-the-last layer:
np.corrcoef(features[:,0], features[:,1])[1,0]

-0.7527464552820806

After adding dropout,
model = keras.Sequential(
    [
        layers.Dense(32, activation="tanh", name="layer1", input_dim=2),
        layers.Dropout(0.2),
        layers.Dense(16, activation="tanh", name="layer2"),
        layers.Dropout(0.2),
        layers.Dense(2, activation="tanh", name="feature_layer"),
        layers.Dense(1, name="out_layer"),
    ]
)
model.summary()

The correlation becomes
-0.8645152290698691

Dropout (2014 paper) is my first thought.
By effectively removing N% of your neurons on each pass through the data, you make it harder for any two neurons to work together. When its buddy disappears it is forced to find another way to learn the patterns in the data. On the next epoch, when its old buddy comes back, it has a new perspective on life, its horizons have been broadened, and it is less likely to re-enter that co-dependent relationship.
The second thought was L2 regularization: it forces the network to not rely on just one neuron in a layer, but to get them all involved.
If you are not convinced, just try it. Try to create a network where, say, only two of the neurons in the final layer carry all the signal. Then re-train it using a dropout of 50%, or a higher L2 value. It may be a weaker model overall, but it is going to be really hard to not have all the neurons share the load.  (Also try it with and without the bias, as mentioned below.)

different and independent activation functions for each neuron within the layer.
Are there such networks?

I'd be interested to hear if you find anything. I suppose you could argue that the bias is creating a unique activation function on each neuron, though I think you are suggesting something like ReLU in some neurons and tanh in others? (A bit tricky to implement efficiently?)
A little bit removed from your question, but the different heads in multi-head attention (in the Transformer model) are another example of getting multiple different "activation functions" into a layer.


what arguments should I pass to dbscan or optic in order to divide the data in a specific way I have thousands of very similar datasets that needs to be divided in diagonal way to two groups.
for example:

and

I tried to play with the argument of dbscan and optic as epsilon and minPoints and even metric and none of them helped me to divide the data properly to 2 groups.

I only succeed to divide the data using dbscan. If I remove the noise between these groups to make them a complete separate 2 groups, I did it using histogram

j = 1
hist, bin_edges = np.histogram(data, bins=500)
max_bin = np.where(np.amax(hist) == hist)[0][0]
max_noise = bin_edges[max_bin+j]
filtered_indicies = data > max_noise
data = data[filtered_indicies]

these lines remove noise from the data, between the groups and also around it when

  j > 1

and that causing me to remove necessary data that I need to reprocess later.

so, I am going back the my main question, how can I know which epsilon, minPoints or other argument of dbscan can help me divide this data properly?
or is there maybe a better way then what I presented here above (histogram) to remove the noise between these groups without removing necessary data?

Start with what the parameters mean.

$\varepsilon$ is the search radius around each point. You need this search radius to be small enough that it can't fully "bridge the gap" between the clusters. If the gap width is variable, $\varepsilon$ needs to be small enough to accommodate the narrowest gap. Note that we can exclude the occasional straying point from consideration when determining the gap width as minPoints can handle a small amount of "bleed" into the gap. However, by inspecting the data you can see that making $\varepsilon$ smaller than the smallest gap would cause the points on the far right side to be excluded from the right cluster.

minPoints is the minimum number of points within $\varepsilon$ to point $x$ in cluster $c$ for $x$ to be included in $c$. When we determined the minimum gap width, we allowed a few points from each cluster to cross into the gap. minPoints must be large enough that those stray points don't bridge the clusters. Specifically, if there are $n$ points from cluster $c1$ within $\varepsilon$ of a point in cluster $c2$, minPoints must be greater than $n$. Note that like with $\varepsilon$, setting minPoints large enough to keep the primary clusters separate would cause some points on the edges to be excluded from these clusters.

DBSCAN can't cleanly cluster all points into the two clusters that you want because it connects using local connectivity, not global coherence. Even one bridge from one bunch of points to the other groups them into a single cluster. In order to separate the two clusters, there will be outliers. If outliers are unacceptable, you could join them to the two main clusters using a simple heuristic, such as joining to whichever cluster has a closer center to the outlier. If post-processing is unacceptable, you might want to use a global clustering method such as spectral clustering; you might also have success with mean shift.


cycle consistency loss explanation I was reading image to image translation papers.
Can someone explain to me in simple language what cycle consistency loss is and why it works in image to image traslation

In normal image GANs, the generator receives random noise and generates an image. As you can see, there is no "visual relationship" between the input and output.
In cycle GANs, generators receive an image and generate another image. The idea is that the input image is of domain A and the output image is a "translation" of the input image into domain B. However, with the experience with normal GANs, why would the "translated image" have any visual relationship with the original one? If the cycle consistency loss were not there, a generator could simply generate images in the target domain that were totally unrelated to the input and the discriminator would be Ok with it.
The role of the cycle consistency loss is to ensure that the generated output image is actually a version of the input image where the domain is what changes, but the "contents" are kept.


Convolution neural network with 11 million parameters unable to overfit on 100 image samples I have been trying to do some sort of image enhancement on grayscale images. I have used both pixel wise loss and perceptual loss (perceptual loss uses classifier between 2 classes trained on the same dataset).

So the input to my network is an bad image and the output is the enhanced image where edges in an image are clearer.

I have written the code in keras

inputs = Input(shape=image_shape)

x = Conv2D(filters=ngf, kernel_size=(7, 7), padding='same')(inputs)
x = BatchNormalization()(x)
x = LeakyReLU(0.2)(x)

n_downsampling = 2
for i in range(n_downsampling):
    mult = 2**i
    x = Conv2D(filters=ngf*mult*2, kernel_size=(3, 3), strides=2, padding='same')(x)
    x = BatchNormalization()(x)
    x = LeakyReLU(0.2)(x)

mult = 2**n_downsampling
for i in range(n_blocks_gen):
    x = res_block(x, ngf*mult, use_dropout=False)

for i in range(n_downsampling):
    mult = 2**(n_downsampling - i)
    x = Conv2DTranspose(filters=int(ngf * mult / 2), kernel_size=(3, 3), strides=2, padding='same')(x)
    x = BatchNormalization()(x)
    x = LeakyReLU(0.2)(x)

#x = ReflectionPadding2D((3, 3))(x)
x = Conv2D(filters=output_nc, kernel_size=(5, 5), padding='same')(x)
x = Conv2D(filters=output_nc, kernel_size=(3, 3))(x)
x = Activation('sigmoid')(x)

outputs = Add()([x, inputs])
#outputs = Lambda(lambda z: K.clip(z, -1, 1))(x)
outputs = Lambda(lambda z: z/2)(outputs)

model = Model(inputs=inputs, outputs=outputs, name='Generator')

Function for res_block

 def res_block(input, filters, kernel_size=(3, 3), strides=(1, 1),
 use_dropout=False):
"""
Instanciate a Keras Resnet Block using sequential API.

:param input: Input tensor
:param filters: Number of filters to use
:param kernel_size: Shape of the kernel for the convolution
:param strides: Shape of the strides for the convolution
:param use_dropout: Boolean value to determine the use of dropout
:return: Keras Model
"""
x = Conv2D(filters=filters,
           kernel_size=kernel_size,
           strides=strides,
           padding='same')(input)
x = BatchNormalization()(x)
x = Activation('relu')(x)

if use_dropout:
    x = Dropout(0.5)(x)

x = Conv2D(filters=filters,
           kernel_size=kernel_size,
           strides=strides,
           padding='same')(x)
x = BatchNormalization()(x)

merged = Add()([input, x])
return merged

Any idea why this wouldn't overfit ? I have been looking for other loss functions but I couldn't find much. I personally tried to make pixel loss penalize the model on pixels where it should have completed a incomplete line in an image by using sobel operator on the enhanced image to get a mask which is multiplied by the difference between the input & target image hoping that the network would focus on completing incomplete edges or so but nothing changed.

You use a sigmoid activation in the last layer, which restricts your possible outputs to be between 0 and 1. This is typically used for binary classification, but it seems like you are not doing classification at all. If your image has values constrained to the range 0-1 then this is likely fine. Otherwise, this could be the source of your problem.

I suggest removing the sigmoid activation in the final convolution and leaving it with no activation.


Difference between subgradient SVM and kernel SVM? What is the difference between subgradient svm and kernel svm?

From my understanding subgradient svm is a linear classifier that uses hinge loss and kernel svm  uses some kernel function for non linear classification. I was wondering, if subgradient is only a linear classifier, could I use a tree of non linear svms to separate non linear data? I would essentially do binary classification separate 1 class vs the rest then the child node of the tree would work on the rest and separate the next class from the rest and so on.  Any general idea or feedback would be great.

You are gravely misunderstanding SVM. Sub-gradient descent algorithm for SVM is a method to solve the underlying optimization problem of SVM.

SVM is always a linear classifier, which can yet by using kernels operate in a higher dimensional space. Therefore in input space, the separating hyperplane (linear!) computed in feature space (kernel!), seems non-linear.

Effectively you are thereby solving a non-linear classification task, but you are projecting into a higher dimensional feature space where the classification task is solved by a linear classifier.

Please read:

Wikipedia on svm

Please watch:

great lecture about SVM


Parameter initialization in a genetic algorithm I'm using a neural network in a genetic algorithm. The neural network has 4 inputs (values between 0 and 1) and 4 outputs, corresponding to the probabilities of different actions. The neural network has 58 parameters.

At first, I create a random population : each individual has 58 random parameters. Parameters are chosen randomly with the default method of keras, on python (values between -1 and 1). Is it the good method ? Maybe the best solution needs to have parameters with values higher than 1 for example, but with my method, only values between -1 and 1 exist in the "gene pool". So, a parameters equal to 3.4 can't appear for example.

I tried to train the same neural network with labeled data and gradient descent, in order to have an idea of parameters range. After training the model, I obtained some parameters with values >1, or <-1. I thought I could use those parameters as an initialization for my genetic algorithm. But how could I get different individuals ? If the 1st parameter of my trained model equals 2.5, do I have to set the 1st parameter of the different individuals to 2.5 +- 20% for example ?

In my opinion you should make the range as large as possible (to a reasonable extent) for the first random initialization.

The genetic algorithm will converge to the appropriate range eventually, but giving it a narrow range could result in a sub-optimal solution because the algorithm doesn't have any way to reach a better solution. The only downside of a large range is that it might take a bit longer (more generations) to converge.

So I would suggest you keep a completely random initialization of the values, for instance in the range [-10,10].


How to plot logistic regression decision boundary? I am running logistic regression on a small dataset which looks like this:

After implementing gradient descent and the cost function, I am getting a 100% accuracy in the prediction stage, However I want to be sure that everything is in order so I am trying to plot the decision boundary line which separates the two datasets.

Below I present plots showing the cost function and theta parameters. As can be seen, currently I am printing the decision boundary line incorrectly.

Extracting data

clear all; close all; clc;

alpha = 0.01;
num_iters = 1000;

%% Plotting data
x1 = linspace(0,3,50);
mqtrue = 5;
cqtrue = 30;
dat1 = mqtrue*x1+5*randn(1,50);

x2 = linspace(7,10,50);
dat2 = mqtrue*x2 + (cqtrue + 5*randn(1,50));

x = [x1 x2]'; % X

subplot(2,2,1);
dat = [dat1 dat2]'; % Y

scatter(x1, dat1); hold on;
scatter(x2, dat2, '*'); hold on;
classdata = (dat>40);

Computing Cost, Gradient and plotting

%  Setup the data matrix appropriately, and add ones for the intercept term
[m, n] = size(x);

% Add intercept term to x and X_test
x = [ones(m, 1) x];

% Initialize fitting parameters
theta = zeros(n + 1, 1);
%initial_theta = [0.2; 0.2];

J_history = zeros(num_iters, 1);

plot_x = [min(x(:,2))-2,  max(x(:,2))+2]

for iter = 1:num_iters
% Compute and display initial cost and gradient
    [cost, grad] = logistic_costFunction(theta, x, classdata);
    theta = theta - alpha * grad;
    J_history(iter) = cost;

    fprintf('Iteration #%d - Cost = %d... \r\n',iter, cost);

    subplot(2,2,2);
    hold on; grid on;
    plot(iter, J_history(iter), '.r');  title(sprintf('Plot of cost against number of iterations. Cost is %g',J_history(iter)));
    xlabel('Iterations')
    ylabel('MSE')
    drawnow

    subplot(2,2,3);
    grid on;
    plot3(theta(1), theta(2), J_history(iter),'o')
    title(sprintf('Tita0 = %g, Tita1=%g', theta(1), theta(2)))
    xlabel('Tita0')
    ylabel('Tita1')
    zlabel('Cost')
    hold on;
    drawnow

    subplot(2,2,1);
    grid on;
    % Calculate the decision boundary line
    plot_y = theta(2).*plot_x + theta(1);  % <--- Boundary line
    % Plot, and adjust axes for better viewing
    plot(plot_x, plot_y)
    hold on;
    drawnow

end

fprintf('Cost at initial theta (zeros): %f\n', cost);
fprintf('Gradient at initial theta (zeros): \n');
fprintf(' %f \n', grad);

The above code is implementing gradient descent correctly (I think) but I am still unable to show the boundary line plot. Any suggestions would be appreciated.

logistic_costFunction.m

function [J, grad] = logistic_costFunction(theta, X, y)

    % Initialize some useful values
    m = length(y); % number of training examples

    grad = zeros(size(theta));

    h = sigmoid(X * theta);
    J = -(1 / m) * sum( (y .* log(h)) + ((1 - y) .* log(1 - h)) );

    for i = 1 : size(theta, 1)
        grad(i) = (1 / m) * sum( (h - y) .* X(:, i) );
    end

end

EDIT:

As per the below answer by @Esmailian, now I have something like this:

[m, n] = size(x);

x1_class = [ones(m, 1) x1' dat1'];
x2_class = [ones(m, 1) x2' dat2'];

x = [x1_class ; x2_class]

Your decision boundary is a surface in 3D as your points are in 2D.

With Wolfram Language

Create the data sets.

mqtrue = 5;
cqtrue = 30;
With[{x = Subdivide[0, 3, 50]},
  dat1 = Transpose@{x, mqtrue x + 5 RandomReal[1, Length@x]};
  ];
With[{x = Subdivide[7, 10, 50]},
  dat2 = Transpose@{x, mqtrue x + cqtrue + 5 RandomReal[1, Length@x]};
  ];

View in 2D (ListPlot) and the 3D (ListPointPlot3D) regression space.

ListPlot[{dat1, dat2}, PlotMarkers -> "OpenMarkers", PlotTheme -> "Detailed"]

I Append the response variable to the data.

datPlot =
 ListPointPlot3D[
  MapThread[Append, {#, Boole@Thread[#[[All, 2]] > 40]}] & /@ {dat1, dat2}
  ]

Perform a Logistic regression (LogitModelFit). You could use GeneralizedLinearModelFit with ExponentialFamily set to "Binomial" as well.

With[{dat = Join[dat1, dat2]},
 model =
  LogitModelFit[
   MapThread[Append, {dat, Boole@Thread[dat[[All, 2]] > 40]}],
   {x, y}, {x, y}]
 ]

From the FittedModel "Properties" we need "Function".

model["Properties"]

  {AdjustedLikelihoodRatioIndex,    DevianceTableDeviances, ParameterConfidenceIntervalTableEntries,
  AIC,    DevianceTableEntries,   ParameterConfidenceRegion,
  AnscombeResiduals,  DevianceTableResidualDegreesOfFreedom,  ParameterErrors,
  BasisFunctions, DevianceTableResidualDeviances, ParameterPValues,
  BestFit,    EfronPseudoRSquared,    ParameterTable,
  BestFitParameters,  EstimatedDispersion,    ParameterTableEntries,
  BIC,    FitResiduals,   ParameterZStatistics,
  CookDistances,  Function,   PearsonChiSquare,
  CorrelationMatrix,  HatDiagonal,    PearsonResiduals,
  CovarianceMatrix,   LikelihoodRatioIndex,   PredictedResponse,
  CoxSnellPseudoRSquared, LikelihoodRatioStatistic,   Properties,
  CraggUhlerPseudoRSquared,   LikelihoodResiduals,    ResidualDeviance,
  Data,   LinearPredictor,    ResidualDegreesOfFreedom,
  DesignMatrix,   LogLikelihood,  Response,
  DevianceResiduals,  NullDeviance,   StandardizedDevianceResiduals,
  Deviances,  NullDegreesOfFreedom,   StandardizedPearsonResiduals,
  DevianceTable,  ParameterConfidenceIntervals,   WorkingResiduals,
  DevianceTableDegreesOfFreedom,  ParameterConfidenceIntervalTable}

model["Function"]

Use this for prediction

model["Function"][8, 54]

0.0196842

and plot the decision boundary surface in 3D along with the data (datPlot) using Show and Plot3D

modelPlot =
 Show[
  datPlot,
  Plot3D[
   model["Function"][x, y],
   Evaluate[
    Sequence @@
     MapThread[Prepend, {MinMax /@ Transpose@Join[dat1, dat2], {x, y}}]],
   Mesh -> None,
   PlotStyle -> Opacity[.25, Green],
   PlotPoints -> 30
   ]
  ]

With ParametricPlot3D and Manipulate you can examine decision boundary curves for values of the variables. For example, keeping x fixed and letting y vary or vice versa.

Manipulate[
 Show[
  modelPlot,
  ParametricPlot3D[
   {x, u, model["Function"][x, u]}, {u, 0, 80}, PlotStyle -> Orange],
  ParametricPlot3D[
   {u, y, model["Function"][u, y]}, {u, 0, 10}, PlotStyle -> Purple],
  PlotLabel ->
   StringTemplate["model[`1`, `2`] = `3`"] @@ {x, y, model["Function"][x, y]}
  ],
 {{x, 6, Style["x", Orange, Bold]}, 0, 10, Appearance -> "Labeled"},
 {{y, 40, Style["y", Purple, Bold]}, 0, 80, Appearance -> "Labeled"}
 ]

Update

You can also plot contours of the probability in 2D.

plot = ListPlot[{dat1, dat2}, PlotMarkers -> "OpenMarkers", PlotTheme -> "Detailed"];

Manipulate[
 db = y /. First@Quiet@Solve[model["Function"][x, y] == p, y];
 Show[
  plot,
  Plot[db, {x, 0, 10}, PlotStyle -> Red]
  ],
 {{p, .5}, 0, 1, Appearance -> "Labeled"}
 ]

Hope this helps.


Character-based word representation using bi-lstm In this blog, it teaches us how to get a word embedding using bi-lstm in character level like the image below:

I am wondering how to optimize the word vector from character-based approach. Is there any loss function or any strategy to update or optimize the word vector trained by a word's characters?

This would generally only be a part of a network, used for some task like classification, entity recognition or machine translation. The loss used for this task will guide the learning of the embedding space. This is different from approaches like word2vec or GLoVe where you use context windows to learn an embedding, which is an unsupervised (or maybe self-supervised) approach.

The approach that you mention here could still be used in these scenarios, by structuring it with a similar loss as word2vec or GLoVe. One advantage would be that you can leverage word similarity (words that end with -ly have a similar function). However these models are usually trained on incredibly large corpora where these character based similarities are not that relevant, and the computational cost increases quite a bit by incorporating bigger models like LSTMs.