Decoder Transformer feedforward I have a question about the decoder transformer feed forward during training.
Let's pick an example: input data "i love the sun" traduction i want to predict (italian traduction) "io amo il sole".
Now i feed the encoder with the input "i love the sun" and i get the hidden states.
Now i have to do multiple feed forwards on the decoder with the input "BOS io amo il"
where BOS is a token that stands for beginning of sentence.
So i have this feedforward i assume

[BOS, IO, AMO, IL] -> decoder -> IO
[BOS, IO, AMO, IL] -> decoder -> AMO
[BOS, IO, AMO, IL] -> decoder -> IL
[BOS, IO, AMO, IL] -> decoder -> SOLE

I think this is the correct way. And what should be applied to differentiate the training i think is the masked attention mechanism maybe(?)
is it right to assume that the masking will be
[1 0 0 0,
 0 0 0 0 ,
 0 0 0 0,
 0 0 0 0]   for the first feed forward

[1 0 0 0,
 1 1 0 0 ,
 0 0 0 0,
 0 0 0 0]   for the second feed forward

[1 0 0 0,
 1 1 0 0 ,
 1 1 1 0,
 0 0 0 0]   for the third feed forward

[1 0 0 0,
 1 1 0 0 ,
 1 1 1 0,
 1 1 1 1]   for the fourth feed forward

is it the correct way? or what should be different?
If you can provide me also a python implementation could be useful, thanks in advance.

There are some problems with your description:

During training, the decoder receives all the shifted target tokens, prepending the BOS token. You removed sole. The actual input would be: [<bos>, io, amo, il, sole]. Note that the output at the position of sole would be the end-of-sequence token <eos>.

During training, there is a single forward pass (not one per token), and all the output tokens are predicted at once. Therefore, only the last one of your attention masks is used.

During inference, we don't have the target tokens (because that is what we are trying to predict). In this case, we have one pass per generated token, starting with <bos>. This way, the decoder input in the first step would just be the sequence [<bos>], and we would predict the first token: io. Then, we would prepare the input for the next timestep as [<bos>, io], and then we would obtain the prediction for the second token. And so on. Note that, at each timestep, we are repeating the computations for the past positions; in real implementations, these states are cached instead of re-computed each timestep.

About some piece of Python code illustrating how the Transformer works, I suggest The annotated Transformer, which is a nice guide through a real implementation. You may be most interested in the function run_epoch for the training and in the function greedy_decode for the inference.


Is it necessary to sort time series data on datetime stamps? Say I have a time-series data frame for a bank that records number of cases filed for 16 districts as below (cases can be for many purposes - loans, credit cards, real estate, etc.):

application_date    branch_id    district    case_count
2017-04-01              1        A           40.0
2017-04-03              1        A           5.0
2017-04-04              1        A           4.0
2017-04-05              1        A           113.0
2017-04-07              1        A           76.0

But if I sort the above data by application_date, like so:

application_date    branch_id    district    case_count
2017-04-01              1         A           40.0
2017-04-01              62        B           5.0
2017-04-01              63        C           4.0
2017-04-01              64        D           113.0
2017-04-01              65        E           76.0

Considering that branch_id and district are categorical variables, does sorting affect performance of standard ML models like random Forests and gradient boosted models as well as their predictive accuracy?

Sorting data won't affect the training of your model, it is similar to changing the random seed.

It can affect the validation that you are doing. In case you do time series you can do sliding window or roll-out-window, that they need the data to be sorted before the splitting.

It seems that you want to do time series regression with supervised learning so what you probably want to do is change your problem to a supervised learning problem. You can see a bit more here. But this is me guessing what you wnat from a couple lines.


Why is averaging the vectors required in word2vec? While implementing word2vec using gensim by following few tutorials online, one thing that I couldn't understand is the reason why word vectors are averaged once the model is trained. Few example links below.

https://www.kaggle.com/ananyabioinfo/text-classification-using-word2vec/notebook?scriptVersionId=11358361&cellId=7

https://www.kaggle.com/varun08/sentiment-analysis-using-word2vec?scriptVersionId=2185653&cellId=15

My questions are:

Is it just to create a single vector instead of vectors of the dimension size or to increase the accuracy or is there any reason behind this?
Is it mandatory to take average of the vectors or are there any alternatives to this.

I have gone through the original paper on word2vec but that doesn't give clear explanation on this.

The reason to average the embedded vectors of the words in a paragraph or document is to obtain a single fixed-size vector that represents the whole text. Then, the document-level vector can be used as input to a document classification model or any other document-level model.
If you explicitly want to compute word-level representations and then combine them into a document/paragraph-level representation, then averaging is the standard approach.
On the other hand, to obtain document/paragraph/sentence-level representations in general, there are many alternatives to combining word-level vectors. Some remarkable examples include doc2vec for paragraph/document-level, or LASER or BERT for sentence-level representations.


Spatial Data Mining and ATM banking transactions I have the following dataset. This dataset contains ATM banking transactions in different places over 30 days. These ATMs are located in different cities and provinces. The problem that I need to solve is to develop a model that can be able to detect which ATMs are in the same city or  in the same province. The only thing I know about this problem is that is related to Spatial Data Mining but I have no idea which techniques, algorithms, and tools I should use to solve this problem.

In this dataset PS ID column corresponds to the ATM IDs, AMOUNT column corresponds to the amount of money transferred.

My question is which techniques, algorithms, and tools I should use to solve this problem???

Should I use clustering or I should use other techniques?

This sounds like an unsupervised learning problem since you are trying to group observations according to some common association rather than trying to predict a target. My first impression is that you are facing an uphill battle here as your data set doesn't look comprehensive enough. As a starting point, this is what I would try:

You could assume that ATMs in the same time zone would exhibit similar seasonality patterns on both daily and weekly scales. For example, on a typical weekday, there would probably be a consistent pattern of an up-tick in usage around say 8am, probably peaking during the lunch break with another peak following the end of the workday (these are just assumptions, I haven't done any analysis on this). So you could convert your TIME column into a factor with 24 levels (one for each hour of the day), but you would need to ensure that the times reported are in some consistent time zone, as opposed to the local time of the ATM from which they come which would of course not make much sense.

That approach may allow you to cluster on the basis of time zone, but what about city? Well, if you assume that different cities have different levels of affluence, the AMOUNT column might yield some useful associations.

So in summary, I'd try a clustering algorithm based on seasonal usage patterns and the value of the transactions. I'd be interested to hear if you can get any useful results from this as my feeling is that your data set probably doesn't have enough useful features to be effective.


Name of this type of cross variable interaction Plot In recent papers of Social Network Analysis , I have been observing this particular type of graph used increasingly. I looked for what they are called to replicate the graph, but couldn't find the name of it?

Plot Reference is here:

Bots increase exposure to negative and
inflammatory content in online social systems


How to apply multiple clustering algorithms to same dataset and make comparison? I've a dataset and I want to implement K-Means, Fuzzy C Means, Gaussian Mixture Model, Spectral Graph. After that, I want to see the clusters that I get from different methods. What is the proper way to do that? Or should I only stick one algorithm and try to maximize correctness of that clustering?

Have a look at the scikit learn API, they have a section dedicated to this topic.
If you want to go further, have a look at chapter 3 "Unsupervised Learning and Preprocessing" of the book Introduction to Machine Learning with Python: A Guide for Data Scientists, there is a section on clustering methods (they also mention the evaluation of clustering methods which is very insightful)


Preprocess list data I got question about preparation data for my ML algorithm.
Raw data has format similar to:

{
  "finances": [
    {
      "assets": 1230.39,
      "investments": 3245.39,
      "netProfit": 8765.45,
      "year": 2017
    },
    {
      "assets": 111.11,
      "investments": 222.22,
      "netProfit": 333.33,
      "year": 2016
    },
    {
      "assets": 1111.11,
      "investments": 2222.22,
      "netProfit": 3333.33,
      "year": 2015
    }
  ],
  "someValue": 123.45,
  "title": "Hello!"
}

And I am wondering what is best way to pass data about finances to my algorithm (Number of years can differ from 0 to 8).

I was thinking about making every parameter for every year new attribute, but this would lead to have many not null values for some cases, because not every record has 8 years of financial history.

My question is: What is best way to handle such data in Input data (In fact every of finances elements would have more values in it (around 10-15).

The ultimate end goal of your modeling is going to affect the way you want to format your data. It's a good practice whenever you start a machine learning project to ask yourself, what is the precise question you want to answer, because whatever model you generate, it's only going to make sense if used in the context of the question asked.

If in your case you want to predict loan approvals, then first you need to check whether that information is even present in the data you have. The JSON you have just shows historical data, but do you know the outcome for each data point? Is that another feature, maybe recorded in the "someValue", "title" area of the file? Without it, you really can't do anything.

If you can get that information, then it's perfectly fine to generate a variable for each year: assets_year1, assets_year2, ... assets_year8, investments_year1, ... etc. True, year8 might have a lot of Null values, but that's not necessarily bad. For example, most models working with text data consist of really sparse training matrices, yet they do very well in practice. Depending on the algorithm, it might weed out those variables anyways. With Null values, you just have to try imputing the Null values differently, and you can also create an indicator variable for whether the column has missing values, and see how different settings change performance.

To take it a step further, you can also create additional features to encapsulate the nuances that are occurring over the years. Create features like the historical average, average change per year, total number of years of history, etc. Creative feature engineering is the hardest part, but can lead to big changes in performance.


Understanding the filter function in Convolution Neural Networks I am trying to follow the following tutorial accessible with this link.
Under the 3rd Heading, "3. Visualize the Activation Maps for Each Filter", we can see the following function:
def apply_filter(img, index, filter_list, ax):
    # set the weights of the filter in the convolutional layer to filter_list[i]
    model.layers[0].set_weights([np.reshape(filter_list[i], (4,4,1,1)), np.array([0])])
    # plot the corresponding activation map
    ax.imshow(np.squeeze(model.predict(np.reshape(img, (1, img.shape[0], img.shape[1], 1)))), cmap='gray')

I understood what they are trying to do. They are applying the filters and trying to show the output after that. But, what I don't understand is the following line:
model.layers[0].set_weights([np.reshape(filter_list[i], (4,4,1,1)), np.array([0])])

What does it mean to assign weights here and also, why are they reshaping the filter which is of 4*4 to 4*4*1*1?

The function set_weights on a Keras layer requires the shape of the inputs to match the shape of the weights which you are replacing. You can find out which dimensions these are by calling the get_weights method on the layer in which you are interested.

In that tutorial, it would look something like the following. We only need the first element returned by get_weights(), hence the [0]. We then see the shape of it:

In [7]: model.layers[0].get_weights[0].shape
(4, 4, 1, 1)

So the person who wrote that tutorial needed to match that shape and, as it is only considering one layer of the defined model, it is able to be hard-coded in their example.

Here are the docstrings for the two main functions I mentioned above:

set_weights()

In [8]: l1.set_weights?

Signature: l1.set_weights(weights)
Docstring:
Sets the weights of the layer, from Numpy arrays.

# Arguments
    weights: a list of Numpy arrays. The number
        of arrays and their shape must match
        number of the dimensions of the weights
        of the layer (i.e. it should match the
        output of `get_weights`).

# Raises
    ValueError: If the provided weights list does not match the
        layer's specifications.

get_weights()

In [9]: l1.get_weights?
Signature: l1.get_weights()
Docstring:
Returns the current weights of the layer.

# Returns
    Weights values as a list of numpy arrays


Latent variables with thresholds There are many ML techniques to estimate latent variables such as the EM algorithm. Is there a technique that allows for thresholds for each of the latent variables?

I have a feature space with 10 variables $(X_1,\dots,X_{10})$ and the outcome $Y$. 7 of the $X$ features are known (I have their observations) and 3 are unknown. Each of the unknown can be within a range from 0 up to a positive constant number.

What ML technique would you recommend for estimating the above latent variables with the setup described above?

re. "estimate latent variables"

Quantities that are trained in order to fit a "best" model within a family of models are called hyper-parameters.  To any instance of the model they are fixed.  To the optimisation routine they are an index into search space.  Adding constraints on the range of a hyper-parameter both reduces the search space of the optimisation and requires extra "feasibility" checks during typical gradient descent.

A variable is "latent" when it is purely internal to the model, i.e. not an observable.  The meaning of its scale would depend on the context and on your interpretation, since it cannot be compared to anything observed.  You rarely want to constrain that range inside the model.

I would suggest leaving the hyper-parameters and latent variables unconstrained and if you want to read an output train a "neuron"-like response to get what you want out : e.g. sigmoid / tanh / softmax


Are CNNs indeed translation invariant? I read that in a hidden layer of a convolutional neural network, all neurons share the same weights and bias. As a result, all the neurons detect the same feature and hence ConvNets become invariant to translation.

And then I read that in 2003 Simard, Steinkraus and Platt achieved better classification in MNIST classification by artificially expanding the data using translation among other operations.

So, my question is are ConvNets really invariant to translation?

Short answer - No, CNNs are not really translation invariant.

I specifically mean the style of image classification network in the paper you mentioned (i.e., (input) > (conv layer) > ...  > (conv layer) > (fully conn layer) ... > (fully conn layer) > (output)).

This is partly because of the difference between translation invariance and translation equivariance (an important distinction, imo). See this question and the answers

Why

The last few layers in the network are fully connected (FC).  FC layers are definitely not translation invariant (they don't give consideration to the spatial relationship between the inputs).  But, the whole network could still be translation invariant if everything before the first FC layer is translation invariant.

We have convolution layers before the FC layers, but convolution layers are translation equivariant, not translation invariant (see below, it'll make sense).  Therefore the whole network is not (really) translation invariant.

Adding pooling layers after convolution makes the network invariant to small translation motions.

Convolution operations are translation equivariant

Translation equivariance of a function means that it's output for a translated version of the input is a translated version of the output.

if $$f(x(i)) = y(i)$$
then $$f(x(i-t)) = y(i-t)$$
where $i$ is the spatial index

Compare to translation invariance

Translation invariance means that the output of a translated version of the intput, is exactly the same as the output for the original input.

if $$f(x(i)) = y(i)$$
then $$f(x(i-t)) = y(i)$$
where $i$ is the spatial index


How do I encode time in high dimensional space? I have a dataset of form text, text, category, category, time, text and I would like to apply the attention mechanism to it. This requires that all inputs be in the same vector space. I am using a particular encoding method (from BERT) for the text-type data and I can build a custom trainable embedding for the category features. However, I don't have a good way of embedding time data.

Currently, my time feature is normalised on [0,1], and represents when over the time period (one year) the post was created. Naively, I would split this up into month, day of week, and time of day features to do feature engineering, but I don't have a good way to embed this in very high dimensional space (say, 500+ dimensional space).

What's the best option here? I would like to avoid tiling or repeating the same feature set to reach the requisite dimensionality -- is there a better way? I could put a trainable embedding layer on top of those three features, too, but this seems suboptimal.

I don't know if it's a good technique or not, but I saw this paper last year that proposes a time2vec, analogous to doc2vec, et al:
S. M. Kazemi et al., “Time2Vec: Learning a Vector Representation of Time,” arXiv:1907.05321 [cs], Jul. 2019.


What is the tag mapping for entity recognition in nltk? When doing entity recognition using NLTK, one gets as a result a Tree with a bunch of words mapped to tags (eg. Mark -> NNP, first -> JJ, ...). It's not at all clear what all the tags stand for at first glance and I was unable to find any documentation about these tags in the NLTK docs.

>>> from nltk import word_tokenize, pos_tag, ne_chunk
>>> sentence = "Mark and John are the first to work at Google from one years old in 39 years."
>>> print ne_chunk(pos_tag(word_tokenize(sentence)))
(S
  (PERSON Mark/NNP)
  and/CC
  (PERSON John/NNP)
  are/VBP
  the/DT
  first/JJ
  to/TO
  work/VB
  at/IN
  (ORGANIZATION Google/NNP)
  from/IN
  one/CD
  years/NNS
  old/JJ
  in/IN
  39/CD
  years/NNS
  ./.)

I ended up looking into the source code to get the mapping. Posting in case anyone else runs into the same problem.

Tag mapping according to nltk source

  'CC':   'Coordinating conjunction',   'PRP$': 'Possessive pronoun',
  'CD':   'Cardinal number',            'RB':   'Adverb',
  'DT':   'Determiner',                 'RBR':  'Adverb, comparative',
  'EX':   'Existential there',          'RBS':  'Adverb, superlative',
  'FW':   'Foreign word',               'RP':   'Particle',
  'JJ':   'Adjective',                  'TO':   'to',
  'JJR':  'Adjective, comparative',     'UH':   'Interjection',
  'JJS':  'Adjective, superlative',     'VB':   'Verb, base form',
  'LS':   'List item marker',           'VBD':  'Verb, past tense',
  'MD':   'Modal',                      'NNS':  'Noun, plural',
  'NN':   'Noun, singular or masps',    'VBN':  'Verb, past participle',
  'VBZ':  'Verb,3rd ps. sing. present', 'NNP':  'Proper noun, singular',
  'NNPS': 'Proper noun plural',         'WDT':  'wh-determiner',
  'PDT':  'Predeterminer',              'WP':   'wh-pronoun',
  'POS':  'Possessive ending',          'WP$':  'Possessive wh-pronoun',
  'PRP':  'Personal pronoun',           'WRB':  'wh-adverb',
  '(':    'open parenthesis',           ')':    'close parenthesis',
  '``':   'open quote',                 ',':    'comma',
  "''":   'close quote',                '.':    'period',
  '#':    'pound sign (currency marker)',
  '$':    'dollar sign (currency marker)',
  'IN':   'Preposition/subord. conjunction',
  'SYM':  'Symbol (mathematical or scientific)',
  'VBG':  'Verb, gerund/present participle',
  'VBP':  'Verb, non-3rd ps. sing. present',
  ':':    'colon',


Keras P/R metrics at different thresholds during training I'm training a binary classifier and I'd like to see Precision/Recall metrics at different thresholds.
Tensorflow 2.3 introduced tf.keras.metrics.Precision and tf.keras.metrics.Recall which take a thresholds parameter, where you can specify one or multiple thresholds for which you want the metrics computed. This all works as advertised i.e.
m = tf.keras.metrics.Precision(thresholds=[0.1, 0.2, 0.3, 0.4, 0.5, 0.6])
m.update_state([0, 1, 0, 1], [0.4, 0.5, 0.3, 0.8])
m.result().numpy()

Returns the precision value at each threshold [0.5, 0.5, 0.6666667, 1., 1., 1.], as per the documentation.
However when passed as metrics to Model.compile I get a single metric regardless of how many thresholds I have.

pr_thresholds = list(np.arange(0.05, 0.95, 0.05))
model.compile(
    'adam',
    'binary_crossentropy',
    metrics=[
        keras.metrics.Precision(thresholds=pr_thresholds),
        keras.metrics.Recall(thresholds=pr_thresholds),
    ]
)

I get
Epoch 34/50
395/395 [==============================] - 22s 54ms/step - loss: 0.4314 - precision: 0.7886 - recall: 0.9008 - val_loss: 0.5113 - val_precision: 0.7434 - val_recall: 0.8769

What's happening here ? Does it always use the default threshold value of 0.5 in this case ?
Is there a way I can get it to display the values for multiple thresholds during training ?

You can see the metrics value for each threshold along the fitting process if you explicitely instantiate the corresponding metric class for each threshold, as follows:
model.compile(
optimizer=keras.optimizers.Adam(learning_rate=1e-2),
loss='categorical_crossentropy',
metrics=[metrics.Recall(thresholds=0.6),
         metrics.Recall(thresholds=0.9)])

model.fit(X_train, y_train, epochs=10, validation_data=(X_test, y_test))

and as you can see in the image below, for each epoch you can see that the first recall value (with threshold 0.6) is higher than the second one (threshold 0.9) as expected:

And for your case, to build the list of metrics objects programatically, where you can now see 3 recalls per epoch:
thresholds = [0.6, 0.7, 0.9]
metrics_objs_list=[metrics.Recall(thresholds=thr) for thr in thresholds]


Why does my Keras Conv Net only return 1? model = Sequential()
   model.add(Conv2D(32, (3, 3), activation='relu', input_shape=(img_h_w,img_h_w,1)))
   model.add(Conv2D(32, (3, 3), activation='relu'))
   model.add(MaxPooling2D(pool_size=(2,2)))
   model.add(Dropout(0.25))
   model.add(Flatten())
   model.add(Dense(128, activation='relu'))
   model.add(Dropout(0.5))
   model.add(Dense(1, activation='softmax'))
   model.compile(loss='binary_crossentropy', optimizer=keras.optimizers.SGD(lr=0.01),metrics=['accuracy'])
   model.fit(x_train, y_train, batch_size=100, epochs=3, verbose=1)
   score = model.evaluate(x_test, y_test, verbose=0)

Running this returns

2500/2500 [==============================] - 5s 2ms/step - loss: 7.5694 - acc: 0.5252
Epoch 2/3
2500/2500 [==============================] - 5s 2ms/step - loss: 7.5694 - acc: 0.5252
Epoch 3/3
2500/2500 [==============================] - 6s 2ms/step - loss: 7.5694 - acc: 0.5252

Which shows the accuracy is at 50%, and when I look at the predictions made, I see it only outputs 1. It is meant to classify images into 0 or 1.

It is only outputting 1 because softmax makes no sense when given a single input. Softmax computed with only one input $x$ is equivalent to $\frac{e^x}{e^x} = 1$.

In other words, the line

model.add(Dense(1, activation='softmax'))

is wrong. Use sigmoid instead:

model.add(Dense(1, activation='sigmoid'))

or read more about one-hot encoding your expected output and the helper function https://keras.io/utils/#to_categorical.


A metric between trees I have certain tree structures. I am not an expert in machine learning.

As I would with take KNN, I would calculate distances via metric function and a new data point and the points from the training set. Okay, understood.

Now let's consider the data points aren't tuples but trees. How would one calculate the distances between two graphs.

For instance, I want the following

I have several stereotypical trees like:

|
 - <form class="xyz">
   |
    - <input type="text">
    - <button value="Click here">

between all these node, there could be arbitrary other nodes, the value of the button-node could be different, the type of the input node could be different, even the button-node could be a different node type, like input, the form-node will be surrounded by arbitrary nodes.

How would one calculate such a distance between trees?

The distance between objects of complex structure can be tricky. Consider a simpler example, when you got two sequences of words. How do you calculate the distance? In speech recognition, the distance between the groundtruth and recognized transcriptions is calculated as the word-error-rate based on the Levenstein distance between the aligned sentences.

GRAAL is an extension of the alignment algorithm to the networks. However, it only considers the network topology. You might want to extend the node similarity degree to encounter the type and attributes of the nodes.


How to explain the connection between the input layer and H1 of this CNN Architecture? I am currently reading the paper proposed by LeCun et al. for handwritten zip code recognition. There is this figure below visualizing the CNN architecture. But I do not really understand how the connection between Layer H1 and input layer makes sense. If there are 12 kernels with size 5x5, shouldn't the layer H1 be 12x144? Or is there any downsampling taking place here too?

Yes, the spatial dimensions (height and width) are reduced: the input is 16x16, H1 is 8x8 and H2 is 4x4.
Also see the first paragraph in the architecture section:

Source
In modern terms you would say that they use a stride of 2. Which reduces the spatial dimensions accordingly.
EDIT (based on your comment)
The formula for the spatial output dimension $O$ of a (square shaped) convolutional layer is the following:
$$O = \frac{I - K + 2P}S + 1$$ with $I$ being the input size, $K$ being the kernel size, $P$ the padding and $S$ the stride. Now you might think that in your example $O = \frac{16 - 5 + 2*2}2 + 1 = 8.5$ (assuming $P=2$)
But take a closer look at how it actually plays out when the 5x5 kernel of layer H1 scans the 16x16 input image with a stride of 2:

As you can see from the light grey area the required and effective padding is actually not 2 on all sides. Instead for the width or height respectively it is 2 on one side and 1 on the other side, i.e. on average $(2+1)/2=1.5$.
And if you plug that into the equation to calculate the output size it gives: $O = \frac{16 - 5 + 2*1.5}2 + 1 = 8$. Accordingly the convolutional layer H1 will have spatial dimensions of 8x8.


Using iGraph to build a Distribution Model I would like to analyze the distribution of the Customers from a Shop, if the Shop is closed or terminated. Consider the following sample data-set;
| ShopID | MonthlyCVisitCount |   Lat     |     Lng    |
--------------------------------------------------------
|  A1    |      15000         |  39.84349 |  116.33986 |
|  A2    |      24560         |  39.84441 |  116.33995 |
|  A3    |      14789         |  39.84615 |  116.34012 |
|  A4    |      35479         |  39.84891 |  116.34039 |

I would like to build a distribution model using NetworkX or iGraph based on the distance (i.e., lat, lng) and determine how the distribution (MonthlyCVisitCount) would spread if I supposedly close the shop A1.
Is it possible to obtain the shortest path from A1 to A2, A3, A4 and based on some business rules, I can analyze the distribution.
Edit 1:
Consider the modified dataset;
| PrimaryShopID | ToShopID |  Dist |
------------------------------------
|     A1        |    A2    | 0.125 |
|     A1        |    A3    | 0.354 |
|     A1        |    A4    | 0.950 |

The dataset above has distance between ShopID A1 to multiple shops, now that I have distance calculated and based on this distance I provide weights, say;

0 to 200 - 50%
201 to 500 - 20%
501 & above - 0%
Meaning that 30% of Customers I have lost.

How do I proceed with this scenario, I am mostly comfortable in working with R, any example or a pointer to the same problem will be helpful

NetworkX does have the ability to obtain shortest paths between points for you. As for how you can encode business rules, NetworkX does allow you to manipulate graph objects easily so you could code up a function containing your rules and apply it to to the graph object.

To model the distribution shift you could give weightings to each shop that would determine how much of the monthly visits they acquire when one of their neighbors close.


Give Variable Length input to LSTM My input data consist of list of list. Both list have dynamic length for every example like below.

X[0] = [[0, 1, 3, 5, 8, 26], [3, 4, 5, 7, 9, 15, 26, 30, 38, 39, 40]]
X[1] = [[1, 2, 4, 16, 19, 29, 48]]
..

My target data consist of dynamic length list like below:

y[0] = [5, 7, 8, 12, 15, 17, 29]
y[1] = [2, 4, 6, 8, 10, 11, 16, 18, 19, 29, 30, 33, 35]
..

I try to train LSTM model with given input X, predict Y.

If I concat each X list to one list and make padding operation LSTM accepts that data type. My example code is below:

X_train = sequence.pad_sequences(X_train, maxlen=padding_size)
X_test = sequence.pad_sequences(X_test, maxlen=padding_size)

model = Sequential()
model.add(Embedding(50, 10, input_length=X_train.shape[1], mask_zero=True))

if isBidirectional:
    model.add(Bidirectional(LSTM(lstm_layer_number)))
else:
    model.add(LSTM(lstm_layer_number))

if isDropout:
    model.add(Dropout(0.5))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy']) # binary crossentropy yields the best results

The problem is that I don't want to give input as a fixed size vector. Instead of given fixed size vector, I want to give input as a dynamic size and list by list.

How can I give these input as sequence(list by list) to LSTM ?

Any help is appreciated.

You can try padding the inputs with zero that is of lesser in length.


PCA method for feature selection - How to solve the raise Exception error ("Data must be 1-dimensional")? #Função que permitirá rankear as features mais importantes em um barhplot
def ranks_PCA (x_train, y_train, features_train, RESULT_PATH='Results'):
    print("\nMétodo PCA")

    pca = PCA(n_components=58)
    pca.fit_transform(x_train)

    imp_array = np.array(pca.components_)
    imp_order = imp_array.argsort()
    ranks = imp_order.argsort()

    # Plot PCA
    imp = pd.Series(pca.components_, index=x_train.columns)
    imp = imp.sort_values()

    imp.plot(kind="barh")
    plt.xlabel("Importance")
    plt.ylabel("Features")
    plt.title("Feature importance using PCA")
    # plt.show()
    plt.savefig(RESULT_PATH + '/ranks_DT.png', bbox_inches='tight')

    return ranks

#Função para predição das features dos dados de teste
def predict_PCA(x_test_sel, k_vetor, y_train):
    model = decomposition.PCA()
    model.fit(k_vetor, y_train)
    y_predict = model.predict(x_test_sel)
    return(y_predict)

#Função que calcula o ranking dos dados de treinamento
ranks4 = frk.ranks_PCA(x_train, y_train, features_train, RESULT_PATH)

I have doubts if this implementation is correct to obtain more important features. When trying to run this code, I get the following error:

  Traceback (most recent call last): File "feat_test.py", line 235, in
  'Results/PDBbind2018_F58_Delta_pKd') File "feat_test.py", line 78, in
  run_experiment ranks4 = frk.ranks_PCA(x_train, y_train,
  features_train, RESULT_PATH) File "C:\Users\Patricia\Desktop\VT-58 -
  Cópia\feature-importance\feature_rank_
  ensemble\Scripts\feature_ranks.py", line 121, in ranks_PCA imp =
  pd.Series(pca.components_, index=x_train.columns) File
  "C:\Users\Patricia\Desktop\VT-58 -
  Cópia\feature-importance\feature_rank_
  ensemble\env\lib\site-packages\pandas\core\series.py", line 305, in
  init data = sanitize_array(data, index, dtype, copy,
  raise_cast_failure=True) File "C:\Users\Patricia\Desktop\VT-58 -
  Cópia\feature-importance\feature_rank_
  ensemble\env\lib\site-packages\pandas\core\construction.py", line 482,
  in saniti ze_array raise Exception("Data must be 1-dimensional")

Can anybody help me?

pca.components_ is a 2d array, so you cannot make a Series out of it.


Markov Chain vs Bayes Net I am learning about Markov Chain and Bayesian Nets. However at this point I am a bit confused about what types of problems are modelled with the two different models presented to us. From what I understand (mostly from the examples I have read) Markov Chains are being used to represent the change in a single type of variable over time. So for example a random variable X representing the weather. Let X = {sun, rain}. Then for a markov chain, at time = 0 we are given P(X) and a transition model $P(X_t/X_{t-1})$. So with this knowledge we could calculate $P(X_\infty)$. Like asking a question, given the initial distribution of a random variable X and a transition model, what would be the value of P(X=x) at time t? The solution of such a question can be answered by mini forward algorithm.
Now for bayesian network, from what I understand, we model dependencies among different random variables. So here essentially we have some random variables that may have a causation relationship with other variables. There are some nice properties about such networks that let us define the joint over all variables easily.
Onto my question -> Often the topic of markov chains is introduced before bayesian networks. What is the relationship between the two? because I can't seem to draw parallels between them, to me I see both as quite different approaches at modelling quite different problems.
In what other contexts can markov chains be used? or are they always used to model a single variable varying over time steps? I hope to gain some clarity to distinguish between the two and hopefully this would help me understand the topics better! Any suggestions/readings/links are much appreciated!

First of all, I'd disagree that Markov Chains are dealing with a "single type of variable". If you look at the formal definition of a Markov Chain, you'll see that variables $X$ are random variables. And random variables are defined over arbitrary (well, measurable) sets of possible outcomes. So your $X$ can not only be from $\{sun,rain\}$ set, but if could be from a Cartesian product of $\{sun,rain\}$ and $\{windy,cloudy,calm\}$ and temperature from $[-60,60]$ interval.
About the relation between Markov Chains and Bayes Nets, I'd say that there is a common framework that lets you understand relationship between those (and, in fact, many other probabilistic structures). In all cases we have a collection of random variables - for Markov Chains these are $X_0,X_1,X_2\dots$ And for Bayes Nets lets call them $A,B,C,D,E,F$.
In both cases we are interested in the probability distributions over the whole collection of variables, called joint probability distribution:
$$P(X_0,X_1,X_2\dots)\qquad\text{and} \qquad P(A,B,C,D,E,F)$$
The problem is that trying to represent this distribution (say on a computer) is impossible for any reasonably complex problem - the number of possible combinations of variables grows exponentially with the number of variables. So we are coming up with a way to factorize these joints into smaller, manageable multipliers.
For the Markov Chains, the factorization property is
$$P(X_0,X_1,X_2\dots) = P(X_1|X_0)P(X_2|X_1)P(X_3|X_2)\cdots$$
For the Bayes Nets, the factorization property could look like this (for a particular dependency graph that I just made up):
$$P(A,B,C,D,E,F) = P_a(A)P_b(B)P_c(C|A)P_d(D|A,B)P_e(E|C,D)P_f(F|E)$$
In my opinion, considering the joint distribution and then seeing what factorization structure is imposed on it by your framework is a good place to start studying it.


Predict method of the perceptron algorithm Can someone explain to me how the predict method of the perceptron algorithm works?

def predict(self, pattern):
    activation = np.dot(self.w, pattern) + self.b
    if activation > 0:
        return 1
    else:
        return -1

In this example b stands for bias.

def train(self, pattern, label, step = 0.01):
    out_label = self.predict(pattern)
    if out_label == label:
        return True
    self.w += step * (label - out_label) * pattern
    self.b += step * (label - out_label)
    return False

This is the train method. But as far as I understand in out_label there will only be either 1 or -1. How could this be possibly equal to label? If we would return activation I (think) I would understand it.

Here is how the code appears to break down:

method parameters:

The train(...) method takes an input vector called pattern along with the target output called label and a weight update coefficient (learning rate) called step.

method execution:

train(...) first passes the input vector through our neuron using the predict(...) method.
The result of predict(...) is stored in a variable named out_label.
The out_label variable is either 1 or -1.
Depending on the pattern that we provide we want the target output (label) to be either 1 or -1.
If out_label is equal to label (1 == 1 or -1 == -1), then we return True
Otherwise, the neuron outputted a value we do not want (-1 != 1 or 1 != -1), so we update the weights accordingly, then return False

example where the neuron is correct:

Let's say we have the following parameters:

input vector (pattern) = [-5,2,8]

target output (label) = 1

The train(...) method executes as follows:

we call the predict(...) which does the following:

passes our vector [-5,2,8] through the neuron using the function np.dot(self.w, pattern) + self.b
stores the result of the above function in a variable called activation (let's say activation = 0.23)
returns 1 because activation > 0

we store the result of predict(...) in the variable out_label, which is 1 in this example
because our target output (label = 1), is equal to the actual output (out_label = 1) of the neuron, we return True and consider the neuron output correct.

example where the neuron is incorrect:

Let's say we have the following parameters:

input vector (pattern) = [-1,-5,3]

target output (label) = 1

The train(...) method executes as follows:

we call the predict(...) which does the following:

passes our vector [-1,-5,3] through the neuron using the function np.dot(self.w, pattern) + self.b
stores the result of the above function in a variable called activation (let's say activation = -0.65)
returns -1 because activation <= 0

we store the result of predict(...) in the variable out_label, which is -1 in this example
because our target output (label = 1), is not equal to the actual output (out_label = -1) of the neuron, we update the weights using:
self.w += step * (label - out_label) * pattern
self.b += step * (label - out_label)
we then return False because the neuron was incorrect


How to improve results from ML model? (spam classification) I am trying to build a model that predicts if an email is spam/not-spam. After building a logistic regression model, I have got the following results:
          precision    recall  f1-score   support

         0.0       0.92      0.99      0.95       585
         1.0       0.76      0.35      0.48        74

    accuracy                           0.92       659
   macro avg       0.84      0.67      0.72       659
weighted avg       0.91      0.92      0.90       659

Confusion Matrix:
 [[577   8]
 [ 48  26]]

Accuracy:  0.9150227617602428

The F1-score is the metric I am looking at.  I am having difficulties in explaining the results: I think are very bad results! May I ask you how I could improve it?
I am currently considering a model that looks at corpus of the emails (subject + corpus).
After Erwan's answer:
I oversampled the dataset and these are my results:
Logistic regression
              precision    recall  f1-score   support

         0.0       0.94      0.77      0.85       573
         1.0       0.81      0.96      0.88       598

    accuracy                           0.86      1171
   macro avg       0.88      0.86      0.86      1171
weighted avg       0.88      0.86      0.86      1171

Random Forest
              precision    recall  f1-score   support

         0.0       0.97      0.54      0.69       573
         1.0       0.69      0.98      0.81       598

    accuracy                           0.77      1171
   macro avg       0.83      0.76      0.75      1171
weighted avg       0.83      0.77      0.75      1171

In your results you can observe the usual problem with imbalanced data: the classifier favors the majority class 0 (I assume this is class "ham"). In other words it tends to assign "ham" to instances which are actually "spam" (false negative errors). You can think of it like this: with the "easy" instances, the classifier gives the correct class, but for the instances which are difficult (the classifier "doesn't know") it chooses the majority class because it's the most likely.
There are many things you could do:

Undersampling the majority class or oversampling the minority class is the easy way to deal with class imbalance.
Better feature engineering is more work but it's often how to get the best improvement. For example I guess that you use all the words in the emails as features right? So you probably have too many features and that probably causes overfitting, try reducing dimensionality by removing rare words.
Try different models, for instance Naive Bayes or Decision Trees. Btw Decision Trees are a good way to investigate what happens inside the model.


How to plot the bar charts of precision , recall and f-measure? I have used 4 machine learning models on a task and now I am struggling to plot their bar charts just like shown below in the image . I am printing classification report to get precision , recall etc . My code is shown:
def Statistics(data):
  # Classification Report
  print("Classification Report is shown below")
  print(classification_report(data['actual labels'],data['predicted labels']))

  # Confusion matrix
  print("Confusion matrix is shown below")
  cm=confusion_matrix(data['actual labels'],data['predicted labels'])

  plt.figure(figsize=(10,7))
  sn.heatmap(cm, annot=True,cmap='Blues', fmt='g')
  plt.xlabel('Predicted')
  plt.ylabel('Truth')
Statistics(data)

How can I plot this type of chart in python
I have saved my actual and predicted labels into csv . There are 4 models applied on data . So I saved their actual and predicted labels into Csv's  just like shown below :
path='/content/drive/MyDrive/CSVs/OriginalCensusRFmodel.csv'
def read_csv(path):
  # Read csv
  data=pd.read_csv(path)
  data=data.drop('Unnamed: 0',axis=1)
  return data
data=read_csv(path)

    def Statistics(data):
      # Classification Report
      print("Classification Report is shown below")
      print(classification_report(data['actual labels'],data['predicted labels']))

      # Confusion matrix
      print("Confusion matrix is shown below")
      cm=confusion_matrix(data['actual labels'],data['predicted labels'])

      plt.figure(figsize=(10,7))
      sn.heatmap(cm, annot=True,cmap='Blues', fmt='g')
      plt.xlabel('Predicted')
      plt.ylabel('Truth')
    Statistics(data)

Try running the below method which uses a cross validation strategy to evaluate the models' performance across different metrics.
Of course it might be improved by for example changing the plot type to box plot so that you will see not only the mean score for each estimator but also the distribution of it.
from functools import reduce

def _get_model_name(model):
    """
            Returns a string with the name of a sklearn model
                model: Sklearn stimator class
    """
    if isinstance(model, Pipeline):
        estimator = model.steps[-1][1]
        name = "Pipeline_" + str(estimator)[:str(estimator).find("(")]
    else:
        name = str(model)[:str(model).find("(")]
    return name

def plot_cv_score(X, y, models_list, cv = 5, scoring_list = None, refit = True, return_scores = False):
    """
            X: numpy_array/pandas dataframe n_rows, m_features
            y: numpy_array/pandas dataframe n_rows
            Plots min, max and avg kfold crosval_score for a list of models

    """

    names, mean_score = list(), list()
    ldf = list()
    mnames = list()

    for i, model in enumerate(models_list):
        name = _get_model_name(model)

        if refit:
            model.fit(X, y)

        for metric in score_list:

            score = cross_val_score(model, X, y, cv = cv, scoring = metric, n_jobs= -1)
            mean_score.append(np.mean(score))

        tmp = pd.DataFrame({name: mean_score}, index = score_list)

        ldf.append(tmp)

        mean_score = list()

    frame_scores = reduce(lambda x,y: pd.merge(x,y, left_index = True, right_index = True), ldf).T

    fig, ax  = plt.subplots(1,1, figsize = (10,5))

    frame_scores.plot.bar(ax = ax, cmap = 'RdYlBu', edgecolor = "black")
    ax.legend(loc = 'best')
    ax.set_xlabel("Score")
    ax.set_title("Cross validation model benchmark")

    if return_scores:
        return frame_scores

Example:
from sklearn.metrics import accuracy_score, f1_score, roc_auc_score
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
from sklearn.svm import SVC
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.pipeline import Pipeline

X, y = load_breast_cancer(return_X_y= True)

models_list =[LogisticRegression(random_state= 42),
              SVC(probability= True),
              RandomForestClassifier(random_state = 42),
              GaussianNB()]

score_list = ["roc_auc", "accuracy", "f1", "precision", "recall"]

t = plot_cv_score(X = X, y = y, models_list = models_list, cv = 5, scoring_list = score_list, refit = True)

Outputs:


Does high accuracy metrics with small (but equally sampled) dataset means a good model? I have been training my CNN with 200 images per class for a classification problem.  There problem is a binary classification one. And with the amount of test data ( 25 per class) I am getting good accuracy, precision and recall values. Does that mean my model is actually good?

You can do a crossvalidation to be sure your testing set is not just very easy to classify.

If it is possible, you could try to augment the size of your training set by doing some rotation, shift, flip ... If you are using Keras, you can read this blog.


Recurrent neural network multiple types of input Keras For a project I want to use recurrent neural networks, however my knowledge on this subject is still somewhat limited. I do have some experience with convolutional nets and traditional neural networks. I need to predict a probability distribution over one of the inputs of the next step. Most of my sequences are relatively short, with some big outliers in there. There is more than enough data so that should not be an issue.

I'm using Keras for this task, using either LSTM or GRU layers.

My input exists of sequences of categorical input $x_A$, $x_B$ and a few numerical values $x_1$ to $x_{11}$. I'm trrying to predict $x_A$ of the last step that is not trained on. For the high cardinality categorical features I'm using an embedding layer to map them to a dense space and then merge the three different types of input into one layer. My issue is merging them together before going to the softmax layer to do the one-hot encoding prediction of the last step. I cannot seem to get the inputs to match. I pad the shorter sequences to 10 steps plus the label and truncate the longer sequences cutting off the start. I've attempted both the Graph and the Sequential interface.

def build_graph_model(n_A, n_B, max_length, batch_size):
    g = Graph()
    g.add_input(name='A', input_shape=(max_length, ), dtype='int')
    g.add_node(Embedding(n_A, 32, input_length=max_length, mask_zero=True), name='embedding_A', input='A')
    g.add_input(name='B', input_shape=(max_length, ), dtype='int')
    g.add_node(Embedding(n_B, 32, input_length=max_length, mask_zero=True), name='embedding_B', input='B')
    g.add_input(name='rest', input_shape=(max_length, 11))
    g.add_node(Dense(30, activation='relu'), name='rest_dense', input='rest')
    g.add_node(LSTM(100), name='lstm', inputs=['embedding_A', 'embedding_B', 'rest_dense'])
    g.add_node(Dense(n_dest, activation='softmax'), name='softm', input='lstm')
    g.add_output(name='output', input='softm')
    g.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

This gives an assertion error for having the wrong input dimensions. If I remove the $x_1$ to $x_11$ thing completely it works. If I change the Dense layer into TimeDistributed(Dense) I still get the same error although I think that should work. Inserting the rest input directly into the merge doesn't work because the merge cannot be the first layer. I'm uncertain how to fix this issue.

EDIT: This guy asks the same question but much more concise (https://groups.google.com/forum/#!topic/keras-users/zCuS4xdv5QY)

Kera's functional API allows arbitrary models to be combined like you are describing.

One option is to define a model very similar to the documentation linked above. First, define a "main input" that is categorical, and then, define an "auxiliary input" that is numerical. Both of these sub-models can be merged into a single network with additional layers.


Multi-touch Attribution Model I would like to develop an algorithm for Multi-Touch Time-Decay Attribution Model for an e-mail campaign. I have the research paper ready and I would convert the paper into R-script. I am struggling to find the data for the purpose.

Can anyone please give me some pointers on where to find such kind of data or what kind of features are considered for such model, so that I can try to simulate the data?

Thank you in advance.

We have recently completed one such exercise using Markov chains. The data that we used for this project was as follows:
For a selected set of users, for a 6 month time period their complete campaign history. Format as follows:
Userid, campaign-id, campaign-date, response tag, response value

The response tag is a binary tag indicating responded or no.

Our markov chain approach did not really care about the time of campaign, it only used the order and getting null state vs success.

What approach are you using in your model? I'd be happy to compare notes. While I'd not be able to share the data, we could run your code on our data.

We had real campaign history data for a client, hence didn't have to simulate it.


Neural Network for classifying humans I am currently searching a neural network that can classify if there is a human in an image or not.

I checked the ImageNet dataset, but the 1000 classes there contain nothing like human or person or such, mostly animals and things.

Segmentation CNNs usually have a lot longer runtime, and all I really need is basically a binary NN, just "Yes = Human in Image", "No = No human in image". Can someone point me to a training set or a pre-trained network?

In Part 1 of this "tutorial/discussion" ther is a good explanation of the problem and a possible solution using OpenCV built in Haar Cascades  or HOG for Human Detection (this is the key-word you should use to search info about your task).

They are both really fast compared to NN and require little training.

In Part 2 are explained more modern approaches using Deep Learning. A notable mention would be AlexNet.

At the end there is even a tutorial on Setting up a Basic Human Detector with Tensorflow.


Libraries for Bayesian network inference with continuous data Is there any good libraries that allow me to:

Construct a Bayesian network manually
Specify the conditional probabilities with any continuous PDF, not just Guassian
Perform inference, either exact or approximate

I looked at the following libraries so far, none of them meet the 3 requirements:

pgmpy: only work on discrete distribution or linear Guassian distribution
bnlearn: same as pgmpy
gRain: only discrete distribution
Huggin: only discrete distribution and Guassian
deal: no support for inference
abn: same as deal
libpgm: only discrete distribution and Guassian

You can use pymc3. I am pretty sure it works for all the 3 requirements.
http://pymc-devs.github.io/pymc3/


Do i need to use hyperparamters from Gridsearch to train on WHOLE training set to get final model? I just want to make sure i am on the right lines so please correct me if wrong. I am testing which hyperparmets are best for logisitic regession on my data X, y where X is featrues and y is target. X, y are made from my training set. I also have a test set.

from sklearn.linear_model import LogisticRegression

# split train into target and features
    y = Train['target']
    X = Train.drop(['target'], axis = 1)
    X = pd.get_dummies(X)
#split test data into target and features

y_test = Test['target']
X_test = Test.drop(['target'], axis = 1)
X_test = pd.get_dummies(X_test)

logistic = LogisticRegression()  # initialize the model
# Create regularization penalty space

    param_grid = {'C': [0.001, 0.01, 0.1, 1, 10, 100, 1000] }

clf=GridSearchCV(logistic,param_grid=param_grid,cv=5)

best_model = clf.fit(X, y)# View best hyperparameters
print('Best Penalty:', best_model.best_estimator_.get_params()['penalty'])
print('Best C:', best_model.best_estimator_.get_params()['C']) #

I will now use these hyper parameters and 'train' it on my training data. Just so i'm sure when we say train do i then take best_model and train on the whole X data. Then i use my X_test which is my hold out data on this new trained model:

bestLog=best_model.best_estimator_
trained_model=bestLog.fit(X,y)
predicted=trained_model.predict(X_test)

then use this output above as my final model to test?

As far as I understand (disclaimer: I'm not very familiar with Python) your approach is correct: the selected hyper-parameters are tested on the hold out test set which is different from the training set, this way there's no data leakage and you can evaluate the true performance of your model before applying it to the test set.

For analysis purposes it could be useful to compare the performance of the best model on X (training set) and X_test (hold out) in order to check for overfitting.

Note that in a case like this where you directly select the best hyper-parameters I would consider it acceptable to skip the testing on the hold out set, however in this case you wouldn't know the true performance of your model (so for instance you wouldn't be able to check if it's overfit). To be clear: I don't think you should do this, it's just a remark to show the difference with/without hold out set.


How should I treat these non-English documents in the NLP task? So I have a small corpus of about 30k documents and about 50 documents in this corpus are in other languages (Persian, Chinese, Arabic, German, Spanish etc). I will be using this corpus for training a machine learning model.

Now the question is: How should these non-English documents be treated?

Should I exclude them from the final corpus and from training the model?
or should I manually translate them (Requesting natives from each language to translate it for me) and include them in the final corpus?
or should I use Google translate/DeepL to translate these non-English documents into English and then include them in the final corpus?

Each document in the corpus under question is not larger than 500 words each.

Any help or hint will be appreciated.

You can use these tips :

  Should I exclude them for the corpus and from training the model?

You can do this if you don't have a lack of data. But I think excluding 500 docs from 30K docs won't make a big difference in training. The model's generalisation power won't be compromised.

  should I manually translate them (Requesting natives from each
  language to translate it for me) and include them in the final corpus?

You should do this only when you need the 500 docs as they hold significant information. I would not recommend this method personally.

  should I use Google translate/DeepL to translate these non-English
  documents into English and then include them in the final corpus?

That is something you can do. Using Google Translate could be a help if the structure of the sentences in the docs is simple and sober. You can get good translations without the need of any natives of other countries.

I think you should opt for this method.

Conclusion:

Translate the docs which hold importance using Google Translate.
Else, omit them if you have an extra of 500 docs. They willl not affect the model's performance significantly.

Tip:

I don't think that you can paste the 500 docs in the Google Translate console. This will be time consuming if the docs belong to different languages. Hence, try to omit these documents. Otherwise, you will require a mini application which could use some translation API to transform the docs.


Help framing a sequence prediction problem I've found lots of tutorial/examples that focus on sequence prediction, which use previous time steps of the input variable(s) in order to create a forecast e.g. predict stock market price based on previous price.

What if I want to make a sequence prediction for a variable which is not part of the inputs (e.g. using sequences of X1 and X2 to predict a sequence for X3 at the same time step(s)). Is this the same kind of problem in the previously mentioned tutorials, just in a different context?

I've been trying to learn how to make a Sequential model in Keras, but first wanted to get some clarity on my problem and make sure i'm thinking about it in the right way.

If anyone can provide some insight into clearly framing the problem for me to start trying to build a model it would be most appreciated.

If I understand your meaning: you are modelling by proxy. In other words, it could be seen as using a common autoregressive, simply with the autoregressive part itself removed. I won't get into the details of RNNs and Keras, but let me first explain with a common model used for time-series analysis, ARIMA:

ARIMAX stands for:

Autoregressive - model a variable using its own historic values
Integrated - use differences between values over timesteps (not the raw values)
Moving Average - include historic values of the actual forecast errors
X - common final letter to show we use exogenous variables, variable apart from our target

ARIMA is characterised by three parameters, which tell us how many previous timesteps of the first three terms above are used: ARIMA(p, d, q). p is for AR terms, d for I terms and q for MA.

Within this framework, your problem would be equivalent to setting p = 0. We can see on the Wikipedia site linked above, or here, that doing so makes the ARIMA model essentially an exponential smoothing model!

You could start by analysing how correlated your variables X1 and X2 are with the target variable X3, in order to assess how well (or badly) they might explain the target.

If you follow some tutorials on setting up an experiment for ARIMA, it will only be technicalities to feed the same data into an RNN, or anything else you might build using Keras. You need to feed blocks of data into the model, leaving out previous timesteps of the target variable itself... although if you have them and can use them for validation/testing, I would wonder why you aren't using them to improve the model!


How to get back stationary data from non stationary data after performing diff operations in var (time series)? I have applied var model on multiple features ,but var model accepts the stationary data so i have converted non-stationary data to stationary data by doing diff() ,given the data to var model.
what my question is ,is their option get back  data normal data from stationary to non-stationary  in time series?

You simply have to add the resulting time series. The cumulative sums are comparable to your original input.


Constructing function - f(x,y) for the given minimums (Python) Problem Statement:
I need to construct a function f(x,y) in which there're 3 minimums. 2 local and 1 global which are written below.

Locals are: z = f(0.2,0.3) = 0.7 | z = f(0.6,0.8) = 0.8
Global is: z = f(0.85,0.5) = 0.6

What I've tried so far
I've tried Polynomial regression to construct features that estimate the given minimum points. However, an estimated function z = f(x,y) just going through all the given minimum points without considering them as minimums. I need an approach by means of which I can construct a function that has 3 minimums that are mentioned above. I'd be more than happy if you could provide a code snippet of your approach. As a programming language, I use Python.
Code
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import cm
from mpl_toolkits.mplot3d import Axes3D

degree = 6

def func(x1, x2):
    arr = []
    for i in range(degree, 0, -1):
        arr.append(x1 ** i)

    for i in range(degree, 0, -1):
        arr.append(x2 ** i)

    arr.append(1)

    return np.array(arr)

def der_func(x1, x2):
    arr1 = []
    arr2 = []
    for i in range(degree - 1, -1, -1):
        arr1.append((i + 1) * x1 ** i)

    for i in range(degree + 1):
        arr1.append(0)

    for i in range(degree):
        arr2.append(0)

    for i in range(degree - 1, -1, -1):
        arr2.append((i + 1) * x2 ** i)

    arr2.append(0)

    return np.array([arr1, arr2])

given_x = np.array([[0.2, 0.3],
                    [0.6, 0.8],
                    [0.85, 0.5]])

given_y = np.array([0.7, 0.8, 0.6])

data_x = []
data_y = []

for i in range(len(given_x)):
    x1x2 = given_x[i]

    data_x.append(func(x1x2[0], x1x2[1]))
    data_y.append(given_y[i])

    if i < 3: # in case there's more than 3 data point in given_x
        data_x.append(der_func(x1x2[0], x1x2[1])[0])
        data_y.append(0)

        data_x.append(der_func(x1x2[0], x1x2[1])[1])
        data_y.append(0)

data_x = np.array(data_x)
data_y = np.array(data_y)

w = np.linalg.inv(data_x.T @ data_x) @ data_x.T @ data_y
pred_y = data_x @ w

cost = np.mean((data_y - pred_y) ** 2)
print("Cost | ", cost)

#######################################

X = np.linspace(0, 1, 100)
Y = np.linspace(0, 1, 100)

X, Y = np.meshgrid(X, Y)
Z = np.zeros(np.shape(X))
for i in range(len(X)):
    for j in range(len(X[0])):
        Z[i, j] = np.array([func(X[i, j], Y[i, j])]) @ w

fig = plt.figure()
ax = fig.gca(projection='3d')
ax.contour(X, Y, Z, 150)
ax.scatter(given_x[:3, 0], given_x[:3, 1], given_y[:3])
plt.show()

One way to go is to define functions that are non-zero only near your 'minimal' points, and add them. The goal is to avoid overlap, such that a function associated with a point won't modify the value of your global function around other points.

import numpy as np

def func0(x1):
    lim = 0.01
    dist = x1*x1
    if dist<lim:
        value = np.exp(-dist/(lim-dist))
    else:
        value = 0
    return value

That will give you a 1D function with nice properties (value 1 at 0, 0 outside the neigborhood of 0) :

x = np.linspace(-1, 1, 1000)
z = np.array([func0(i) for i in x])
Z = z.reshape(1000)

plt.plot(Z)
plt.show()

Modify it to get a 2 D function centered in (a,b) :

def func(x1,x2):
    a = 0
    b = 0
    lim = 0.01
    dist = (x1-a)*(x1-a) + (x2-b)*(x2-b)
    if dist<lim:
        value = np.exp(-dist/(lim-dist))
    else:
        value = 0
    return value

This will give you a 2d function :

x = y = np.linspace(-1, 1, 100)
z = np.array([func(i,j) for j in y for i in x])
Z = z.reshape(100, 100)

x, x = np.meshgrid(x, y)

plt.imshow(Z,extent=(x.min(), x.max(), y.min(), y.max()))
plt.colorbar()
plt.show()

Then build the threes functions associated with your points.

def func1(x1,x2):
    a = 0.2
    b = 0.3
    lim = 0.01
    dist = (x1-a)*(x1-a) + (x2-b)*(x2-b)
    if dist<lim:
        value = np.exp(-dist/(lim-dist))
    else:
        value = 0
    return value

def func2(x1,x2):
    a = 0.6
    b = 0.8
    lim = 0.01
    dist = (x1-a)*(x1-a) + (x2-b)*(x2-b)
    if dist<lim:
        value = np.exp(-dist/(lim-dist))
    else:
        value = 0
    return value

def func3(x1,x2):
    a = 0.85
    b = 0.5
    lim = 0.01
    dist = (x1-a)*(x1-a) + (x2-b)*(x2-b)
    if dist<lim:
        value = np.exp(-dist/(lim-dist))
    else:
        value = 0
    return value

And add them (up to coefficients that determine the value on points of interest) to form a global function :

def global_func(x,y):
    return 1-0.3*func1(x,y)-0.2*func2(x,y)-0.4*func3(x,y)

This will give you :

x = y = np.linspace(0, 1, 1000)
z = np.array([global_func(i,j) for j in y for i in x])
Z = z.reshape(1000, 1000)

x, x = np.meshgrid(x, y)

plt.imshow(Z,extent=(x.min(), x.max(), y.min(), y.max()))
plt.colorbar()
plt.show()

Finally you can check that :

print(global_func(0.2,0.3))
print(global_func(0.6,0.8))
print(global_func(0.85,0.5))

returns :

0.7
0.8
0.6

You have a continuous function with all expected properties by construction.

If you are unhappy with it being constant on a lot of the domain, you can modify the individual functions so that they don't stop at the limit with :

value = np.exp(-dist)

And in your global function, determine which 'minimal' point is the nearest and only apply a function of the form 1-coef * funci based on which point is nearest. This way the individual functions won't overlap and thus won't modify value of the global function around other 'minimal' points.


I had such a problem earlier and that what I did is to
separate the date to year, month and day.
I used logistic regression.
For my side, it was helpful and the test percentage was 77%.


How L2 Regularization penalizes weights in TensorFlow? I know that L2 Regularization technique is used to reduce over-fitting and penalizing large weights. In more than one place, I saw that it is used like the code below in TensorFlow library:
reg = tf.nn.l2_loss(w_conv1) + tf.nn.l2_loss(w_conv2) + \
      tf.nn.l2_loss(w_conv3) + tf.nn.l2_loss(w_conv4) + \
      tf.nn.l2_loss(w_conv5) + tf.nn.l2_loss(w_fc1) + \
      tf.nn.l2_loss(w_out)

    loss = tf.reduce_mean(loss + reg * beta)

I am confused about how this code penalizes weights? It adds L2 to weights and then the result to the loss. Could anyone explain this for me?

L2 loss is based on the square of the weights of your Network. As a given weight increases in size, the loss will increase exponentially (quadratically, to be precise). Neural Networks are therefore "pushed" to spread the weight values more evenly across each layer, since for such a quadratic loss factor it's better to have a lot of smaller weights, rather than few very big ones. The exponential relationship between weight value and loss is the reason why large weights get penalized. L2 loss is typically useful when your model suffers from overfitting, since weights redistribution across layers is good to prevent few neurons from staling all the "explanatory power" and generating overfitting.

(The main drawback of L2 loss is that the model becomes more sensitive to outliers, therefore less robust.)

Hope this helps, otherwise let me know.


Make classification and prediction at the same time I am working on the detection and prediction of epileptic seizures and I was thinking about something : would it be possible to apply classification and prediction at the same time. I mean, having seen some data points of EEG, it would be possible to predict the (few) next values of the time series while saying "it's part of a seizure" (or at the opposite, "nothing happens, remain quiet") at the same time ?

If yes, would it be possible to do both using the same neural net ?

Thanks for any help

I think the most intuitive solution would be to have two networks (i.e. one for predicting the next values in the time series and one for classifying if it is or isn't a seizure), because these are two very different tasks and there are different models that excel at each.
The classification network could even be on top of the time series predictor (include the predicted values to assess if it is a seizure or not).
However, what you want (one model to solve both tasks) is most certainly possible and is usually referred to as Multi-Task Learning. In this framework there usually is a large network that branches off to $M$ smaller ones (one for each task). The shared layers are supposed to learn from all $M$ tasks and outperform networks trained individually. If you would like to see some research done in the area I would suggest following MIT's affective computing group (example MTL applications: 1, 2, 3, 4, 5).
In your case you would have a network with two outputs, one for time series prediction and one for classification. Many machine learning libraries allow you to define two cost functions (one for each task) and even have a hyper-parameter you can tune to select the importance of each task. I've done a similar thing with keras' functional api.


Input Pipeline for Tensorflow on GPU The tensorflow example CIFAR10 uses input pipelines to load data from the disk to a queue. I would like to implement this for my own models, but I run into an error that I can't fix somehow. My minimal example looks like the following (taken from the tutorials: https://www.tensorflow.org/programmers_guide/reading_data):

def read_sample_of_csv(filename_queue):

   # Define Reader
   reader = tf.TextLineReader()
   key,value = reader.read(filename_queue)

   # Decode CSV
   record_defaults = [[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.],[1.]]
   col0, *col = tf.decode_csv(value, record_defaults=record_defaults)
   features = tf.stack([*col])
   reference = col0

   # Return
   return features, reference

def input_pipeline(filenames, batch_size, ...):
   # Reader for samples
   slipangle_filename_queue = tf.train.string_input_producer(filenames, shuffle=shuffle)
   features, reference = read_sample_of_csv(slipangle_filename_queue)

   # Batch samples
   capacity = min_after_dequeue + (num_threads+3) * batch_size
   feature_batch, reference_batch = tf.train.shuffle_batch([features,reference], batch_size=batch_size)

   # Return
   return feature_batch, reference_batch

with tf.device(MY_DEVICE):
   feature_batch, ref_batch = input_pipeline(filenames, batch_size, ...)

This is basically what is done in CIFAR10, just with a csv. The script runs with MY_DEVICE = '/cpu:0', but when I try to run it on the GPU with MY_DEVICE = '/gpu:0' I get the following error:

InvalidArgumentError (see above for traceback): Cannot assign a device to node 'shuffle_batch/random_shuffle_queue': Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available.
Colocation Debug Info:
Colocation group had the following types and devices:
QueueEnqueue: CPU
QueueSize: CPU
QueueClose: CPU
QueueDequeueMany: CPU
RandomShuffleQueue: CPU
[[Node: shuffle_batch/random_shuffle_queue = RandomShuffleQueue[capacity=1500, component_types=[DT_FLOAT, DT_FLOAT], container="", min_after_dequeue=1000, seed=0, seed2=0, shapes=[[9], []], shared_name="", _device="/device:GPU:0"]()]]

Can I not use input pipelines on GPUs? Isn't this what is done in CIFAR10?

Solution to my question: If you run a tensorflow session you can choose if the placement of nodes on the different devices is hard or soft coded. In the specific case of CIFAR10 this means:

The Queues be placed on the GPU and that's why the error popped up, but:
If you turn on soft placement it works. This means, that even though the placement on the GPU of several parts of the model can't be done, Tensorflow solves this problem automaticly by placing them somewhere else
I don't actually know where the nodes get placed. I guess on the CPU, but I'm not sure. In tensorboard the color of the input pipeline nodes claims a placement on the GPU. Very confusing!

The soft placement can be done by starting the session like this:

sess = tf.Session(config=tf.ConfigProto(allow_soft_placement=True, log_device_placement=True))

I found this on the following tutorial page: https://www.tensorflow.org/tutorials/using_gpu


Why does the classic Neural Network perform better than LSTM in Sentiment Analysis My goal is to predict the polarity of some reviews (negative, positive or neutral). I tried two different neural networks:

    left_branch = Input((7000, ))
    left_branch_dense = Dense(512, activation = 'relu')(left_branch)

    right_branch = Input((14012, ))
    right_branch_dense = Dense(512, activation = 'relu')(right_branch)
    merged = Concatenate()([left_branch_dense, right_branch_dense])
    output_layer = Dense(3, activation = 'softmax')(merged)

    model = Model(inputs=[left_branch, right_branch], outputs=output_layer)
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    model.fit([np.array(review_matrix), np.array(X_train)], labels,epochs=2, verbose=1)
    model.save('model.merged')

    #############################################################################

    #############################################################################

    #We will try to merge two different models in a different way: Accuracy: 70

    # Prepare the review column for embedding:
    review_matrix_for_embedding = prepare_for_encoding(train_set[4].tolist(), 7000) # Shape: (1503,100)

    second_matrix = np.array(pd.concat([onehot_category, aspect_matrix],axis=1))

    left_branch = Input(shape=(100,), dtype='int32')
    # input_dim: Size of maximum integer (7001 here); output dim: Size of embedded vector;
    # input_length: Size of the array
    left_branch_embedding = Embedding(7000, 300, input_length=100)(left_branch)
    lstm_out = LSTM(256)(left_branch_embedding)
    lstm_out = Dropout(0.7)(lstm_out)
    lstm_out = Dense(128, activation='sigmoid')(lstm_out)

    right_branch = Input((7012, ))
    merged = Concatenate()([lstm_out, right_branch])
    output_layer = Dense(3, activation = 'softmax')(merged)

    model = Model(inputs=[left_branch, right_branch], outputs=output_layer)
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    model.fit([review_matrix_for_embedding, second_matrix], labels,epochs=5, verbose=1)

The first one does 80% accuracy while the second one does 70%, with embedding vectors and LSTM layer. How is it possible? Is there anything wrong in my architecture?

First of all, I have noticed that you have used sigmoid activation function for your LSTM Dense Layer and in your ANN you used relu, maybe, MAYBE, this can be a reason for your lower performance. That is could be happening because sigmoid functions suffer from two problems:
 - Saturation of gradients: sigmoid functions have tail distributions, meaning that they saturate in this 'flat' regions practically diminishing the gradient to zero and affecting backpropagation/training process.
 - Sigmoid outputs are not zero-centered: This is an issue due the gradient calculation during backpropagation. Either having all enters positivo or negative will add a 'zigzag' effect difficulting the training process.

My comments above are taken from this excellent tutorial that you should read: http://cs231n.github.io/neural-networks-1/
 I tried to summarize it but they did a master work and I think you should read it.

Second, we must consider other factors such as: are you analyzing your train/test/val losses? Maybe your LSTM networks just takes longer to train and reach its minimum. You need to work a little more on these parameters before taking any conclusions. Plot a graph showing your training and validation loss so we can assess if your model is underfitting.

Lastly, I have a question for you: Why should your LSTM network perform better than a simple ANN? Although LSTM + Embeddings are powerful techniques that have gained attention in a lot of fields, essentially NLP, that will be not every task that they beat classical approaches. I myself have tried with different data sets, and depending on the application, simple ML algorithms such as SVM would easily beat the more complex ones, including sentiment analysis.

So to conclude, try these things and let us know your results. Also, if anyone disagrees with my answer, I would like to discuss it. I hope it helps.


Suggestion for a better way to organize data to generate frequent item-sets? I have a data of a bag of words in a document. The data has 3 columns: {document number, word number, count of the word in the number}. I am supposed to generate frequent item-sets of a particular size.
I thought that I would make list of all words that appear in a document, create a table of this list, and then generate frequent item-sets using Mlxtend or Orange . However, this approach does not seem to be efficient.

If the size is reasonable (i.e. not too many documents and not too many words in a document), you could try to build a map for each possible itemset, for instance like this:

// Assuming data is an array of size N containing all the documents
// clusters is a map associating each itemset with a set of documents
for i=0 to N-1
  for j=i+1 to N-1
    group = overlap(data[i], data[j])
    add data[i] to the set clusters[group]
    add data[j] to the set clusters[group]

An alternative version if the number of different values and size of the sets allow it and/or if it's possible to precompute the itemsets of interest:

for i=0 to N-1
  for every subset S of data[i]
    add data[i] to the set clusters[S]

(adapted from https://datascience.stackexchange.com/a/60609/64377)


How do I build a model to predict which service a customer will use on an app? There's an app with over 50 services. I have the data on the type of service a specific customer (they have a unique customer number) does on the app, the date, location, time, duration on a service and the volume of that service.
I want to build a ML model to personalize the services the customer gets daily when they open the app. i.e. Top 6 services they're likely to use for that day
Which machine learning technique would be optimal to achieving this?

Start with the most frequent used ones... then explore the need to build a more complex model. Chances are the distribution of service is skewed and you don’t really need to build a model outside a top n.


How to conclude, regression results can't be improved anymore? I am trying to forecast sector's 1 month forward return using macroeconomic variables (170 variables). I tried a few things:

For variable Selection

Run regression with each one of the independent variables and dependent variable separately. Then, using independent variables whose p-values are less than 0.05 to:
Run stepwise regression for variable selection
Finally, run elastic net with best variables (chosen using the above steps i.e., step 1 and step 2)

My observation:
Results (r2) for test set largely depends on how data is split (I don't know why) and even in the best split, r2 is very less (as per my client) in range of 0.12-0.20. How can I either improve this result or conclude that it can't be improved with linear regression?
I have attached a couple of diagnostic plots for independent variables which have the lowest p-values

In general, it is very hard to prove a negative. So, concluding that a result cannot be somehow improved with some unspecified technique tends to be almost impossible. Often the best one can do is to follow state-of-the-art good practices and try out several sensible approaches. Then it is at least unlikely that the results can be easily improved.

For example, using p-value cut-offs from univariate regressions to eliminate variables and using stepwise model selection is known to be a poor practice, so I would assume that one could improve over your current results. For example, without knowing more details of your data, I would expect that elastic net without pre-screening variables with the hyperparameters selected using cross-validation (within the training set) would perform better on a hold-out test set in many cases.

Additionally, if you primarily care about predictive performance then linear regression is unlikely to achieve the best predictive performance. Some form of boosted tree or similar approach will likely be substantially better, variables may tend to be non-linearly related to the outcome. With some subject matter knowledge/good feature engineering you may also achieve pretty good results with penalized (i.e. ridge/elastic net or similar) linear regression - but perhaps the outcome needs to be transformed (e.g. to the log-scale), perhaps some predictors need transforming (e.g. also to the log-scale) and perhaps we can expect certain interactions to be potentially important.

It is also not clear how you are evaluating performance (cross-validation? a hold-out set?). Evaluating model performance on data that directly entered the modeling is likely to be overoptimistic, particularly when you have many potential predictors and presumably not a lot of data. You mentioned a split, which I speculated might be a train-test split, but then it seemed like you kept changing the split to find good performance metrics. If you do that, you can no longer trust the resulting performance metrics.


Synthetic time series generation according to some distribution I'm trying to develop a change detection model that uses sliding windows. Given a time series with some features I've a sliding widows that analyses that time period and compares with a successive time window. If the distribution of data has changed than it is notified. Anyway, I don't have enough real data about my problem, so I need to generate some artificial time series but with different distributions, no matter what distributions, the important thing is that they are different, so that I can test when my model detects change in the data. How can I generate those data?

If I understand you correctly, I think I have a simple way and you do not need to overthink it.

Take a Gaussian (Normal) distribution random number generator, it could even be just downloading a set from https://www.random.org/gaussian-distributions/
Multiply by the desired standard deviation, add the desired mean.
From a certain timestamp, change the standard deviation and mean to a new value to simulate the distribution change. This is the same as concatenating smaller sets with different mean and standard deviation.
See if/when your model detects it.


Forecasting - 3 independent variables I have 3 independent input variables: time, temperature and pressure, and one output - usefulness.

Example:

Number of the sample; Time; Temperature; Pressure; Usefulness
1;1;30;50;9%
2;3;50;50;80%
3;5;50;40;75%
4;7;50;40;?%

I want to train my network and predict the usefulness for the last sample. Which model of forecasting should I use? ARIMA? For coding, I use python.

Thanks!

The name of the model is ARMAX, which is a model dependent of exogenous and endogenous (lagged) variable.

https://stats.stackexchange.com/questions/351146/fit-an-armax-model-in-r


On Python is the `i` variable defined inherently? I am new to this whole landscape, please need some hand holding at first, I can't find a beginner section. Can someone please tell me if i is defined inherently in Python. Can i test for this: what type of object is i?

squares=['red', 'yellow', 'green', 'purple', 'blue']

for i, square in enumerate(squares):
    print(i, square)

Square i is not even defined? There is no programming error.

No, i is not predefined. If you use enumarate(), i (or whatever you may call it) is a count: see the docs.

squares=['red', 'yellow', 'green', 'purple', 'blue']

for i, square in enumerate(squares):
    print(i, square)

Gives you:

0 red
1 yellow
2 green
3 purple
4 blue

You can also set a start value for the count (Python by default starts counting by 0):

for i, square in enumerate(squares,10):
    print(i, square)

Gives you:

10 red
11 yellow
12 green
13 purple
14 blue

In case you just want to loop over a list, you can write:

for i in squares:
    print(i)

Which gives you:

red
yellow
green
purple
blue


How to manually calculate AUC and Accuracy How to calculate AUC using some formula? What are the parameters required and what formula to use.

For accuracy, $$ \frac{TP+TN}{Total} $$ is this right way to calculate AUC?

For accuracy, TP+TN/total is it right way to calculate?

If your problem is binary classification, then yes.

  How to calculate AUC using some formula? What are the parameters
  required and what formula to use?

Check this question.


Too low accuracy on MNIST dataset using a neural network I am beginning with deep learning. This is an implementation of a simple neural network with just 1 hidden layer on MNIST dataset. Why is it that the loss doesn't change at all after any epoch? It clearly means that it is not learning at all. The accuracy is approx. 11% that is like random guessing. But should it be so less?
I have used Adam optimizer and cross_entropy loss.

input_nodes = 784
hl1_nodes = 64
output_nodes = 1

from keras.datasets import mnist
(X_train, y_train), (X_test, y_test) = mnist.load_data()

X_train_reshaped = X_train.reshape(X_train.shape[0],784)

model = Sequential()
model.add(Dense(hl1_nodes, activation='relu', input_shape=(input_nodes,)))
model.add(Dense(output_nodes, activation = 'sigmoid'))

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

history = model.fit(x=X_train_reshaped, y=y_train, validation_split=0.33, verbose=1,epochs=10)

#output
Train on 40199 samples, validate on 19801 samples
Epoch 1/10
40199/40199 [==============================] - 4s 87us/step - loss: -55.0254 - acc: 0.1142 - val_loss: -55.1361 - val_acc: 0.1088
Epoch 2/10
40199/40199 [==============================] - 3s 76us/step - loss: -55.0284 - acc: 0.1141 - val_loss: -55.1361 - val_acc: 0.1088
Epoch 3/10
40199/40199 [==============================] - 3s 74us/step - loss: -55.0284 - acc: 0.1141 - val_loss: -55.1361 - val_acc: 0.1088
Epoch 4/10
40199/40199 [==============================] - 3s 75us/step - loss: -55.0284 - acc: 0.1141 - val_loss: -55.1361 - val_acc: 0.1088
Epoch 5/10
40199/40199 [==============================] - 3s 75us/step - loss: -55.0284 - acc: 0.1141 - val_loss: -55.1361 - val_acc: 0.1088
Epoch 6/10
40199/40199 [==============================] - 3s 75us/step - loss: -55.0284 - acc: 0.1141 - val_loss: -55.1361 - val_acc: 0.1088
Epoch 7/10
40199/40199 [==============================] - 3s 75us/step - loss: -55.0284 - acc: 0.1141 - val_loss: -55.1361 - val_acc: 0.1088
Epoch 8/10
40199/40199 [==============================] - 3s 75us/step - loss: -55.0284 - acc: 0.1141 - val_loss: -55.1361 - val_acc: 0.1088
Epoch 9/10
40199/40199 [==============================] - 3s 75us/step - loss: -55.0284 - acc: 0.1141 - val_loss: -55.1361 - val_acc: 0.1088
Epoch 10/10
40199/40199 [==============================] - 3s 75us/step - loss: -55.0284 - acc: 0.1141 - val_loss: -55.1361 - val_acc: 0.1088

What am I missing?
Edit: It is same upto the 4th digit even after 90th epoch.

What am I missing?

Incorrect architecture for the classification task. You have a single binary output, trained using binary_crossentropy, so the NN can only classify something as in a class (label 1) or not (label 0). Instead, you most likely want 10 outputs using softmax activation instead of sigmoid, and categorical_crossentropy as the loss, so that you can classify which digit is most likely given an input.
Incomplete processing of MNIST raw data. The input pixel values in X_train range from 0 to 255, and this will cause numeric problems for a NN. The target labels in y_train are the digit value (0,1,2,3,4,5,6,7,8,9), whilst for classification you will need to turn that into binary classes - usually a one-hot coding e.g. the label 3 becomes a vector [0,0,0,1,0,0,0,0,0,0].

Scale the inputs - a quick fix might be X_train = X_train/ 255 and X_test = X_test/ 255
One-hot code the labels. A quick fix might be y_train = keras.utils.to_categorical(y_train)

I made those changes to your code and got this after 10 epochs:

val_loss: 0.1194 - val_acc: 0.9678


How to save Tensorflow predictions to data frame? I am new to Tensorflow. I have trained Tensorflow model, but I need to take model predictions and add them to my original test set as a column. How can I do that?

def model(self, layers_dims, X_train, Y_train, X_test, Y_test, learning_rate=0.00001,
          num_epochs=1000, print_cost=True):
    """
    Implements a three-layer tensorflow neural network: LINEAR->RELU->LINEAR->RELU->LINEAR->SOFTMAX.

    Arguments:
    X_train -- training set, of shape (input size = 12288, number of training examples = 1080)
    Y_train -- test set, of shape (output size = 6, number of training examples = 1080)
    X_test -- training set, of shape (input size = 12288, number of training examples = 120)
    Y_test -- test set, of shape (output size = 6, number of test examples = 120)
    learning_rate -- learning rate of the optimization
    num_epochs -- number of epochs of the optimization loop
    minibatch_size -- size of a minibatch
    print_cost -- True to print the cost every 100 epochs

    Returns:
    parameters -- parameters learnt by the model. They can then be used to predict.
    """

    ops.reset_default_graph()  # to be able to rerun the model without overwriting tf variables
    tf.set_random_seed(1)  # to keep consistent results
    seed = 3  # to keep consistent results
    (n_x, m) = X_train.shape  # (n_x: input size, m : number of examples in the train set)

    n_y = Y_train.shape[0]  # n_y : output size
    #print('Ytrain shape', Y_train.shape)
    costs = []  # To keep track of the cost

    # Create Placeholders of shape (n_x, n_y)
    X, Y = self.create_placeholders(n_x, n_y)

    # Initialize parameters
    parameters = NN_predict_trading_decisions.initialize_parameters(layers_dims)

    # Forward propagation: Build the forward propagation in the tensorflow graph
    ZL = NN_predict_trading_decisions.forward_propagation(X, parameters)

    # Cost function: Add cost function to tensorflow graph
    cost = NN_predict_trading_decisions.compute_cost(ZL, Y)

    # Backpropagation: Define the tensorflow optimizer. Use an AdamOptimizer.
    #optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)
    optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)

    # Initialize all the variables
    init = tf.global_variables_initializer()

    # Start the session to compute the tensorflow graph
    with tf.Session() as sess:

        # Run the initialization
        sess.run(init)

        # Do the training loop
        for epoch in range(num_epochs):

            epoch_cost = 0.  # Defines a cost related to an epoch
            current_cost = sess.run([optimizer, cost], feed_dict= {X: X_train, Y: Y_train})
            epoch_cost +=  current_cost[1]

            # Print the cost every epoch
            if print_cost == True and epoch % 100 == 0:
                print("Cost after epoch %i: %f" % (epoch, epoch_cost))
            if print_cost == True and epoch % 5 == 0:
                costs.append(epoch_cost)

        # plot the cost
        plt.plot(np.squeeze(costs))
        plt.ylabel('cost')
        plt.xlabel('iterations (per tens)')
        plt.title("Learning rate =" + str(learning_rate))
        plt.show()

        # lets save the parameters in a variable
        parameters = sess.run(parameters)
        print("Parameters have been trained!")

        # Calculate the correct predictions
        correct_prediction = tf.equal(tf.argmax(ZL), tf.argmax(Y))

        # Calculate accuracy on the test set
        accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))

        print("Train Accuracy:", accuracy.eval({X: X_train, Y: Y_train}))
        print("Test Accuracy:", accuracy.eval({X: X_test, Y: Y_test}))

        return parameters

Thanks to Ankit Seth I found the answer.

#this returns output layer of forward prop with trained parameters of the model.
pred =  forward_propagation(X, parameters)
predictions_sigm = pred.eval(feed_dict = {X: X_test})
predictions_sigm = sigmoid(predictions)

#pred.eval returns linear regression layer for each class separately, so we need to pick index of maximum of them. I don't use softmax layer, since the output should be the same.

#init list of classes
y_list = []

for i in range(predictions_sigm.shape[1]):
    class_list = []
    class_list.append(predictions_sigm[0][i])
    class_list.append(predictions_sigm[1][i])
    class_list.append(predictions_sigm[2][i])
    class_list.append(predictions_sigm[3][i])

    #get index of maximum value of sigmoid. it will correspond to class
    y = np.argmax(class_list)
    y_list.append(y)

   #then append y_list to dataframe
   df['predicted_y'] = y_list


Time series decomposition For me, the original data looks to have like a decreasing or constant trend but stl() is giving a different trend altogether. Can someone here please explain why?
The decomposition plot is as follows:

To my eye, the second half of 2018 is substantially higher than the prior two years, so the trend that stl() is giving seems not unreasonable.

Trying to fit a time series model to only three years of data is going to require you to take a pretty strong hand guiding the model; I wouldn't expect a turn-key solution to give satisfactory results.

One obvious problem with the decomposition from stl() is that the seasonal figure is wildly overfit.  I would try using the Fourier series approach to fitting the seasonal figure that Rob Hyndman describes in this blog post.  I have applied that technique in exactly your situation and gotten decent results.

The closest to a turn-key solution is probably going to Facebook's prophet library.  But with only three years of data, I still suspect you'll overfit the seasonal component if you call it with the defaults.

If you post your actual data and describe your goals more completely, folks might be able to help more.


How to create multi-hot encoding from a list column in dataframe? I have dataframe like this
Label   IDs
    0   [10, 1]
    1   [15]
    0   [14]

I want to create a multihot encoding of the feature IDs. It should look like this
Label ID_10 ID_1 ID_15 ID_14
  0     1     1    0     0
  1     0     0    1     0
  0     0     0    0     1

The goal is to use them as features in Keras. So it is also acceptable to make this transformation using a Keras API.

You can try using sklearn's MultiLabelBinarizer (https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MultiLabelBinarizer.html):
mlb = MultiLabelBinarizer()
mlb.fit(d['IDs'])

new_col_names = ["ID_%s" % c for c in mlb.classes_]

# Create new DataFrame with transformed/one-hot encoded IDs
ids = pd.DataFrame(mlb.fit_transform(d['IDs']), columns=new_col_names)

# Concat with original `Label` column
pd.concat( [d[['Label']], ids], axis=1 )


Optimum minimum number of instances in weka's j48 There is a parameter named minnumobj in the options of the j48 tree algorithm in weka. This parameter indicates the minimum number of participants to be in a leaf. Does this parameter have an optimum value regarding the instances or attributes? Or should I set the value that achieves the highest classification success? For example, I have 6500 instances and 9 attributes and I got highest success when minnumobj=100.

If there were an optimum in terms of number of instances or attributes, then that would be set in the program.  No, such a parameter generally has an optimum that depends on the data itself, and needs to be tuned to get optimal performance.