I don't understand RidgeCV's fit_intercept, and how to use it for my data Alright, I have an assignment that makes me calculate weights for a function with different terms. At first, I thought I might just leave the weight for the term $1$ out, and instead use the intercept. I have decided to use RidgeCV, as I had a large amount of multicolinearity.

However, now I have appended my $x$ by a row of $1$'s, and did the following:

RidgeCV(fit_intercept = False).fit(x, y)

Alright, now I have an array of weights. For comparison, I have tried also

RidgeCV(fit_intercept = True).fit(x, y)

As expected, the weight for $1$ became 0. However, all other values changed too - and the intercept is different from the weight for $1$ from before. I have also a different .score() - the first one is higher.

Why is that the case? I thought all fit_intercept was doing is adding a row of $1$ to my $x$, which obviously can't be true. Also, should I try centering my data myself, or is that now unnecessary?

The weight on the constant column was also penalized by Ridge, whereas the intercept fitted directly is not.

To your second question: Centering shouldn't be necessary (except maybe for computational reasons), though scaling is still recommended so that coefficient penalties are applied more equally.


Confusion about the Bellman Equation In some resources, the belman equation is shown as below:
$v_{\pi}(s) = \sum\limits_{a}\pi(a|s)\sum\limits_{s',r}p(s',r|s,a)\big[r+\gamma v_{\pi}(s')\big] $
The thing that I confused is that, the $\pi$  and $p$ parts at the right hand side.
Since the probability  part -  $p(s',r|s,a)$- means that the probability of being at next state ($s'$), and since being at next state ($s'$) has to be done via following a specific action, the $p$ part also includes the probability of taking the specific actions inside it.
But then, why the $\pi(a|s)$ is written at the beginning of the equation? Why do we need it? Isn't the possibility of taking an action stated at the $p(s',r|s,a)$ part already?

$p(s', r | s, a)$ is the probability of arriving at state $s'$ and obtain reward $r$  given that the environment was in state $s$ and the agent took action $a$. Therefore, this probability is defined assuming action $a$ is taken. There is no probability of taking $a$ included there.
The probability of the agent taking an action is provided by the policy $\pi$, and that is why we need it in the equation.
You can think of the interaction of these two terms with the law of total probability: $p(A)=\sum _{n}p(A\mid B_{n})p(B_{n})$, where $p(B_{n})$ is analogous to $\pi(a|s)$ and $p(A\mid B_{n})$ is analogous to $p(s', r | s, a)$.


Why not continue next epoch with best weights I've seen the following happen in our training: (numbers are paraphrased)

epoch 10 - val loss 0.0500 - loss 0.0450
epoch 11 - val loss 0.0400 - loss 0.0400
epoch 12 - val loss 0.0420 - loss 0.0410

Now clearly epoch 11 is better than epoch 12. I assumed epoch 13 would start with the weights of epoch 11, but have been told they do not.

In my opinion worst of all, our training pipeline is set up with ReduceLROnPlateau lowering the learning weight after 4 epochs of no improvement. Now say epoch 15 has a validation loss of 0.0415 and a loss of 0.0405, we could've better trained on with the weights of epoch 11.

Why are new epochs not started with the weights of the previous best epoch?

A new epoch starts with the same weights as the previous epoch but with different mini-batch of data. Model performance will have small oscillations across mini-batches for that reason.

Always starting each epoch with the best overall weights might result in staying in a local optimum. Gradient descent is already prone to staying in a local optimum, at the expense of finding a better optimum. Encouraging continued exploration of weights is one of the goals of training.

There is always a chance that both training and validation loss will do down. It is okay that training performance gets worse, there is no need for monotonic improvement in training.

At the end of the training, you should select the weights that have the lowest validation loss.


Adding Features To Time Series Model LSTM have been reading up a bit on LSTM's and their use for time series and its been interesting but difficult at the same time. One thing I have had difficulties with understanding is the approach to adding additional features to what is already a list of time series features. Assuming you have your dataset up like this:

t-3,t-2,t-1,Output

Now lets say you know you have a feature that does affect the output but its not necessarily a time series feature, lets say its the weather outside. Is this something you can just add and the LSTM will be able to distinguish what is the time series aspect and what isnt?

For RNNs (e.g., LSTMs and GRUs), the layer input is a list of timesteps, and each timestep is a feature tensor. That means that you could have a input tensor like this (in Pythonic notation):

# Input tensor to RNN
[
    # Timestep 1
    [ temperature_in_paris, value_of_nasdaq, unemployment_rate ],
    # Timestep 2
    [ temperature_in_paris, value_of_nasdaq, unemployment_rate ],
    # Timestep 3
    [ temperature_in_paris, value_of_nasdaq, unemployment_rate ],
    ...
]

So absolutely, you can have multiple features at each timestep. In my mind, weather is a time series feature: where I live, it happens to be a function of time. So it would be quite reasonable to encode weather information as one of your features in each timestep (with an appropriate encoding, like cloudy=0, sunny=1, etc.).

If you have non-time-series data, then it doesn't really make sense to pass it through the LSTM, though. Maybe the LSTM will work anyway, but even if it does, it will probably come at the cost of higher loss / lower accuracy per training time.

Alternatively, you can introduce this sort of "extra" information into your model outside of the LSTM by means of additional layers. You might have a data flow like this:

TIME_SERIES_INPUT ------> LSTM -------\
                                       *---> MERGE ---> [more processing]
AUXILIARY_INPUTS --> [do something] --/

So you would merge your auxiliary inputs into the LSTM outputs, and continue your network from there. Now your model is simply multi-input.

For example, let's say that in your particular application, you only keep the last output of the LSTM output sequence. Let's say that it is a vector of length 10. You auxiliary input might be your encoded weather (so a scalar). Your merge layer could simply append the auxiliary weather information onto the end of the LSTM output vector to produce a single vector of length 11. But you don't need to just keep the last LSTM output timestep: if the LSTM outputted 100 timesteps, each with a 10-vector of features, you could still tack on your auxiliary weather information, resulting in 100 timesteps, each consisting of a vector of 11 datapoints.

The Keras documentation on its functional API has a good overview of this.

In other cases, as @horaceT points out, you may want to condition the LSTM on non-temporal data. For example, predict the weather tomorrow, given location. In this case, here are three suggestions, each with positive/negatives:

Have the first timestep contain your conditioning data, since it will effectively "set" the internal/hidden state of your RNN. Frankly, I would not do this, for a bunch of reasons: your conditioning data needs to be the same shape as the rest of your features, makes it harder to create stateful RNNs (in terms of being really careful to track how you feed data into the network), the network may "forget" the conditioning data with enough time (e.g., long training sequences, or long prediction sequences), etc.
Include the data as part of the temporal data itself. So each feature vector at a particular timestep includes "mostly" time-series data, but then has the conditioning data appended to the end of each feature vector. Will the network learn to recognize this? Probably, but even then, you are creating a harder learning task by polluting the sequence data with non-sequential information. So I would also discourage this.
Probably the best approach would be to directly affect the hidden state of the RNN at time zero. This is the approach taken by Karpathy and Fei-Fei and by Vinyals et al. This is how it works:

For each training sample, take your condition variables $\vec{x}$.
Transform/reshape your condition variables with an affine transformation to get it into the right shape as the internal state of the RNN: $\vec{v} = \mathbf{W} \vec{x} + \vec{b}$ (these $\mathbf{W}$ and $\vec{b}$ are trainable weights). You can obtain it with a Dense layer in keras.
For the very first timestep, add $\vec{v}$ to the hidden state of the RNN when calculating its value.

This approach is the most "theoretically" correct, since it properly conditions the RNN on your non-temporal inputs, naturally solves the shape problem, and also avoids polluting your inputs timesteps with additional, non-temporal information. The downside is that this approach often requires graph-level control of your architecture, so if you are using a higher-level abstraction like Keras, you will find it hard to implement unless you add your own layer type.


How can decision trees be tuned for non-symmetrical loss? Suppose we use a decision tree to predict if a bank customer can pay back a credit. So it is a two class classification problem. Now we can make two mistakes:

$\alpha$ error: The customer can back the credit, but we predict he can't.
$\beta$ error: The customer can't pay back the credit, but we predict he can.

Now we know that $\beta$ errors are 123.4 times as expensive as $\alpha$ errors. But we only have a given set of data. In this set we have $n_1=10000$ customers who paid back the credit and $n_2 = 100$ customers who didn't.

How can the training of the decision tree be adjusted to account for the fact that $\beta$ errors are more expensive?

(Note: This is a theoretical question to learn about decision trees. I know about other classifiers like neural networks and I know of ensembles. However, I only want to know about decision trees here.)

Theoretically, decision tree algorithms specify the feature as well as the threshold that maximize the separation between classes at each node. This can be done by solving optimization problem related to the entropy at the children nodes.
You can modify this optimization problem by including the miss- classification costs you have in the optimization problem so the algorithm will be biased toward the most expensive class


How to process multivariant questions in Orange can you explain how to process multivariant questions in Orange? For example:
I have sociological survey database. Among the others there is question:

How would you like to spend your weekend? (pick only 3):

At home
With friends
With family
At work

And here is database example:

Database logic: max three options to pick - 3 columns. Respondent answer's order doesnt matter. Is there a possibility to make frequency histogramm in Orange based on these three columns?

Since you say the order of picks doesn't matter, I think your data will be more easily manageable (in any tool, including Orange) if you first transform the database into a wide table of one-hot encoded features, with every option its own column:

id    weekend_home    weekend_friends    weekend_family    weekend_work
1     1               1                  1                 0
2     1               1                  0                 0
3     0               1                  0                 0
4     1               1                  0                 0
5     1               1                  1                 0


What are h(t-1) and c(t-1) for the first LSTM cell? I know in a LSTM chain you should connect the h(t) of the previous cell to the h(t+1) of the next cell, and doing so for c(t). But what about the first cell? What does it get as h(t-1) and c(t-1)?

I also like to know, if we want to make a multi layer LSTM, What should we give to the first cell of the second layer as h and c? Also will we throw away the h and c of the last cell of the each layer?

And a little other question: will we give the y(output) of each cell to the x(input) of it's corresponding cell in the top layer?

Assuming that the LSTM is going to be used for sequence generation (e.g. in a language model or the decoder part of an encoder-decoder NMT architecture), we have the following:

In supervised learning setups:

In a language model, the LSTM's  $h_{-1}$ and $c_{-1}$ are initialized to zeros, for all the layers. If the LSTM is the decoder part of an encoder-decoder NMT architecture, they will be initialized with the hidden state and context vector of the LSTM from the encoder part, either from the last position of from a combination of all positions by using some attention mechanism.

During training, the inputs $x$ are not fed with the LSTM predictions for the previous tokens because this presents convergence problems; instead, you use teacher forcing, which consists in feeding the gold tokens (the tokens from the training data) as inputs.

During inference, you have to use the network's own predictions as inputs for the next tokens.

The mismatch between the training and inference regimes is said to present exposure bias, that is, during training you make the network see only "good" previous tokens and it never learns to recover from its own bad predictions. However, according to recent research, exposure bias may actually not be a serious problem.

In more exotic learning setups:

In textual GANs, $h_{-1}$ and $c_{-1}$ are sometimes initialized with a latent vector $z$ that follows some prior distribution.

Also, in textual GANs and other learning setups where there are no gold tokens, teacher forcing is not an option, so you need to use the LSTM predictions as previous token inputs.


Learning token dictionary When processing texts written in a human-readable language, typically a first step is tokenization, where we have a dictionary of tokens (words) and we collapse each substring in that dictionary into a single token.  For a language like English, the dictionary is known in advance.

In my case, I want to parse files with a format not based on any human language.  I don't have a dictionary of known tokens; instead, I want to learn this dictionary.  I do have lots of sample data I can analyze.

Are there techniques for generating a dictionary, given many example files?

I expect that any common substring that appears in many files is a good candidate for a token, so an obvious approach is to look for all common substrings and add them to the dictionary.  However, a complication is that there is no equivalent of whitespace that separates tokens, so it seems like some filtering might be needed to avoid treating phrases (sequences of multiple tokens) as a new token.  For example, imagine if we had a large corpus of English writing, except that all spaces and punctuation are omitted, and we tried to infer a dictionary of English words from that.  We might naturally infer that "ARE", "THE", and "HER" are reasonable tokens, as they occur frequently as substrings.  All well and good so far.  However, we wouldn't want the algorithm to add "ARETHE" to the dictionary, even though "ARETHE" appears fairly frequently, as "ARETHE" is just the concatenation of the two tokens "ARE" and "THE".  I suspect it might be possible to detect this, because the frequency of "ARETHE" is not much more than the product of the frequencies of "ARE" and "THE", but I'm not sure.  Is there a clean way to handle this wrinkle?

Actually you can often just split on the white space, remove punctuation, and lowercase. This will provide your tokens. For example, if I had a string "jdf asdsa sdfr" (no English), then I could derive the tokens jdf, asdsa, and sdfr. The only thing that is known before hand would be the stopwords (and, be it the), which do come from the English dictionary. However, in this case, it does not sound like you would need stopwords. I would suggest looking at the following libraries available in python:

SpaCy

NLTK

Scikit-learn


Reinforcement learning, pendulum python I'm having trouble finding a good reward function for the pendulum problem, the function I'm using: $-x^2 - 0.25*(\text{xdot}^2)$
which is the quadratic error from the top. with $x$ representing the current location of the pendulum and $\text{xdot}$ the angular velocity.

It takes a lot of time with this function and sometimes doesn't work.
Any one have some other suggestions?
I've been looking in google but didn't find anything i could use

You could use the same reward function that Openai's Inverted Pendulum is using:

$costs=-(\Delta_{2\pi}\theta)^2 - 0.1(\dot{\theta})^2 - 0.001u^2$

where $(\Delta_{2\pi}\theta)$ is the difference between current and desired angular position performed using modulo $2\pi$. The variable $u$ denotes the torque (the action of your RL agent). The optimal is to be as close to zero costs as it gets.

The idea here is that you have a control problem in which you can come up with a quadratic 'energy' or cost function that tells you the cost of performing an action at EVERY single time step. In this paper (p.33 section 5.2) you can find a detailed description.

I have tested RL algorithms in this objective function and I did not encounter any problems for convergence in both MATLAB and Python. If you still have problems let us know what kind of RL approach you implemented and how you encoded the location of the pendulum.

Hope it helps!


Hyperparameter tuning with Bayesian-Optimization I'm using LightGBM for the regression problem and here is my code.

    def bayesion_opt_lgbm(X, y, init_iter = 5, n_iter = 10, random_seed = 32, seed= 100, num_iterations = 50,
                          dtrain = lgb.Dataset(data = X_train, label = y_train)):
      def lgb_score(y_preds, dtrain):
        labels = dtrain.get_labels()
        return 'r2', r2_score(labels, y_preds), True

      def hyp_lgb(num_leaves, feature_fraction, bagging_fraction, max_depth, min_split_gain, min_child_weight):
          params = {'application': 'regression',
                    'num_iterations': 'num_iterations',
                    'early_stopping_round': 50,
                    'learning_rate': 0.05,
                    'metric': 'lgb_r2_score'}
          params['num_leaves'] = int(round(num_leaves))
          params['feature_fraction'] = max(min(feature_fraction, 1), 0)
          params['bagging_fraction'] = max(min(bagging_fraction, 1), 0)
          params['max_depth'] = int(round(max_depth))
          params['min_split_gain'] = min_split_gain
          params['min_child_weight'] = min_child_weight
          cv_results = lgb.cv(params,
                              train_set = dtrain,
                              nfold = 5,
                              stratified = False,
                              seed = seed,
                              categorical_feature = [],
                              verbose_eval = None,
                              feval = lgb_r2_score)
          print(cv_results)
          return np.max(cv_results['r2-mean'])
      bounds = {'num_leaves': (80,100),
                'feature_fraction': (0.1, 0.9),
                'bagging_fraction': (0.8, 1),
                'max_depth': (5,10,15,20),
                'min_split_gain': (0.001, 0.01),
                'min_child_weight': (10,20)
                }

      optimizer = BayesianOptimization(f = hyp_lgb, pbounds = bounds, random_state = 32)

      optimizer.maximaze(init_points= init_iter, n_iter = n_iter)
    bayesion_opt_lgbm(X_train, y_train)

When I run my code, I get an error something like that, Please help me where am i missing
TypeError                                 Traceback (most recent call last)
TypeError: float() argument must be a string or a number, not 'tuple'

The above exception was the direct cause of the following exception:

ValueError                                Traceback (most recent call last)
<ipython-input-57-86f7d803c78d> in <module>()
     40   #Optimize
     41   optimizer.maximaze(init_points= init_iter, n_iter = n_iter)
---> 42 bayesion_opt_lgbm(X_train, y_train)
     43

2 frames
/usr/local/lib/python3.6/dist-packages/bayes_opt/target_space.py in __init__(self, target_func, pbounds, random_state)
     47         self._bounds = np.array(
     48             [item[1] for item in sorted(pbounds.items(), key=lambda x: x[0])],
---> 49             dtype=np.float
     50         )
     51

ValueError: setting an array element with a sequence.

The pbounds must all be pairs; you cannot specify a list of options for max_depth.
The package cannot deal with discrete hyperparameters very directly; see section 2, "Dealing with discrete parameters", of their "advanced tour" notebook about this.


Orange, K-means clustering, memory error I have a dataset of over 67 000 records, and I'm trying to run a k-means cluster analysis on that. Orange returns a memory error.
The data is in an excel file, but I also tried to load it from a csv. Omitting some records and downsizing it under 65k records also couldnt help.
Hierarchical clustering also doesnt work for the same reason.

Does anyone have any ideas on what to do here?

Error code:

  MemoryError
  Traceback (most recent
  call last):
  File
  "C:\Python34\lib\site-packages\Orange\widgets\gui.py", line 2220, in

      b.button = btn = button(b, master, label, callback=lambda: do_commit())   File
  "C:\Python34\lib\site-packages\Orange\widgets\gui.py", line 2191, in
  do_commit
      commit()
  File "C:\Python34\lib\site-packages\Orange\widgets\unsupervised\owkmeans.py",
  line 246, in run
      self.run_optimization()
  File "C:\Python34\lib\site-packages\Orange\widgets\unsupervised\owkmeans.py",
  line 225, in run_optimization
      self.optimization_runs.append((k, kmeans(self.data)))
  File "C:\Python34\lib\site-packages\Orange\projection\base.py", line 28, in
  call
      clf = self.fit(data.X, data.Y)
  File "C:\Python34\lib\site-packages\Orange\clustering\kmeans.py", line 23,
  in fit
      proj = proj.fit(X, Y)
  File "C:\Python34\lib\site-packages\sklearn\cluster\k_means_.py", line 821,
  in fit
      n_jobs=self.n_jobs)
  File "C:\Python34\lib\site-packages\sklearn\cluster\k_means_.py", line 277,
  in k_means
      tol = _tolerance(X, tol)
  File "C:\Python34\lib\site-packages\sklearn\cluster\k_means_.py", line 162,
  in _tolerance
      variances = np.var(X, axis=0)
  File "C:\Python34\lib\site-packages\numpy\core\fromnumeric.py", line 2938,
  in var
      keepdims=keepdims)
  File "C:\Python34\lib\site-packages\numpy\core_methods.py", line 102, in
  _var
      x = asanyarray(arr - arrmean)
  MemoryError

Only 32-bit Orange is currently available for Windows (the filename says win32, if you look). So your per-process RAM is limited to 4 GB. The exception says you ran out of RAM.

Use 64-bit Anaconda installer to get 64-bit Python and libraries on Windows, and then install Orange 3 there.


Can RNN be replaced with non-recurrent classifier for Sequence Classification problem? Setup:
We have sequence of events that are not evenly spaced (not a time series). Length of the sequence is constant.
Goal:
Predict class of the event that is most probable to follow this sequence.
Background:
I know that RNN would probably a good fit for this task, but at the same time I wonder whether parameters sharing in our U,W,V matrices actually hurt accuracy ( even though training process is cheaper). Let's say we are ok to spent more time(and data) for training and don't want to compromise accuracy.
Question:
Is it true that by using regular MLP we can achieve better or at least same performance if we just combine/flatten all features from those sequence events and pass them alltogether as an input? I believe model should still be able to learn interactions between features(that represent different events in a sequence) but not sure how good it will be at it and if not then why?

Conceptually, the sound of dropping a metal chain on the floor is different from the sound of dropping the separated links of that chain.
Feedforward NN
In a feed forward neural network all of sequential features would be consumed independently:
$f(x)=WX+b=w_1x_1+..+w_nx_n+b$
This is all good, as far as you’re ready to sacrifice the step-wise dependency between x1, x2 etc.
Recurrent NN
To be able to utilise the temporal or sequential signal in your dataset you need a method that “chains” each feature with its past/future state, right?
We "connect" these sequential events with the alleged "hidden state":
$a_n = f(W_n, a_{n-1}, x_n)$
By exploding the above equation you can see how past information is accumulated in the subspace of the hidden state:
$a_n = f(W_n, a_{n-1}, x_n) = f(W_n, f(W_{n-1}, a_{n-2}, x_{n-1}), x_n)$, since $ a_{n-1}=f(W_n, a_{n-2}, x_n)$.
You may be able to see some issues with the above especially when it comes to sequences of larger length, usually tackled with LSTM and attention pooling architectures, but this is a different discussion!
Hope it helps.


Expectation number of points in initial clustering for LSH I have a very skewed, 10-dimensional data set. I need approximate nearest neighbours for my use case and I was looking into Locality senstive hashing. However after scaling and randomly generating hyperplanes through the origin and coding the data points like that I got very skewed buckets because of the nature of the data. After thinking about it for a bit I came up with an idea to take random points from the data and use these as cluster centers for the hash. Every point will be mapped to the ID of the closest randomly picked center. My question is if the expected size of the bucket of a specific point is the same for all the other data points. I think that is the case but others say it shouldn't. My rationale is that more dense areas have more randomly decided cluster points while outliers will not be picked very often. I can't find anything about this in literature.

Edit: I did run some tests and they seem to support my hypothesis somewhat but variance is relatively high because there is a high dependence between sizes of clusters (data points wise)

If you want to compute (approximate) nearest neighbors in 10-dimensional data, I would recommend using a k-d tree.  A k-d tree can handle skewed/non-uniform data sets and adjusts naturally to that.

More generally, you could look at data structures for nearest neighbor search, including binary space partitioning trees and metric trees.  But k-d trees are a good starting point, and they support both exact nearest neighbor queries as well as approximate nearest neighbor queries.

I would expect this to work better than locality-sensitive hashing (LSH).


What's the good index to choose number of clusters so that obtained clusters are homogeneous? I perform a clustering on one-dimensional dataset and I need a way to automatically decide what's the optimal number of clusters from $k \in \{2, 3, 4, 5, 6\}$. The number of observations to cluster is low (usually around 10-13). I think I'd need to check optimising for one of two goals (or both at the same time) and see what works best:

to achieve partitioning with the lowest within-cluster variances. Intuitively, I would go for something like average within-cluster variance, but I'm actually ok with the situation when some clusters would be formed out of single observation (it's actually desirable for outliers and that's why I check for relatively high number of clusters). And average within-cluster variance would always favour lower number of clusters.
to achieve partitioning with the most similar distances between pairs of observations within a cluster. For example, if I have objects $a, b, c, d$ in my cluster, I'd like to have $d(a, b) \approx d(b, c) \approx d(c, d)$ where $d$ is euclidean distance and $a, b, c, d$ are sorted.

I have studied scikit-learn options and none of them seems appropriate to my case.

Your problem is not appropriate for machine learning. Machine learning will not give robust answer to clustering itself (parameter) or automatic number of clustering (hyperparameter). The number of examples are too few (10-13) and the number of examples to the number of groups (2-5) is also too low.


why do transformers mask at every layer instead of just at the input layer? working thru the annotated transformer, I see that every layer in both the encoder (mask paddings) and decoder (mask padding + future positions) get masked. Why couldn't it be simplified to just one mask at the first layers of encoder and decoder?

If the masking were only applied in the first layer, the self-attention in the subsequent layers would bring to each position information from future tokens.
Let's break it down with numbers:

At layer $i$, if causal masking is applied, the output at position $t$ contains information about layer $i-1$ at positions $1..t-1$, that is, $L_{i,t} = f_i(L_{i-1,1},...,L_{i-1,t-1})$.

If no causal masking is applied, then the output at position $t$ contains information about layer $i-1$ at all positions in the sequence of length $T$, that is, positions $1..T$ $L_{i,t} = f_i(L_{i-1,1},...,L_{i-1,T})$

If causal masking is applied at layer 1 (the first layer) but not at layer 2 or 3, we obtain that for position t at layer 3 we would have: $L_{3,t} = f_3(L_{2,1},...,L_{2,T}) = f_3(L_{2,1},...,f_1(L_{1,1},...,L_{1,T}))$, which means that position $t$ contains information from future tokens, as $T > t$.

Note: The original answer was wrong and was completely edited. To check the original answer, refer to the post timeline.


Prediction with multiple features (attributes) I recently used a prediction algorithm to try to predict how much an user is willing to interact with an item using ALS (Matrix Factorisation) but I had to "condense" multiple attributes into one, deciding some arbitrary weights for the values to be able to calculate the rating to use.

It works (sort of) but I think it is definitely wrong because weighting the attributes manually is static and do not adapt to the data themselves dynamically.

I was thinking to switch to a Distributed Random Forest algorithm but even if it would use all the attributes (randomly) they would not be used all together and this makes it not fit for my purposes.

During my researches I found out that the Recurrent neural networks may be fit for what I am trying to achieve but before focusing on them, because I am not really an expert in the field, I would like to have suggestions :)

Potentially I could take advantage of the time-series feature of the RNNs but for now I prefer to focus on supporting the set of attributes I have.

Currently the "rating" for the ALS is calculated weighting some attributes like:

how many times the user viewed the item page
how many times the user viewed the category page
how many times the user viewed the sub category page
how many times the user bought something from the category
how many times the user bought something from the sub category
how much time ago the user bought the item
how much time ago the user bought something from the same category
how much time ago the user bought something from the same sub category

(and similar)

Is it worth trying with an RNN? If yes, should I try to go with tensorflow or caffe?

Thank you!

There is nothing "wrong" about weighting different actions to get a measure of preference between a user and an item, and then using ALS to compute a latent feature space. This is quite common in my experience. However, the consequence of ALS is that it can detect categories and sub-categories using individual user/item interactions. So, I don't see why you need to be adding higher level preference data into your user/item matrix. The weighting should be done using only behavior that indicates preference between a specific user and specific item, not user/category or user/sub-category interactions - ALS will detect if a user likes a category on its own.

How to weight the preference behavior is somewhat tricky, but you can use heuristics or information from your data. For example, a page view is not as strong as a product purchase and should be weighted less. You might look at the distribution of page views vs purchases to get an idea how common one is vs the other, to develop a weighting scheme.

I am sure that you could also use an RNN in a classification setting, to predict the next item that a user will purchase given an input sequence of the last n items they've purchased or interacted with. The framework you use depends on your preference, for Python based Keras, MXNet, PyTorch, and Tensorflow are all worthy. For easy Spark integration and JVM-based, DL4J is a good choice.


Data Augmentation Multi Outputs This question is asked several times here on SE, but I havent been able to find the right answer.
I'm trying to build a network with 1 input and 2 outputs. I don't have a lot of data so I would like to use a generator for augmentation (preferably with imgaug).
My code:

seq = iaa.Sequential([ ....
])

gen = ImageDataGenerator(preprocessing_function=seq.augment_image)
batch_size = 64

def generate_data_generator(generator, X, Y1, Y2):
    genX = gen.flow(X, batch_size=batch_size, seed=42)
    genY1 = gen.flow(Y1, batch_size=batch_size, seed=42)
    while True:
            Xi = genX.next()
            Yi1 = genY1.next()
            Yi2 = function(Y2)
            yield Xi, [Yi1, Yi2]

H = model.fit_generator(generate_data_generator(gen, trainX, trainY1, trainY2),
                epochs=epochs,
                steps_per_epoch=len(trainX) // batch_size,
                validation_data=(testX, [testY1, testY2]))

With this, I get the error:

'ValueError: ('Input data in `NumpyArrayIterator` should have rank 4. You passed an array with shape', (115, 16))'

a normal fit like this works just fine, so I don't there is anything wrong with the normal input

H = model.fit(trainX, {"output1": trainY1, "output2": trainY2},
    validation_data=(testX, {"output1": testY1, "output2": testY2}),
    epochs=epochs)

This is my input, I have no idea how I get a shape of (115,16) here

trainX.shape
trainY1[0].shape
trainY2[0].shape

Out[181]:
(115, 158, 100, 3)
Out[181]:
(16,)
Out[181]:
(9,)

thank you

You should pass X and Y collectively to the ImageDataGenerator.flow() method.

Please refer to this answer in case you are looking for a multi-output classification model using ImageDataGenerator in Keras.

https://datascience.stackexchange.com/a/75034/98109


Does class_weight solve unbalanced input for Decision Tree? I've read in sklearn's documentation that we have to take special care in balancing the input for a decision tree, but it doesn't tell you what function to use. However, I've found the parameter class_weight. If I use class_weight: balanced as a parameter, will that mean that I can omit balancing the input by hand?

Yes you don't need to balance your train data by hand. But your test data can still be (truly) unbalanced.


Why we use transposed filter as the deconvolution operation instead of the pseudo inverse of filter? I am trying to visualization CNN by the method in this paper "Visualizing and Understanding Convolutional Networks"

According to this tutorial A guide to convolution arithmetic for deep
learning (P.19)

We can always rewrite convolution as a matrix multiplication

For example, $Y$ is a feature map &  $X$ is input & $W$ is a filter & $\otimes$ is convolution operation

$$
\begin{bmatrix}
y_{11} & y_{12} \\\\
y_{21} & y_{22}
\end{bmatrix}
=
\begin{bmatrix}
x_{11} & x_{12} & x_{13} & x_{14}\\\\
x_{21} & x_{22} & x_{23} & x_{24}\\\\
x_{31} & x_{32} & x_{33} & x_{34}\\\\
x_{41} & x_{42} & x_{43} & x_{44}
\end{bmatrix}
\otimes
\begin{bmatrix}
w_{11} & w_{12} & w_{13} \\\\
w_{21} & w_{22} & w_{23} \\\\
w_{31} & w_{32} & w_{33}
\end{bmatrix}
$$

In the matrix multiplication form

$$
\begin{bmatrix}
y_{11} \\\\
y_{12} \\\\
y_{21} \\\\
y_{22}
\end{bmatrix}
=
C
\begin{bmatrix}
x_{11}\\\\
x_{12}\\\\
\vdots \\\\
x_{44}
\end{bmatrix}
$$

$$
C=
\left[\begin{smallmatrix}
w_{11} & w_{12} & w_{13} & 0 & w_{21} & w_{22} & w_{23} & 0 & w_{31} & w_{32} & w_{33} & 0 & 0 & 0 & 0 & 0\\\\
0 & w_{11} & w_{12} & w_{13} & 0 & w_{21} & w_{22} & w_{23} & 0 & w_{31} & w_{32} & w_{33} & 0 & 0 & 0 & 0\\\\
0 & 0 & 0 & 0 & w_{11} & w_{12} & w_{13} & 0 & w_{21} & w_{22} & w_{23} & 0 & w_{31} & w_{32} & w_{33} & 0\\\\
0& 0 & 0 & 0 & 0 & w_{11} & w_{12} & w_{13} & 0 & w_{21} & w_{22} & w_{23} & 0 & w_{31} & w_{32} & w_{33}\\\\
\end{smallmatrix}
\right]
$$

So we can visualize filter by multiplying $C^{-1}$.

OK, my question is why we use $C^{T}$ instead of the pseudo inverse $(C^{T}C)^{-1}C^{T}$?

First, transposed convolution isn't the inverse operation of convolution.
It only takes the output shape of origin convolution as its input shape and takes the input shape of origin convolution as its output shape.

Second, transposed convolution can be seen to a learnable upsampling operation. It just likes bilinear or bicubic upsampling but all of the parameters are learned by gradient descent.

https://www.youtube.com/watch?v=nDPWywWRIRo&index=12&list=PL3FW7Lu3i5JvHM8ljYj-zLfQRF3EO8sYv&t=1632s


Stateful LSTM for time-series prediciton - should each input sequence be shifted by 1 time step or by `sequenceLength` time steps I am building an LSTM to attempt to learn the historic trend of some time-series data set (e.g. the daily share price of a company). When training my network, I am taking batches of size 1, each consisting of 25 sequential daily closing prices, where it then makes a prediction for the price on the 26th day.

Say the first sequence in an epoch used to train begins at $t=0$ and ends at $t=24$. I am using a stateful LSTM and hence taking the output state of one batch and inputting it to the state of the next batch, does this mean that my 2nd batch must be $[t=25, t=49]$?

Can I instead "slide" each batch by 1 time step so that the 2nd batch is $[t=1, t=25]$, or does this defeat the purpose of passing the state between batches?

The second batch should be [t=25,t=49] because a Stateful LSTM must be sequential, because as you rightly say, the state of the network is at the state of t=24, if you then feed t=2 (or anything other than t=25) then the sequencing is messed up.

With stateless LSTM you can create sequences from any starting point, but with stateless LSTM the sequence must be observed.

In subsequent epochs you could change your start index so that different sequences are inputted. Nothing wrong with that. I personally generate a random number between 0 and the sequence length.


How to use MNIST dataset to make predictions on similar images (colorblindness charts)? I am trying to use the MNIST dataset to train a convolutional neural network to classify digits written in colorblindness charts. As some people have suggested, I have tried playing with the brightness and contrast, as well as converting to grayscale, but all of these have inconsistent results because all charts are very different.

I am looking for general ideas about approaches I can try. Would style transfer make sense? Here is an example of an easy chart I want to be able to classify:

I tried various transformations and got results such as these. Although to me, a mildly colorblind person, these are easier to see, it is not nearly close to MNIST.

TLDR:
I don’t have a dataset of color blindness charts. MNIST is readily available however. I’m trying to somehow use MNIST (probably after transforming it) to make the neural network classify the limited color blindness charts I have. MNIST and the charts are different and I need to make them similar so that the NN after training on modified MNIST can predict color blindness charts.

EDIT:

I tried the suggestions and applied various OpenCV Transformations to images. At the end, everything is resized and converted to grayscale. My accuracy went up from 11% (on non processed images) to 33% on the new images.

My problem is finding a set of transformations that are universal. My transformations work well for some images:

Some are not so good:

How can I improve this? Here are the transformations I do:

image = cv2.imread(imagePath)
    image = imutils.resize(image, height=400)
    contrasted_img = CONTRASTER.apply(image, 0, 60) #applies contrast
    median = cv2.medianBlur(contrasted_img,15)
    blur_median = cv2.GaussianBlur(median,(3,3),cv2.BORDER_DEFAULT)
    clustered = CLUSTERER.apply(blur_median, 5) #k-means clustering with 5
    gray = cv2.cvtColor(clustered, cv2.COLOR_BGR2GRAY) #RESULT

SECOND EDIT:

I used written advice and calculated that the digit usually takes up between 0.1 and 0.35 ish of the image. So I increase threshold for black and white until that happens. This resulted in images like this:

My accuracy of the NN went up to 45%! Another amazing improvement. However, my biggest issue is with images where the digit is darker than the background:

This results in incorrect thresholds. I also have some noise issues, but these are less common and can be fixed with tuning my percent white and threshold:

I think the $RGB$ color components of the pixels form vectors that if you see them in spherical coordinates, angles are colors, and radius is intensity.

I tried it and ended up with:

Build a 3-class classifier for the angles calculated for each pixel (background and two colors).

If you end up with only 2 classes (the third one has very few members), you have an intensity problem, and you should reclassify for radius (or just take a threshold value).

If you have 3 classes, you have a color problem. Drop the class of the (black) background color (probably just the class of the top left pixel. Also drop the class with the largest mean distance to the center of the image, the second background.

The remaining pixels will be white, the others black.

You still have a picture with circles forming the digit, but a pooling filter will even it out.

3-way classification directly on the image was unsuccessful, because the distance between light gray and dark gray is similar to light gray/light purple. The angles are much more separable.


Which visualization tool should I use? We consider the below function :
$f(x,y,z,a) = x*y*z*a$, where $x,y \in \mathbb{Q}\cap[0,1000],$ $ a\in\{2,3\} $
and $z=z(x)$, taking values from the below table based on the level-range that $x$ belongs to.

For instance, if $x=150$, then $z=5.$
I am looking for the proper visualization tool that can produce all possible 2-dim and 3-dim graphs of the above function.\
For instance,

if I fix $x=150$ and $a=2$, I want the 2-dim graph $(y,f(150,y,5,2))$
if I fix $a=2$, I want the 3-dim graph $(x,y, f(x,y,z(x),2) )$
and if I fix $y=10.5$, I want the two 2-dim graphs $(x,f(x,10.5,z(x),2))$ and $(x,f(x,10.5,z(x),3))$

I would like your suggestions on the above. Would Tableau do the job here ? I am not familiar with any tool and I am trying to chose the most situable to start with. Should I consider some other tool maybe? (Power BI or ??)
In addition, please advise on the functionality and the advanced options provided by the above tools.

Based on your explanation, $z$ is not a parameter so your function can be simplified like this:
$$f(x,y,a) = x*y*z(x)*a\text{, where } x,y \in \mathbb{Q}\cap[0,1000], a\in\{2,3\} $$
Note that $z(x)$ is a piecewise function. Also $a$ can have only two values so $f$ can be divided into $f_2$ and $f_3$, each of these having only two arguments so it simplifies things.
I don't know Tableau or Powerbi so I can't advise about that. I use R and I played a bit with your function. For a global visualization I came up with a kind of heat map like this:

In case you're interested, I did it like this with R:
library(ggplot2)

myrandomdata <- function(n) {
  a2 <- data.frame(x=runif(n,0,1000),y=runif(n,0,1000),z=20,a=rep(2,n))
  a3 <- data.frame(x=runif(n,0,1000),y=runif(n,0,1000),z=20,a=rep(3,n))
  df <- rbind(a2,a3)
  df$z[df$z<=500] <- 10
  df$z[df$z<=200] <- 5
  df$z[df$z<=100] <- 2
  df$f <- df$x *df$y * df$z * df$a
  df
}

df <- myrandomdata(50000)
ggplot(df,aes(x,y,colour=f))+geom_point()+facet_grid(a~.)+scale_color_gradient(low="blue", high="red")

(using random sampling is probably not the standard way but it was the easy way)


padding on mnist for LeNet Architecture LeNet accepts 32X32 image. So, to use LeNet for MNIST dataset,we have to change the size from 28X28 to 32X32. I came across This implementation. i am confused about how the following line of code work.

np.pad(X_train, ((0,0),(2,2),(2,2),(0,0)), 'constant')

Above line of code pad 28X28 pixed image to become a 32X32 image. Can anyone help me understand how exactly its done.

Basically, it does exactly what you specify. The used numpy function appends values in each dimension. The amount of "pads" on each axis is specified by ((0,0),(2,2),(2,2),(0,0)), given the dimension of your dataset, which is:

10000 (samples) x 28 (image dimension 1) x 28 (image dim. 2) x 1 (grayscale value!)

Now let's see what your specification means in that regard:

(0,0)Pad 0(as in the amount) values before and after each row
(2,2)Pad 2 before and 2 after each value of dim. 1 of your image data: 28 values -> 32
(2,2)Pad 2 before and 2 after each value of dim. 2 of your image data: 28 values -> 32
(0,0)Pad, again, nothing in the grayscale value dimension

That means you will end up with a 32x32 image in the respective dimension. Now, the only thing that's left is: Which values do we pad? The answer is quite simple, you do not specify any constant_values, meaning it will pad with the default constant_values (which is specified on the above linked page). Namely this value is 0.

To sum it up, simply imagine you have a 32x32 image, your 28x28 is in the middle, and on the outside you have a 2-value-thick border of 0's.


Can Generative Adversarial Network be run on any embedded / edge device? I am using DCGAN ( Deep Convolution GAN ) to generate images. However, I want to run it on embedded devices, such as Intel Movidiusor Jetson.

But, while going through several internet articles, I came to know that embedded devices are only used for inference in case of deep learning.

I have a few questions :

GAN is basically a generative network. How can it have inference ? Will the inference and training be same thing for GAN ?
Can it (GAN) be run on embedded devices , where memory is really an issue?

1. Mini introduction to Generative Adversarial Networks (GANs)

During the training phase, a GAN model essentially makes two neural networks compete against one another; the generator and the discrimator. As always we start with a training dataset, e.g. of images. Then:

The discrimator network is given images and must deicde whether or not they are real images
The generator learns how to generate new images, with the aim of tricking the discriminator into thinking the image is real

Here is a simple sketch from this article:

Because both networks are fighting over the same metric (the loss of the discrimator network), this idea is also described as a min-max problem. The discriminator is minimising it's own error, while the generator is maximising the same error.

More resources:

To see where GANs all started, have a skim of the seiminal paper by Ian Goodfellow. For a nice video introduction:

A general overview from Numberphile
A more detailed explanation from Ian Goodfellow himself, explaining GAN-related concepts, such as:

mode collapse, where the generator produces copies of the same image
using GAN to generate super-resolution images

2. Inference using GANs

Using the process outlined above, we have trained two neural networks. Performing inference could actually mean using either one of those (or a combination of both, I suppose). In your case, it would seem you only require the generator part:

  I am using DCGAN ( Deep Convolution GAN ) to generate images

This means you would take your trained generator model and use it to generate random samples by providing different random noise states as input (as is shown in the diagram above). Hopefully the learned distribution of your generator during training will be diverse enough to produce different samples. If it ends up always producing the same image, given different random noise samples as input, you are seeing an example of mode collapse (see linked video above).

3. GANs on embedded devices

To end up with a generator that is really of any practical use, you will need to do a lot of training, with many examples. State-of-the-art models use databases of images such as ImageNet, which contains 1 million images over 1000 categories.

Inference:

To perform this kind of training from scratch, I really don't think you will have any luck on an embedded or micro device, such as a Jetson.
To perform inference, however, you might be able to do it. This would simply require loading the pre-trained generator model onto the device (along with the deep learning framework required e.g. Tensorflow). This might work, and I have seen people manage to do this - there were frame rates of around 1 images per second on a conservative convolutional network.

Memory constraints are just a problem as computation itself. On an embedded device, this will likely be a painfully slow process, but could work out if you are able to optimise many parts of the network and training process. You could e.g. compress images to smaller scales, use a final output layer which has fewer neurons, or even go down the path of using less floating point precision in your calculations! See this paper on 8-bit training (as opposed to standard 32-bit). There are also smaller versions of well-known models such as tiny-YOLO.

Training:

Another approach to edge-training (online training) you might take would be to start with transfer learning. You could take a model that was already trained on a task similar to your own, which you will have to search for in open-source projects. This model could then be loaded onto the embedded device and made to work as was done for inference, however we want to train. We do this by keeping the weights from the pre-trained model, freezing the majority of the layers and just allowing the final one or two layers to actually be fine-tuned with your new edge data.


Beyond one-hot encoding for LSTM model in Keras I have an LSTM model in Keras for categorical classification (20 possible categories). In many cases, my data can fit multiple categories.

Obviously, my current model uses one-hot encoding and fits on that - that gives me accuracy and validation rates in the 50-60% but I want to improve that by comparing how the model does against the top 3 categories that the algorithm chooses.

Right now, I use Keras with categorical_crossentropy. I presume that this checks to see if the label is the top match and bases the accuracy on that matching. How can I modify the fit/training of the model to allow the labeled category to be in the top 3 (or top X-number) of matches for the accuracy score?

You are looking for top-k categorical accuracy. This is actually implemented in keras, but you might want to change the k, which can be done with partial function. This is the related link.


When is something a Deep Neural Network (DNN) and not NN? When would a neural network be defined as a Deep Neural Network (DNN) and not a NN?

A DNN as I understand them are neural networks with many layers, and simple neural networks usually have fewer layer... but what a many and a few in numbers? or is there some other definition?

What are networks trained used Tensorflow, Caffee as such? I haven't (as far I know) seen anybody manually design a network with many many layers.
They seem to promote their tools for creating DNN, but is it actually DNN if you only make a network with two layers?

You are right. Mainly any network with more than two layers between the input and output is considered a deep neural network. Libraries like tensorflow provide efficient architecture for deep learning applications such as image recognition, or language modelling using Convolutional neural networks and Recurrent neural networks. Another thing to keep in mind, is the depth of the network also has to do with the number of units being used in the layer. Mainly, as your non-linear hypotheses get complex you will need deep neural networks.


Artificially expanding the datasets through rotation of images in MNIST I came across this question from the 3rd chapter of the book Neural Networks and Deep Learning by Michael Nielsen, this is a question given in his exercise.

  One way of expanding the MNIST training data is to use small rotations
  of training images. What's a problem that might occur if we allow
  arbitrarily large rotations of training images?

I would happy if someone explain why large rotations would be problematic.

The problem is that numbers are not invariant to rotations.

For example, see what happens when you rotate a 4 in steps of 90 degrees:

So unless your task includes recognizing numbers which are written sideways or upside-down this does not provide a proper data augmentation.


Is my loss function right? WGAN I am new to GANs, but I was able to train a DCGAN decently. I decided to try a WGAN (not the improved one). I seem to get outputs, but my loss doesn't seem to converge for the generator.

I am using the loss function

from keras import backend as K
def wasserstein_loss(y_true, y_pred):
    return K.mean(y_true * y_pred)

for both the critic and generator. I am labelling real images as -1 and fake ones as 1.

Here's my loss over time:

Is the generator loss supposed to behave like that? Even though the discriminator loss seems to behave as expected, is a swing that large normal?

If anyone's wondering, yes it is right. The weight-clipping WGAN is very sensitive to changes in the weight-clipping hyperparameter, which is probably causing this. Upgrading to the gradient descent version fixed it. Loss should look like this btw


How to combine dictionary of word-forms into a single list without duplicates https://github.com/gutfeeling/word_forms

From the URL, I find out how to convert word into all the forms(verb, noun, adjective). I need that after conversion, list format.

e.g:

from word_forms.word_forms import get_word_forms

a=get_word_forms("review")
print(a)

output:

{'n': {'reviewer', 'reviewers', 'reviews', 'review'}, 'a': set(), 'v': {'reviewed', 'reviews', 'reviewing', 'review'}, 'r': set()}

my expected output is:

['reviewer', 'reviewers', 'reviews', 'review','reviewed', 'reviews', 'reviewing', 'review']

how this is possible?

You can get a list of all possible words (including the nouns, verbs, adjectives and so on) by using a comprehension. This loops over all elements of a sequence, as with a for-loop, but can be faster.

In you case, you have a dictionary, so we loop over each key-value pair. The keys are n (noun), a (adjective), v (verb) and r (??). For each of these keys, we have a list of words, which we can append to your final desired output list, if they aren't already in the list.
Because we loop over a dictionary, and then over each of the lists, with becomes nested:

from word_forms.word_forms import get_word_forms
word_forms = get_word_forms("review")

desired_words = []
for word_type, word_list in word_forms.items():
    for word in word_list:
        if word not in desired_words:
            desired_words.append(word)

print(desired_words)
# ['reviewers', 'review', 'reviewer', 'reviews', 'reviewing', 'reviewed']

This can be compressed into a nested comprehension:

desired_words = []
{desired_words.append(word) for word_type, word_list in word_forms.items() for word in word_list if word not in desired_words}

print(desired_words)
# ['reviewers', 'review', 'reviewer', 'reviews', 'reviewing', 'reviewed']

We didn't need to save the resulting object, we only want to side-effect of appending to desired_words

Another way to do this would be to simply put all the words into a list, then convert it into a set() object, which would remove all duplicates and order the list alphabetically:

desired_words = []
# the 'if' statement is removed in the dictionary comprehension:
{desired.words.append(word) for word_type, word_list in word_forms.items() for word in word_list}
# Set removes duplicates - we then wrap it back into a list
desired_words = list(set(desired_words))


Is there a RandomForest implementation that handles categorical data without encoding in python? I am working on a binary classification project with both continuous and categorical features. I know that the R implementation of RandomForest can handle categorical data passed in as factor type data. Python's scikit-learn implementation requires encoding of categorical data however (e.g. one-hot). I'm curious about the difference in the results I receive using the two implementations, and I'm wondering if anyone knows of a python implementation of RandomForest that can handle categorical data without encoding.

Because sklearn uses the CART algorithm it cannot accept categorical data as-is (as you have pointed out). There is an existing ticket out to change this.
The issue really should be rephrased as supporting categorical splitting on Decision Trees - not Random Forest, as Random Forrest is simply the ensemble method using these decision trees as fitters.
I don't know of a python package that supports this functionality but I do suspect that it would help increase performance because it would avoid a common pitfall of using random forest on hot-encoded data; see One-Hot Encoding is making your Tree-Based Ensembles worse, here’s why?
In the meantime - I use TargetEncoder when I know I need to use Random Forrest. It allows the categorical data to be encoded in a "smart" fashion without needing to use one-hot encoding.


How do we decide which kernel needs to be used in SVM? How do we decide which kernel needs to be used for a particular dataset?
Is there any criteria needs to be followed?
And also what is the criteria to select C and gamma values?

Kindly excuse me if this question looks silly.

Thank you.

Without knowing the geometry of your data, determining a kernel is generally achieved through trial and error.  The radial basis function (RBF) kernel is a good starting choice because most data are not linearly separable.  Fortunately training an SVM is fast, so brute-forcing the kernel search is not a terrible method.

For selecting the C and gamma values, we generally use a grid search:

Every cell in the graph represents a SVM model trained on your data given the respective C and gamma parameters.  The color represents its accuracy depicted by the scale on the right.  Note that our scales are logarithmic, so we are able to search a broad solution space while only training 169 models.

From this example, we can predict that the optimal (C,gamma) value combination would be in the region around $( 10, 0.1 )$ or $( 10^{10}, 1e^{-7}) $

We can further refine our C and gamma parameters by "zooming" into the regions with greatest validation accuracy.  In other words, we can repeat the grid search with finer granularity and with boundaries determined by the regions of interest until we find the optimal C/gamma combination.


How to convert a mel spectrogram to log-scaled mel spectrogram I was reading this paper on environmental noise discrimination using Convolution Neural Networks and wanted to reproduce their results. They convert WAV files into log-scaled mel spectrograms. How do you do this? I am able to convert a WAV file to a mel spectrogram

y, sr = librosa.load('audio/100263-2-0-117.wav',duration=3)
ps = librosa.feature.melspectrogram(y=y, sr=sr)
librosa.display.specshow(ps, y_axis='mel', x_axis='time')

I am also able to display it as a log scaled spectrogram:

librosa.display.specshow(ps, y_axis='log', x_axis='time')

Clearly, they look different, but the actual spectrogram ps is the same. Using librosa, how can I convert this melspectrogram into a log scaled melspectrogram? Furthermore, what is the use of a log scaled spectrogram over the original? Is it just to reduce the variance of the Frequency domain to make it comparable to the time axis, or something else?

I think you're wrongly interpreting what the authors meant by log-scaled. When the authors mention log-scaled, they are not referring to the frequency (y) axis, although spectrograms are typically log-scaled here. They are instead referring to the scale of the 3rd dimension in the spectrogram. In your case, the raw spectrogram is displaying power in color. What you want is instead decibels, which are log-scaled.

In your case, the code would look like this:

y, sr = librosa.load('audio/100263-2-0-117.wav',duration=3)
ps = librosa.feature.melspectrogram(y=y, sr=sr)
ps_db= librosa.power_to_db(ps, ref=np.max)

lr.display.specshow(ps_db, x_axis='time', y_axis='mel')

Note: Each spectrogram will be scaled based off of the ref within librosa.power_to_db. If you do not supply anything, librosa just shoves a 1 in there, which may or may not be what you're looking for. You can also try out np.median.


Training Neural network classifier using string inputs My thesis topic is about building a (deep) neural-network classifier to classify the type of a place. I am given both labels and some inputs in string type. So for example the label "Supermarket" might have a feature like "Food".

How should I feed my string input features to the neural network?

In other words, is there any efficient way to substitute those string inputs with numerical values knowing that there are A LOT of them?

You always avoid feeding direct strings into neural networks. This thread here explains why you should avoid doing this : Neural Network parse string data?

Once you convert the strings you have into vectors or any other form of numerical representation and encoding your labels as categorical, it will solve the problem you have at hand.

If you need me to elaborate more on this, I would be more than happy to do so.


Imbalanced dataset - Undersampling & multiple classifiers Let's suppose that my dataset in a classification problem looks like that:

class A: 50000 observations
class B: 2000 observations
class C: 800 observations
class D: 200 observations

These are some ways which I considered to deal with this imbalanced dataset:

I reject straight away oversampling because it usually makes the model overfit (in the minority classes) by a lot.
Secondly, if I run the classifier with the data like that then it will be overclassifying documents in class A so I reject this method too.
Another approach is to do undersampling and reduce class A to let's say 4000 documents (where I tested it and it gives the best results so far).
However, in this way I am losing quite a lot of information. So I am wondering if building multiple classifiers with 4000 documents each for class A (different at each time) is a better solution (although I think that this approach resembles quite a lot the oversampling approach which I rejected).

What do you think of method (4) comparing to method (3)?

Maybe your (3) can be complemented to an oversampling of the other classes. I don't think every oversampling generates overfitting. I agree with you, it will make you lose information. But the only way to know if it's that bad is by checking it.

Regarding (4), I don't see how you'll manage the predictions of these multiple classifiers. You could test binary classifiers for each class, like classifier1 for A/not-A, classifier2 for B/not-B. In this case, undersampling could be applied.

However, all these advices are just speculation. You must test on your data, and see the results with more evaluation tools, such as learning curves and feature importance analysis.


How can I handle missing categorical data that has significance? I have a data set that is highly categorical and has a lot of missing values. For instance:

i | A_foo | A_bar | A_baz | outcome
--+-------+-------+-------+--------
0 |  nan  |  nan  |  nan  |   1
1 |   0   |   1   |   0   |   1
2 |  nan  |  nan  |  nan  |   0
3 |   1   |   0   |   0   |   0

The problem is that 1 and 0 have a different meaning that nan. I don't want to impute the data because assigning values of 0 or 1 will bias my data, but many machine learning algorithms will not work on a dataset with missing values. How can I handle this?

Imputation and dealing with missing data a broad subject; you should start by researching standard material on this subject.

The first question to figure out is Why is some data missing?  and What is the process that causes data to be missing?  It's important to understand how this happens, because this will affect what solution is appropriate.

Randomly missing data

If data is missing totally at random (whether a value is missing does not depend on any of the feature values of that item), then imputation can be appropriate.  It should not create bias, if you do it appropriately.  There are many techniques for imputation.  You don't mention what you tried or why you think it will bias your results, but in general, if you use an appropriate method of imputation, there is no reason why it needs to bias your data.

Alternatively, you can use a classifier that can tolerate missing data.  Some classifiers are designed to handle missing data and can tolerate it.  However, I don't know of any reason to use them over imputation.

Non-randomly missing data

In contrast, if the chance for data to go missing for some object depends on the value of the features of that object, then you have a bigger problem.  In that case imputation can create bias -- as can any other method.  Your best hope is to understand in greater depth the random process that causes data to be missing and the probability distribution (probability that data goes missing, as a function of feature values), and try to design a procedure that is appropriate for that process.

Your specific situation: all features missing

Your specific situation is especially weird: it appears in your case either all features are missing, or none are.  That's a weird one.  For instances where the features are missing, you have absolutely no information about those instances.  So, the best classification decision in that case is probably a very simple rule: take whichever class appears most frequently in your training set (or, most frequently among instances with missing data).  Run the classifier on the remaining instances, i.e., the instances with no missing data.

But in real life this situation is pretty rare.  It's more typical that some features are missing and others are present, and that requires more work to handle.


Semantic text similarity using BERT Given two sentences, I want to quantify the degree of similarity between the two text-based on Semantic similarity.
Semantic Textual Similarity (STS) assesses the degree to which two sentences are semantically equivalent to each other.
 say my input is of order:

index    line1                       line2
0        the cat ate the mouse      the mouse was eaten by the cat
1        the dog chased the cat     the alligator is fat
2        the king ate the cake      the cake was ingested by the king

after application of the algorithm , the output needs to be

index    line1                       line2                           lbl
0        the cat ate the mouse      the mouse was eaten by the cat    1
1        the dog chased the cat     the alligator is fat              0
2        the king ate the cake      the cake was ingested by the king 1

Here lbl= 1 means the sentences are semantically similar and lbl=0 means it isn't.
How would i implement this in python ?
I read the documentation of bert-as-a-service but since i am an absolute noob in this regard I couldn't understand it properly.

BERT is trained on a combination of the losses for masked language modeling and next sentence prediction. For this, BERT receives as input the concatenation of the special token [CLS], the first sentence tokens, the special token [SEP], the second sentence tokens and a final [SEP].

  [CLS] | First sentence tokens | [SEP] | Second sentence tokens | [SEP]

Some of the tokens in the sentences are "masked out" (i.e. replaced with the special token [MASK]).

BERT generates as output a sequence of the same length as the input. The masked language loss ensures that the masked tokens are guessed correctly. The next sentence prediction loss takes the output at the first position (the one associated with the [CLS] input and uses it as input to a small classification model to predict if the second sentence was the one actually following the first one in the original text where they come from.

Your task is neither masked language modeling nor next sentence prediction, so you need to train in your own training data. Given that your task consists of classification, you should use BERT's first token output ([CLS] output) and train a classifier to tell if your first and second sentences are semantically equivalent or not. For this, you can either:

train the small classification model that takes as input BERT's first token output (reuse BERT-generated features).
train not only the small classification model, but also the whole BERT, but using a smaller learning rate for it (fine-tuning).

In order to decide what's best in your case, you can have a look at this article.

In order to actually implement it, you could use the popular transformers python package, which is already prepared for fine-tuning BERT on custom tasks (e.g. see this tutorial).


What are examples for XOR, parity and multiplexer problems in decision tree learning? In scikit-learn documentation and in decision tree learning Wikipedia article there is mention of "There are concepts that are hard to learn because decision trees do not express them easily, such as XOR, parity or multiplexer problems."

I can't recall knowing such problem types.

What are examples (and possibly related datasets) of such problems?

Below is an example of XOR dataset for classification. As you can see, decision trees perform pretty poorly on this dataset. Reason is decision trees splits space into rectangular regions. Therefore they are not pretty good with this kind of distributions.

If you really want to use trees in that sort of situation, it is interesting to use so-called rotation trees. Rotation is about performing PCA (principal components analysis) on input features while learning trees. Using it, decision trees can then build non-rectangular regions.

Also, here is a playground to test gradient boosting algorithms including on XOR dataset. It is really interesting ! You can click on "rotate trees" to activate rotation.
Gradient boosting playground


Information Extraction from Free-form text to create Transactions I'm working on a use case where the user will be provided a text box to enter the details of the transaction application.
For Example user might enter the below text and I have to parse the data and accordingly create the transaction. (I took basic example as a startup)

  Ex 1 : Transfer $100 from Account 1 into Account 2
  Ex 2: Move Money of total 400 dollars into account owned by Mr.xxxx from Acct#
  Ex 3: Deposit an amount of five thousand dollars into acct#

For above examples I have to parse and convert into a tabular format like
Trx_Type (Deposit or Transfer), Amt, From_Acct#, To_Acct#, Acct_Holder_Name,etc.

I'm trying to look options by doing Named Entity (NER) or Classifier modelling using NLTK in python. Can someone please share ideas from where to start and guide me with a draft way to proceed.
Looking forward for a response. Thanks!

My answer is based on couple of assumptions:

user input is more or less standard, so there won't be "Ex 20000"
you have at least majority of forms of input covered

In every representative example of transaction description you would need to mark words of interest, be it name of holder and account number. You can start small with 10-20 examples for start, and then, when you have all required fields marked, you can train a sequence labelling model, or, say it, custom Named Entity Recognition model, which will parse new text and extract required data for you.

How to train actual model is answered in that question: Help regarding NER in NLTK , and how to mark up the data - is more of a question you should answer, since only you do know, what should be marked as account number, account holder etc.

If you would be digging into NER training, I would advise you not to rely only on current word features, but add some regexp-alike features and dictionaries as features, since task seems very limited in terms of context variation.


How to find the size of a tensor in bytes? Supposing I have a tensor named encoding which is of the shape (1,32) Now I would like to find out the size of the tensor in bytes not the shape of it. So basically I want to know how much memory it takes.

I found this StackOverflow question which does this exact thing but in TensorFlow 1. So how can I accomplish the same?? Here is the link ===> https://stackoverflow.com/questions/36966316/how-to-get-the-dimensions-of-a-tensor-in-tensorflow-at-graph-construction-time

  The SO question uses the Tensorflow.contrib module which (sadly) has been removed. So anyone know a way?

Step 1 :Get the dtype of the tensor. This will tell you about the number of bytes e.g.float64 is 64 bits = 8 Bytes.

Step 2 Get the shape of the Tensor. This will give you the number of place-holders of the dtype. lets's assume shape = m x n x p
Count of the placeholders is C = m * n * p
Memory = 8 * C => Memory = 8 *m * n * p Bytes.

See this excerpt from Deep Learning with Python, by Francois Chollet

  A 60-second, 144 × 256 YouTube video clip sampled at 4 frames per second would have 240 frames. A batch of four such video clips would be stored in a tensor of shape (4, 240, 144, 256, 3). That’s a total of 106,168,320 values! If the dtype of the tensor was float32, then each value would be stored in 32 bits, so the tensor would represent 405 MB


Can BERT/ELMo be used (or retrained) to generate a text in both directions? Text generation is perhaps one of the fun things to do with old NGram or new BERT/ELMo models. I am wondering can BERT be used to generate text from the end of a sentence, or better in both directions. That is instead of giving some starting words, we give some ending words.

Neither BERT nor ELMo can be used as-is for next word (or previous word) predictions. BERT is trained on a masked language model (LM) task and can therefore only be used to guess masked words, obtaining this way a contextual representation of them. There have been some attempts to use BERT for text generation, but they have been unsuccessful up to now. ELMo is a bidirectional LM, so you need the whole sentence to use it as input.

Both BERT an ELMo use the whole sentence (except masked tokens for BERT) by construction, so they can hardly be trained on "causal" LM tasks without modifying the models:

BERT would need to be added a self-attention mask to force causality. This would be equivalent to using BERT's pre-trained weights as initial values for a normal Transformer (causal) language model or any text-generation model, like a neural machine translation (NMT) one; actually, this has been attempted for NMT here and here.
ELMO should be stripped from one of the directions' LSTM. This would be equivalent to using one of the direction's parameters to initialize an LSTM for text generation.

Another option would be to use them for knowledge distillation. This has been done with BERT.


How to retrieve tweets from all cities using Twint package in python? Using Twint, I'm able to get tweets from each city in the list, but while creating a dataframe I only get tweet from the last city in the list, is it possible to append the tweets based on cities?
india = ["chennai","pune","mumbai"]
for city in india:
    print(city)
    c = twint.Config()
    c.Search = "phone"
    c.Lang= "en"
    c.Pandas = True
    c.Since = "2021-01-01"
    c.Near = city
    c.Limit = 10
    twint.run.Search(c)
data = twint.output.panda.Tweets_df(["date","tweet","near"])
data.head()

        date                              tweet                              near
0   2021-03-23 18:47:09 Cameron FaulknerThe Black Shark 4 is a high-en...   mumbai
1   2021-03-23 18:42:21 Asad just break Zoya's phone so she won't expo...   mumbai
2   2021-03-23 18:40:13 @Babi3Kat Send you phone number an                  mumbai
3   2021-03-23 18:34:12 @sahilbaani @SamsungIndia ????? 5 Saal purane ...   mumbai
4   2021-03-23 18:33:52 @91mobiles They are just started to make more ...   mumbai

You are only seeing the last one since you overwrite your object df at each iteration and when the loop is finished it will store only the last value store which happens to be mumbai
If you want to create a DataFrame with all the results you should do something like this:
ldf = list()
india = ["chennai","pune","mumbai"]
for city in india:
    print(city)
    c = twint.Config()
    c.Search = "phone"
    c.Lang= "en"
    c.Pandas = True
    c.Since = "2021-01-01"
    c.Near = city
    c.Limit = 10
    twint.run.Search(c)
    tmp = twint.output.panda.Tweets_df(["date","tweet","near"])
    ldf.append(tmp)
data = pd.concat(ldf, ignore_index = True)
data.head()


Cluster titles or ingredients of food into n-categories I have a dataset which has information about food recipes (in german), that looks like this:

Here is a link to a small .csv file (first 1000 rows of my data)
https://drive.google.com/file/d/1C7thFlOnDn-oTc6AaDWA3CXXcX8m9NRu

The idea is to cluster the recipe names into n-categories so that afterwards I can assign every recipe to a category. Of note, there are tags and ingredients for every recipe, maybe this information helps to refine the clusters?
For example the algorithm (maybe a semantic analysis?) should output: categorised recipes into 200 biggest found categories: (hamburger, soup, pizza, ...)

Is there a way to do this?

Note: I have for every recipe min. 1 image. The idea is to label my images with n-categories, afterwards to train a convolutional neural network with my data.
The input would be an food image, the output would be a category.

This sounds like a job for latent dirichlet alocation (i.e. topic modeling).

https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation
https://radimrehurek.com/gensim/wiki.html


Where can I find datasets with large number of features (100000) but less samples(100-1000)? I am working on a dimensionality reduction algorithm for datasets with high feature to sample ratio. I need datasets with around 100000 features or more and 100-1000 samples with class labels to evaluate my algorithm's performance.

Moreover the features should be real valued and not binary like in the Dorothea dataset.

Just take a dataset that has a large number of features and ignore as much data as you need to get the ratio you desire. E.g. take imagenet, but randomly select 100 samples per class.


Why my classification accuracy is high for both training and testing data? I have a dataset with 10 features and 1 binary classification target. I tested this dataset with decision tree classifier. I did some basic check like missing values but the data looks clean. My classification accuracy for both training and testing data is really high and it's looks suspicious. I would like to know whether I am doing any mistake or is there any way to explain why the accuracy is too high?
Can anyone advise me here?
import pandas as pd
from sklearn.model_selection import KFold, StratifiedKFold, RepeatedKFold, RepeatedStratifiedKFold, cross_validate, train_test_split
from sklearn.tree import DecisionTreeClassifier, DecisionTreeRegressor

df = pd.read_csv('https://docs.google.com/spreadsheets/d/e/2PACX-1vSvUmbtHUh2e0iYj7nMDaP8Tf_pCnCa-HrWwAmaxrERxxvd2y_5qxuSP10t6db4RUSjTdOi9WshZhoR/pub?output=csv')

input_selected_features = df.drop(labels = 'Target', axis = 1)
target_selected_feature = df['Target']
X_train, X_test, y_train, y_test = train_test_split(input_selected_features, target_selected_feature, test_size = 0.2, train_size = 0.8, random_state = 101)

# k fold cross validation scores
kf = KFold(n_splits = 10, shuffle = True)
cv_results = cross_validate(estimator = DecisionTreeClassifier(), X = input_selected_features, y = target_selected_feature, cv = kf, scoring = ['accuracy', 'f1'], return_train_score = True)

# print(cv_results)
print('training accuracy - ',cv_results['train_accuracy'].mean())
print('testing accuracy - ',cv_results['test_accuracy'].mean())
print('training f1 score - ',cv_results['train_f1'].mean())
print('testing f1 score - ',cv_results['test_f1'].mean())
```

Your data contains 7621 records and 3873 of them are unique records.
When there is duplicates in different folds, cross-validation fails. And these duplicates causes to high train&test set performance metrics. Removing the duplicates decreases the test set's accuracy and f-1 scores:
import pandas as pd
from sklearn.model_selection import KFold,cross_validate, train_test_split
from sklearn.tree import DecisionTreeClassifier

df = pd.read_csv('https://docs.google.com/spreadsheets/d/e/2PACX-1vSvUmbtHUh2e0iYj7nMDaP8Tf_pCnCa-HrWwAmaxrERxxvd2y_5qxuSP10t6db4RUSjTdOi9WshZhoR/pub?output=csv', index_col=0)
df.drop_duplicates(inplace=True)

input_selected_features = df.drop(labels = ['Target'], axis = 1)
target_selected_feature = df['Target']

# k fold cross validation scores
kf = KFold(n_splits = 10, shuffle = True, random_state=42)
cv_results = cross_validate(estimator = DecisionTreeClassifier(), X = input_selected_features, y = target_selected_feature, cv = kf, scoring = ['accuracy', 'f1'], return_train_score = True)

# print(cv_results)
print('training accuracy - ',cv_results['train_accuracy'].mean())
print('testing accuracy - ',cv_results['test_accuracy'].mean())
print('training f1 score - ',cv_results['train_f1'].mean())
print('testing f1 score - ',cv_results['test_f1'].mean())

> training accuracy -  0.9990828974566813
> testing accuracy -  0.7454174325368284
> training f1 score -  0.9990071171229987
> testing f1 score -  0.727174422877005

Note that, also the first column in the original csv file is an index column (1,2,..). It is better to read that column with index_col argument.


How to use plot model in keras? I'm trying to draw the keras model with the plotmodel.

Setup:

I installed graphviz binaries with:

choco install graphviz

added path to the bin folder, and then I did:

pip install pydotplus
pip install graphviz

Code:

Getting this error when I try to execute:

from keras.models import Sequential
from keras.layers import Dense
from keras.utils.vis_utils import plot_model
model = Sequential()
model.add(Dense(2, input_dim=1, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
plot_model(model, to_file='model_plot.png', show_shapes=True, show_layer_names=True)

Error:

  ImportError: Failed to import pydot. You must install pydot and graphviz for pydotprint to work.

Can someone suggest some alternative to draw the keras model? I need to implement this for a CNN.

In addition to what you have already done, in my notebook I added two lines:

import os
os.environ["PATH"] += os.pathsep + 'C:/Program Files (x86)/Graphviz2.38/bin/'

(Source)


How to find which features have been selected by PCA algorithm? I used PCA function in MATLAB to decrease features on my data set.

By this code I can reduce features from 12 to 8(as an example). It works good but my question is that how can I found with feature have been removed or which feature are selected on the result?

[~, pca_scores, ~, ~, var_explained] = pca(myDataSet, 'NumComponents', 8);

PCA doesn't remove any specific feature.

What PCA does it to calculate linear combinations of your variables in such way that they get "summarized" in fewer variables.

You don't eliminate variables, you reexpress them.


R Rpart taking too much time on data set I am trying to build a CART model using  rpart on a data set with around 7k rows and 456 columns

cmodel2=rpart(DV ~ .,data=teltrain2,method="class")

This has never returned a result yet. I am using a 16GB ram machine with R 3.2.3. Every time on running this, R execution goes on and on, I have to kill the R GUI process to finally stop it.

Description of the columns

id --> unique id (which I have removed from the training set)
Location --> number
col3 to col400 --> these have been hot encoded by me and include just 0 or 1 as values
col400-col456 --> these are features which have numeric values like 23,144 etc, all less than 1000
DV --> dependent variable is categorical with more than 2 values.

I have tried running rpart with a smaller set of 1000 rows and 456 cols, still no luck. I don't want to do any kind of feature elimination on the first run as as of now I have no idea whether they affect the dependent variable or not.

I have tried mentioning minbucket=25,20,15,10 with no luck either.

My objective is to get the resulting model of rpart on the entire data set. What are my options?

Have you tried using randomForest instead of rpart? For example, let's assume you have two data.frames:

train_data and test_data

In my example, the last column is the class (and is a factor variable), and all other variables (all other columns) are numeric. For example, one training example might look like:

4 5 10 12 1 0

where 0 is the class label. You can do the following to run a random forest with 50 trees:

library(randomForest)
model_pl_1 <- randomForest(train_data[,-ncol(train_data)], train_data[,ncol(train_data)], test_data[,-ncol(test_data)], test_data[,ncol(test_data)], ntree=50)


How to save google colab notebook outputs and variables I want to save variables and outputs in text file with colab!

Any bright idea?

Save your content in the current directory and execute !dir to see the content. Then follow the following code snippet:

from google.colab import files
files.download('your typical text file or what ever.txt')

For more information take a look at here.


application of CNN in genomics Since CNN has been widely applied in DNA sequence data, I'm wondering why CNN is not often used for predicting phenotype from SNP data, given that
 SNPs are essentially parts of DNA sequences and retain the nature of ordering.

For instance, this paper https://arxiv.org/abs/1611.09340 states that "... most of these techniques(CNNs) are based on sequence data where convolutional or recurrent networks are appropriate. When the full DNA sequence is unavailable, such as when data is acquired through genotyping, other methods need to be used..."

Please explain regarding the algorithms rather than the biology side(e.g. SNPs only explains a small portion of genetic variation, etc)

Hey guys, I just wanna remind that I'm addressing the use of SNPs(or you can include other types of variants as well)...not full DNA sequence data. It's very common to use CNN on full DNA sequences to predict variation loci. And it's easy to understand as it's natural to use CNN on sequential data.

But what I don't understand is why not using CNN (1)on SNPs (since SNPs retain the major spatial structure of DNA sequences) (2) to predict phenotype?

CNN has been used many times in genomics, what you described is not new. A 5s quick search on Google gave me this paper:

  https://www.biorxiv.org/content/biorxiv/early/2017/12/31/241414.full.pdf

  ...predict phenotypes from genotypes by using a deep convolutional neural
  network (CNN)...

"Genotype" is a better terminology than "SNP", as short indels are also important. Your idea is certainly not new, and in fact insufficient as a research topic anymore.

Predicting phenotype from just genotype is old and quite limitied. No good journal will take a topic like that anymore. No matter how good your model is; no matter how many layers you have (e.g. 1000000 trillion layers), how many GPUs you have (e.g. every single GPU in the world), how many engineers you have (e.g. every single ML engineer in the world). CNN will never solve problem perfectly as it neglects the vital information from proteins. Whatever you see from a paper would be limited in applications (e.g. strong assumptions). Genotype data don't tell you how exactly your genes are transcribed.

Instead, the current research trend is on applying deep learning networks integrating mutliple data sources (proteins, transcripts, genes, genotypes etc), and form a single unifed model. This is why we have AI for precision medicine.


invalid character in identifier I ran this code and this doesn't work, I'm using python 3 btw, I have checked the syntax a million times. I have installed all the necessary packages and all of them are up to date, here is the code I ran:

from sklearn import tree
features = [[140, 1], [130, 1], [150, 0], [170, 0]]
labels = [0, 0, 1, 1]
clf = tree.DecisionTreeClassifier()
clf = clf.fit(features, labels)
print(clf.predict([[150, 0]]))

Here is the console error message (I don't know what it's exactly called, please tell me if you know):

pydev debugger: starting

Traceback (most recent call last):
  File "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Extensions\Microsoft\Python\Core\ptvsd_launcher.py", line 111, in <module>
    vspd.debug(filename, port_num, debug_id, debug_options, run_as)
  File "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Extensions\Microsoft\Python\Core\Packages\ptvsd\debugger.py", line 36, in debug
    run(address, filename, *args, **kwargs)
  File "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Extensions\Microsoft\Python\Core\Packages\ptvsd\_main.py", line 47, in run_file
    run(argv, addr, **kwargs)
  File "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Extensions\Microsoft\Python\Core\Packages\ptvsd\_main.py", line 98, in _run
    _pydevd.main()
  File "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Extensions\Microsoft\Python\Core\Packages\ptvsd\pydevd\pydevd.py", line 1628, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Extensions\Microsoft\Python\Core\Packages\ptvsd\pydevd\pydevd.py", line 1035, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Extensions\Microsoft\Python\Core\Packages\ptvsd\pydevd\_pydev_imps\_pydev_execfile.py", line 25, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:\Users\Sanjay\Documents\python files\SLNforVt\VisualTest\VisualTest.py", line 6
    print(clf.predict([[150, 0]]))
                                  ^
SyntaxError: invalid character in identifier

I am using Visual Studios here, I do not know if that affects this program in anyway but I also tried it using the python idle. Other Python programs I write work fine on Visual Studios without any errors.

Your text has some characters that the editor does not show.  Using an online UTF-8 encoder, you can see that the end of the line has:

\x29\xef\xbb\xbf\x0a

When it should simply be a close paren and a new line like:

\x29\x0a

You need to delete the offended characters.  Best way to handle these situations, is to select an area of the text on either side of the compiler complaint and delete it all as a block and then retype the text.

After the cleanup your code produces:

[1]

PS: That error output is called a stack trace, and it was exactly right, there was an invalid character present at the location indicated.


Wouldn't it make more sense to give less importance to gradient far away in past in AdaGrad? This is the update equation of a weight by AdaGrad:
$$w_{new} = w_{old} - \frac{lr}{\sqrt{G_{}+E}}.G_{w_{old}}$$
Where $G$ is the sum of the gradients of the same weight at previous iterations, $E$ is just a small value to prevent division by zero and $G_{w_{old}}$ is the gradient of $w_{old}$
Here $G$ is the sum of all the gradients, and so we are giving equal importance to all the gradients irrespective of the timestep.
Now, wouldn't it make more sense to give more importance to gradients than the one in distant past. Because, this is what we do in SGD with momentum. We give more importance to the gradients than the ones in the distant past.
So, why don't we do the same thing over here, Giving more importance to the gradients than the ones in the distant past?

One small correction- G here is the sum of squares of the gradient of a particular parameter which is kept track of and is monotonic in nature.
That being said, the way I see the equation functions is, it lets the past dictate the future or simply put, the past gradients of the parameter decides by how much it's corresponding learning rate drops.
In SGD no matter what they gradients were in the distant past their importance drops exponentially(assuming $\eta < 1$). But AdaGrad, seems to be meant for data with sparse features and here, though the update rule gives equal importance to the past gradients, the update is sporadic instead of being regular.
I understood your question and for me at least the concept of sparse features makes the update rule of AdaGrad sense.
Quoting from the paper for quick reference-
In many applications of online and stochastic learning, the input instances are of very
high dimension, yet within any particular instance only a few features are non-zero. It is
often the case, however, that infrequently occurring features are highly informative and
discriminative.
.
.
.
In contrast, our algorithms dynamically incorporate knowledge of the geometry of the data
observed in earlier iterations to perform more informative gradient-based learning.
Informally, our procedures give frequently occurring features very low learning rates and
infrequent features high learning rates, where the intuition is that each time an
infrequent feature is seen, the learner should “take notice.”


Syntax error in function but everything seems correct I am custom defining a function and following is my function but I get error as below
def  saveSlice(img, fname, path):
     img  = np.uint8(img * 255)
     fout = os.path.join(path, f'{fname}.png')
     cv2.imwrite(fout, img)
     print(f'[+] Slice saved: {fout}', end='\r')

I get error message as below
File "ipython-input-79-e8d731623c8b", line 3
fout = os.path.join(path, f'{fname}.png')
                                                                           ^

SyntaxError: invalid syntax
What am I doing wrong?

If you use Python earlier than 3.6 f-strings are not available, and that code has an incorrect syntax. In that case you can use the format method of the strings instead like that (2 lines are modified):
def  saveSlice(img, fname, path):
     img  = np.uint8(img * 255)
     fout = os.path.join(path, '{}.png'.format(fname))
     cv2.imwrite(fout, img)
     print('[+] Slice saved: {}'.format(fout), end='\r')


Issue with output dimensions in keras I'm currently trying to build and train a model for CIFAR data using keras.
My labels should be one-hot encoded.

data.y_train.shape

is (45000, 10).
My model is defined like this:

model = keras.models.Sequential()
model.add(keras.layers.Conv2D(32, (3, 3), activation='relu', input_shape=(32, 32, 3)))
model.add(keras.layers.MaxPooling2D((2, 2)))
model.add(keras.layers.Conv2D(64, (3, 3), activation='relu'))
model.add(keras.layers.MaxPooling2D((2, 2)))
model.add(keras.layers.Conv2D(64, (3, 3), activation='relu'))
model.add(keras.layers.Flatten())
model.add(keras.layers.Dense(64, activation='relu'))
model.add(keras.layers.Dense(10, activation='softmax'))

Yet, when I try to train the model, I get this error:

ValueError: Error when checking target: expected dense_10 to have shape (1,) but got array with shape (10,)

Does someone know, why the dimensions do not fit?

Answer:
The error states that the target needs to be in single row i.e. (1,) where as you are giving (10,)

ValueError: Error when checking target: expected dense_10 to have shape (1,) but got array with shape (10,)

SOLUTION 1:
Use "keras.utils.np_utils.to_categorical" to convert your model data labels to categorical one hot vectors.

from keras.utils import to_categorical
label = array(label_column)
# one hot encode
encoded_label = to_categorical(label)

Then you can have 10 labels in target which your last layer expects.

model.add(keras.layers.Dense(10, activation='softmax'))

SOLUTION 2: If above does not work, then change the last layer of your CNN network to have one output as given below.

 model.add(Dense(1, activation='sigmoid'))


Difference between a target and a label in machine learning If I have a supervised learning system (for example for the MNIST dataset) I have features (pixel values of MNIST data) and labels (correct digit-value).

However sometimes people use the word target (instead of label).

Are target and label interchangeable? Is label just used for classification? Target both for classification and regression?

Target: final output you are trying to predict, also know as y. It can be categorical (sick vs non-sick) or continuous (price of a house).

Label: true outcome of the target. In supervised learning the target labels are known for the trainining dataset but not for the test.

Label is more common within classification problems than within regression ones. Nonetheless, they are often used interchangeably without great precision.


Derivation of dz[1] for backpropagation Can anyone mathematically prove this equation given the values of $dz^{[2]}$, $W^{[2]}$, $z^{[1]}$ and the activation function $g^{[1]}$

$dz^{[1]} = w^{[2]T}dz^{[2]} * g^{[1]'}(z^{[1]})$

For the formal proof I think it's best to post on maths stackexchange instead.

Here are one of the best videos on backpropagation:

(link)

(link)

Your equation shows the chain rule.

hadamard-multiplying any matrix (like $w^T$) by a vector-with-n-entries (in your case by $dz^{[2]}$) is basically saying "I scale each stripe of my matrix by the corresponding entry in vector $dz$"

regarding the "stripes":

each row of $W^{[2]}$ represents a weights from one of our neurons to all neurons in preceding layer
each column of $W^{[2]}$ represents weights from all our neurons towards one particular neuron in preceding layer.
your activations ($g$) each depends on the weights that flow into it (the weights are the parameters of these activations).
your activations also depend on the values of neurons that flow into them. Notice, although neurons are the same entities used by the activations, they are treated as separate branches on the tree diagram (link)

If you watch the "tree diagram link", here is one more hint: in the video, the girl was summing up the terms from different branches. This is exactly what the dot product is doing during backprop.