What Is Saturating Gradient Problem Can anyone explain what is Saturating Gradient problem? It would be nice if anyone can provide math details as well. Thank you in advance!

If you use sigmoid-like activation functions, like sigmoid and tanh, after some epochs of training, the linear part of each neuron will have values that are very big or very small. This means that the linear part will have a big output value regardless of its sign. Consequently, the input of sigmoid-like functions in each neuron which adds non-linearity will be far from the center of these functions.

In those locations, the gradient/derivative value is very small. Consequently, after numerous iterations, the weights get updated so slowly because the value of the gradient is very small. This is why we use the ReLU activation function for which its gradient doesn't have this problem. Saturating means that after some epochs that learning happens relatively fast, the value of the linear part will be far from the center of the sigmoid and it somehow saturates, and it takes too much time to update the weights because the value of gradient is small. You can take a look at here as a solution for this problem.
If I want to explain the math part, suppose that you are using sigmoid as the activation function. If $\sigma$ represents sigmoid, its gradient is $\sigma (1-\sigma$). Now suppose that your linear part, the input of sigmoid is a positive number which is too large, then sigmoid which is:

$$ \frac{1}{1+e^{-x}} $$

will have a value near to one but smaller than that. On the other hand $\sigma$ will be so close to zero, multiplying $\sigma$ and $1 - \sigma$ will result in a small value, which means the value of the gradient is very small. If the value of the linear part is very small, then $\sigma$ will be close to zero and $1- \sigma$ will be close to 1 but smaller than that. Again, multiplying these will lead to a small value as the gradient.


LSTM: Taking previous output values as feature As far as I know, there is practically no limit on the number of dimensions of input feature for LSTM. And it apparently can learn the sequence of data.

My question is does LSTM by nature, also take previous output values besides feature vector as new feature element for the next output values? i.e. If we have:

[t]  -> [x1, ..., xn] [y_t]
[t+1] -> [x1, ..., xn] [y_(t+1)]

Is it necessary to manipulate data like this

[t]  -> [x1, ..., xn, y_(t-1)] [y_t]
[t+1] -> [x1, ..., xn, y_t]    [y_(t+1)]

or LSTM is already handling this for us?

LSTMs handle this for you.

You can search for some tutorial on the LSTMs that explains how they works. One that I like is this.


How to suggest medical treatments based on injury subjective and objective notes using ML? There are 30k+ history injury subjective and objective note records in the dataset, each one with correct corresponding treatments data, the simplified history dataset structure is like:
injury subjective note | GP's objective note | treatments note
I cut my finger, ... | The wound is bleeding, ... | Clean the wound, treat with antibiotic ointment, ...
Fell from cycling, forearm hurts, ... | Bone fracture, ... | Bone X-ray, splint, braces, casts, slings, ...
Now how to using ML/AI to suggest (output) treatments based on new (input) subjective and objective note?
Would like to know:

What algorithms could be best fit?
What tools/platforms are good candidates to choose from?
Any resource links/pages with similar project/purpose that could learn from?

There is a lot of research being done around exploiting biomedical text data in general and clinical notes in particular. I'm not up to date with the whole domain (it's a big one) but let me sketch a few possible directions.

The standard text classification approach: consider every possible treatment as a class, the goal being to predict for each instance of subjective+objective notes the correct class(es). Note that the design can either be standard multiclass classification or multi-label classification. All the regular text classification techniques can be used, from traditional methods like decision trees to using DL with word/text embeddings.
The sequence to sequence approach: the principle here is for the model to represent how an input sequence (here subjective+objective notes) is transformed into an output sequence (treatment). The standard example for this kind of task is machine translation, but this design is used in many other problems.
The semantic representation with a third-party ontology. This approach relies on existing biomedical resources, especially normalized vocabularies like Mesh and/or UMLS. One would use an automatic annotation tool (e.g. cTakes) to extract the medical terms from the subjective+objective notes, and the treatments could also be encoded with the controlled vocabulary. This can greatly facilitate the job of the model by providing it with normalized features and classes. Once encoded, the data can be used in a classification setting similar to the first approach.

About software/data resources: there are a lot, but as far as I know nothing which does exactly what you need out of the box. SciSpacy is a Python module for processing biomedical data. There are datasets available (e.g. shared tasks data), and various research prototypes and resources. See also this answer for pointers to various biomedical resources.


meaning of fine-tuning in nlp task There are two types of transfer learning model. One is
feature extraction, where the weights of the pre-trained model are not changed while training on the actual task and other is the weights of the pre-trained model can be changed.

According to those categorizations, static word vector like word2vec is a feature extraction model, where each vector encodes the meaning of the word.

The meaning of the word changes context. For example, "Bank of the river"  vs "Bank as a financial institute". These word2vec vectors do not differentiate between these meaning.

Current models like Bert consider the context. Bert is a language representation model. That means, it internally can represent word by contextual word vectors.

By default, Bert is a fine-tuning model. This is where my imagination about fine-tuning started to fall apart.
Let's say, on top of Bert model we created some task-specific layer. Now, if we fine tune, by definition, the weights of the lower level (language representation layer) will change at least a bit that means, the vector of the word will also change (if we compare before and after fine tune). That means the meaning of the word change a bit because of the new task.
If my above interpretation is correct, I can not comprehend this phenomenon for example, the word vectors of task sentiment analysis are different that the word vectors (of the same word) of task question answering. Can anybody help me?

Please correct me if anything above is wrong. Thanks

In my understanding, when you are fine-tuning for any task you use additional data (not used during pre-training) and those examples will change the weights on lower levels so that your model is better prepared for the context in which you will use it. A good example is a Twitter sentiment classifier.


Wordnet Lemmatization I tried finding about exception lists in wordnet lemmatizers. "Morphy() uses inflectional ending rules and exception lists to handle different possibilities" which I read from http://www.nltk.org/howto/wordnet.html . Can you explain what is an exception list. Thank you.

The exception list files are used to help the processor find base forms from 'irregular inflections' according to the man page.  They mean that some words, when plural or a different tense, can't be algorithmically processed to find the base/root word.  More details can be found in the morphy man.  I'm not a language processing expert, but this is likely a result of English words that 'break the rules'.  If you think about the code like a human trying to learn English: the student learns rules to use (algorithm) and then has to memorize exceptions for the rules (exception lists).  An over-simplified analogy that does not involve endings/conjugation would a spell checking program.  An algorithm might check for 'i before e, except after c' but would first have to check the word against an exception list to make sure it isn't 'weird' or 'caffeine' - please don't start a linguistics fight about this rule, I am not commenting on the validity of it/that's not the point I'd like to make.


converting a Tensorflow model for Tensorflow.js (Warning, I'm a Tensorflow/ML noob)

I'm trying to convert and load a TensorFlow model into tensorflow.js

The model is saved:

saver.save(sess, checkpoint_path, global_step=model.global_step)

and potentially restored:

saver.restore(session, ckpt.model_checkpoint_path)

This (and similar lines) creates a directory structure:

model_tmp
  checkpoint
  model.ckpt-100.data-00000-of-00001
  model.ckpt-100.index
  model.ckpt-100.meta
  ...
  model.ckpt-500.meta

...So I should be able to convert to a JS model, right?

#TODO --output_node_names='???' \
tensorflowjs_converter \
    --input_format=tf_saved_model \
    --saved_model_tags=serve \
    model_tmp/model.ckpt-500 \
    model_web

Wrong! When I run the script above I get an error:

IOError: SavedModel file does not exist at: model_tmp/model.ckpt-500

I've tried model_tmp, model_tmp/checkpoint but neither of them work either.

Also, I'm not sure what to provide at output_node_names in the converter call.

You can build from source and use the summarize_graph tool which is distributed as a part of Tensorflow graph_transforms tools. Detailed docs are available here. Quick install:

First, make sure you have bazel build system installed (for MacOS):

brew install bazel

Next get Tensorflow sources, build and run summarize_graph:

git clone git@github.com:tensorflow/tensorflow.git
cd tensorflow
bazel build tensorflow/tools/graph_transforms:summarize_graph
bazel-bin/tensorflow/tools/graph_transforms/summarize_graph --in_graph=path/to/model_graphdef.pb

Note that summarize_graph requires GraphDef format as input.

If during the build attempt bazel can't find Xcode, configure it:

bazel clean --expunge
sudo xcode-select -s /Applications/Xcode.app/Contents/Developer
sudo xcodebuild -license
bazel clean --expunge


Detecting low-quality, user-created text content I run a website that allows visitors to publish text content. It is very similar to a forum in terms of functionality. I'd like to automatically exclude or flag, submitted text content that is "spammy" or "low quality" (subjective I know). Obviously I want to detect content that is offensive or classic spam. I also want to detect content that is meaningless or nonsense. I know this must be a common problem for those that run web apps that allow user-submitted content, but I haven't found an obvious solution yet.

Ideally I'd like to use open source software that can run on Linux, suitable for me to install on my webservers, or an AWS service.

I've tried AWS Comprehend which, without any custom training, can detect the sentiment of text accurately, It doesn't seem to detect low-quality or spam out-of-the box. It supports training of custom classifiers, you simply provide a csv containing class/document pairs. This could be a good way to go, can anyone suggest a free dataset that I could use to train it?

It seems like text quality detection would be such a common requirement that I thought there might be some software out there that is pre-trained for that exact purpose, perhaps a Linux package. Does anyone have any suggestions?

I wasn't sure wether to post this in Data Science or Programming, I apologize if this is misplaced.

There might not be a off-the-shelf package for detecting "spammy" or "low quality" because is it is relative to the context (as you already recognized).
One option is to use rules-based approach. Have a list of spam keywords. If any of them appear then stop the text. This is simple but also a limited approach.
Another option is to use machine learning learning to train a system to detect spam. The first step is collect thousands of human-labeled examples of both spam and not spam. Then apply natural language processing text classifier, naive Bayes classifier is a good baseline.


How implement a feed forward network with arbitrary node connections in keras? I'm new to using the keras framework. I have read some examples about how to construct deep learning models with the Sequential and Graph classes in keras. However, I see that, independently if I use Sequential or Graph, it is assumed that each hidden unit of a layer is fully connected with all the hidden units of the other layer, isn't it?

If I want to construct a deep feed forward network that it is not fully connected, for instance the first hidden unit of the second layer is not connected to the second unit of the third layer...etc, even I want add connections (skip connections) between hidden units that belong to non-consecutive layers, how can I implement this in keras?

A possible solution can be found at https://stackoverflow.com/questions/48111217/how-implement-a-feed-forward-network-with-arbitrary-node-connections-in-keras?noredirect=1#comment83198128_48111217

"The suggestion is simply work with many little layers and make as many skip connections as you want. Many parallel layers with 1 node represent well a single layer with many nodes. So you can, for instance, create lots of Dense(1) layers and treat them as nodes. Then you connect them in any way you like."


Linear Regression model visualization I am trying to visualize, how does the linear regression model draw a straight line when we have multiple features and one label to predict. Like when we have 1 feature and 1 label, we can easily plot them on a 2-D graph and draw a straight line. Even if we have two features and one label, we can still plot them on a 3-D graph but I couldn't able to visualize, what will happen when we have 20 features. Will algorithm use 20-D graph or am I missing something.

I am new to machine learning. So this question can come as naive.

If there are 20 independent variables and 1 dependent variable, a linear regression model can be viewed as a 20-dimensional hyperplane in 21-dimensional space. The hyperplane is not "vertical" with respect to any independent variable. So if you pick a value for each of the 20 independent variables and draw a vertical line at the point consisting of those values, it will cross the hyperplane once at the height that the model outputs when you plug in the values of the independent variables.


Is it possible to make F1_Score differentiable and use it directly as a Loss function? One of the metrics that is widely used in binary classification is the F1 score:

$F_1 = 2\cdot \frac{recall \cdot precision}{recall+precision}$

The problem of the F1-score is that it is not differentiable and so we cannot use it as a loss function to compute gradients and update the weights when training the model. The F1-score needs binary predictions (0/1) to be measured.

I am seeing it a lot. Let's say I am using per example a Linear regression or a gradient boosting.

Is there any way that it can be minimized directly?

Yes there is, let's take $F_1$ score  base definition, with :
$$
F_1 = 2 \times \frac{precision \times recall} {precision + recall} \\
F_1 = \frac{2 \times TP} {2 \times TP + FP + FN}
$$
And this is the same as the Sørensen-Dice coefficient, also known as Dice coefficient or Bray-Curtis distance. This is a statistical indicator that measures the similarity of two samples :

$$ Dice(X,Y) = \frac{2|X \cap Y|}{|X| + |Y|}$$

Concerning the implementation of this loss, we can approximate $|X \cap Y|$ as the sum of the matrix obtained using Hadamard product ($\odot$, also known as the element-wise product) between the ground truth ($y$) and the prediction ($\hat{y}$). We can then define $L_{Dice}$ as follows:

\begin{align*}
L_{Dice} &= 1 - Dice \\
L_{Dice}\left(y, \hat{y}\right) &= 1 -
        \frac{ 2\sum y \odot \hat{y}}
             {\sum y + \sum \hat{y}}
\end{align*}

You will often find this loss in the context of segmentation problems, as well as others quite close, such as the Jaccard index (IoU).


Can identifiers be used to train a model? I recently participated in some Machine Learning competition where we were asked to decide whether a rider should accept or not a course (~2k riders and ~140k courses). It came up that some of the winners used the identifier of the rider (an integer number unique for each rider) in their features which was discarded on default notebook and it greatly improved their score.
Is it legitimate? can identifier be used to train a model?

You can use any variable during model training if this variable is available during inference time. This is the only technical restriction.
Another question is should you include this column or not. If this column is completely unique and does not have any relevant information in it then you can discard it. Also, categorical columns with high cardinality (number of unique values) might also have a negative impact.
On the other hand, if including this identifier improved the score a lot (not a random fluctuation) then there are might be more to it and you need to share more info on the dataset and used models.


Does binning a time series with pd.qcut (using quantiles) create data leakage? Let's say I want to predict whether a company will default on it's debt at some point in time (so binary classification) and one of the time series variables I'm using is the "revenue" of that company thought time.
Can I binned this variable "revenue" using quantiles cut (like so => pd.qcut(df['revenue'],bins=10)) without creating a data leakage ?
I'm under the impression that I can not really do that since the quantile cut is made by knowing the entire distribution of the variable "revenue" throughout the period. That is, the bin attributed to "revenue" at any point in time in my training data will carry information about the future.
Am'I correctly assuming that this will create a data leakage for this time series prediction problem ?
If so, can I safely use pd.cut instead with no quantile ?
Thank you

I think that indeed you may have leakage by using pd.qcut. A solution to avoid that leakage is to do it in a time-series cross-validation fashion. The idea is to derive the quantile values in a training fold, and, with those values, do the cut in its validation fold.
This is a little complex and if you want it simpler you can use pd.cut, which will have no leakage issues as long as you don't derive the cuts from the full distribution. If you use some default cuts defined by you, the risk of leakage will be very low.


Understanding logistic regression loss function equation I am writing a scientific paper that - among other things - deals with logistic regression in the context of machine learning.
I read this article where the author states that, given a set of instance-label pairs $(\boldsymbol{x_i},y_i),\  i=1,...,l\ \ \boldsymbol(x_i) \in R^n,\ y_i \in \{-1,+1\}$, Logistic Regression solves the following optimization problem:

$$
\begin{equation}\label{eq:log}
  \min_{w} \frac{1}{2}\boldsymbol{w}^T\boldsymbol{w} + C\sum\limits_{i=1}^l log(1+e^{-y_i\boldsymbol{w}^T \boldsymbol{x_i}})
\end{equation}\ \ \ \ \ \ \ \ \ \  (Eq.1)
$$

Is it correct if I say that: "The first summand of Equation 1 is a regularization term while the second represents the negative log-likelihood loss function"?

Thank you for your kind assistance.

Yes the first is the L2 regularization term to keep the norm of weights parameter as small as possible.


built-in cov in pandas DataFrame results ValueError array is too big So, I have a dataset with 39.949 variables and 180 rows. dataset is successfully
saved in DataFrame but when I try to find cov() it result an error.
here is the code

  import pandas as pd
  cov_data=pd.DataFrame(dataset).cov()

Here is the error

File "/home/syahdeini/Desktop/FP/pca_2.py", line 44, in find_eagen
cov_data=pd.DataFrame(data_mat).cov()
File "/usr/lib/python2.7/dist-packages/pandas/core/frame.py", line 3716, in cov
baseCov = np.cov(mat.T)
File "/usr/lib/python2.7/dist-packages/numpy/lib/function_base.py", line 1766, in cov
return (dot(X, X.T.conj()) / fact).squeeze()
ValueError: array is too big.

Christopher is right about the size of the array. To be simplistic about it, if this translates to 1.6B floats, at 16 bytes per float (32-bit version; 64-bit is bigger), then you're trying to create an array of about 26 GB. Even if you have the RAM for that, I'd imagine that it's probably going to overload something else along the way.

(Maybe not, but generally speaking, any operations that are that computationally intensive should first raise the question of whether you are doing the right calculation in the first place. And if you do need to do something of that magnitude, you should then try to break it down into more manageable chunks that can be run in parallel or distributed across machines.)

But given that you are describing a very, very wide dataset (~40k columns x 180 rows), I wonder whether you really want to take the covariance matrix of the transposed dataset (so 180x180 = 32,400 covariances)? That would be a far more tractable problem, and it's easier to see how it might be useful.

In any case, you're probably far better off calculating each pairwise covariance (or at least, the vector of cov(x_i,x_k) for all x_k != x_i) at the point where you'll actually use it, rather than calculating a giant matrix initially then referring back to it later. Memory issues aside, it'll make your life much easier if you start running things in parallel, and will help ensure you don't waste resources on unnecessary calculations.


How to interpret dummy variable in ML prediction? I am working on a binary classification problem where I have a mix of continuous and categorical variables.

Categorical variables were created by me using get_dummies function in pandas.

Now my questions are,

1) I see that there is a parameter called drop_first which usually is given the value True. Why do we have to do this?
Let's say for the purpose of example, we have 2 values in gender column namely Male,Female. If I use drop_first=True, it returns only one column. like gender_male with binary 1 and 0 as values
For example, If my feature importance returns gender_male as an important feature, Am I right to infer that it is only Male gender that influences the outcome (because male is denoted as 1 and female is 0) and female (0's) don't impact the model outcome? or 0's in general doesn't play any role in ML model predictions?

2) Let's say my gender has 3 values for example Male,Female,Transgender. In this case if I use drop_first=True, it would only returns two columns

gender_male with 1 and 0  - Here 0 represents Transgender right?

gender_female with 1 and 0 - Here 0 represents Transgender right?

3) What's the disadvantage of not using drop_first=True? Is it only about the increase in number of columns

Can you help me with the above queries?

1) Using drop_first=True is more common in statistics and often referred to as "dummy encoding" while using drop_first=False gives you "one hot-encoding" which is more common in ML. For algorithmic approaches like Random Forests it does not make a difference. Also see "Introduction to Machine Learning with Python"; Mueller, Guido; 2016:

  The one-hot encoding we use is quite similar, but not identical, to
  the dummy encoding used in statistics. For simplicity, we encode
  each category with a different binary feature. In statistics, it is common
  to encode a categorical feature with k different possible values
  into k–1 features (the last one is represented as all zeros). This is
  done to simplify the analysis (more technically, this will avoid making
  the data matrix rank-deficient).

However, using dummy encoding on a binary variable does not mean that a 0 has no relevance. If gender_male has high importance that does not generally say anything about the importance of gender_male==0 vs gender_male==1. It is variable importance and accordingly calculated per variable. If you, for example, use impurity based estimates in Trees it only gives you the average reduction in impurity achieved by splitting on this very variable.

Moreover, if your gender variable is binary, gender_male==1 is equivalent to gender_female==0. Therefore from a high variable importance of gender_male you cannot infer that being female (or not) is not relevant.

2) In this case gender_male==0 AND gender_female==0 means Transgender is true.

3) see 1). For algorithmic approaches in ML there is no statistical disadvantage using one-hot-encoding. (as pointed out in the comments it might even be advantageous since tree-based models can directly split on all features when none is being dropped)


Pandas Merge question I have a dataset  where for each date I have two values say A and B, there are dates when one of the values is missing

Date       C1   C2
1/1/2019    A   10
1/1/2019    B   20
1/2/2019    A   25
1/2/2019    B   30
1/3/2019    A   23
1/4/2019    A   32

I want to make sure for every date, I have the date for A and B.
One way I can think is to have driver table with all distinct dates and data for A and B and then do left join.
Is there any better way of doing this with pandas merge or iterating through rows? I can't think of any but feel like there might be better way of doing it. Thanks!

I think you have listed the main ways of doing it - you can achieve it by iterating or merging. What is "best" will depend on your use case.

Here is a way to do it by iterating over a dataframe. This way gives you more control about what gets filled i.e. you could add more conditions on which values to fill in. I first make a new dataframe that has all the complete dates and alternating A B A B values in column C1:

import pandas as pd
import numpy as np

dates = pd.date_range(start="1/1/2019", end="1/10/2019")
repeated_dates = np.repeat(dates, 2)
df = pd.DataFrame(index=repeated_dates, columns=["C1", "C2"])
df["C1"] = (len(df) // 2) * ["A", "B"]

# See first 5 rows
print(df.head())

           C1   C2
2019-01-01  A  NaN
2019-01-01  B  NaN
2019-01-02  A  NaN
2019-01-02  B  NaN
2019-01-03  A  NaN

We will later fill the values of the C2 column.

Next make a dataframe (it would actually be your starting data), by dropping a few rows from our "results" dataframe above:

df_missing = df.drop(df.index[[3, 9]])

C2_col = []

# grouping by the index (i.e. the date) gives us two rows at a time
for date, group in df.groupby(df.index):
    try:
        # see which values your data has for this data and extract them
        day = df_missing.loc[date, ["C1", "C2"]]
        C2_A, C2_B = day.C2.values

    # If the date wasn't there, we can catch the error and give any values we want
    except KeyError as e:
        # Could now use more condition e.g. on the date or previous values, etc.
        C2_A = C2_B = "was_missing"

    # Keep the values in a list
    C2_col.extend([C2_A, C2_B])

# Overwrite the column that was full of NaN values
df["C2"] = C2_col

We can see in the final result that all dates are present, along with the A B A pattern, and we could insert whatever we wanted into those dates with missing values:

print(df)

           C1           C2
2019-01-01  A          NaN
2019-01-01  B          NaN
2019-01-02  A  was_missing
2019-01-02  B  was_missing
2019-01-03  A          NaN
2019-01-03  B          NaN
2019-01-04  A          NaN
2019-01-04  B          NaN
2019-01-05  A  was_missing
2019-01-05  B  was_missing
2019-01-06  A          NaN
2019-01-06  B          NaN
2019-01-07  A          NaN
2019-01-07  B          NaN
2019-01-08  A          NaN
2019-01-08  B          NaN
2019-01-09  A          NaN
2019-01-09  B          NaN
2019-01-10  A          NaN
2019-01-10  B          NaN


How to learn Machine Learning I want to get into machine learning. I've been in information security for the last 10 years so i have an IT background.

Where is the best place to start:
Can anyone recommend a good book? And also a platform i can use to practice (preferably free)
Also if there are any online courses someone could recommend that would be great.

I looked into AWS's offering of machine learning but that is not included in the free tier.

Any help/advice would be much appreciated.

Thank You.

Courses: Andrew Ng Machine Learning Course from Coursera is what introduced me to machine learning, and I cannot think of a better way to do it. The course focuses on the theory rather than practice, but it offers solid theoretical foundations. fast.ai Machine Learning course is more programming oriented, I haven't taken it but it looks good.

Books: Hands on machine learning with Scikit-learn and Tensorflow by Aurélien Géron has a decent amount of theory explained in a very simple way, and it complements it with lots of code examples. If you want to dig deeper in statistics and theory, The Elements of Statistical Learning might be a good resource.

Practice: Kaggle is for sure the best place to practice and learn. After reading something or doing some course I advise to download the Titanic dataset and try to submit predictions.


Predict future value with time period using non linear regression model Here I have dataset import from csv file. I want to predict the next value with the time series. Can we use nonlinear regression model to predict the value for next time period or Is there any regression model can we use to predict the value?

Here I upload the subset of my original dataset.

 Time     x     x1   x2     y
0:06:00  63    NaN   NaN   63
0:07:00  63    NaN   20    104
0:08:00  104   11     0     93
0:09:00  93    0      0     ?

If your data is time series data then you should analyze it accordingly, with a time series model. Alternatively, you could build a regression model with polynomials or splines, depending on your data and how "curvy" it is. Another option would be a GAM model (Generalized Additive Model), which works very well with curves of various shapes.


Reinforcement Deep Learning for object detection After reading the state of the Art of object detection using the CNN's(R-CNN,Faster R-CNN,YOLO,YOLOv2,SSD) I was wondering if there is an efficient method that use deep learning with reinforcement learning for object detection ?

To my extent of my knowledge, RL is used as a model for attention mechanism in object detection field (particularly the REINFORCE algorithm which is a 'flavor' of policy gradient methods). You can take a look at one of the papers that first proposed the method: RAM


Can not install spacy package on windows 10 via pip I have below environment.
OS: Windows 10
Python: Python 3.7.4
PIP: pip 19.3.1
I am trying to install spacy in my windows 10 OS. It gives me below error.
 ERROR: Command errored out with exit status 1:
   command: 'd:\rajesh\python\env1\scripts\python.exe' 'd:\rajesh\python\env1\lib\site-packages\pip' install --ignore-installed --no-user --prefix 'C:\Users\rajesh.das\AppData\Local\Temp\pip-build-env-vna552d_\normal' --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'thinc<7.4.0,>=7.3.0' 'cymem<2.1.0,>=2.0.2' 'preshed<3.1.0,>=3.0.2' wheel 'cython>=0.25' 'murmurhash<1.1.0,>=0.28.0'
       cwd: None
  Complete output (460 lines):
  Collecting thinc<7.4.0,>=7.3.0
    Using cached https://files.pythonhosted.org/packages/d4/38/f79bb496ced36f8d69cdbdfe57a322205582ed9508bda5bd0227969d5a77/thinc-7.3.1.tar.gz
  Collecting cymem<2.1.0,>=2.0.2
    Using cached https://files.pythonhosted.org/packages/ce/8d/d095bbb109a004351c85c83bc853782fc27692693b305dd7b170c36a1262/cymem-2.0.3.tar.gz
  Collecting preshed<3.1.0,>=3.0.2
    Using cached https://files.pythonhosted.org/packages/5f/14/de231123ddbe0bf12bd9b1993122d67f22859643bee4dad3b6ce91986336/preshed-3.0.2.tar.gz
  Collecting wheel
    Using cached https://files.pythonhosted.org/packages/00/83/b4a77d044e78ad1a45610eb88f745be2fd2c6d658f9798a15e384b7d57c9/wheel-0.33.6-py2.py3-none-any.whl
  Collecting cython>=0.25
    Using cached https://files.pythonhosted.org/packages/6d/da/b48f8a1778a82d24d1c3a543d9db8461374f0c70c0145f0b5a8664dbc122/Cython-0.29.14-cp37-cp37m-win32.whl
  Collecting murmurhash<1.1.0,>=0.28.0
    Using cached https://files.pythonhosted.org/packages/22/e9/411be1845f1ac07ae3bc40a4b19ba401819baed4fa63b4f5ef28b2300eb4/murmurhash-1.0.2.tar.gz
  Collecting blis<0.5.0,>=0.4.0
    Using cached https://files.pythonhosted.org/packages/98/5a/f9b8a78e3d1fdde1b0215413d88ab55d907ab81f95b62418a6e9cda30dec/blis-0.4.1.tar.gz
  Collecting wasabi<1.1.0,>=0.0.9
    Using cached https://files.pythonhosted.org/packages/ff/ef/e8266e158ed32bf5f723fac862b6518833d0b53ca183165a8718f212c0d5/wasabi-0.4.2-py3-none-any.whl
  Collecting srsly<1.1.0,>=0.0.6
    Using cached https://files.pythonhosted.org/packages/1e/5b/f83e478fbf44c0d04f182e56db0871a53f5096febb8eb0be7cec75dca73b/srsly-0.2.0.tar.gz
  Collecting numpy>=1.7.0
    Using cached https://files.pythonhosted.org/packages/ce/ad/2e88f36b56f64f70c081b32fa5512dacedf12005ccb0c2d300d44dcc1215/numpy-1.17.4-cp37-cp37m-win32.whl
  Collecting plac<1.2.0,>=0.9.6
    Using cached https://files.pythonhosted.org/packages/86/85/40b8f66c2dd8f4fd9f09d59b22720cffecf1331e788b8a0cab5bafb353d1/plac-1.1.3-py2.py3-none-any.whl
  Collecting tqdm<5.0.0,>=4.10.0
    Using cached https://files.pythonhosted.org/packages/7f/32/5144caf0478b1f26bd9d97f510a47336cf4ac0f96c6bc3b5af20d4173920/tqdm-4.40.2-py2.py3-none-any.whl
  Building wheels for collected packages: thinc, cymem, preshed, murmurhash, blis, srsly
    Building wheel for thinc (setup.py): started
    Building wheel for thinc (setup.py): finished with status 'error'
    ERROR: Command errored out with exit status 1:

Also I was trying to install dependency package of spacy thinc, It also gives same error.
Can anyone give some idea?

I was getting this exact same error trying to install spaCy into a 32-bit Python virtual environment. The latest version of spaCy I was trying to install (2.3.2) requires 64-bit Python.  When I rebuilt my virtual environment as 64-bit and then tried to install spaCy, it worked fine.
Just make sure you are trying to install spaCy into a 64-bit Python environment and you should be good to go.


Building a search tool and classifying text using NLP and ML Im a newbie in information Retrieval. Currently Im reading a book entitled "An Introduction to Information Retreival" by Christopher D. Manning and Prabhakar Raghavan. Im trying to implement an ai based search tool to search for some relevant information from a private dataset. ( say a chemical or mathematical dataset that invloves more number of numericals and unstructed messy representation of unit of measure)

Following is a row in my dataset in JSON format where keys are the coloumn name and value are the values corresponding to it.

  "ABC Project": {
    "In/Out diameter": " Both in and out are 1”  ",

    "Design Pressure (barG)": {
      "Max": "116 psiG (7.99 barG)",
      "Minimum": "79.7 psiG (5.49 barG)",
      "Design": "174 psiG (11.99 barG)"
    },

    "C02 %": "0.671",
    "MW": "16.68 kg/kmol",
    "TITLE": null,
    "Mothiram": "There is  very dense forest and their lived a king. The name of the king was Pandidhurai. He was very brave ",
    "Thooval delivery material": "- thooval delivery material is panam patta 316/316L  "

The sample search query that will fetch the above mentioned row from my dataset would be

" The project in which molecular wight is nearly 16 kg/kmol and Thooval delivery material panam patta 316l is used in which Mothiram is King Pandidhurai "

What I have done.

Preprocessing

Read each row from table(say $T$) as $R_{i}$ and search query as $Q$ do  the following preprocessing

convert_lower_case(data)
remove_punctuation(data)
remove_apostrophe(data)
remove_single_characters(data)
convert_numbers(data)
remove_stop_words(data)
stemming(data)
remove_punctuation(data)
Convert Word to Vector for coloumn $C_i$ and do it for each corresponding row in table $R_i$
Build Vector Space Model for coloumn $C_i$ and do it for each corresponding row in table $R_i$
Build a vector space model for query $Q$.
Compute cosine score to select Row having high score

Challenges I'm facing

When I do 'remove_punctuation(data)' Im loosing vital informations (say  $"$ denotes inches in the diameter coloumn)
A way to interpret unit of measure in the data
Unable to keep the relation between value ( say 16.78 kg/k mol  is stored in multi diamentions and unable to find it is related to MW).
I think it could be solved by building a ML classifier and train it to identify the entity MW. Say < 16.78 kg/k mol , MW > and using the unit of measure as a feature. But there are values like percentage composition of different chemical components, IN and OUT diameter , Inside and Outside Temperature. etc
Difficulty in interpreting scientific terms and mapping to one root form ( say MW, mol wt, molwt etc are all different ways of representing molecular weight)
Difficulty in finding the close proximity of neumerical values

Question

  Please suggest me a step by step approach to build a search tool using this dataset (Please also suggest me an apt algorithm that would
  be useful in each step)

I have read from here that some documents can be tagged as < document, class > and train a Navie Base Classifier Model for better search performance. Is it possible in this case , if so what could be class label you would be suggesting ? UPDATE : I think Navie Base cannot be used as the coloumn or key would  be greater than 1000. Im looking for a scalable approach to the problem.

  Is there any better approach to solve this problem than using a vector space model and  computing cosine similarity ?

Here are a few general thoughts about your project:

As far as I understand, you're trying to extract very specific information from a semi-structured database using free-form natural language queries, correct? If yes it's important that you realize that this is a quite ambitious project, reaching a decent stage of quality is probably going to take a lot of work, and the performance is unlikely to be perfect.

Apparently numerical values and units are important pieces of information for matching the query. In this case you should probably implement a special process for those, because standard text processing is not going to work very well.

Detection: if there is only a small number of possible ways in which these values are written, it's probably more efficient to use ad-hoc regular expressions. If not, you could try to train a custom NER model.
Representation: that's the tricky part imho. For general text vectors are fine, but with vectors it's difficult/impossible to represent special values like those. Given that you have a semi-structured database, you might want to try a more semantic representation adapted to your data: that could involve techniques from semantic role labeling, relationship extraction, etc.
Matching: the advantage of a semantic representation is that you could convert the query to a semantic representation and then apply a detailed matching procedure suited to your data (in particular you can compare corresponding numerical values and use a threshold, or return the difference to represent how far they are).


Unblanced classes: classifier only predict one class I am trying to use a multiclass classification using python. For that I used few algorthims like Random Forest, Xgboost, Logitic regression.

My problem is simple, I have users, Images, and people ratings on those images. I devided the ratings into 3 classes:

class 1: good marks

class 2: bad marks

class 3: medium evaluations

At first I got these results

  I have 70% of bad marks (class2) 18% of good marks (class1) 10% of bad
  marks (class 3)

so  all was good I did the classification and I got a good accuracy (75%).

I have collected more data and more (good marks) and the accuracy kept decreasing.

So I understood that the accuracy was good only because the algorithms where predicting that almost all the marks are bad, so basically only one class- class 2, and when I got more data in class 1 and class 3 the accuracy decreased.

This is example of the confusion matrix I was having

Predicted Marks     1    2  3
Actual Marks
1                  48   85  3
2                  17  250  4
3                  10   89  1

I understood that I was getting this problem because I have unbalanced data, so I was predicting bascially the highest frequency class.

I did some researches and I found that there are option called `

  class_weight='balanced'
  So I used it in the classification algorithms.

And I got this result

Predicted Marks   1    2   3
Actual Marks
1                  53   61  29
2                  66  161  53
3                  24   40  17

So my question is:

First I know that the option balanced try to  requilibrate the data
but I don't understand how. I found in the explanations that it"considers each class as important as the other"` but still don't understand how.
Does it duplicate rows of the minor categories ?
When I used that option, it was obvious that the classifier were predicting other Mark classes more frequently, but there are more number of right predictions than the wrong ones.
( example in the second confusion matrix the class 1 was predicted as class 2 66 times , but in the confusion matrix 1 only 17 times)

Am I analysing the problem right? and what are your suggestions for such a problem?

Sorry for making this long, any help will be appreciated !!

I found in the explanations that it "considers each class as important as the other" but still don't understand how. Does it duplicate rows of the minor categories?

No, it adds a weight to each example, depending on its class. The majority class will be assigned a small weight while the minority ones will be assigned larger weights. This weight are considered during the training phase of your model, so that each example of a minority class impacts the parameter updates more than one from the majority class.

For example, suppose class 1 has a weight of $0.5$, class 2 has a weight of $1.2$ and class 3 has a weight of $1.5$. Each example from class 3 will impact the parameter updates 3 times more than an example from class 1, etc.

You can see what weights scikit-learn has selected through this function. You can also select your own weights (if you feel that some class should be more important than what scikit-learn has selected) by modifying the dictionary.

  When I used that option, it was obvious that the classifier were predicting other Mark classes more frequently, but there are more number of right predictions than the wrong ones.

I'm not sure what the question is, but if you are asking "why do I have more misses now than before?".

Because you used that option, your model has learned to classify examples while treating the classes of equal importance. But the classes don't have the same frequencies. For example if you miss-classify 1 out of 4 examples, you would miss more predictions from the majority class than from the minority ones.

Is this good or bad?

Depending on your goal.

If you want to simply maximize your algorithm's correct predictions, then you might get a better result without balancing your classes. In this case you should use a metric like accuracy.

If your goal something else, for instance to have the same accuracy in each class, you should select a more appropriate metric (e.g. a macro-averaged one).

Is there an alternative to class_weights?

Yes, in most cases it is preferable to balance the number of samples in your classes. This is done by either over-sampling the minority classes, under-sampling the majority ones, or a combination of both.

If you want to try out this approach, I'd suggest using imbalanced-learn.


How to find the probabilities of certain events occurring in a defined sequence? Good day, everyone!

I have a problem where I have to program something like this:

I have some arbitrary number of events, let's call then Event A, B, C, D, E, F, ... and so on.

Now they occur in well defined sequences, just like below:

[A, A, C, A, A, A, F, B, A]

That's just a random sample, I have a data of thousands of such event sequences, and there are going to be 9 events in every sequence.

What I'm trying to do is to calculate the probability of each individual event happening. Like:

what's the probability of event A occurring first
what's the probability of event B occurring, given that the previous event was A
what's the probability of event C occurring, given that the previous two events were A and B

So far I've gone through the maths behind conditional/joint probabilities, markov chains, but I can't get to come up with the programming part. Are there any leads to where I can get my hands on some framework, library, or a tool which will help me run the simulation with all of my sequences, and give any finalised probabilities?

Thanks!

P.S: For those who are aware of MLB, I'm just trying to find the probabilities of certain scores being made in all the 9 innings based on past scores. If I calculate the simple probability of a score being made, I get a very high probability of zero, which is not the case. I only have the data for the box scores.

My answer is inspired by this question in Code Review SE

import random
import string
from collections import Counter

def test_data(n, m, choices):
    return [[random.choice(choices) for _ in range(m)] for _ in range(n)]

data = test_data(50000, 9, ["A", "B", "C", "D", "E", "F"])

def subsequence_counts(sequences):
    return Counter([seq for seq in map(''.join, sequences)])

counter = subsequence_counts(data)

print(counter['BCEEECFAF'])

What the above code does:

1) The test_data function, creates a n lists with m items using elements of the choices parameters.

In specific, we create 50000 lists with 9 items length using the capital letters A, B, C, D, E, F

This is just for testing the code

2) The function subsequence_counts uses a list comprehension and a Counter to keep track of every single combination of the sublists we have.

3) Finally, you can access the Counter and get the frequency of any event you want.


xgboost cannot identify perfectly fitting regression line For a dataset I want to use xgboost for the optimal ensembling of $n$ forecasts instead of just using their arithmetic mean for combination. I found that xgboost generates forecasts that are worse than many of the $n$ individual forecasts the moedl could choose for combination.

I do not know why this can be the case. For the illustration of my observation I created the toy dataset below. The artificial target variable is generated by $$y = \frac{x_1+x_2}{2} \, \,\mbox{with } x_1, x_2 \sim N(0,1) $$
Given the deterministic relationship between $y$ and the two explanatory variables $x_1$ and $x_2$, xgboost could make perfect forecasts, but it does not. The linear model easily does. Since this is the most simple multivariate linear regression model I can think of and xgboost fails I wondering about the implications.

Why is this the case? What are the limitations of tree models for regression?
Why is then xgboost used for stacking and ensembling of forecasts if it cannot reproduce the MSE minimizing arithmetic mean as optimal combination mechanism?

Note that the parameters of xgboost do not affect that. I tried many parameter settings and the results are never perfect.

Data Generation

library(tidyverse)
library(xgboost)
n <- 1000
param0 <- list("objective"  = "reg:linear", "eval_metric" = "rmse")
set.seed(1)
df <- tibble(x1 = rnorm(n), x2 = rnorm(n), y = (x1+x2)/2)

xgboost

xgtrain <- xgb.DMatrix(as.matrix(df[1:900,c("x1","x2")]), label = df$y[1:900], missing = NA)
xgtest <- xgb.DMatrix(as.matrix(df[901:1000,c("x1","x2")]), missing = NA)
#Crossvalidation just to illustrate that the algorithm
#learns something that is not correct since the test data
#cannot be forecasted with 0 error.
#xgb.cv(nrounds = 100,nfold = 10, params = param0, data = xgtrain)
#nrounds and other parameters do not not get you to the prefect forecast
model <- xgb.train(nrounds = 100, params = param0, data = xgtrain)
preds_xgb <- predict(model, xgtest)
#no perfect forecasts
sqrt(mean((preds_xgb-df$y[901:1000])^2))
0.04654448

Linear regression

model <- lm(y ~ x1+x2, data = df[1:900,])
#0.5 and 0.5 for x1 and x2 as expected
model$coefficients
preds_lm <- predict(model, df[901:1000,c("x1","x2")])
#perfect forecasts
sqrt(mean((preds_lm-df$y[901:1000])^2))
1.389314e-15

I think that the reason for this to happen is that tree-based methods have problems with linear problems. This is because tree-based methods do partitions of the variables, and not on combinations of the variables.  To fit a linear regression, a tree-based method will have to do a lot of partitions to obtain low error. However, in principle, using enough deep trees you should be able to overfit your training data, although it might take many trees.

If your concern is to make a perfect forecast, no tree based method is able to do a perfect forecast, and this happens with most kind of data. As your data is linear, you happen to be able to do a perfect forecast with linear regression, but this won't happen in real life.


How to approach the numer.ai competition with anonymous scaled numerical predictors? Numer.ai has been around for a while now and there seem to be only few posts or other discussions about it on the web.

The system has changed from time to time and the set-up today is the following:

train (N=96K) and test (N=33K) data with 21 features with continuous values in [0,1] and a binary target.
The data is clean (no missing values) and updated every 2 weeks. You can upload your predictions (on the test set) and see the log-loss. Part of the test data is even live data and you get paid for good predictions.

What I would like to discuss:

As the features are totally anonymous I think that there is not much feature engineering we can do. So my approach is very mechanical:

inspired by this I use a classification algorithm to filter out those training data which fit to my test data best.
Figure out some nice preprocessing
train nice classification algorithms
build ensembles of them (stacking, ..).

The concrete question:

Concerning step 1: Do you have experience with such an approach? Let's say I order the probability of train samples to belong to test (usually below 0.5) and then I take the largest K probabilities. How would you choose K? I tried with 15K .. but mainly to have a small training data set in order to speed up training in step 3.

Concerning step 2: The data is already on a 0,1 scale. If I apply any (PCA like) linear transformation then I would break this scale. What would you try in preprocessing if you have such numerical data and no idea that this actually is.

PS: I am aware that because numer.ai pays people discussing this could help me make some money. But as this is public this would help anybody out there...

PPS:
Today's leaderboard has an interesting pattern:
The top two with logloss of 0.64xx, then number 3 with 0.66xx and then most of the predictors reach 0.6888x.

Thus there seems to be a very small top field and lot of moderately successful guys (including me).

I've looked at the approach and I'd select K by trying a range, i.e. 5k, 10k, 15k etc and then exploring the range in which the best result falls, say the best is 15k then I might do 13, 14, 15, 16, 17 and so on.

So far I've not found any pre-processing to be effective.

Answering the comment:

I've tried using LogisticRegression, SVM, Neural Networks, RandomForests, Multinomial NB, Extra Trees. All except Neural Networks using the implementations in sklearn. PyBrain for the NN.


Performance measure: Why is it called recall and sensitivity? precision is the fraction of retrieved instances that are relevant, while recall (also known as sensitivity) is the fraction of relevant instances that are retrieved.

I know their meaning but I don't know why it is called recall? I am not a native-speaker of English. I know recall means remember, then I don't know the relevance of this meaning to this concept! maybe coverage was better because it shows how many instances were covered...or any other term.

Moreover sensitivity is also insensible to me!

Could you please help me to associate these words to the concept and have a sense of them?

I think the term "sensitivity" comes from the world of medical tests. A very sensitive test will test positive for most or all people who take the test and really have a disease, as well as for many people who don't. This corresponds to high recall, which means the query retrieves most or all of the relevant documents, as well as many that may not be relevant.


How to make output dimensions match input dimensions in CNN? I have training samples of the following shape: (1000,2). These are numeric sequences, each of length = 1000, dimensions = 2.

I need to build a Convolutional Neural Network to output Predictions/Sequences of the same shape (1000, 2). From what I understand, after applying convolution and pooling, the height and width of the input is reduced.

How should I then set up the fully connected layer(s) and an output layer in my CNN, so that the output dimensions match the input dimensions? Or in general, how should I set up my CNN architecture to achieve this?

You have (at least) two options. You can either:

Use a bottleneck architecture where you use pooling layers to reduce the dimensions of your data and then upcast it again using deconvolutional/upsampling layers to go back to the original dimensions. The advantage of this approach is that you use a bigger receptive field for the final output, which means it can look at a bigger part of your input sequence. You could combine this with a highway connection like structure. A very similar concept is used in 2d with segmentation, here the spatial dimensions are preserved at the output. One of the best models is called a U-Net.
Use no pooling and only padding-same convolutional layers with non-linearities. You can increase the depth but you will keep the (1000, 2) dimensions everywhere. The advantage is that it is very easy to implement, and if applicable to your problem only uses relatively local information.


Prediction vs causation in a ML project I am performing a classification task and was able to identify significant predictors (important features using Random Forest) that can help separate the classes or influence the outcome.

But I read online that prediction models are not causal models.

Let's say if my prediction model says that Age is one of the significant factors that influence outcome (death), how can I prove that Age is the cause of death.

I read that any intervention/change on strong predictors of your models, will not necessarily impact the outcome.

How can I find out the list of factors that really cause change in the outcome?

Currently what I do is run a RF model to identify the important features and communicate that these are the top 5 features that seem to influence the outcome.

How can I prove that it is causation and not just correlation?

ML questions are concerned mainly around predictions, but we can extrapolate to (in a certain sence) causality.

First of all these are two different modelling approaches:

Causal inference is focused on knowing what happens to $Y$ when you change $X$.  Prediction is focused on knowing the next $Y$ given $X$

Some of the current causal approaches are randomised testing, do-calculus etc...

So how can we extrapolate to causal inference of standard predictive ML models?

Counterfactual Explanations we can simulate counterfactuals for predictions of machine learning models where we simply change the feature values of an instance before making the predictions and we analyze how the prediction changes. Read more about it here, and there is python library called alibi that implements it.


What metric does `Scalar test loss` in AutoKeras mean? I'm using the following code:

import autokeras as ak

regressor = ak.StructuredDataRegressor(max_trials=100)
regressor.fit(x=train_dataset.drop(columns=['MPG']), y=train_dataset['MPG'])
# Evaluate the accuracy of the found model.
print('Accuracy: {accuracy}'.format(
    accuracy=regressor.evaluate(x=test_dataset.drop(columns=['MPG']), y=test_dataset['MPG'])))

The result of the evaluate function is Scalar test loss. I just want to clarify what metric would that be. What is the default loss function?

Since you are using StructuredDataRegressor, the default loss function is mean squared error. That information can be found in the __init__ function with the default keyword argument `loss='mean_squared_error'.


How do I calculate a similarity matrix with a Student-t kernel? As the title says, how do I calculate a similarity matrix with an un-normalized Student-t kernel? I'm attempting to calculate Kullback-Leibler divergence for different t-SNE runs, but need a Q-matrix for that. A few steps before the Q-matrix, I need the similarity matrices made using the un-normalized Student-t kernel.

I'm using r, not sure if that's relevant to an answer.

You can use dt from the stats to get the density of a Student-t distribution. See the help page for extra information about this, and related, functions.

An example, showing the Student-t distrib

library(stats)

xs = seq(-5, 5, .1)
density = dt(xs, df=1)

plot(xs, density)


LSTM number of units for time series My question is on the number of units in an LSTM cell.

I've come across the following example which is a model for predicting a value in a series based on its 2 lag observations. I'm wondering why the number of units in the LSTM cell is 100 which is much higher than the number of features. In general, wouldn't be more logical to set the number of units to the number of input features? What is the advantage of having a number of units higher than the number of features? And what does it intuitively mean?

My code:

series = array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])

n_features = 1

series = series.reshape((len(series), n_features))

n_input = 2

generator = TimeseriesGenerator(series, series, length=n_input, batch_size=8)

model = Sequential()

model.add(LSTM(100, activation='relu', input_shape=(n_input, n_features)))

model.add(Dense(1))

model.compile(optimizer='adam', loss='mse')

There are very few resources that justify number of cells proportional to input. The intuition though is clear from colah's blog. The longer the sequence you want to model, the more number of cells you need to have in your layer. For e.g. if you are using the LSTM to model time series data with a window of 100 data points then using just 10 cells might not be optimal. The goal of any RNN (LSTM/GRU) is to be able to encode the entire sequence into a final hidden state which it can then pass on to the next layer. So even if you look at the older NMT models (sequence to seq for language translation), at a very basic level, it had a layer of an lstm encoding all info from the sequence that was passed to it and then using it to kick start the decoding. So basically, if the number of units are relatively lesser, the ability to encode all the info MIGHT not be optimal.

Having said that, we regularly use

a) multiples of 32 (https://svail.github.io/rnn_perf/) so inspite of their explanation being vague at best, the way to look at it is that when u declare an output size of say, 100, the RNN will generate square matrices of 100 x 100 with weights in them (that will be adjusted during back prop to give you a final model) and that matrix multiplications of such a matrix will be unwieldy as opposed to a matrix thats a multiple of 32 ( this is totally my intuition again, please correct, if im mistaken)

b) also if you use more than a certain number of hidden units, you will end up with the vanishing gradient problem (exploding gradients typically dont occur due to relu activation functions that keep activations between 0 and 1). Hence you are better off going "deeper" than "wider" , where you can stack 2-3 layers of 128 hidden cells "vertically", instead of using 512 hidden cells in one layer (this , i have actually experienced in almost all forms of experiments with text as well as image data)

im hoping this clarified matters somewhat


How can I compare my regressors? I am trying to build a regressor for a dataset which gives info about students' school performance and the probability of getting admitted in the University of their choice.

The first 5 observations look like this :

    GRE_Score  TOEFL_Score  Uni_Rating  LOR   CGPA  Research Chance_of_admit
0      337         118           4      4.5   9.65     1          0.92
1      324         107           4      4.5   8.87     1          0.76
2      316         104           3      3.5   8.00     1          0.72
3      322         110           3      2.5   8.67     1          0.80
4      314         103           2      3.0   8.21     0          0.65

I have build the following regressors so far : linear regressor, knn regressor and a recurrent neural network. ( I will try a few more later. )

So far, in order to chose among my models, I used the "score" method on the test set for the first two regressors ( it returns the $R^2$ for each one of them ) and the "evaluate" method on the test set for the network ( it returns the MeanSquaredError ).

So, keeping in mind that $R^2$ and MeanSquaredError have different formulas, how can I compare my network with the other two models ??

Any help is much appreciated.

If you up to a predictive model, you look for a model which performs well on the test set and the metric of interest is the mean squared error which indicates by how much you fail to predict $y$ on average. So don't use $R^2$. Just compare all models based on MSE.


How can I increase the number of iterations per epoch in MATLAB? I am training a deep learning network using MATLAB and would like to increase the number of iterations per epoch.

Using trainingOptions

options = trainingOptions('sgdm', ...
    'Momentum',0.9,...
    'MaxEpochs',100, ...
    'ValidationData',{XValidation,YValidation}, ...
    'MiniBatchSize', 64,...
    'Verbose',false, ...
    'Plots','training-progress');

I get the following, where the number of iterations per epoch is set automatically to 1. Therefore 1 pass of 64 training examples is needed for one epoch. How can I increase this to, for example 10 iterations per epoch?

According the the docs:

  An iteration corresponds to a mini-batch.

As I understand, the number of iterations is the number of passes, each pass using 64 (batch size) number of examples. So in the above, I have 1 pass of 64 training examples for 100 epochs.

Does this mean the iterations per epoch are set automatically according to how many training examples there are present? Can I alter this parameter?

I think you misunderstand the difference between the terms:

Epoch - A single pass over the entire training dataset.

Mini-batch - Number of samples to pass thru the model before performing a single update of its weights.

Iterations - Number of required mini-batches to complete a full epoch.

Example:
you have a training dataset with 256 samples, a mini-batch size of 32 and epoch number 100.
Each training sample will be viewed by the model 100 times (as per the epoch number).
Each epoch will be done in 8 iterations (i.e. the model will be updated 8 times in an epoch).
Each iteration will include 32 training samples passed thru the model. The back propagation algorithm will calculate the required change in the model weights for every single sample, but the actual update will take place on the mean (or sum) of all 32 samples in the mini-batch.

As you said at the end, "iterations per epoch are set automatically according to how many training examples there are present".
There is no reason to alter this parameter, if you want to change the number of passes over the data, use the epoch number.


Balancing data-sets for regression problems Unbalanced data-sets are a well described problem for classification-problems.

However, for regression similar problems can arise. An example is the data-set where target variable has a very irregular histogram.

When fitting a model on such data, the main focus will be on the peak around the value 120. However, I want to create a model which works over the entire interval from 40 to 160.

How would you typically approach these problems?
Are you aware of any literature regarding this topic?
Is balancing the data-set a good approach and how can this be implemented?

First, the histogram you plotted is that of the marginal distribution. The residuals might look perfectly fine once you regress on the variables. Linear regression is an estimation of Y conditional on the variables X; thus, in order to determine whether the distribution is a good fit, you examine the residuals which are assumed to be gaussian.

If you have data that is bounded between an interval, one approach would be modeling it using a beta regression. If the data is constrained to be positive, assuming a gamma distribution could work as well. You must, however, check the diagnostic plots afterwards to ensure that the distribution assumptions that you are making are correct.

See Beta Regression by Ferrari, A Better Lemon Squeezer by Smithson '06, and Extending the General Linear Model by Faraway. Also, for Bayesian implementations of regression models, you could look at Gelman's book.


what does one Shot learning mean? do they only need one image to train for some new class detection? Being new to deep learning I am somewhat struggling to grasp the idea of one shot learning.
Let us say I have a class to detect which didn't exist in training dataset such as COCO or Image NET. Can I train model for that class using only image or the training set must be large as for YOLO or RCNNs?

One-Shot Learning refers to the problem when you only have very few or a single sample for some classes in your training dataset. A common application is, for example, face recognition. Here you may have only a single image per person in your dataset. Nevertheless, you'd like your neural net to be able to recognize that person from new images. A good intro is provided by Andrew Ng here.
A popular example are Siamese Nets introduced by Koch et al. The basic idea is to learn a latent representation of the images in your training set. When a new image is presented during inference the net calculates the latent representation of that new image and searches for the image in your training set whose latent representation is most similar to the one of the new image and predicts the corresponding class.
When you do not have any sample of a specific class available this is considered Zero-Shot Learning. However, in this case the neural net requires the dataset to include some auxiliary information for all images. The most popular being the Animals with Attributes (AWA) dataset:

Initial approaches to solve the Zero-Shot Learning problem did so in two steps:

For a given image learn to predict the attributes
Learn to predict the class based on the attributes

For details you may refer to the paper Attribute-Based Classification for Zero-Shot Visual Object Categorization. More recent approaches take another route though.
However, both approaches require, as always in Deep Learning, large datasets to learn.


Data science without knowledge of a specific topic, is it worth pursuing as a career? I had a conversation with someone recently and mentioned my interest in data analysis and who I intended to learn the necessary skills and tools. They suggested to me that while it is great to learn the tools and build the skills there is little point in doing so unless i have specialized  knowledge in a specific field.

They basically summed it to that I'd just be like a builder with a pile of tools who could build a few wooden boxes and may be build better things (cabins, cupboards etc), but without knowledge in a specific field I'd never be a builder people would come to for a specific product.

Has anyone found this or have any input on what to make of this ? It would seem if it was true one would have to learn the data science aspects of things and then learn  a new field just to  become specialized.

Drew Conway published the Data Science Venn Diagram, with which I heartily agree:

On the one hand, you should really read his post. On the other hand, I can offer my own experience: my subject matter expertise (which I like better as a term than "Substantive Expertise", because you should really also have "Substantive Expertise" in math/stats and hacking) is in the retail business, my math/stats are forecasting and inferential statistics, and my hacking skills lie in R.
From this vantage point, I can talk to and understand retailers, and someone who doesn't have at least a passing knowledge of this field will have to face a steep learning curve in a project with retailers. As a side gig, I do statistics in psychology, and it's exactly the same there. And even with quite some knowledge of the hacking/math/stats part of the diagram, I would have a hard time getting up to speed in, say, credit scoring or some other new subject field.
Once you have a certain amount of math/stats and hacking skills, it is much better to acquire a grounding in one or more subjects than in adding yet another programming language to your hacking skills, or yet another machine learning algorithm to your math/stats portfolio. After all, once you have a solid math/stats/hacking grounding, you could if need be learn such new tools from the web or from textbooks in a relative short time period. But the subject matter expertise, on the other hand, you will likely not be able to learn from scratch if you start from zero. And clients will rather work with some data scientist A who understands their specific field than with another data scientist B who first needs to learn the basics - even if B is better in math/stats/hacking.
Of course, all this will also mean that you will never become an expert in either of the three fields. But that's fine, because you are a data scientist, not a programmer or a statistician or a subject matter expert. There will always be people in the three separate circles who you can learn from. Which is part of what I like about data science.

EDIT: A little while and a few thoughts later, I'd like to update this post with a new version of the diagram. I still think that Hacking Skills, Math & Statistics Knowledge and Substantive Expertise (shortened to "Programming", "Statistics" and "Business" for legibility) are important... but I think that the role of Communication is important, too. All the insights you derive by leveraging your hacking, stats and business expertise won't make a bit of a difference unless you can communicate them to people who may not have that unique blend of knowledge. You may need to explain your statistical insights to a business manager who needs to be convinced to spend money or change processes. Or to a programmer who doesn't think statistically. Caulcutt (2021, Significance) is a short article that says much the same, but gives more detail than I do.
So here is the new data science Venn diagram, which also includes communication as one indispensable ingredient. I have labeled the areas in ways that should guarantee maximum flaming, while being easy to remember.
Comment away.

R code:
draw.ellipse <- function(center,angle,semimajor,semiminor,radius,h,s,v,...) {
    shape <- rbind(c(cos(angle),-sin(angle)),c(sin(angle),cos(angle))) %*% diag(c(semimajor,semiminor))
    tt <- seq(0,2*pi,length.out=1000)
    foo <- matrix(center,nrow=2,ncol=length(tt),byrow=FALSE) + shape%*%(radius*rbind(cos(tt),sin(tt)))
    polygon(foo[1,],foo[2,],col=hsv(h,s,v,alpha=0.5),border="black",...)
}
name <- function(x,y,label,cex=1.2,...) text(x,y,label,cex=cex,...)

png("Venn.png",width=600,height=600)
    opar <- par(mai=c(0,0,0,0),lwd=3,font=2)
        plot(c(0,100),c(0,90),type="n",bty="n",xaxt="n",yaxt="n",xlab="",ylab="")
        draw.ellipse(center=c(30,30),angle=0.75*pi,semimajor=2,semiminor=1,radius=20,h=60/360,s=.068,v=.976)
        draw.ellipse(center=c(70,30),angle=0.25*pi,semimajor=2,semiminor=1,radius=20,h=83/360,s=.482,v=.894)
        draw.ellipse(center=c(48,40),angle=0.7*pi,semimajor=2,semiminor=1,radius=20,h=174/360,s=.397,v=.8)
        draw.ellipse(center=c(52,40),angle=0.3*pi,semimajor=2,semiminor=1,radius=20,h=200/360,s=.774,v=.745)

        name(50,90,"The Data Scientist Venn Diagram",pos=1,cex=2)
        name(8,62,"Communi-\ncation",cex=1.5,pos=3)
        name(30,78,"Statistics",cex=1.5)
        name(70,78,"Programming",cex=1.5)
        name(92,62,"Business",cex=1.5,pos=3)

        name(10,45,"Hot\nAir")
        name(90,45,"The\nAccountant")
        name(33,65,"The\nData\nNerd")
        name(67,65,"The\nHacker")
        name(27,50,"The\nStats\nProf")
        name(73,50,"The\nIT\nGuy")
        name(50,55,"R\nCore\nTeam")
        name(38,38,"The\nGood\nConsultant")
        name(62,38,"Drew\nConway's\nData\nScientist")
        name(50,24,"The\nperfect\nData\nScientist!")
        name(31,18,"Comp\nSci\nProf")
        name(69,18,"The\nNumber\nCruncher")
        name(42,11,"Head\nof IT")
        name(58,11,"Ana-\nlyst")
        name(50,5,"The\nSalesperson")
    par(opar)
dev.off()


Sharing Jupyter notebooks within a team I would like to set up a server which could support a data science team in the following way: be a central point for storing, versioning, sharing and possible also executing Jupyter notebooks.

Some desired properties:

Different users can access the server and open and execute notebooks that were stored by them or by other team members. The interesting question here is what would be the behavior if user X executes cells in a notebook authored by user Y. I guess the notebook should NOT be changed:
Solution should be self-hosted.
Notebooks should be stored either on the server or on Google drive or on self-hosted instance of owncloud.
(Bonus) Notebooks will be under git versioning control (git may be self-hosted. Cannot be bounded to GitHub or something of that sort).

I looked into JupyterHub and Binder. With the former, I didn't understand how to allow cross users access. The latter seems to only support GitHub as the storage of the notebooks.

Do you have experience with either of the solutions?

JupyterHub does not provide version control system nor facilitates sharing of Notebooks. You mentioned yourself limitation of Binder.

Try Zeppelin. Version 0.7 should be released within a few next days.

As you can see from the roadmap, this version delivers "enterprise" features which are exactly about collaboration.
Version control system (git) is integrated.
It's self-hosted.

In essence, I think it meets all requirements you posted. On top of that it delivers richer visualisation capabilities and plethora of other features (works with Shiro, Knox, Kerberos - secure Spark anyone?).


ValueError: Input 0 of layer sequential is incompatible with the layer: expected ndim=3, found ndim=4. Full shape received: [None, 25, 25, 1] I am trying to use conv1D but getting that error.
My dataset's is batched and has a shape of [None, 25, 25, 1]
I am using input_shape=(25,25)
I am not able to figure out what should I change so I can get it to work.
My model:
model = Sequential()
model.add(Conv1D(32, kernel_size=3, activation='relu', input_shape=(25,25))
model.add(Flatten())
model.add(Dense(1, activation='sigmoid'))

I have solved the problem by changing the shape of my dataset using:
tf.reshape(data, [25, 25])


How should classification be done for a very small data set? I am looking at data from the London Data Store based on social characteristics between London boroughs.

Since there are only about 30 London boroughs, the data sets I am looking at are naturally very small. For example, I might be fitting regression/correlations to a plot of about 30 points.

What are appropriate ways to conduct classification on such small data sets, and why? 'Why' is important.

I was thinking of something like SVM, or Naive Bayes. Or regression if the data is continuous.

What are very inappropriate ways to conduct classification here?

I don't think you need some classification algorithm, you can use your basic understanding on data/ Business Knowledge to do the classification. As the number of data points are too low, the model cannot give you good/generalised results.

Even if you try applying some complex algorithm like SVM/NN, it is of no use as the data is too low.

If you still want to apply some machine learning algorithm and then you can apply Naive Bayes, Decision Tree as these are the basic algorithms, can do the job.


Machine Learning Method For Creating Chemical Formula We work at chemical company . We have nearly 3000 chemical formulas which is composed with chemical raw materials.
Our chemical formulas is composed of 20-25 raw materials. As you guess, amount of these raw materials are very important for our formulas. As a result we obtain different products via using different raw materials and different amount of raw materials.
Here is some examples:
Sample Formula 1:
%10 Raw material A
%25 Raw material B
%5 Raw material C
%60 Raw material D

Result product has such properties :
Property X: 3
Property Y: 10
Property Z: 2

What kind of machine learning method or methods should we use? I have looked for CNN but as you see there are many input and output properties as parameters.
Our goal is training a network via using our exist formulas then we will want that our network will create new formulas according to input parameters which we will give.

When picking an ML method, a great place to start is the sklearn algorithm cheat sheet:

You are predicting quantities, so regression class of variables is what you're looking for. Specifically you have multiple continuous inputs and multiple continuous outputs, which is a multivariate (linear) regression family.
As highlighted in the comments, start with a simple baseline model and build up in complexity to attempt to improve performance, towards CNNs or other.


Adding time as a feature with xgboost/random forests I am trying to use xgboost for performing some regression and the features I have are rather simple and limited. I have the time stamp associated with some measurements. The measurements are customer counts and the dependent variable is predicting the average customer wait time. There is dependence on time of the day and weekends as well. For example, I notice that wait times are longer in thee afternoon than in the morning and evenings. There is also a dependency on weekday, holiday or weekends. So I also added a boolean variable to indicate whether the given day is a weekend or holiday or not.

The time data I have is in 10 minutes interval and is a regular python time stamp with the date, hours, minutes and second granularity. How should such time features be included with xgboost or random forests or indeed any such modelling paradigms?

You have already constructed features that indicate relevant events like weekday, holiday and so on, this is good. Create as many such features you can come up with.

As for the time of day, I'd try using the hour only and combination of hour+minute. Probably less important is the minute alone, unless there is something that causes spikes at the same times of every hour.

Since you use tree-based models, you don't need to normalize. Mark your time features as categorical (can you do that in xgboost? I forgot, but I think you can in lightgbm) or as boolean where applicablen. And then check feature importances, which you can with gradient boosted tree type of models.

Finally, make sure your validation splits are good, which can be a challenge whenever you have some kind of time involved. Make sure your don't leak information and that your validation set is representative.


Website navigation path using graph theory In web analytics, one of the main issues is to find the optimal path of navigation. Question: Is there any literature in graph theory area, which is focused on studying optimal website navigation path?

My Comment to Tom was pretty large. So putting it here:

But in network theory there's a concept of main node (forgot the exact terminology). This node is most important because it has most connection to other pages. Home page is a very simple kind of main node. Also different set of visitors might have different main node. So, if we could capture what is the main node, then if we directly drop the visitor to this page or some other page, then the conversion might increase. Because the visitor who will come to this main page, will able to easily navigate to his required page. And then able to get what he is looking for. Another method people use for shortest path is MCMC methodology. But I'll prefer network method. Figuring out the entire method is pretty cumbersome and I don't have much time. So I was wondering if somebody wrote some journal paper on it, I can refer that.

I've done a lot of web analytics work and while we did do path analysis, we never focused so much on whether the user took the absolute shortest path to the page.

Unlike building a road system, a website should be built to provide different paths to the same content depending on the intent of the user. For example, some users know exactly what they want and will do a search. Other users are exploratory and want to browse.

Finally, sometimes you specifically want the user to step through a handful of pages to sell them on a conclusion. This is called a conversion funnel. If the user landed directly on the conversion point without seeing the info leading up to it, they'd bounce off the site and we'd lose the lead.


Incrementally Train BERT with minimum QnA records - to get improved results We are using Google BERT for Question and Answering. We have fine tuned BERT with SQUAD QnA release train data set (https://github.com/google-research/bert , https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v1.1.json)

It generated new checkpoints and BERT is giving good answers for most of questions we asked on our text documents. However, there are some questions which it is answering wrong, so we are trying to further fine tune with our Question and known answer on our text document. We further trained based on last generated checkpoint and got new checkpoint.

With new checkpoint when we are asking the same question, the answer did not got corrected! Previously BERT was giving wrong answer with 99% confidence and now also giving same wrong answer with 95% confidence.

Can someone suggest, if they have same or similar experience, and suggest please.
Following are questions in BERT github Issues, and are unanswered for quite some time:

BERT accuracy reduced after providing custom training..The answer is
also not correct : https://github.com/google-research/bert/issues/492
Unable to incrementally train BERT with custom training: https://github.com/google-research/bert/issues/482
Little training has no impact:    https://github.com/google-research/bert/issues/481
Custom Domain Training: https://github.com/google-research/bert/issues/498

Remember that BERT was first pre-trained using the concatenation of BooksCorpus (800M words) and English Wikipedia (2,500M words). Then the fine-tuning in your case uses the SQuAD dataset consisting of 100,000+ questions (based on Wikipedia articles) with a learning rate in the order of e-5.

So when you add let's say 100 new domain-specific question/document/answer to your input during the fine-tuning, this will not have a major effect on the learned parameters especially if your new input has out-of-Bert-vocabulary words or uses different English style, as opposed to Wikipedia/Books style.

  Can someone suggest, if they have same or similar experience, and
  suggest please.

Following are some suggestions that might help:

Increase the amount of new input data to help the model converge to your domain-specific use case.
Before fine-tuning,  map recurrent out-of-Bert-vocabulary words of your new input to existing words in Bert vocabulary which have almost the same meaning.
Increase the learning rate for your new input. But this is not very safe as the model may "forget" what he learned and not converge. But as the fine-tuning doesn't take much time you can try it out and see the results.


How to extract data from an API every hour in Python? I am new to Python and I tried writing a script that extracts air quality json data from an API every hour and logs it into a same excel file. My code doesn't return anything.
Is my code correct ? and how can I make it log into the excel file every hour please ? Thank you very much.

Here is the script:

def write_to_excel():
    request = requests.get("https://api.waqi.info/feed/paris/?
     token=?")
    request_text = request.text
    JSON = json.loads(request_text)

    filterJSON = {
        'time': str(JSON['data']['time']['s']),
        'co': str(JSON['data']['iaqi']['co']['v']),
        'h': str(JSON['data']['iaqi']['h']['v']),
        'no2': str(JSON['data']['iaqi']['no2']['v']),
        'o3': str(JSON['data']['iaqi']['o3']['v']),
        'p': str(JSON['data']['iaqi']['p']['v']),
        'pm10': str(JSON['data']['iaqi']['pm10']['v']),
        'pm25': str(JSON['data']['iaqi']['pm25']['v']),
        'so2': str(JSON['data']['iaqi']['so2']['v']),
        't': str(JSON['data']['iaqi']['t']['v']),
        'w': str(JSON['data']['iaqi']['w']['v']),
    }

    liste.append(filterJSON)
    try:
        os.remove("airquality.xlsx")
    except:
        pass

    pd.DataFrame(liste).to_excel('airquality.xlsx')
    print(liste)

if __name__ == "__main__":
     schedule.every(3).seconds.do(write_to_excel)
     while True:
         schedule.run_pending()

'''

Assuming that example is working for you, trying to write the data every 3 seconds, you need to just change the scheduling to be

schedule.every(1).hour.do(write_to_excel)

You are currently writing the data at each interval to the same file, so you will overwrite the file every time. You could do a few things here:

Open the excel file (e.g. into a pandas DataFrame), append the new data and save it all back to disk. This is pretty inefficient though and will have problems once you have a lot of data.
Write the data to a database, extending it every hour. This is the most professional solution.
Write a new file to disk each hour, including e.g. the timestamp of the hour in the filename to make each file unique. This is simple and just means you iterate over the files one-by-one when reading them later to do analysis or plotting etc.

You could change your function to be like this, implementing the first option above:

def request_data_and_save(excel_file: str = "air_quality.xlsx"):
    request = requests.get("https://api.waqi.info/feed/paris/?token=?")
    request_text = request.text
    JSON = json.loads(request_text)

    filterJSON = {
        'time': str(JSON['data']['time']['s']),
        'co': str(JSON['data']['iaqi']['co']['v']),
        'h': str(JSON['data']['iaqi']['h']['v']),
        'no2': str(JSON['data']['iaqi']['no2']['v']),
        'o3': str(JSON['data']['iaqi']['o3']['v']),
        'p': str(JSON['data']['iaqi']['p']['v']),
        'pm10': str(JSON['data']['iaqi']['pm10']['v']),
        'pm25': str(JSON['data']['iaqi']['pm25']['v']),
        'so2': str(JSON['data']['iaqi']['so2']['v']),
        't': str(JSON['data']['iaqi']['t']['v']),
        'w': str(JSON['data']['iaqi']['w']['v']),
    }

    # Get a writer and append data to the file
    with pd.ExcelWriter(excel_file, mode="a") as xl:
        df = pd.DataFrame.from_dict(filterJSON)
        df.to_excel(xl)

if __name__ == "__main__":
    schedule.every().hour.do(write_to_excel)
    while True:
        schedule.run_pending()

NOTE: this code is not tested


Disadvantage of decision tree I read in a blog that the decision tree has this disadvantage:

Not fit for continuous variables

If this is true, then why?

Decision trees work well with categorical variables because of the node structure of a tree. A categorical variable can be easily split at a node. For example, yes or no or 0 or 1. A continuous variable is just that, continuous along a range which cannot be easily split at a node. For example, a floating point number between 0 and 100.


Dealing with normalized regression output I have a regression model that is trained on a bunch of features and normalized targets so naturally when I use the model to predict on a new input, the output is also normalized (well not normalized per se but not what I expect to see). How should I deal with this? I tried using the inverse_transform function in sklearn that is in most scalers but it's not giving me correct results. This is probably because it wasn't already fitted on this data and therefore doesn't know how to inverse it. What can I do?

In linear regression, you don't have to normalize the output variable. This is actually why, for example, StandardScaler ignores y input even if you had entered. Also, inverse_transform is for the input variable. The prediction you get should be in your actual output domain.


How to mantain the nested structure of a tf.dataset after applying map? I'm creating a tf.dataset object containing 2 images as inputs and a mask as target. All of them are 3D in grayscale. After applying a custom map, the shape of the object changes from  ((TensorSpec(shape=(), dtype=tf.string, name=None), TensorSpec(shape=(), dtype=tf.string, name=None)), TensorSpec(shape=(), dtype=tf.string, name=None)) to (TensorSpec(shape=<unknown>, dtype=tf.float32, name=None), TensorSpec(shape=<unknown>, dtype=tf.float32, name=None), TensorSpec(shape=<unknown>, dtype=tf.int32, name=None)), losing the nested structure. When I fit the data, my model throws an error because it only detects one input instead of 2.
Here is what I'm doing:
x, y = get_filenames(train_data_path, img_type='FLAIR')
z = get_filenames(train_data_path, img_type='mask')

path_dataset = tf.data.Dataset.from_tensor_slices((x, y))
mask_dataset = tf.data.Dataset.from_tensor_slices(z)
dataset = tf.data.Dataset.zip((path_dataset, mask_dataset)).shuffle(50).repeat(10)

ds = dataset. \
    map(lambda xx, zz: ((tf.py_function(load, [xx], [tf.float32, tf.float32])),
                        tf.py_function(load_mask, [zz], [tf.int32])),
        num_parallel_calls=tf.data.AUTOTUNE)

ds = ds.map(lambda xx, zz: (tf.py_function(random_crop_flip, [xx, zz],
                                           [tf.float32, tf.float32, tf.int32])),
            num_parallel_calls=tf.data.AUTOTUNE)
ds = ds.batch(2)
ds = ds.prefetch(tf.data.AUTOTUNE)

I can't map separately the images and the masks because they need the same seed for the random cropping and flipping. Is it possible to change the shape after the map so that I can feed it to my 2 input model?
My random_crop_flip function is as follows:
def random_crop_flip(images, mask, width=128, height=128, depth=128):
    img_bl, img_fu = images
    img_bl = img_bl.numpy()
    img_fu = img_fu.numpy()
    mask = mask.numpy()
    x_rand = random.randint(0, img_bl.shape[2] - width)
    y_rand = random.randint(0, img_bl.shape[1] - height)
    z_rand = random.randint(0, img_bl.shape[3] - depth)
    img_bl_f = img_bl[:, y_rand:y_rand + height, x_rand:x_rand + width, z_rand:z_rand + depth, :]
    img_fu_f = img_fu[:, y_rand:y_rand + height, x_rand:x_rand + width, z_rand:z_rand + depth, :]
    mask_f = mask[:, y_rand:y_rand + height, x_rand:x_rand + width, z_rand:z_rand + depth, :]
    flip_x = random.choice([True, False])
    flip_y = random.choice([True, False])
    flip_z = random.choice([True, False])

    if flip_x:
        img_bl_f = np.flip(img_bl_f, axis=2)
        img_fu_f = np.flip(img_fu_f, axis=2)
        mask_f = np.flip(mask_f, axis=2)

    if flip_y:
        img_bl_f = np.flip(img_bl_f, axis=1)
        img_fu_f = np.flip(img_fu_f, axis=1)
        mask_f = np.flip(mask_f, axis=1)

    if flip_z:
        img_bl_f = np.flip(img_bl_f, axis=3)
        img_fu_f = np.flip(img_fu_f, axis=3)
        mask_f = np.flip(mask_f, axis=3)

    return (img_bl_f, img_fu_f), mask_f

The tuple in the output isn't solving my problem. Is it possible to modify the return to get my desired output?

I have managed to solve this problem by "flattening" (eliminating the parenthesis) the return of the random_crop_flip, and applying another map on top of them, where I specified the shapes and returned my desired structure (x ,y), z:
def _set_shapes(img_bl, img_fu, mask):
    img_bl.set_shape([128, 128, 128, 1])
    img_fu.set_shape([128, 128, 128, 1])
    mask.set_shape([128, 128, 128, 1])

    return (img_bl, img_fu), mask

Then my code looks like this:
x, y = get_filenames(train_data_path, img_type='FLAIR')
z = get_filenames(train_data_path, img_type='mask')

path_dataset = tf.data.Dataset.from_tensor_slices((x, y))
mask_dataset = tf.data.Dataset.from_tensor_slices(z)
dataset = tf.data.Dataset.zip((path_dataset, mask_dataset)).shuffle(50).repeat(10)

ds = dataset. \
    map(lambda xx, zz: ((tf.py_function(load, [xx], [tf.float32, tf.float32])),
                        tf.py_function(load_mask, [zz], [tf.int32])),
        num_parallel_calls=tf.data.AUTOTUNE)

ds = ds.map(lambda xx, zz: (tf.py_function(random_crop_flip, [xx, zz],
                                           [tf.float32, tf.float32, tf.int32])),
            num_parallel_calls=tf.data.AUTOTUNE)
ds = ds.map(_set_shapes)
ds = ds.batch(2)
ds = ds.prefetch(tf.data.AUTOTUNE)


How can I classify specific types of words in a document given I have the full text of the document and the labels I am working on a project that involves picking out specific kinds of objects from text. The documents I am going though are life sciences and biomedical in nature, and in these documents there are specific biomedical "objects" I want to pick out. The nature and variety of the text objects means I can't use regex or string matching. It has to be some kind of classification.
These text objects can be one word, or multiple words, but they are always in sequence.
An example sentence would be like
During the process of protein synthesis, X was used.

I need to pick out X. Luckily, I have plenty of labeled documents, and plenty of labels to go along with it. So I know a human can pick out these objects. So the challenge now is to get a machine to be able to pick out these types of objects from unseen text. I am working under the assumption that these specific text objects all fall under somewhat similar grammatical and textual context, so given enough labeled data, a machine should be able to learn how to pick out the text object.
Two Main Questions.

How do I label specific words in a document such that some model will understand that given a sequence of text, the object at position Y is a labeled and what we should be trying to classify.

Does anything I just said make any sense? Is there any research on what I've been talking about, because I've looked around and have not been able to find much.

The task you describe corresponds exactly to Named Entity Recognition (NER). This is a standard task for which there are many available libraries. NER is usually done with sequence labelling models such as Conditional Random Fields. Sequence labelling implies that the data is provided as a sequence which looks like this:
During    <features ...>  O
the                       O
process                   O
of                        O
protein                   O
synthesis                 O
,                         O
X_token1                  B_classX
X_token2                  I_classX
X_token3                  I_classX
was                       O
used                      O
.                         O

Here I'm using the common BIO format (Begin, Inside, Outside an entity) but there are variants. The model is trained with data annotated in this way, where there can be additional features (very often POS tag and others). Then when fresh text is provided (with the features) the model predicts the BIO tag for every token.
There has been a lot of research and resources produced for extracting specific entities in the specific context of biomedical data, so you might be interested in exploring these specific resources as well.

Medline, PMC are huge collections of biomedical abstracts/papers
There are many tools for extracting biomedical annotations based on Medline/PMC data: PubTator, cTakes, SciSpacy,etc.


How do I implement the sigmoid function in Octave? so given that the sigmoid function is defined as hθ(x) = g(θ^(T)x), how can I implement this funcion in Octave given that g = zeros(size(z)) ?

This will compute the sigmoid of a scalar, vector or matrix.

function g = sigmoid(z)
%   SIGMOID Compute sigmoid function
%   g = SIGMOID(z) computes the sigmoid of z.

% Compute the sigmoid of each value of z (z can be a matrix,
% vector or scalar).

SIGMOID = @(z) 1./(1 + exp(-z));

g = SIGMOID(z);

end


Why has Hadoop failed to become popular? Apache Hadoop was once considered one of the tools you should have as a Data Scientist. Around 2012 to 2014, it gained the popularity steeply among Data Scientists, and seemed to be considered one of the toolsets on the same boat as Python, R, and SQL.

But these days I have not heard its story. In fact, the Google Trend also agrees with me, that the popularity is peak at May, 2015.

Why has Hadoop failed to become one of the necessary tools as a Data Scientist, like Python or R?

I wouldn't say Hadoop failed to become popular rather I would say, it is still the base of any production big data system.

Python or R are handy at the beginning when you just need to try out things but when it comes to putting things in production, Hadoop is the way to go. It does not directly provide any data scientist tools what it does provide is the base where data would be stored, processed and can be used to apply machine learning algorithms using Spark.

So to summarize, I see Hadoop being used in the following contexts

For Data Storage in Production Systems
Hadoop YARN as a cluster manager for tools like Spark, Flink
If someone still wants to code in R/Python/SAS - Hadoop being used as backend.

Hope this helps.