Masked Language Modeling on Domain-specific Data My goal is to have a language model that understands the relationships between words and can fill the masks in a sentence related to a specific domain. At first, I thought about pretraining or even training a language model(like BERT) from scratch, but unfortunately, my data isn't that big to help the previous model learn new connections, let alone learn the embeddings from scratch.
Now what I have in mind is creating a transformer model with my own vocabulary which consists of words in my domain-specific data (after separating them with spaces and not using transformer tokenizers). This way the vocab size would be smaller and the positions and relations would be learned faster and more easily. Although I'm a bit confused about implementation.
Can I use this architecture (that is for NMT) and give plain text for both the input and output? or should I mask some tokens in the input and give the complete sentence as the label?
Any other suggestions?

First I suggest reading the transformers paper. Couple of quick notes is that this model consists of an encoder and a decoder, and the original task the paper is trained on is machine translation. Datasets (benchmarks) they used to train and evaluate this model from scratch were WMT 2014 Engligh-to-German, WMT 2014 English-to-French (section 5.1 of the paper). The conclusion is that you cannot train transformers from scratch unless you have your sentence pairs in 2 languages.
MLM on the other hand is something that BERT used for training. So if you want to go in this direction, you can use the pre-trained BERT and fine-tune it with Masked language model head on top of your BERT model using your own dataset. You need to stick to the tokenizer that BERT was originally trained on but at least you are taking advantage of some general context that was learned during training BERT from scratch. If you want to train BERT either from scratch or fine-tune with MLM head you can follow this tutorial from hugging face


How to understand backpropagation using derivative Before I was learning about gradient descent, but now I understand this. Now, I have a problem with the backpropagation algorithm. I know the idea - minimalize error in multilayer neural network using chain rule. However, I don't understand the role of the derivative of the sigmoid function. This derivative is described in the algorithm. What is the point of this? Can you explain this step by step using simple language?

This is a continuation to this answer. The first part of that answer covered gradient descent. In short this algorithm finds a set of parameters which results in a minimum of a function. From the description of the algorithm you can see where the derivative of the function is needed.

How this applies to a neural network

Let's first consider a single neuron network. This is called the perceptron. We will use the sigmoid activation function. It has inputs $x$ and an associated output

$\hat{y} = \frac{1}{1+e^{w^Tx+b}}$.

The model is parametrized by the weights $w$ and the biases. These are the model parameters we need to tune. So this single neuron takes a vector $x$ and using the weights vector and biases will output some value $y$.

If the weights are randomly chosen then the output will also be random. This is of no use to us. We need to determine a way of calculating a measure of correctness or wrongness in order to know how we should change our weights. I will use a measure of how wrong we are and call this the cost

$C = \frac{1}{2N} \sum_{i=0}^{N}(\hat{y} - y)^2$.

This is takes the squared difference between the predicted output $\hat{y}$ and the ground truth $y$. We will take the sum of this wrongness for all the instances that we try to predict, hence the summation. And we divide by $N$ to normalize the error. The additional division by 2 is simply to make the derivative easier to compute you will see soon.

So we want to minimize the cost function using gradient decent. Thus we need to take the derivative of the cost function with respect to the parameters we can tune, these being the weights $w$. We must thus compute $\frac{\partial C}{\partial w}$. Using calculus we can use chain rule in order to simplify our derivative as

$\frac{\partial C}{\partial w} = \frac{\partial C}{\partial \hat{y}}\frac{\partial \hat{y}}{\partial w}$

Our gradient descent calculation thus becomes

$w^{new} = w^{old} - \nu \frac{\partial C}{\partial \hat{y}}\frac{\partial \hat{y}}{\partial w}$

where $\nu$ is the learning rate.

Let's explicitly spell out this derivative

$\frac{\partial C}{\partial \hat{y}} = \hat{y} - y$

and, the second is the derivative of the activation function which we are using which is the sigmoid function.

$\frac{\partial \hat{y}}{\partial w} = \frac{1}{1+exp(w^Tx + b)} (1 - \frac{1}{1+exp(w^Tx + b)})$


convert some observations into variables I have a table formatted like the following table:

Feature amount  ID  Location
Feat1      2    1   US
Feat2      0    1   US
Feat3      0    1   US
Feat4      1    1   US
Feat2      2    2   US
Feat4      0    2   US
Feat3      0    2   US
Feat6      1    2   US

Let's say I have 200 different IDs.
I want to convert all different features into variables and amount into observations, so I combine rows with same ID into one row. For example,

Feat1 Feat2 Feat3 Feat4 Feat5 Feat6 ID Location
  2     0     0     1    NA    NA   1    US
 NA     2     0     0    NA    1    2    US

Is there a good way to do it either in Python (pandas) or R?

Thanks in advance!

Assume your table can be put into a pandas DataFrame object data with 4 columns as above. One way to achieve what you want is to do a GROUPBY using ID and Location. Then gradually assign values to each row of new table:

    newdata = pd.DataFrame(columns=['ID', 'Location', 'Feat1', 'Feat2', 'Feat3', 'Feat4', 'Feat5', 'Feat6'])
    grouped = data.groupby(['ID', 'Location'])

    for index, (group_name, d) in enumerate(grouped):
        newdata.loc[index, 'ID'] = group_name[0]
        newdata.loc[index, 'Location'] = group_name[1]
        for feature, amount in zip(d['Feature'], d['amount']):
            newdata.loc[index, feature] = amount


Dealing with categorical variables I have a panel data set. My dependent variable is total costs, and almost all of my independent variables are categorical variables. For instance, age is "old","new". Now i have some questions.

Should i use a dummy for all of them? For example, only type variable has 33 values itself, or i can use clustering and reduce them? Or any other way if you know)
Is there a difference in terms of behaviour between categorical variables which have a rank or not? For example type is "A","B",..."S" so no rank between A and B but quality is "A1","A2","A3" which A1 means highest quality.

I don't know why, I can't find enough information about variable selections and making data ready. So now i have lots of variable and I think I should choose between them and also reduse number of dummies.

You should convert the categorical variables to dummies. For each individual variable in general you want to have equal number of elements of each class, or at least the numbers should be close. If not, you can cluster smaller classes to form a larger one. For example, let's assume you have a categorical variable with 5 different categories. You want each class to be approximately %20 of the data. If it is not, you can define a new class which combines smaller classes to make each class approximately equal.

For the second part, if you can actually quantify how much A1 is better than A2, or able to assign a relative value to them based on some heuristics; you can convert them to numerical variables.

You can find an example of this in this notebook (section titled "Aggregating categorical variables"). It is from the course "Principles of Machine Learning: R Edition" on edX. You can watch the videos on audit mode for free; and the notebooks are on github.


How do I use multilevel regression models? I have crime event data rows:

dayofweek1, region1, hour1, crimetype1
dayofweek2, region2, hour2, crimetype2 ...

and I want to use them as factors to model crime rates/probabilities at the region level.

I also want to use the resulting  model to be able to input factor values to produce crime probabilities. e.g. on Sunday, in region1 there is a .03 chance of burglary at 3pm.

I think I should use a multilevel model, but everything I have found assumes a y value at the individual data row level which I do not have. All the row data are crimes.

Does anyone have an example of such a model (obviously not necessarily crime data, and preferably using python)?

Can the prediction bit be done?

I think what you're looking for is Survival analysis.

From Wikipedia:

  Survival analysis is a branch of statistics for analyzing the expected duration of time until one or more events happen, such as death in biological organisms and failure in mechanical systems.

In your case, you'd like to predict the time a crime will occur in a given region.

Here is a good talk about survival analysis in Python.
This is a python library dedicated to survival analysis and the one used in the video mentioned above. The lib itself has some examples which will also help you understand survival analysis as a whole.
An introduction to the concepts of Survival Analysis and its implementation in lifelines package for Python.

Hope this helps!

EDIT1:

As of the 'per group prediction': your problem is this:

  everything I have found assumes a y value at the individual data row level

There's no such thing as an algorithm to predict a target variable per group. You have to transform your data so it has a 'y' value per group, and then train some model based on that. It may be seen as a regression problem, or maybe as some kind of survival analysis. But you can't escape from the fact that you'll need one target value per group to start doing stuff.


Deriving Feature Map formula for Inhomogeneous Polynomial Kernel How do we derive/extrapolate the feature map for inhomogeneous polynomial kernels for degree $d$, define as $$K(x, x^\prime) = \bigg( (x \cdot x^\prime) + c\bigg)^d$$

I know we don't need to know a valid feature map to use the inhomogeneous polynomial kernels, I'm simply curious as to how one would derive/extrapolate it.

Let $x, y \in \mathbb R^N, \mathbf \alpha \equiv (\alpha_0, \dots, \alpha_N) \equiv (\alpha_0, \mathbf \alpha'_0), \; \mathbf z^\alpha \equiv z_0^{\alpha_0} \dots z_N^{\alpha_N}$ and ${d \choose \alpha}$ be the multinomial coefficient. Then

$$\left(\left<\mathbf x, \mathbf y \right> + c\right)^d = \sum_{|\mathbf \alpha| = d} {d \choose \alpha} c^{\alpha_0} (\mathbf x \mathbf y)^{\mathbf \alpha'_0}$$

If you redistribute the coefficient ${d \choose \alpha} c^{\alpha_0}$ between $\Phi(\mathbf x)$ and $\Phi(\mathbf y)$, it follows that

$$\Phi(\mathbf z) = \left(\sqrt{ {d \choose \alpha} c^{\alpha_0}} \mathbf z^{\alpha'_0} \right)_{\forall |\mathbf \alpha| = d}$$

References

A Tutorial on Support Vector Machines for Pattern Recognition (cf. 6.1. The VC Dimension for Polynomial Kernels)
Data Mining and Analysis: Fundamental Concepts and Algorithms, Ch.5, pp 149-151.


Crawling customer reviews from Amazon I want to know if there is any way that I can crawl customer reviews for particular products from amazon without being blocked. At the moment, my crawler is blocked after a few times. Any idea will be appreciated.

You are getting blocked because people do not want to waste server bandwidth on someone who is trying to exploit it without bringing significant profits.

Try to make your crawling less predictable.

Slow down the frequency with which you ping the server and vary the actions of your crawler. This will make it harder to detect as it will act less predictably and may be wrongfully identified as being a very quick human.


My training accuray is 1.0 but the predictions on the training data are wrong My neural network is not working right, and I am trying to find out what is up.

I inserted just three images to a transfer learning (mobilenet) neural network. The three images' classes are:
array([[0., 0., 0., 1.],
       [0., 1., 0., 0.],
       [0., 1., 0., 0.]])

I did 50 epochs on these pictures and by the 20th epoch or so, the training accuracy stayed at 1.0:

Epoch 50/50
3/3 [==============================] - 6s 2s/step - loss: 1.3671 - acc: 1.0000 - val_loss: 1.3770 - val_acc: 0.0000e+00

Then when I went to predict the outcome of the same three images like so:
predictions_test_2 = model_mn.predict(X, batch_size=1, verbose=1)

the predictions were:
array([[0.2473848 , 0.25099277, 0.251868  , 0.24975444],
       [0.24154082, 0.25245225, 0.25358915, 0.25241777],
       [0.24333884, 0.25127387, 0.25357786, 0.25180945]], dtype=float32)

How could that be if the training accuracy is 1.0?!

This is the code:
def mobilenet(img_rows, img_cols, channel=1, num_classes=None):

model = MobileNet( include_top=True,weights='imagenet')

model.layers.pop()

model.outputs = [model.layers[-1].output]

model.layers[-1].outbound_nodes = []

x=Dense(num_classes, activation='softmax')(model.output)

model=Model(model.input,x)

#To set the first 8 layers to non-trainable (weights will not be updated)

for layer in model.layers[:8]:

   layer.trainable = False
model_new = Sequential()
for layer in model.layers[:-1]: # just exclude last layer from copying
    model_new.add(layer)
model=model_new
model.add(Dense(256,activation='relu',input_shape=(1000,)))

model.add(Dense(64, activation='relu'))
model.add(Dense(4,activation='softmax'))

# Learning rate is changed to 0.001
sgd = SGD(lr=1e-6,decay=1e-1,momentum=0.95, nesterov=True)
adam=Adam(lr=1e-6, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0000001, amsgrad=True)
model.compile(optimizer=adam, loss='categorical_crossentropy',metrics=['accuracy'])

# checkpoint
filepath="weights-improvement-mn-{epoch:02d}-{val_acc:.2f}.hdf5"
checkpoint = ModelCheckpoint(filepath, monitor='val_acc', verbose=1, save_best_only=True, mode='max')
callbacks_list = [checkpoint]
return model

model_mn = mobilenet(img_rows, img_cols, channel, num_classes)

model_mn.fit(X, Y,batch_size=3,epochs=50,shuffle=True,verbose=1,validation_data=(X_vall, Y_vall))

Chances are, you are overfitting your data, you need to closely monitor the validation accuracy, as if it diverges from the training one too much, then you enter overfitting territory.

Also, using just 3 pictures for training is just too little, your network won't be able to generalize properly with such an small input.


"Optimal range for loss function": Myth or Truth? I am currently working on a regression problem using a deep neural network that given a volume 32x32x256 in input need to generate a second volume of the same dimensions in output, this is not a segmentation problem and the volume generated by the net is formed by floats (to be more specific, im tryng to remove noise from a noisy x-ray beam).
As a loss function i am using the MSE beetwen the volume generated by the net and a ground volume already calculated without noise.
The volumes have been normalized on values between 0 and 1, before giving them to the net.
When we started the first trainings we were getting very low values in MSE (about 3*10^-4), but the deep-net was converging (on values near 5*10^6).
One of my co-worker stated that those values (of the loss) were far to low to get an optimal result with a framework as Keras adn Adam as optimizer, and proposed to change the loss function to get result within a range of 100 - 0.01, he proposed to do the sum of the square difference instead of the mean.
At first i was really confused by his statement, but a did a test:
Instead of normalizing the volumes on values between 0 and 1 i normalized them between values 0 and 1000, before giving them to the net.
In this test the MSE went from values of about 30'000 and coverged of values near 600, after denormalizing the data this new neural network gave result much more accurate of the first network, and just by changing the starting normalization!
Do you have any idea of the cause of this behaviour, was my coworker right?

Very interesting statement you got there, i was not aware of this 'optimal loss value'.
the only way i could explain it is with the gradient computation : gradient is computed using :
$Grad = loss*lr$
So to me, changing the loss range is the same as changing the learning rate of your optimizer, it could be a good experiment to increase your lr instead of increasing the loss function range and see what happens to the training. I'm not aware of the subtilties of Adam optimizer so my explanation may be completly wrong.


CNN only performs well when split into 2 models I have 2 groups of data of equal shape (the main difference between the 2 are that one has half as many features - and consequently different labels of course)that perform better when they are trained independently (both using the same code). Is there any canonical approach to create a single model that performs well on both from here? Of course, I have some ideas such as increasing the number of layers and neurons but was wondering if there is something I am ignorant of. (p.s. this is for a regression problem)

You could try saving the model after training the one it performs best on. Then restore it and try feeding in the 2nd set.

saver = tf.train.Saver()

Then at the end of training, add the line

save_path = saver.save(sess, "/your/file/path.ckpt")

Then to restore.

with tf.Session() as sess:
      saver.restore(sess, "/your/file/path.ckpt")


keep_dims is deprecated, use keepdims instead I downloaded:

!git clone https://www.github.com/matterport/Mask_RCNN.git
os.chdir('Mask_RCNN')

And I've got an error:
which version I should have of Keras?

WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/keras/backend/tensorflow_backend.py:1154: calling reduce_max (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/keras/backend/tensorflow_backend.py:1188: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/keras/backend/tensorflow_backend.py:1290: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead

Futhermore:

totalMemory: 5.94GiB freeMemory: 5.44GiB
2019-04-03 22:37:38.374934: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0, 1
2019-04-03 22:37:40.343417: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-04-03 22:37:40.344366: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971]      0 1
2019-04-03 22:37:40.344373: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0:   N N
2019-04-03 22:37:40.344377: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 1:   N N
2019-04-03 22:37:40.345556: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 11435 MB memory) -> physical GPU (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:02:00.0, compute capability: 5.2)
2019-04-03 22:37:40.450785: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 5220 MB memory) -> physical GPU (device: 1, name: GeForce GTX TITAN Black, pci bus id: 0000:01:00.0, compute capability: 3.5)
2019-04-03 22:37:42.518519: W tensorflow/core/framework/allocator.cc:108] Allocation of 51380224 exceeds 10% of system memory.
2019-04-03 22:37:42.601229: W tensorflow/core/framework/allocator.cc:108] Allocation of 51380224 exceeds 10% of system memory.
2019-04-03 22:37:51.648032: W tensorflow/core/framework/allocator.cc:108] Allocation of 51380224 exceeds 10% of system memory.
2019-04-03 22:37:51.678817: W tensorflow/core/framework/allocator.cc:108] Allocation of 51380224 exceeds 10% of system memory.
2019-04-03 22:37:51.706928: W tensorflow/core/framework/allocator.cc:108] Allocation of 51380224 exceeds 10% of system memory.
[I 22:37:55.611 NotebookApp] Starting buffering for fa2cd5ca-20f3-4472-b6ca-6821e2f56118:02508f46d629494ab46babe6d7611656

It's not an error, it's simply notifying you that the code is written using past versions of Tensorlfow and some of the arguments of special methods were going to be deprecated in the future releases of the library. It's ok to be used, but you can also check what version is used and make a virtual environment and install the specified version of the library you want.

You can run your cell twice in order not to see the warning again.


Match between objective function and evaluation metric Does the objective function for model fitting and the evaluation metric for model validation need to be identical throughout the hyperparameter search process?
For example, can a XGBoost model be fitted with the Mean Squares Error (MSE) as the objective function (setting the 'objective' argument to reg:squarederror: regression with squared loss), while the cross validation process is evaluated based on a significantly different metric such as the gamma-deviance (residual deviance for gamma regression)?
Or should the evaluation metric match the the objective function as much as possible, hence the root mean square error need to be selected as the evaluation metric?

The evaluation metric for model validation has to be the same throughout the the hyperparameter search process in order fairly compare different models.
The objective function for model fitting can be different throughout the hyperparameter search process. During the hyperparameter search process, you can compare different algorithms and each of those algorithms can have different objective functions.
The objective function and evaluation metric should be thought of as complete separate concepts. There can only one objective function per algorithm. However, there can be many evaluation metrics. Objective functions are chosen for computers so they can efficiently and effectively fit the training data. Evaluation metrics are chosen for human stakeholders so they can better understand the impact of the model.
The confusion between the two concepts often comes from just measuring a machine learning system on minimizing error. If the measurement of a machine learning system broadens to include other requirements, then the difference between objective function and evaluation metric is more clear. The objective function is typically setup only to minimize error over the training dataset and work well with the chosen optimization technique. Evaluation metrics can also assess model error but can also include prediction speed, model size, model fairness, and many other requirements.


Sklearn Pipeline for mixed features: numerical and (skewed) categorical I am working on a dataset from Kaggle (housing price prediction). I have done some pre-processing on the data (missing values, category aggregation, selecting ordinal vs one-hot). I am trying to implement a pipeline to streamline the code. The pipeline consists of a ColumnTransformer with two components: one component contains a standard scaler applied to numerical and ordinal features; the second component has a one-hot encoder for the remaining set of features. I am passing this transformer to a GridSearchCV object to tune hyperparameters. In this case, it is a LASSO model. So, I am trying to tune the coefficient of the penalty term. The problem is some of the one-hot encoded features are highly skewed with the count in mostly one category. When GridSearchCV tries to run cross-validation, it raises an error saying that unknown categories are found while validating the model. I think this happens because while fitting the one-hot encoder the train set doesn't contain data points with specific labels that show up in the validation set. One obvious way to handle this would be to fit a one-hot encoder, keep it aside and then build a pipeline and carry on with the grid search (/validation) steps. This seems a bit disconnected to me considering the notion of pipeline was defined for exactly this purpose. Maybe I am missing something here. Is there a better (/efficient) way to achieve the above rather than separating the one-hot encoder from the pipeline?

For reference, the data for the above histogram,

Category   Counts
CompShg    1434
Tar&Grv      11
Tar&Grv      11
WdShngl       6
WdShake       5
Roll          1
ClyTile       1
Metal         1
Membran       1

You can use handle_unknown='ignore' in the OneHotEncoder; levels present in the test set but not the train set will be encoded as all-zeros rather than raising an error.

But... that example you provide, I rather doubt it's worth it to keep all the levels.  The coefficient learned for such a small level's dummy variable will be overly specialized (overfit).  Consider alternatives.


Finding similarity between two histogram plots I have data census(name, sex, age, capital_gain) and I want to plot all possible views (in a histogram) and find the most interesting view based on the distinction to other views.

So, I have to compute the similarity between two histogram plots, for example:

Average Capital Gain vs Sex (which computed by aggregate Capital gain to Sex)

Average age vs Sex (which computed by aggregate average age to sex)

My questions is, what kind of measurement that can be used to check the similarity between those two histogram plots? I consider the features that want to be compared are (x-axis, y-axis, and pattern of the histogram)

I really not sure about distance function that can be used for this case. I know Jaccard distance, but not sure it can be used in this situation or not.

Thank you,

I'm curious what other people will say, but one option is to use KL-divergence.  If your two histograms have the same x-axis, you can divide every column by the total count to convert counts to proportions. Then you could treat the histograms as probability mass, and use the KL-divergence.  KL-divergence is really a measure of the distance between two probability distributions, but histograms are an approximation of a discrete distribution. You'd pick one of your distributions to be P and one to be Q, then calculate $-\sum_x P(X)\log{\frac{P(X)}{Q(X)}}$ where $x$ is each bin on the x-axis of the histogram.  Be aware that this distance metric is directional, meaning you'll get a different answer if you swap P and Q.  The reason is that the KL-divergence is a measure of information transfer (like, the information you gain by moving from Q to P), but I think that's irrelevant to your question.


neural network for binary classification of xor gate i have written this neural network for XOR function.the output is not correct.it is not classifying the test inputs correctly.can anyone please let me the reason why.

import numpy as np
import pandas as pd
x=np.array([[0,0],[0,1],[1,0],[1,1]])
y=np.array([[0],[1],[1],[0]])
np.random.seed(0)
theta1=np.random.rand(2,8)
theta2=np.random.rand(8,1)
np.random.seed(0)
b1=np.random.rand(4,8)
b2=np.random.rand(4,1)
alpha=0.01
lamda=0.01

for i in range(1,2000):
    z1=x.dot(theta1)+b1

    h1=1/(1+np.exp(-z1))
    z2=h1.dot(theta2)+b2
    h2=1/(1+np.exp(-z2))

    dh2=h2-y
    #back prop

    dz2=dh2*(1-dh2)
    H1=np.transpose(h1)
    dw2=np.dot(H1,dz2)
    db2=np.sum(dz2)

    W2=np.transpose(theta2)
    dh1=np.dot(dz2,W2)
    dz1=dh1*(1-dh1)

    X=np.transpose(x)
    dw1=np.dot(X,dz1)

    db1=np.sum(dz1)

    dw2=dw2-lamda*theta2

    dw1=dw1-lamda*theta1
    theta1=theta1-alpha*dw1
    theta2=theta2-alpha*dw2
    b1+=-alpha*db1

    b2+=-alpha*db2

#prediction
#test inputs
input1=np.array([[0,0],[1,1],[0,1],[1,0]])
z1=np.dot(input1,theta1)

h1=1/(1+np.exp(-z1))
z2=np.dot(h1,theta2)

h2=1/(1+np.exp(-z2))

expected output=[0],[0],[1],[1]
actual output=[[ 0.95678049]
 [ 0.99437206]
 [ 0.98686979]
 [ 0.98628204]]

all are ones here.

There are a few mistakes in the code, so I am going to present a revised version here with comments.

Setup

import numpy as np
import pandas as pd
x=np.array([[0,0],[0,1],[1,0],[1,1]])
y=np.array([[0],[1],[1],[0]])
np.random.seed(0)

# Optional, but a good idea to have +ve and -ve weights
theta1=np.random.rand(2,8)-0.5
theta2=np.random.rand(8,1)-0.5

# Necessary - the bias terms should have same number of dimensions
# as the layer. For some reason you had one bias vector per example.
# (You could still use np.random.rand(8) and np.random.rand(1))
b1=np.zeros(8)
b2=np.zeros(1)

alpha=0.01
# Regularisation not necessary for XOR, because you have a complete training set.
# You could have lamda=0.0, but I have left a value here just to show it works.
lamda=0.001

Training - Forward propagation

# More iterations than you might think! This is because we have
# so little training data, we need to repeat it a lot.
for i in range(1,40000):
    z1=x.dot(theta1)+b1
    h1=1/(1+np.exp(-z1))
    z2=h1.dot(theta2)+b2
    h2=1/(1+np.exp(-z2))

Training - Back propagation

    # This dz term assumes binary cross-entropy loss
    dz2 = h2-y
    # You could also have stuck with squared error loss, the extra h2 terms
    # are the derivative of the sigmoid transfer function.
    # It converges slower though:
    # dz2 = (h2-y) * h2 * (1-h2)

    # This is just the same as you had before, but with less temp variables
    dw2 = np.dot(h1.T, dz2)
    db2 = np.sum(dz2, axis=0)

    # The derivative of sigmoid is h1 * (1-h1), NOT dh1*(1-dh1)
    dz1 = np.dot(dz2, theta2.T) * h1 * (1-h1)
    dw1 = np.dot(x.T, dz1)
    db1 = np.sum(dz1, axis=0)

    # The L2 regularisation terms ADD to the gradients of the weights
    dw2 += lamda * theta2
    dw1 += lamda * theta1

    theta1 += -alpha * dw1
    theta2 += -alpha * dw2

    b1 += -alpha * db1
    b2 += -alpha * db2

Prediction

This is where you can kick yourself, you forgot to use the biases!

input1=np.array([[0,0],[1,1],[0,1],[1,0]])
z1=np.dot(input1,theta1)+b1
h1=1/(1+np.exp(-z1))
z2=np.dot(h1,theta2)+b2
h2=1/(1+np.exp(-z2))

print(h2)

When I run the above code I get a correct-looking output

[[ 0.01031446]
 [ 0.0201576 ]
 [ 0.9824826 ]
 [ 0.98584079]]

In summary your three big errors were the wrong dimension for the bias vectors in setup, incorrect derivatives for sigmoid function (using correct form, but with wrong variable) and forgetting to use bias at all when predicting at the end. Other details are still worth noting, but would not have prevented you getting something working.


What can I do with a Decision Tree with poor ROC Let's say I do a Decision Tree analysis.

But the performance characteristics are nothing great (e.g. ROC is nothing great).

Is there anything I can do with this "not so great" tree. Or do I typically need to trash it and try something else (either a new data set or a new analysis on the same data)?

Decision trees has one big quality and one big drawback. Their big quality is that they are what is known as glass-box models. What I mean by that is that they expose what they learn in a very clear and intuitive way. The name comes from the fact that you can see through a glass box. So, because of that, decision trees are very useful for analysis, they are a nice support to understand the relations between variables, which variables are important, in which way they are important. And even if they does not provide crystal clear information, they can bring you ideas about that. This might be very helpful especially if you have domain expert knowledge and you can put things together in a meaningful manner.

Their main drawback is their high variability. This problem is mainly caused by their greedy approach. Each decision in the first level nodes shape differently the tree. You can even go further and see that a single additional data point is enough in many cases to get a totally different tree, especially if the sample is small or if the data is noisy.

There are two types of approaches to solve this issue. The first type of approach tries to improve the single tree you built. This kind of approaches are known as pruning. A simple example would be reduced error pruning. This approach is simple and produce good results. You train your tree on a sample of data. Then you take another sample of data, you fit the new data on the tree, and then evaluate again the nodes of the tree from the perspective of the new data. If a non-leaf node get at least the same error if would not be split than if it would be split, then you can decide to cut the child nodes and transform that node into a leaf node. There are however much nicer pruning strategies, which are stronger, perhaps based on cross validation or some other criteria, mostly statistical based. Notice however that for reduced error pruning you need additional data, or to split your original sample in two, one for training, the other for pruning. If you go further to estimate the prediction error you need a third sample.

The second approach would be either to build multiple times some trees and chose some based on cross validation, bootstrapping or whatever method you you, or use a tree ensembles like bagging or boosting algorithms. Note that for boosting and bagging you loose glass box property.

Ultimately you have to choose between understanding and performance, having as a decent compromise the pruning procedure.


Try this. It is a bit ugly, as I couldn't manage to get a solution without iterating through one of the dataframes. It could be done by iterating over the original data, or the new df with the list of all dates.

I had a similar problem, any cleaner ways of doing this would be appreciated.

import pandas as pd

df = pd.DataFrame(data={'Item':['A','A','B','A'],
                        'StartDate':['2018-01-01','2018-01-07', '2018-01-03','2018-01-03'],
                        'EndDate': ['2018-01-04','2018-01-08','2018-01-05','2018-01-05']})
df['StartDate'] = pd.to_datetime(df['StartDate'])
df['EndDate'] = pd.to_datetime(df['EndDate'])

index = pd.MultiIndex.from_product([['A','B'],pd.date_range(start = df.StartDate.min(), end = df.EndDate.max())], names=['item', 'date'])
new_df = pd.DataFrame(index = index, data = {'counts':0})

dates_list = new_df.index.get_level_values(1)
items_list = new_df.index.get_level_values(0)
for row in df.itertuples():
  new_df.counts[(items_list == row.Item) & (dates_list>=row.StartDate) & (dates_list<= row.EndDate)] += 1

print(new_df)

Which gives this. You can remove the multi-index later if needed

                 counts
item date
A    2018-01-01       1
     2018-01-02       1
     2018-01-03       2
     2018-01-04       2
     2018-01-05       1
     2018-01-06       0
     2018-01-07       1
     2018-01-08       1
B    2018-01-01       0
     2018-01-02       0
     2018-01-03       1
     2018-01-04       1
     2018-01-05       1
     2018-01-06       0
     2018-01-07       0
     2018-01-08       0


Why is training take so long on my GPU? Details:
GPU: GTX 1080
Training: ~1.1 Million images belonging to 10 classes
Validation: ~150 Thousand images belonging to 10 classes
Time per Epoch: ~10 hours
I've setup CUDA, cuDNN and Tensorflow( Tensorflow GPU as well).
I don't think my model is that complicated that is takes 10 hours per epoch. I even checked if my GPU was the problem but it wasn't.
Is the training time due to the Fully connected layers?
My model:
model = Sequential()
model.add()
model.add(Conv2D(64, (3, 3), padding="same", strides=2))
model.add(Activation('relu'))
model.add(Dropout(0.25))
model.add(Conv2D(64, (3, 3), padding="same", strides=2))
model.add(Activation('relu'))
model.add(Dropout(0.25))
model.add(Conv2D(32, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(3, 3), strides=2))
model.add(Flatten())
model.add(Dense(256))
model.add(Activation('relu'))
model.add(Dense(4096))
model.add(Activation('relu'))
model.add(Dense(10))
model.add(Activation('softmax'))

model.summary()

opt = keras.optimizers.rmsprop(lr=0.0001, decay=1e-6)

model.compile(loss='categorical_crossentropy',
          optimizer=opt,
          metrics=['accuracy']
          )

Because there is a lot of data I used the ImageDataGenerator.
gen = ImageDataGenerator(
 horizontal_flip=True
)

train_gen = gen.flow_from_directory(
        'train/',
        target_size=(512, 512),
        batch_size=5,
        class_mode="categorical"
)

valid_gen = gen.flow_from_directory(
        'validation/',
        target_size=(512, 512),
        batch_size=5,
        class_mode="categorical"
)

That's about expected. If you divide the number of seconds by the number of images you processed, you get 33 milliseconds per image, which seems about right for such a small network. Larger networks usually take in the ballpark of 50 to 200 milliseconds per image.

Yes, a large dense layer is likely to hurt your performance, since that's a huge matrix (256 by 4096) and a large matrix multiplication to go along with it every time you run the network.


What is the difference between tensorflow saved_model.pb and frozen_inference_graph.pb? I've re-trained a model (following this tutorial) from the google's object detection zoo (ssd_inception_v2_coco) on a WIDER Faces Dataset and it seems to work if I use frozen_inference_graph.pb from python, but if i take saved_model.pb and put it to tensorflow serving, it predicts a lot of detections all with confidence less than 0.10.

Another thing that I do not understand is that both files are 52Mb however the original files downloaded from the object detection zoo take 98Mb each.

Here's an example of predictions made with the saved_model and tensorflow serving:

And here's example of predictions by the same model, but using frozen_inference_graph.pb:

So what is the actual difference between the saved_model.pb and the frozen_inference_graph.pb?
Are there any ways to inspect what's wrong with the saved_model.pb i use?
How the much smaller size of the re-trained model vs the original could be explained?

saved_model.pb may represent multiple graph definitions as MetaGraphDef protocol buffers. Weights and other variables usually aren't stored inside the file during training. Instead, they're held in separate checkpoint files

.

frozen_inference_graph.pb has its variables converted into inline constants so everything’s in one file and ready for serving on any platform including mobile. Freezing process includes loading the GraphDef, pull in the values for all the variables from the latest checkpoint file, and then replace each Variable op with a Const that has the numerical data for the weights stored in its attributes It then strips away all the extraneous nodes that aren't used for forward inference. Reference: Tensorflow Model Files


What is the difference between sentiment analysis (e.g. on Facebook) and Twitter sentiment analysis? I'm bit confused with the names of sentiment-analysis and twitter-sentiment-analysis.I want to do sentiment analysis for facebook fan page posts.so I decide to use an api but I did not found any api which gives sentiment analysis for facebook Posts.Every api service is made twitter.I found datumbox api which gives results for sentiment-analysis and twitter-sentiment analysis here.

for facebook sentiment analysis, do I need to go with sentiment-analysis api or twitter sentiment-analysis.(I know Name itself suggests sentiment is for twitter tweets).

Is their any api that provides sentiment analysis for facebook posts data ?

or

facebook trained datasets with the classes of POS,Neg,Neu?

There is nothing specifically called Twitter sentiment analysis, rather its colloquial. Sentiment analysis is a common term and could be applied on any dataset depending upon the use-case considered.

I have not came across any facebook api for scraping posts (maybe there are privacy concerns, as sometimes you cannot see fb pages when you are not an user), nevertheless there are several apis to scrap tweets from twitter, even specifying person, geographic location, language.

here is a github gist for reference, it uses python tweepy module to grab current tweets.


What kinds of learning problems are suitable for Support Vector Machines? What are the hallmarks or properties that indicate that a certain learning problem can be tackled using support vector machines?

In other words, what is it that, when you see a learning problem, makes you go "oh I should definitely use SVMs for this'' rather than Neural networks or Decision trees or anything else?

SVM can be used for classification (distinguishing between several groups or classes) and regression (obtaining a mathematical model to predict something). They can be applied to both linear and non linear problems.

Until 2006 they were the best general purpose algorithm for machine learning.  I was trying to find a paper that compared many implementations of the most known algorithms: svm, neural nets, trees, etc. I couldn't find it sorry (you will have to believe me, bad thing). In the paper the algorithm that got the best performance was svm, with the library libsvm.

In 2006 Hinton came up with deep learning and neural nets. He improved the current state of the art by at least 30%, which is a huge advancement. However deep learning only get good performance for huge training sets. If you have a small training set I would suggest to use svm.

Furthermore you can find here a useful infographic about when to use different machine learning algorithms by scikit-learn. However, to the best of my knowledge there is no agreement among the scientific community about if a problem has X,Y and Z features then it's better to use svm. I would suggest to try different methods. Also, please don't forget that svm or neural nets is just a method to compute a model. It is very important as well the features you use.


Very low probability in naive Bayes classifier 1 I have some training data (TRAIN) and some test data (TEST). Each row of each table contains an observed class (X) and some columns of binary (Y). I'm using a Python script that is intended to predict the probability (Pr) of X given Y in the test data based on the training data. It uses a Bernoulli naive Bayes classifier. Here is my script:

https://stackoverflow.com/questions/55187516/look-up-bernoullinb-probability-in-dataframe

It works on the dummy data that is included with the script.

On the real data, I know from experience which class some of the Y columns are indicative of. My script however is giving probability predictions like "1" where I don't think that the class is correct and "6e-77" on correct classes.

Any advice on what I can try please?

Edit

There are two problems. The very low probability is caused by the naive assumption that nothing is related to anything else. This is described here:
https://scikit-learn.org/stable/auto_examples/calibration/plot_calibration_curve.html#sphx-glr-auto-examples-calibration-plot-calibration-curve-py

The incorrect answers are caused by my code getting confused about which class is which, as described on my Stack Overflow post.

Each column of binary (Y) is a feature. The Bernoulli naive Bayes classifier could identify the class (X) where the number of features (Y) was less than 17. The real data had more features than that. I found that another method could classify it accurately. That was:

Trainining:

(1) Count which features (Y) are in each class (X) in the training data

Testing:

(2) Give each row a score (Z) with a starting value of 0.5

(3) For each row:

If each feature (Y) is in the class (X) in the training data then add 1 to the score (Z).
If each feature (Y) is not in the class (X) in the training data then subtract 1 from the score (Z).
If the class (X) is not in the training data then don't do anything

The score (Z) was a good classifier for my data.


source and target functions in igraph package in R What are the source and target functions in igraph package in R?  so that source(s->t) = s and target(s->t) = t

I am working on this directed graph example:

require(igraph, quietly=TRUE)
bsk <-read.table("http://www.dimiter.eu/Data_files/edgesdata3.txt", sep='\t', dec=',', header=T)

bsk.network <- graph.data.frame(bsk, directed=T)

edit: I am trying to write a function like this:

f_source <- function(str,network){
 result <- head_of(network, E(network)[str]);
}

f_source(AA%->%DD,bsk.network)

The problem is the string. It includes quotations while E(network)[str] does not work with quotations.

Given

library(igraph)
set.seed(1)
g <- sample_pa(5)

you got

ends(g, E(g))
#      [,1] [,2]
# [1,]    2    1
# [2,]    3    1
# [3,]    4    1
# [4,]    5    2

and

head_of(g, E(g))
# + 4/5 vertices:
# [1] 2 3 4 5

tail_of(g, E(g))
# + 4/5 vertices:
# [1] 1 1 1 2

source() is an important base function.


Will the job of Data Science is going to be at risk? Data Science is the sexiest job of 21st Century according to the different sources with a high payment salary . But it seems that there will be over production of this profession . According to U.S. Bureau of Labor Statistics there will be about 11.6 million jobs in 2026 but looking at the current trend there may be over production of data science professional till 2026 which may cause unemployment of Data science professionals .

According to a research , the data science syllabus is harder than non Data Science syllabus in engineering.

The skills required for a Data Scientist are:

Computer Science Skill
Mathematics (Calculus , Linear Algebra , Statistics and Probability , Optimization , Numerical Analysis , etc)
Domain Specific Skills

The above 3 points looks simpler but if we start go for mastery it takes a lot of energy and time.

I am 19 year old from a undeveloped country seeking as a professional in Data science but the above mentioned statement makes me to think whether to choose a Data Science as a profession or not ? If I am wrong then it will be my pleasure to give some guidance to a 19 year old boy who completed his 10th grade and passionated in machine learning and Data Science as a profession .

Waiting for your open minded reply  and stay safe in global pandemic☺

I highly doubt that the job itself will going to be at risk. It is rather the other way around: Data science and machine learning will replace a lot of other jobs. In the end there, at least, always needs to be someone providing the data to the machine.
It seems like your question rather should be "will there be enough data science positions in the future to have good chances of getting a job". While I can not give you a concrete answer to this question, I can tell you that, I think, most other jobs have a way higher risk. Data science or, maybe, even more machine learning, seems like a good and future-proof choice for me.


Does Batch Normalization make sense for a ReLU activation function? Batch Normalization is described in this paper as a normalization of the input to an activation function with scale and shift variables $\gamma$ and $\beta$. This paper mainly describes using the sigmoid activation function, which makes sense. However, it seems to me that feeding an input from the normalized distribution produced by the batch normalization into a ReLU activation function of $max(0,x)$ is risky if $\beta$ does not learn to shift most of the inputs past 0 such that the ReLU isn't losing input information. I.e. if the input to the ReLU were just standard normalized, we would lose a lot of our information below 0. Is there any guarantee or initialization of $\beta$ that will guarantee that we don't lose this information? Am I missing something with how the operation of BN and ReLU work?

That is known a problem with the ReLU activation functions. It is often called a "dying ReLU". Given an input over the zero boundary, the unit is now almost always closed. A closed ReLU cannot update its input parameters, a dead ReLU stays dead.

The solution is to use variants of ReLU for the activation function such as Leaky ReLU, Noisy ReLUs, or ELUs.


Prediction with non-scalar output (label) I have recently confronted with a (at least for me) new kind of ML problem, where the output of the model should be a vector/matrix (depending on the interpretation, but there is no difference actually), not a scalar as usual. This is totally unknown for me. What kind of approach should one apply here? Are the "usual" (scalar-based) models applicable on this problem?

(Just for the sake of completeness, the problem is an image segmentation task where the model should decide first: if there a given pattern on the picture?, second: if so, where is it? - In latter case, it should define the borders of the subset pixels).

Neural Networks can have a vector or matrix as output layer, image segmentation is a well researched topic and deep learning (as most things concerning images) are the state-of-the-art. You will need (a lot of) training examples where the pattern is found, and where. This could be a bounding box, or per pixel if it is part of the pattern or not (this will generate a matrix equally sized to your input). To see if the pattern is found you could construct a second network that is just a binary classifier, or you could try to see if your pixel-based network will output almost only zeros in case of no pattern. In this case you will need negative examples as well.


Self-attention mechanism did not improve the LSTM classification model I am doing an 8-class classification using time series data.
It appears that the implementation of the self-attention mechanism has no effect on the model so I think my implementations have some problem. However, I don't know how to use the keras_self_attention module and how the parameters should be set.
The question is how to utilize keras_self_attention module for such a classifier.
The first confusion matrix is 2 layers of LSTM.

    lstm_unit = 256

    model = tf.keras.models.Sequential()
    model.add(Masking(mask_value=0.0, input_shape=(X_train.shape[1], X_train.shape[2])))
    model.add(Bidirectional(LSTM(lstm_unit, dropout=dropout,return_sequences=True)))

    model.add(Bidirectional(LSTM(lstm_unit, dropout=dropout,return_sequences=True)))

    model.add(keras.layers.Flatten())
    model.add(Dense(num_classes, activation='softmax'))

The second confusion matrix is 2 lSTM + 2 self-attention.
    lstm_unit = 256

    model = tf.keras.models.Sequential()
    model.add(Masking(mask_value=0.0, input_shape=(X_train.shape[1], X_train.shape[2])))
    model.add(Bidirectional(LSTM(lstm_unit, dropout=dropout,return_sequences=True)))

    model.add(Bidirectional(LSTM(lstm_unit, dropout=dropout,return_sequences=True)))

    model.add(SeqSelfAttention(attention_type=SeqSelfAttention.ATTENTION_TYPE_MUL,
attention_activation='sigmoid'))

    model.add(keras.layers.Flatten())
    model.add(Dense(num_classes, activation='softmax'))

I have further tried different functions from the module, such as
1.MultiHead
    model.add(MultiHead(Bidirectional(LSTM(units=32)), layer_num=10, name='Multi-LSTMs'))

Residual connection

 inputs = Input(shape=(X_train.shape[1],X_train.shape[2]))
    x = Masking(mask_value=0.0)(inputs)

    x2 = SeqSelfAttention(attention_type=SeqSelfAttention.ATTENTION_TYPE_MUL,
    attention_activation='sigmoid')(x)

    x = x + x2

    x = Bidirectional(LSTM(lstm_unit, dropout=dropout,return_sequences=True))(x)
    x = Flatten()(x)

    output = Dense(num_classes, activation='softmax')(x)

    model = Model(inputs=inputs, outputs=output)

But they are more or less the same, no much effect on the MAR MAP and ACC.
I have 2 Titan Xp so computation power is less problem for me, is there a way to make the model more accurate?

appears that the implementation of the self-attention mechanism has no effect on the model so I think my implementations have some problem.

The first thing I noticed is your base model is doing badly, with low F1 scores, especially for the classes 1..7. Could it be there is just not that much signal in the training data, and the 2-layer LSTM has already sucked it all out?
My second thought is that self-attention is good at finding connections between items however far apart they are in the sequence. But an LSTM is also fairly good at that. So maybe more of the same is not what you need. Self-attention is an essential part of a transformer, because it is the only component that works across the sequence; the only other component is the FFN, which operates on each item in isolation.
Having got those two things off my chest, one thing I've noticed in experiments (and, again, these are in the context of transformers) is the importance of the residual connection. If I take out the residual connection that sits between self-attention and the FFN I get a significantly worse model. (Even though I still have the residual connection across the whole of each layer.)
So you could try adding a residual connection across the self-attention.
The other idea is to move self-attention earlier. Put it before the LSTMs, rather than after them. (Again with a residual connection across it.)
Here is the idea, in (untested) code. I've moved the self attention module to before the first LSTM layer, and then take the sum of the input and the output - that is what makes the residual connection across it.
...

x = Masking(mask_value=0.0, input_shape=(X_train.shape[1], X_train.shape[2])))

x2 = SeqSelfAttention(attention_type=SeqSelfAttention.ATTENTION_TYPE_MUL,
attention_activation='sigmoid')(x)

x = x + x2

x = Bidirectional(LSTM(lstm_unit, dropout=dropout,return_sequences=True))(x)

...

Note that you can't use the Keras sequential class (because the data flow is no longer sequential).
Ref: https://keras.io/examples/nlp/text_classification_with_transformer/ The residual part is hard to spot at first, as it is done as the input to the two layer norm calls.


Do I need validation data if my train and test accuracy/loss is consistent? I am trying to understand the purpose of a 3rd split in the form of a validation dataset. I am not necessarily talking about cross-validation here.

In the scenario below, it would appear that the model is overfit to the training dataset.

Train dataset {acc: 97%, loss: 0.07}
Test dataset {acc: 90%, loss: 8.02}

However, in this scenario it appears much more balanced.

Train dataset {acc: 95%, loss: 1.14}
Test dataset {acc: 93%, loss: 1.83}

Do I need validation data if my train and test accuracy/loss is consistent? Is the purpose of setting a validation split of 10% to ensure this kind of balance before evaluating the model on the training set? What does it prove?

You don't always need 3 separate datasets. You usually split a dataset into 3 if you are doing some parameter or hyperparameter tuning before choosing a final model. Tuning will usually add bias from the 2nd dataset into your model, decreasing it's performance.
For instance:

If you are manually tuning a model over several iterations and using
the results from the 2nd dataset to find the optimal parameters. By
doing so, you built some information from the 2nd dataset into your
model. This will make the 2nd dataset not a good, unbiased benchmark for your final
model. Therefore, you will want to use a 3rd untouched dataset, to
give you unbiased final performance measurements of your model

Some models use a validation dataset internally while building model
to evaluate loss, etc. This will cause the same problem with
including bias into the model. Example:
model.fit(
    train_features,
    train_labels,
    batch_size=20,
    epochs=20,
    validation_data=(val_features, val_labels), # <- here
    verbose=0)


How to handle associated features in machine learning I am working on a classification project in which some features are linked and I'm not sure how to handle them.

I will simplify my project like that :

There are different jobs, and multiple persons working on those jobs.
Those persons are fit for their job.
Replacing someone can only have negative effect or no effect.

I have a person1 that can be replaced by a person2 for a job, and the goal is to predict if it has a negative impact or not.

Each person has is own properties like : weight, age, height, IQ,..
A job also has properties like : manualJob, localization, temperature,..

When I list my features I have something like that :
(Person1 = P1, Person2 = P2, and the data has been normalized)

     P1_weight   P2_weight   P1_IQ    P2_IQ    manualJob   tempJob   neg_impact
0       0.25       0.50      0.25     0.25        1        0.25       1
1       0.75       0.25      0.50     0.25        0        0.50       0
2       0.50       0.75      0.75     0.50        1        0.25       1
...

There should be a high interaction between P1_weight && P2 weight features, (as well between IQs features) that we want to capture in order to predict the neg_impact feature.

1. Now the change between P1_weight and P2_weight is important, but does a classic model as RandomForest can capture the link between those 2 features ? Same for other subject properties (P1_IQ && P2_IQ, P1_height && P2_height,..)

2. I'm afraid that if I reduce the difference between P1_weight and P2_weight into a single feature as the diff (P1_weight - P2_weight), I will lose some information. For example the P1_weight would probably be correlated to the 'manualJob' feature, and if I delete P1_weight this information would be lost?

3. I was thinking that maybe I could preprocess those linked features with out-of-fold prediction, and use those predictions as input with the rest of the features original features (manualJob, tempJob, ..). Is it a good idea ? Which kind of model would be better for the preprocessing to capture the correlation of those linked features ?

NB : my set only contains ~1000 elements

The RandomForest algorithm may capture such interaction by splitting
an upper node by some condition on p1 and a lower node by another condition on p2, the link that you are expecting is captured if this happens under the same branch (p2 is on the path of p1). However, the ability to capture such information depends on the structure of the whole model (depth and number of trees) and the number of labels that you use.
It is a common practice to extract a third feature from coupled
features to explicitly express the linkage between them, for example, the ratio of change or the product. For not losing important information as you suggested (since a diff of 0.1 may be derived by subtracting 0.9-0.8 and 0.2-0.1) you may add to the model the third feature in addition to one of the original features.
I'm not sure that I understand the meaning of out-of-fold, but anyway note that using your labels twice (once for features preprocessing and once for training) can lead you to overfit the model since the labels leaked to the features in the preprocessing step. It is ok to explore your features in advance and in separate to the labels for understanding the links, subcategories, missing values, measuring the correlations and so.

2 last note:

there might be a confusion here between the meaning of correlation and interaction.
you can consider using linear statistical modeling for your needs and number of data points. these models enabling to capture interactions and with their output you can learn about the significance of the correlation and to interpret them with their features importance.
-- If you do so, you could also consider trying GLMM to see if there
are job effects, i.e there is a difference in the interaction between
different jobs.


time serie with only two values Could any one help me know about different approaches, methods or algorithms to build a model to forcast a time serie which has only two values ( 0, 1 ) but they last over time each time .
basically I've some on/off data, it tell me if there is an object in a place and how much time that object was in that specific place . So the output is a rectangular wave but it's not periodic.

so if we have the object in that place field we will get 1 every x seconds then when there is nothing we will get 0 .
Thanks in advance of your response!

First you should define your goal for the prediction. Do you want to predict also a time series output? Then the best thing to do is to use any machine learning model that is sequence focused (e.g. is LSTM), and with the time series that you have as input. Also study the time scale of your model. Study if it should be hour, minutes or any other scale.


Is what I did supervised or unsupervised Machine learning? My goal is to get a smartphone names from Twitter. So this is what I followed:

1- I extracted 100K tweets using the keyword “smartphone”.

2- I Applied LDA after applying ngram tokenization and cleaning.So, I got noisy results such as:(giveway, international, apple, iphone_6, samsung_s5,news…)

3- I filtered the results using a smartphone list (iphone_6, samsung_s5,iphone_4s,…) extracted from DBpedia in order to remove the noise.
Is what I did supervised or unsupervised Machine learning?

The basic idea behind supervised and unsupervised learning is:
In supervised learning, one possesses a training data set that includes the desired target output feature. Examples of popular supervised learning algorithms are linear regression, and support vector classification etc.
In unsupervised learning, we do not possess a training data set with the target output feature i.e. we do not know what our output or results look like. An example is clustering. There are really good explanations out there about this.

LDA is an unsupervised learning algorithm and the process you described can be classified as unsupervised learning. The filtering step that you describe does not make the algorithm supervised because the target smartphones have not been directly correlated to the training data and therefore is only serving as a guide to restrict the cardinality of the target variable.


Sort a data frame column based on another sorted column value in R I have a data frame that is sorted based on one column (numeric column) to assign the rank. If this column value is zero then arrange the data frame based on another character column for those rows which have zero as a value in a numeric column.

But to give rank I have to consider var2 that is the reason I sorted based on var2, if there is any identical values in var2 for those rows I have to consider var3 to give rank. Please see the data frame 2 and 3 rows, var2 values are identical in that case I have to consider var3 to give rank. In case var2 is zero I have to sort the var1 column (character column) in alphabetical order and give rank. If var2 is NA no rank. Please refer the data frame given below.

Below, the data frame is sorted based on var2 column descending order, but var2 contains zero also if var2 is zero I have to sort the data frame based on var1 for the rows which are having zero in var2. I need sort by var1 for those rows which are having var2 as zero and followed by NA in alphabetical order of var1.

    example:
    #      var1    var2    var3    rank
    # 1     c      556      45       1
    # 2     a      345      35       3
    # 3     f      345      64       2
    # 4     b      134      87       4
    # 5     z       0       34       5
    # 6     d       0       32       6
    # 7     c       0       12       7
    # 8     a       0       23       8
    # 9     e      NA
    # 10    b      NA

below is my code
df <- data.frame(var1=c("c","a","f","b","z","d", "c","a", "e", "b", "ad", "gf", "kg", "ts", "mp"), var2=c(134, NA,345, 200, 556,NA, 345, 200, 150, 0, 25,10,0,150,0), var3=c(65,'',45,34,68,'',73,12,35,23,34,56,56,78,123))

# To break the tie between var3 and var2
orderdf <- df[order(df$var2, df$var1, decreasing = TRUE), ]

#assigning rank
rankdf <- orderdf %>% mutate(rank = ifelse(is.na(var2),'', seq(1:nrow(orderdf))))

The expected output is sort the var1 in alphabetical order if var2 value is zero (for those rows with var2 value is zero).

    expected output:
    #      var1    var2    var3    rank
    # 1     c      556      45       1
    # 2     a      345      35       3
    # 3     f      345      64       2
    # 4     b      134      87       4
    # 5     a       0       34       5
    # 6     c       0       32       6
    # 7     d       0       12       7
    # 8     z       0       23       8
    # 9     b      NA
    # 10    e      NA

You can simply arrange by two columns:

library(dplyr)

df %>%
   arrange(desc(var2),var1)

Edit:

To clarify why this works, in your example simply arranging the df by var2 in descending order will already put all 0 and NA values at the bottom (since NA is "worth" less than 0). However since these  rows all have the same value their order will be "random" because we do not specify a tie method. By supplying a second column for arrange, we sort all ties by a second criteria, var1.

This will actually also re-arrange rows with a duplicate var2 value as well but that should be a bonus.


Handling a combined dataset of numerical and categorical features for Regression I have a dataset that has a large number of categorical features and a few numerical features, and I want to predict the probability that any given input is one of two types of a certain binary output feature. I don't want to solve a classifier problem for reasons that are well outlined in this link .

I have seen many tutorials about how to handle them independently but am less sure how to handle them together.

One could approach this in two general ways:

1) bottom up: thinking about unifying the data somehow to begin with

2) top down: deciding how the data needs to look based on the final model you wish to use

Do you already know which model you will use? If that is fixed (for whatever reason), you already know you need to get your data into the correct form, be it numerical or categorical.

As you pinned your question with the tag regression, I can tell you that you need to make your data all numerical, so regression can work.

An example of making numerical data categorical would be to put it into bins. Imagine we have values ranging from zero to ten: [0.173, 7.88, 3.91, ...]. You could simply say that values between 0.00 and 0.99 are category A, values between 1.00 and 1.99 are category B, and so on.

[Edit:]

A slightly more sophisticated way of defining the bins to use would be to define the bins based on some characteristic statistics of your dataset. For example, have a look at the possible ways possible implemented within python's Numpy. Of the available methods there, I have found the Doane method to work best - it will depend on your data though, so read the descriptions.

Making categorical values numerical in a meaningful way depends a little more on you data. It is easy to make them numberic, but you should focus on doing it in such a way as to retain as much of the information each variable contains as well as the relative relationships between each of the categories that you started with. E.g. converting colours into integers would allow you to perform regression, but if yellow becomes 1 and purple 10, the model needs to be able to learn that purple isn't necessarily 10 times bigger than yellow, and that is difficult in the context of regression!


Find the consecutive zeros in a DataFrame and do a conditional replacement I have a dataset like this:

Sample Dataframe

import pandas as pd

df = pd.DataFrame({
    'names': ['A','B','C','D','E','F','G','H','I','J','K','L'],
    'col1': [0, 1, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0],
    'col2': [0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0]})

I'd like to replace some of the 0's in col1 and col2 with 1's, but not replace the 0's if three or more 0's are consecutive in the same column.  How can this be done with pandas?

Original Dataset:

names   col1    col2
A   0   0
B   1   0
C   0   0
D   1   0
E   1   1
F   1   0
G   0   1
H   0   0
I   0   1
J   1   0
K   0   0
L   0   0

Desired Dataset:

names   col1    col2
A   1   0
B   1   0
C   1   0
D   1   0
E   1   1
F   1   1
G   0   1
H   0   1
I   0   1
J   1   0
K   1   0
L   1   0

Consider the following approach:

def f(col, threshold=3):
    mask = col.groupby((col != col.shift()).cumsum()).transform('count').lt(threshold)
    mask &= col.eq(0)
    col.update(col.loc[mask].replace(0,1))
    return col

In [79]: df.apply(f, threshold=3)
Out[79]:
       col1  col2
names
A         1     0
B         1     0
C         1     0
D         1     0
E         1     1
F         1     1
G         0     1
H         0     1
I         0     1
J         1     0
K         1     0
L         1     0

Step by step:

In [84]: col = df['col2']

In [85]: col
Out[85]:
names
A    0
B    0
C    0
D    0
E    1
F    0
G    1
H    0
I    1
J    0
K    0
L    0
Name: col2, dtype: int64

In [86]: (col != col.shift()).cumsum()
Out[86]:
names
A    1
B    1
C    1
D    1
E    2
F    3
G    4
H    5
I    6
J    7
K    7
L    7
Name: col2, dtype: int32

In [87]: col.groupby((col != col.shift()).cumsum()).transform('count')
Out[87]:
names
A    4
B    4
C    4
D    4
E    1
F    1
G    1
H    1
I    1
J    3
K    3
L    3
Name: col2, dtype: int64

In [88]: col.groupby((col != col.shift()).cumsum()).transform('count').lt(3)
Out[88]:
names
A    False
B    False
C    False
D    False
E     True
F     True
G     True
H     True
I     True
J    False
K    False
L    False
Name: col2, dtype: bool

In [89]: col.groupby((col != col.shift()).cumsum()).transform('count').lt(3) & col.eq(0)
Out[89]:
names
A    False
B    False
C    False
D    False
E    False
F     True
G    False
H     True
I    False
J    False
K    False
L    False
Name: col2, dtype: bool


Does NER work on large documents around 1500 - 3000 words or so? Let's say I have a resume and I have segmented the work section.
Usually work section of resume contains company name, designation, work period and job description. Now I have 1000's of resumes and I have manually annotated the work section of each resume with those 4 labels.
But the problem here is each work section is quite large around 1400- 3000 words?
Also job-description annotation is not one word like company name etc...  the entire responsibilities of the job is annotated as one entity.
So, will this work? please let me know what are the things I should consider for such NER entity extraction?
Any suggestions would really help. Thank you in advance.

The size of the document on which NER is run shouldn't be a problem at all, a standard NER system scans the document sequentially and just marks any entity it finds.
The size of the entities to find might be more of an issue, because typical NER systems rely on the previous few words to detect the boundaries of an entity. If the entity spans a large sequence of text, it's harder for the system to detect where it ends. In case it actually causes a problem with your data, it might be possible to specify exactly which features to use in the CRF model (this depends on the implementation I guess).


What are some resources to test your data science skills? I can't get a job to save my life so I am guessing my lack of skills is an issue. I've been doing a lot of reading on statistics and I am getting antsy - I want to move from theory to application and test my skills to see how I measure up with real practitioners.

Most books I have don't have solutions. They have questions and labs but without some semblance of an answer, I don't know if I am doing it right. I know when it comes to fitting models, there isn't a "right" answer, but there are definitely bad answers. I just want to make sure my answers are good enough.

What are some resources where I can practice my data science, statistical inference, machine learning skills - with answers? I know of https://www.kaggle.com/competitions has tasks but those seem beyond my ability right now - and obviously won't come with right answers.

Are there websites that test your ability to build a predictive model?

Any suggestions?

If you want to move from theory to application then I suggest to do exactly that: get your handy "dirty"!

UCI Machine Learning Repository has some easier datasets to get started. Kaggle is great too but before going for any competition look for an easier dataset from their repository.

If you prefer something with more guidance the book "Introduction to
Machine Learning with Python" by Mueller and Guido contains many examples starting with the classic Iris dataset.

Finally, you could consider an online course. The Udacity Nanodegrees include projects which you need to submit so it gives some basic hands-on experience. Though, in your specific case, I would only go this route if you think it might help your CV.


What are some good books on Machine Learning and AI like Krugman, Wells and Graddy's "Essentials of Economics" I am a Logistics student. I like the book "Essentials of Economics" by Krugman, Wells and Graddy in that it is concise, easygoing and not only a beginners book (it gradually approaches advanced subjects thus paving the way for further rigorous Economics course) so any one interested in Economics can learn it even if he/she never studied the subject before. Also, I am very interested in AI and Machine Learning and acknowledge their importance in this our postmodern era and I am self learning Real Analysis and web site development. What are some good introductory books on Machine Learning and AI like Krugman, Wells and Graddy's "Essentials of Economics"?

The two books that come into my mind are:

Artificial Intelligence: A Modern Approach
The Deep Learning Book

They both start from the basics and escalate while moving on.

Also thanks for your recommendation, I'll take a look at it because I want to jump to finance at some point in my career:)


When to use linear or logistic regression? Still I have no clear concept when I should choose linear OR Logistic regression.
 In addition, when I can say either linear or logistic regression performing better?

In particular, I want to know When should I choose Linear Regression and When to choose Logistic regression?

Linear Regression is used for predicting continuous variables.

Logistic Regression is used for predicting variables which has only limited values.

Let me quote a nice example which can help you make the difference between the both:

  For instance, if X contains the area in square feet of houses, and Y
  contains the corresponding sale price of those houses, you could use
  linear regression to predict selling price as a function of house
  size. While the possible selling price may not actually be any, there
  are so many possible values that a linear regression model would be
  chosen.

  If, instead, you wanted to predict, based on size, whether a house
  would sell for more than 200K, you would use logistic regression. The
  possible outputs are either Yes, the house will sell for more than
  $200K, or No, the house will not.


Is Elastic Search recommended if attribute getting search is not a huge text document? We are currently developing a system with MEAN stack with Mongodb at backend. We have employees name, and Ids in our system and our client wants to get pretty good (Read: Google Like) search in our system to search for employees' records. He needs our system to recommend employees even if he has misspelled the name, etc.

One of the suggestions from our development lead was that we should use elastic search but from what I have seen, elastic search is preferred especially in scenarios where we are searching in a large text documents. I am having a feeling that using elastic search in our case will only increase the overhead as the attributes getting search in our case just names, emails or ids, etc.

Currently, our system only uses some basic regrex but I believe, we can improve our search by putting in some effort and without moving towards elastic search (though the development lead feels that we will only be Reinventing the wheel and we should use technologies that do this stuff for us.)

So, I wanted some guidance (Suggestions) on whether to move towards elastic search or should we stick with our current resources to work on a more optimized search than what we have right now?

As you mentioned you are searching  just names, emails or ids, etc which is not large text.

So consider a case where you have 6 documents/records having names as below then you can understand better whether how much the large text could matter.

Rohit kumar Bhatnagar
Shilpa Shinde
Manoj kumar
Rohit Bhatnagar Kandaswamy
Rohit kumar shindey Bhatnagar
Rohit Bhatnagar

If user comes and perform a search for Rohit Bhatnagar, then using the regex you will be showing results in two ways:

Case I: Where regex is strict match

Rohit Bhatnagar Kandaswamy
Rohit Bhatnagar

Case II: When regex is relax

Rohit kumar Bhatnagar
Rohit Bhatnagar Kandaswamy
Rohit kumar shindey Bhatnagar
Rohit Bhatnagar

If we examine case Ist, you are missing two things relaxation(Record 1 & 5 will be missed) and in ranking exact match should come on top that might be more relevant.

In case 2, we relaxed but still exact match is below in ranking

So if need a relevant search then yes you can use a search engine. You could also tweak like whether you need the record 5 or not because if you see that might looks irrelevant so you can control like how many words should be in between to consider a document match. If we say consider 1 word in between then record 5 will be eliminated from results.

Other than search relevance, you can horizontal scale your search if QPS is high. You could also use machine learning techniques(learn to rank), also you could apply synonyms, stemming etc. There are still many other benefits join, streaming, sharding, etc which you could learn from the documentation

Solr: http://lucene.apache.org/solr/guide/7_6/

ES: https://www.elastic.co/guide/en/elasticsearch/reference/6.4/index.html

If above mentioned is required or in near Future then you can use use ElasticSearch or Solr as search engine otherwise you should be good with the current system.


How to deal with evaluations in a multi step ML pipeline Apache Spark provides us the pipeline to construct a ML pipeline. Moreover, we can evaluate the regressors in the pipeline, now here is the point I dont understand: how to apply to testing data.

Ok, lets say I have one dataframe and one regressor, here what I do:

Split the data.
Apply the regressor to the training data.
Apply the model to testing data to get predictions.
Evaluate the prediction using an evaluator (e.g., binary classifier).

Now, lets say I have 3 steps and the last one is the regressors. I can not do the above because the first two also need to be applied to the testing data as well, how to deal with this? How my steps will look like?

A pyspark.ml.PipelineModel is the result of calling the .fit() method of a pyspark.ml.Pipeline. This estimator is a sequence of Transformers and/or Estimators, all of which have a .transform() method. When you call the .transform() method of the pyspark.ml.PipelineModel-- say, when you want to make predictions using the trained model-- the .transform() method of each of the objects in the PipelineModel are called in sequence.

In other words, the transformations for each step are applied by calling the .transform() method of the fitted pipeline- the API does this for you, or more accurately, gives you concise tools to do this as you need to.

What you're talking about is slightly outside the scope of this type of object, however. Train/test splitting occurs outside the Pipeline, and importantly so, for methodological reasons.

A PipelineModel couples data preparation steps with an estimator, but not training operations. Splitting the data, for example, is something you'd have to do outside the Pipeline.

The workflow might go:

Split the data into training and test datasets.

Define your data preparation steps and regression estimator, and combine these into a Pipeline.

Train the entire pipeline on your training data:

fit_pipeline = Pipeline.fit(train)

Make predictions on the training set:

train_preds = fit_pipeline.transform(train)

Use something like a pyspark.ml.evaluation.RegressionEvaluator to evaluate the performance on the training dataset:

evaluator = RegressionEvaluator()
evaluator.evaluate(train_preds)

Then, you can make predictions on the test dataset:

test_preds = fit_pipeline.transform(test)

And evaluate them:

evaluator.evaluate(train_preds)

But you have this completed model and data preparation object that can be used to make predictions on similarly structured data now, called fit_pipeline, which is an instance of pyspark.ml.PipelineModel.


Can the size of a pooling layer be learned? As far as I understood it, the pooling layer doesn't learn anything. It has several parameters, most important its pool_size and stride (see Lasagne documentation for more), but none of those is learned.

Is it possible to learn these two (or one of those) parameters? Are there papers about it?

(I would guess that it is not possible to add this to the objective function in a meaningful way... but I'd like to see if people thought about it.)

No it is not possible to learn those meta parameters from a data set using a learning algorithm. There is no way to calculate a gradient of any objective function with respect to pool_size and stride params. Even if there were, they are typically discrete values and relatively small integers, so cannot be updated in the same way as e.g. the weights are.

What you can do is search through different values looking for optimal ones, using cross-validation to pick the best. The size and stride meta parameters can also have a large effect on memory use and speed of learning, so it may be the case you are looking to find a good compromise. With stride in lower layers for instance, it may be more effective (in terms of training time required for similar level of accuracy) to have a higher stride value but increase the overall training set size by adding random x/y translations to the training set images.


What is the Time Complexity of Linear Regression? I am working with linear regression and I would like to know the Time complexity in big-O notation. The cost function of linear regression without an optimisation algorithm (such as Gradient descent) needs to be computed over iterations of the weight combinations (as a brute force approach). This makes computation time dependent on the number of weights and obviously on the number of training data.

If $n$ is the number of training data, $W$ is the number of weights and each resolution of the weight space is set to $m$ meaning that each weight will iterate through $m$ number of possible values. Then the time complexity of this linear regression is

$O(m^Wn)$

Is this correct?

It highly depends on the "solver" you are using.

Calling $n$ the number of observations and $p$ the number of weights, the overall complexity should be $n^2p+p^3$.

Indeed, when performing a linear regression you are doing matrices multiplication whose complexity is $n^2p$ (when evaluating $X'X$) and inverting the resulting matrix. It is now a square matrix with $p$ rows, the complexity for matrix inversion usually is $p^3$ (though it can be lowered).

Hence a theoretical complexity : $n^2p+p^3$.

Side notes

However, numerical simulations (using python's scikit library) seem to have a time complexity close to $n^{0.72} p^{1.3}$

This may be due to the fact that no implementation actually perform the full inversion (instead, the system can be solved using gradient descents) or due to the fact that there are other ways to calibrate the weights of a linear regression.

Source

An article from my blog : computational complexity of machine learning algorithms


What's the best strategy to train a CNN with images that only have labels for positive characteristics? I have a large database of images that are only partially labeled for multiple, non-exclusive characteristics or objects present on them. For instance, an underwater scene might feature the labels water, swimsuit and fish on it.

Problem is it's only partially labeled, meaning that the fact the label cat is absent doesn't mean there's no cat on the image.

What'd be the best strategy to train a CNN able to tell the missing labels on the dataset?

The image base has 230 thousand images on it, but given that multiple labels are possible, it's hard to tell the ratio of images that are fully labeled. I'm able to code python and managed to use keras to train on mnist dataset on gpu.

I think your best bet would be transfer learning. Start with a model that has already been trained with a wider dataset such as the ones presented here. From there you can train the model with your specific dataset. You can then use output nodes for the labels which you have available to you, and you can get the predictions for the other images from the pre-trained model which are usually trained for thousands of different classes.

Alternatively, you can train a model with all the output classes you have in your label set and another output for "other". Then when an output node is selected you can pass that same input to the pre-trained model which was trained with your data as well.


Categorical Variable Reduction using NN I was trying to categorical variable engineering following this paper. The code is the following:

import random
import pandas
import numpy as np
import tensorflow as tf

from tensorflow.contrib import layers
from tensorflow.contrib import learn
from __future__ import print_function

from sklearn.preprocessing import LabelEncoder

My dataset looks like the following. It's has 2 independent variable ('X1' & 'X2')and 1 dependent variable ('lable'). 'X2' is the categorical variable. I want to create an embedding vector for this variable and run the simple linear regression to predict 'label'using Tensorflow.  I could use any other method. But since linear regression is easiest to understand, I'm trying that.

df = pd.DataFrame({'X1': np.array(["A","A","B","C","B","C","B","C","C","B",
                         "A","B","A","C","A","A","C"]),'X2': np.array([3.3,4.4,5.5,6.71,6.93,4.168,9.779,6.182,7.59,2.167,
                         7.042,10.791,5.313,7.997,5.654,9.27,3.1]),
                       'label': np.array([1.7,2.76,2.09,3.19,1.694,1.573,3.366,2.596,2.53,1.221,
                         2.827,3.465,1.65,2.904,2.42,2.94,1.3])})

For variable 'X1', I'm creating levels.

encoder = LabelEncoder()
encoder.fit(df.X1.values)
X = encoder.transform(df.X1.values)

Recreating dependent variable list.

y = np.asarray([1.7,2.76,2.09,3.19,1.694,1.573,3.366,2.596,2.53,1.221,
                         2.827,3.465,1.65,2.904,2.42,2.94,1.3])

Setting Hyper-parameters

training_epochs = 5
learning_rate = 1e-3
cardinality = len(np.unique(X))
embedding_size = 2
input_X_size = 1
n_hidden = 10

Setting up variables:

embeddings = tf.Variable(tf.random_uniform([cardinality, embedding_size], -1.0, 1.0))

h = tf.Variable(tf.truncated_normal((embedding_size + len(df.X1), n_hidden), stddev=0.1))

W_out = tf.get_variable(name='out_w', shape=[n_hidden],
                            initializer=tf.contrib.layers.xavier_initializer())

Embedding:

embedded_chars = tf.nn.embedding_lookup(embeddings, x)
embedded_chars = tf.reshape(embedded_chars, [-1])
embedded_chars= embedded_chars + np.array([3.3,4.4,5.5,6.71,6.93,4.168,9.779,6.182,7.59,2.167,
                         7.042,10.791,5.313,7.997,5.654,9.27,3.1])

Multiplying with Hidden Layers:

layer_1 = tf.matmul(embedded_chars,h)
layer_1 = tf.nn.relu(layer_1)
out_layer = tf.matmul(layer_1, W_out)

# Define loss and optimizer

cost = tf.reduce_sum(tf.pow(out_layer-y, 2))/(2*n_samples)
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

Run the graph

init = tf.global_variables_initializer()

# Launch the graph
with tf.Session() as sess:
    sess.run(init)

    for epoch in range(training_epochs):
        avg_cost = 0.

        _, c = sess.run([optimizer, cost],
                        feed_dict={x: X, y: Y})
print("Ran without Error")

While running the code, I'm getting the following error.

  ValueError: Shape must be rank 2 but is rank 1 for 'MatMul_1' (op:
  'MatMul') with input shapes: [17], [19,10].

I'm not able to add the continuous variable with embedding variable.

Can anyone please guide me how to do it?

Thank you!

You are multiplying matrices, so the dimensions have to match up.

The following line reshapes the embedding layer to a rank 1 matrix, hence causing the error.

embedded_chars = tf.reshape(embedded_chars, [-1])

You may want to rethink the embedding portion of the code (the function's docstring), as that is what needs to be understood perfectly in order to define the computational graph.


PCA algorithm problems - Python I have implemented PCA algorithm and I understood it very well but still I have some questions. My code is below and it's very simple implementation.

import numpy as np

x = np.loadtxt('CCPP', delimiter=',')
row, column = x.shape

# Mean normalization
for i in range(column):
    x[:,i] = (x[:,i] - x[:,i].mean()) / (x[:,i].max() - x[:,i].min())

sigma = x.transpose().dot(x) / row
u, s, v = np.linalg.svd(sigma, 0)
z = x.dot(u[:,:3]) ## new features

new_x = z.dot(u[:,:3].transpose()) ##reconstruction

First Question

As you can see above my sigma variable is

  x.transpose().dot(x) / row

It's giving to me an nxn matrix (n is number of features). But sigma's formula is $$\Sigma = \frac{1}{n} \sum_{i=1}^n x^{(i)} {x^{(i)}}^T$$

Why there is a summation symbol in formula? I mean, if I use this sigma formulation then sigma is going to be a number, not a matrix. I have to get nxn matrix, right? So is my sigma implementation correct? or am I missing something about the formula?

Second Question

When we are reconstructing X (at the bottom in the code), should new_x equal to my first X? I mean, I reduced dimension of the data set, then I reconstructed it, original dataset and reconstructed dataset must be the same, right? This is my second question.

Third Question

This one is easy. Should I use data compression for each of my dataset which has 1000, 100000 or more features? I mean, can I always use it? Is it a good choice to use it every time?

First question: computing $\Sigma$

Actually, you perform the same computation but using a matrix operation instead of a scalar operation. You might be mislead by your notation of the feature matrix $X$ that you write $x$ because in fact
$$X = (x_j^{(i)})_{i,j} =\Big(x^{(1)} ... x^{(i)} ... x^{(n)}\Big)^T$$
that is to say, $i$-th row of the matrix $X$ contains $x^{(i)}$ the $i$-th sample.

So, when you want to compute the empirical covariance matrix $\Sigma$ (which is indeed a $n \times n$ matrix), you have:
$$\Sigma = \frac{1}{n} \sum_{i=1}^n x^{(i)} {x^{(i)}}^T = \frac{1}{n} X^T X$$
you can check that this is exactly the same computation.

In your code, you are actually computing directly $\Sigma$ using the matrix operation (i.e. $\Sigma = \frac{1}{n} X^T X$) and this does give you a $n \times n$ matrix. So, both your implementation and the formula is correct.

Second question: reconstructing $X$

Your new feature matrix $Z$ is a $n \times 3$ matrix according to your code. In fact, as your code does not show the original size of the feature space, we will see the two cases here:

$X$ is also a $n \times 3$ matrix, then you do not perform any kind of dimension reduction and you should have $X_{new} = X$ (at least in theory, in practice you might have a very small numerical approximation error but it will basically be the same)
$X$ is a $n \times d$ matrix with $d > 3$, then you do perform a dimension reduction and in this case you get rid of some information contained in the original data and in this case $X_{new} \neq X$

In your case, I guess that you do have $d > 3$ and so you're in the second situation.

Third question: when to apply PCA

Well that depends in fact...

First of all PCA performs an SVD which can become very expensive if you have a lot of features. In fact, there are several way to compute PCA, your way is the closer to the mathematical theory but in practice computing an SVD directly on $X$ avoids computing $X^T X$ which is expensive and allows to retrieve the same space. But in any case, in some situation, it can be expensive and time consuming and thus not practical.

As PCA sorts the new feature space according to the value of the eigenvalues, i.e. of the variance of the new directions, removing the last direction might in some cases remove some noise which can help. However, at the same time, it can throw away valuable discrimative information, this is why other methods such as LDA can be interesting.

So to summarize, the answer to the question is no, it is not a good choice to use it everytime, it depends of your problem.


When training NN, how do data loaders work on large datasets? How do you normally organize large data-sets for easy loading when training Neural Networks? I have a largeish data-set which cannot fit into memory, it consists of 200,000 samples, with 10k samples being stored in grouped files. In total this is roughly 50GB of data.

I can separate the grouping to produce 200,000 individual files but I'm not sure if this is the correct course of action as the system will need to do many calls to the file system when training.

How do systems which train massive data-sets work? (For example image-net).

For large data sets the limitations of available memory prevents you from loading all the data simultaneously. What is typically done is to provide your data to the network in batches. Batches are simply groupings of your data. The batch size maximum value is limited by available memory. If you use Keras the ImageDataGenerator.flow from directory provides a convenient way to present your data to the network in batches. Documentation is here. There is also a benefit to presenting your data in batches. For example if your data consists of 50,000 samples and it fits into memory you can provide the 50,000 samples to the network and after the data is forward passed do 1 step of back propagation to adjust the weights. A better solution is to partition your data into batches, say 10 batches of 5,000 samples. In this case if the network is enabled to train on batches you can get 10 steps of iteration of the weights versus the single iteration. This enables the network to converge faster at a lower computational cost.


TypeError: Expected binary or unicode string, got array I am trying to convert an array containing images into a tensor, but I'm getting error "TypeError: Expected binary or unicode string, got array"

images.shape = (10,)

images[0].shape = (375,500,3) and
images[1].shape = (400,500,3).. i.e. the images[i] have various shapes.

imgs = tf.convert_to_tensor(images)

I'm not sure how to solve this, please help.

Try this:

imgs = tf.convert_to_tensor(images[0])

If all of your images are of the same shape. Otherwise you have to resize them to a same dimension then call the code.


is there any way to plot ROC curves from weka i am using some algorithms from weka . i was willing to plot some algorithms' roc curve for comparison . Is it possible and how ?

In the Weka explorer, go to the classify tab and train/test your algorithm.
The result buffer appears in the bottom left box under the section labeled "result list"
Right click the result buffer and click visualize threshold curve, then select the class you want to analyze
to save the ROC curve as an image, hold shift + alt and left click on the graph


How to model anomaly data using Gaussian distribution assuming variables are dependent? (In Python) I have some data which contains anomalies as well.

I want to model data using Gaussian distribution assuming variables are dependent in Python.

How can I model this? Should I use the PDF formula as a function, figure out the probabilities and take out the accuracy?

Note: This has been given as an assignment in my college. I am trying to figure out the solution.

If you assume variables are dependent, in a Gaussian distribution this means the covariance matrix is not diagonal.


Python giving memory error while fitting the text into vector using various vectorizer While training the IMDB movie review dataset for sentiment analysis, the model will give a memory error if I set the features to more than 20,000. Is there any way to pass the memory error?

to answer that question you need to get and give us more information, no chance to advise without knowing where the OOM is thrown and how the memory profile looks like. You may tune the memory usage of your program also in other parts than the part where the error came from, to raise the limit.

Just one hint for a cause that can be resolved without getting more physical memory:
Sometimes you can "lazy load" large data sources. Get it record by record or chunk by chunk, then process it, and write results to the sink, before you get the next record into memory. This is done with python generators, watch out for the keyword "yield" that is used instead of return.
https://stackoverflow.com/questions/1756096/understanding-generators-in-python

further you may be better off using low level libraries for certain tasks, see i.e. http://www.nltk.org/book/ch06.html  2.4