Recommendations and Missing Data in Deep Learning In this research paper, it is discussed how to combine deep learning with wide (shallow) learning to achieve both generalisation and the ability to learn correlation/association rules.

The input vector of such network is an n-dimensional vector of features $\mathbf{x} = (x_1, x_2, x_3,\dots, x_n)$. While training, the following objective function is maximised:
$$  P(Y=1|\mathbf{x})   $$

Within the context of recommender systems (which is one of the main applications of this paper) how does one cope with the problem of missing data (unseen products)?

This depends on what you mean by 'unseen products'. If your definition is like in the paper

[...] new  feature  combinations  that have  never  or  rarely
occurred  in the  past.

then you're thinking along the lines of algorithms like collaborative filtering. Those see the tastes of people as missing values in a matrix and try to complete it using a low-dimensional representation. Now these can be overly generic, depending on how low you choose that dimension to be.

Another, older, approach is something like association rules, which can be deducted using algorithms like apriori, which however rely on the product combinations seen in the past, so they won't ever be able recommend product combinations not seen in the past.

While the former set of algorithms can deal with what under the definition above would be called 'missing data', the latter cannot. Those however tend to give better recommendations. The paper you quote uses neural networks in what looks like a successful attempt to get the best of those two worlds.

If hover you mean by 'missing data' completely new products, than you're out of luck using any of the algorithms described above. If you don't have meta-data about those products, it's near impossible to recommend anything. Just imagine you have a list of products A, B, and C and who bought them. Now you get product D. Who will buy it? Without any additional information on D, you're dead in the water.


Scaling of variables considering the values of a single column or the whole dataset I read many time that for machine learning and data mining algorithms the multi-dimensional input data should be scaled (e.g. normalized or standardized). Now my question is whether the average, min or max value shall be calculated for each row (variable) or for the dataset as a whole.
Let's say we have 3 variables
X=[x_1, x_2,.., x_n]
Y=[y_1, y_2,...,y_n]
Z= [z_1, z_2,...,z_n]
I use the following code in python
def standardize (data, train_split):
    data_mean = data[:train_split].mean(axis=0)
    data_std = data[:train_split].std(axis=0)
    return (data - data_mean)/ data_std

def normalize(data, train_split):
    data_max= data[:train_split].max(axis=0)
    data_min = data[:train_split].min(axis=0)
    return (data - data_min) / (data_max -  data_min)

If I saw it correctly, the code (that I copied from a tutorial) calculated the min,max, average and standarddeviation for the whole dataset. Meaning that the maxium value is the maxium of X,Y and Z. Is this correct to do or shall one only normalize and standardize a variable by considering its own values? So the min, max etc. should be calculated for every variable.
I'd appreciate every comment.

Both can make sense depending on your data :

Example 1 : if you have a feature that varies between -100 and -50 on the whole dataset you would want to normalize that column based on the values on the whole dataset
Example 2 : if each sample is a list of values from different sensors and you only care about their relative values (for anomaly detection for instance) then you might want to normalize each sample

That being said, I think 9 times out of 10 you will want to normalize based on the statistics of the whole dataframe.
Note also that normalizing each sample based on the statistics of the whole dataset is an operation that can be reversed using the normalized data and the dataset statistics and therefore no information is lost : your data is basically the same but rescaled. A neural network could very well learn to reverse this operation.
Normalizing each sample based on this sample statistics can also be useful but this operation actually transforms the representation of your data. As an example, let's say you normalize each sample to a norm of 1, you can't recover the initial data without knowing the initial norm of that specific sample.


Does cross_validate in scikit-learn automatically fits and train the model? # from the titanic dataset
X = df.drop(columns="survived")
y = df.survived
scoring = ['accuracy','precision','roc_auc','f1',]
from sklearn.model_selection import cross_validate
from sklearn.linear_model import (LogisticRegression)
def model_LR(): #logstic Regression
    index = ["kfold-1","kfold-2","kfold-3","kfold-4","kfold-5"]
    s = model_selection.cross_validate(LogisticRegression(), X, y, scoring = scoring, cv = 5 )
    s = pd.DataFrame(data = s, index = index)
    display (s)
    print ("The mean scores for the above:\n", s.mean())

model_LR()

# OUTPUT :
    fit_time    score_time  test_accuracy   test_precision  test_roc_auc    test_f1
kfold-1 0.003998    0.006969    0.774809    0.711340    0.823673    0.700508
kfold-2 0.003990    0.005005    0.820611    0.778947    0.856481    0.758974
kfold-3 0.003003    0.003989    0.774809    0.715789    0.796667    0.697436
kfold-4 0.002992    0.003992    0.767176    0.709677    0.841852    0.683938
kfold-5 0.001994    0.003989    0.819923    0.819277    0.877081    0.743169
The mean scores for the above:
 fit_time          0.003195
score_time        0.004789
test_accuracy     0.791466
test_precision    0.747006
test_roc_auc      0.839151
test_f1           0.716805
dtype: float64

My Question:
In my code above, by calling on the cross_validate with LogisticsRegression, I get scores (such as roc_auc) as if the model has been fitted and trained.
I did not use any fit or train function.
Does that mean that cross_validate does that automatically?
Thx.

Yes, you do not need to fit or train explicitly when using cross_validate(). Also see this example from the SKLearn documentation:

from sklearn import datasets, linear_model
from sklearn.model_selection import cross_validate
from sklearn.metrics import make_scorer
from sklearn.metrics import confusion_matrix
from sklearn.svm import LinearSVC
diabetes = datasets.load_diabetes()
X = diabetes.data[:150]
y = diabetes.target[:150]
lasso = linear_model.Lasso()
cv_results = cross_validate(lasso, X, y, cv=3)

Just as in your case you can already get the results with no further steps, e.g.

cv_results['test_score']

gives this:

OUT: array([0.33150734, 0.08022311, 0.03531764])


How can I train a model for localizing objects(classification not required) in Python I need to make a model that creates bounding box around objects(but does not classify them) for a competition. Which libraries or pre-trained models should I use. I need values of x1,x2(x1+w),y1,y2(y1+h), where all dimensions are measured from top left corner. The accuracy is measured by IOU(Intersection over Union).

The mainstream algorithms used today for finding the bounding boxes of objects in an image includes:

R-CNN (and related Fast R-CNN and Faster R-CNN). This family of the algorithms actual does the localization (finding the bounding boxes) and classification in two separate steps. It finds the bounding box then does the classification. So if you don't need classification, just skip the second step.
YOLO stands for 'You only look once'. The name comes from the fact this algorithm actually does the localisation and classification in one go.

Both algorithms can be implemented in the mainstream python deep learning library like tensor-flow or pytorch etc


Successful ETL Automation: Libraries, Review papers, Use Cases I'm curious if anyone can point to some successful extract, transform, load (ETL) automation libraries, papers, or use cases for somewhat inhomogenious data?

I would be interested to see any existing libraries dealing with scalable ETL solutions.  Ideally these would be capable of ingesting 1-5 petabytes of data containing 50 billion records from 100 inhomogenious data sets in tens or hundreds of hours running on 4196 cores (256 I2.8xlarge AWS machines). I really do mean ideally, as I would be interested to hear about a system with 10% of this functionality to help reduce our team's ETL load.

Otherwise, I would be interested to see any books or review articles on the subject or high quality research papers. I have done a literature review and have only found lower quality conference proceedings with dubious claims.

I've seen a few commercial products advertised, but again, these make dubious claims without much evidence of their efficacy.

The datasets are rectangular and can take the form of fixed width files, CSV, TSV, and PSV. Number of fields range from 6 to 150 and contain mostly text based information about entities.  Cardinality is large for individual information (address), but smaller for specific details like car type (van, suv, sedan).

Mappings from abbreviated data to human readable formats is commonly needed, as is transformation of records to first-normal-form.

As is likely obvious to the cognoscenti, I am looking for techniques that move beyond deterministic methods, using some sort of semi-supervised or supervised learning model.

I know this is a tall order, but I'm curious to assess the state-of-the-art before embarking on some ETL automation tasks to help guide how far to set our sights.

Thanks for your help!

I don't think you'll find anything that checks all of your requirements, but here are some things to look at:

Automated ETL mapping: There is a tool called Karma started by a team at USC's Information Sciences Institute. It learns from your ETL mappings and helps automate future mappings. It's the only open source tool I'm aware of that helps automate the ETL process, but I would be very interested if there are others out there.
Large scale ETL: There are many many tools you could look at for the scalability you are looking for. I can personally recommend looking at Storm and Spark. Storm is excellent for stringing a connection of processing steps together that, given enough resources, can compute in near real time on streaming data. Not too dissimilar Spark has a streaming component with a similar use case, but standard Spark may fit your needs better if the data you are needing to ETL is a fixed set to be processed once.
Data storage: You may also need to consider where all this data will live during the ETL lifetime. You may need something like Kafka to deal with large streams of data. Or maybe HDFS to store a static collection of files.


CNN model low accuracy I have 1299 images in 4 classes (374/269/284/372). I want to use the VGG19 model, add a dense layer at the top and fine-tune it with my images. As I only have 1299 images, I also want to use data augmentation.
Here is the code (it is not exactly the code I used everytime, as I tried several things. When I did not use preprocess_input, I did X = X/255.) :
drive.mount('/content/gdrive')

DATADIR = '/content/gdrive/My Drive/'
CATEGORIES = ['A','B','C','D']
training_data = []

def create_training_data():
    for category in CATEGORIES:
        path = os.path.join(DATADIR,category)
        class_num = CATEGORIES.index(category)
        for img in tqdm(os.listdir(path)):
            img_array = cv2.imread(os.path.join(path,img))
            img_array_RGB = cv2.cvtColor(img_array, cv2.COLOR_BGR2RGB)
            new_array = cv2.resize(img_array_RGB, (224,224))
            training_data.append([new_array,class_num])

create_training_data()

X = []
Y = []

for features, label, in training_data:
    X.append(features)
    Y.append(label)

X = np.array(X)
Y = np.array(Y)

IMG_SIZE=224
X = np.array(X).reshape(-1, IMG_SIZE, IMG_SIZE, 3)

Y_class = Y
Y = to_categorical(Y, num_classes=4)

X_train, X_val, Y_train, Y_val = train_test_split(X, Y, test_size = 0.3, random_state=0, stratify=Y)

Y_train_int, Y_val_int = [np.where(r==1)[0][0] for r in Y_train], [np.where(r==1)[0][0] for r in Y_val]

from keras.preprocessing.image import ImageDataGenerator
datagen = ImageDataGenerator(
        horizontal_flip=True,
        vertical_flip=True,
        brightness_range=[0.2,0.6],
        fill_mode='wrap')

datagen.fit(X_train)

vgg_model = VGG19(weights='imagenet', include_top=False)

X = preprocess_input(X)

x = vgg_model.output
x = GlobalAveragePooling2D()(x)

x = Dense(512, activation='relu')(x)
x = Dropout(0.3)(x)

predictions = Dense(4, activation='softmax')(x)

model = Model(inputs=vgg_model.input, outputs=predictions)

for layer in vgg_model.layers:
    layer.trainable = False

layer_num = len(model.layers)
for layer in model.layers[:21]:
    layer.trainable = False

for layer in model.layers[21:]:
    layer.trainable = True

#class_weights = class_weight.compute_class_weight('balanced',
#                                                 np.unique(Y_train_int),
#                                                 Y_train_int)

model.compile(optimizer='Adam', loss='categorical_crossentropy', metrics=['accuracy'])

history = model.fit(datagen.flow(X_train,Y_train, batch_size=64), epochs=30, batch_size=128, shuffle=True, validation_data = (X_val,Y_val))

However this does not produce relevant results.
I tried several modifications (changing the learning rate, batch size, loss as sparse_categorical_crossentropy by changing from one hot encoding to integer, adding weights to classes, and I also tried to build a CNN from scratch with keras, but nothing seems to give better results (predicting the class with the most images)
Here is an example of plots : https://i.stack.imgur.com/KODAr.png
I don't really know what else I could try, but I really think a good model could classify these images better than this.
Edit :
I changed some parameters :

Data augmentation (I deleted the brightness)
batch_size = 32 for the data augmentation and 64 for the model
added weights to the classes
added x = Dense(256, activation='relu')(x) after x = Dense(512, activation='relu')(x)

And now the learning curve and confusion matrix look like this (only 10 epochs) : https://i.stack.imgur.com/IsJTh.png
The learning curve is really fluctuating and the model doesn't seem to learn, but the accuracy is higher than before.
I don't know what else to do now.

So, the question is asking for advice on what to do next, given the model's performance.
Firstly, in terms of confusion matrices, it is often useful to display proportions, rather than number of examples. It makes it easier for us to gauge where the error is occurring.
Looking at the most recent confusion matrix, it is clear that a lot of the examples are classified into classes 2 and 3. This could be due to a couple of things, potentially class imbalance, especially since the class weights is commented out in the code. Also, looking that the loss over epochs, it looks like the loss could continue to decline if you gave the model more epochs (unless you put a callback on it to prevent it from continuing training, which doesn't look to be the case).
Try that out and edit the post accordingly with the updated results!


Big data analytics references I'm looking for a good introductory book or course to big data analytics.
For the practical part, I'm particularly interested into using big data tools in R.

I would prefer a book, but other references are welcome.

Thanks!

I tried to explore some of the best available resources, which includes online courses (Free/Paid), Books etc.
Books

Big Data and Analytics (WIND)
Hadoop for Dummies
Big Data for Dummies
Hadoop: The Definitive Guide
Learning Spark: Lightning-Fast Big Data Analysis
MapReduce Design Patterns

Online Courses
These are some best platforms that provide lots of courses with rich content and hands-on labs. You can go from beginner to expert level, followed by intermediate. (courses will be - Free/Paid  )

Udemy - Big Data Analytics Courses
Coursera - Big Data Analytics Courses
edX - Big Data Analytics Courses
Udacity - Data Analysis

Other References
For more thoughts on that you might like to explore these websites-

Quora
Big Data - Made Simple
From Dev
KDnuggets

Update [BigData in R]
You can go through these references-

Wikipedia - Programming with Big Data in R
RStudio - Working with BigData in R
InfoWorld - Learn To crunch BigData with R

That's all from my side. Hope it helps. Cheers!


Finding groups of friends in social network data I have some social network data in which I'd like to identify people who belong to a group of 5 friends.  These are users who are friends with 4 other people who all are also friends with each other.

In the graph sense, this task involves identifying components of the graph where each node has an edge between every other node in the component and where the component contains at least 5 nodes.

Is there a well known algorithm to do this?  I was looking at measures like centrality, clustering coefficients, and strongly connected components but none of those are quite what I want.

Thanks!

Community Detection and Clique Percolation:

This is a community detection problem.  Here is a very detailed review article surveying the state of the art.

The Clique Percolation Method is also useful to explore as it pretty much solves what you may need to know.  You can also go through what Matching is, and link the concept with Blossom Algorithm. Though, this finds the maximum matching, you may slightly modify the algorithm to get the minimum matchings of 5 at least.


How to use a RBF kernel to create a "Kernel Space" using the similarity of each pair of point? I am trying to use Semi-Unsupervised clustering using reinforcement learning following this paper.

Assume I have n data-points each of which has d dimensions. I also have c pairwise constraints of whether two elements are supposed to be in the same cluster or not.

The paper states that "the original input dimension of the dataset is
appended to a kernel space with a similarity metric to each
pairwise point in the set of constraints" creating a d + 2c dimensional space. They also say the kernel they use is an RBF.

Can anybody explain to me what do they mean? how can you use the similarity of a pair of elements to generate a whole new dimension for every data point in the dataset? what is a "kernel space" exactly in this case? if I try to google it I only get results relevant to operating systems kernels.

Let's take a simple example, a binary classification using only 2 dimensions with only 8 observations. Consider the first 8 rows of the dataset. It is impossible to linearly separate the data using a hyper-plane.

So, we can use a kernel transform. The books usually drop you off right here(no help). It does not tell us how to transform. Transform using what? How many dimensions? What constraints?

We could consider the simplest transformation, f(x,y)=xy. We could take this transformation and now consider it a new dimension. So we started with (D2+Class) features and we could add 1 dim (to get D3+Class) if we chose.

Q. Does this help?  I suggest either plotting this dataset out by hand. Using f(x,y)=xy is Not very helpful.

So let's try another, f'(x,y)=x^2 + y^2. We started with (D2+Class) features and added another dimension. We could conceivably plot D4+C, BUT using dimensions 1,2 & 4 are easier to visualize. I suggest plotting by hand (for effect) {d1,d2,d4} to your graphic or plot it using 3D software.

Now ask yourself, Is this new situation linearly separable?

As for the constraints, Do you remember LaGrange multipliers? Well if you want to use a gaussian like constraint we could use a Radial Basis Function (RBF).
Where:

RBF: K(x,y) = exp(-gamma * (||x−y||)^2)), gamma > 0

The first part is kernel 101. The LaGrangian is kernel 400. lol

dataset

        d1  d2      C          d3         d4
| row | x | y | class |f1(x,y)=xy|f'(x,y)=x^2 + y^2|
| ---:|--:|--:| -----:|---------:|----------------:|
|   1 |  1|  0|      0|         0|                1|
|   2 |  0|  1|      0|         0|                1|
|   3 | -1|  0|      0|         0|                1|
|   4 |  0| -1|      0|         0|                1|
|   5 |  2|  0|      1|         0|                4|
|   6 |  0|  2|      1|         0|                4|
|   7 | -2|  0|      1|         0|                4|
|   8 |  0| -2|      1|         0|                4|


The exact meaning of cost function ? (Machine Learning) I'm reading the "Python Machine Learning" book by Sebastian Raschka, and we use different cost functions.
For Adaline model (with a linear activation function) we use the MSE error : (Phi is the activation function and y the target)
$$
Err(w) = \frac{1}{2}\sum(\phi(z) - y)^2
$$
But for the Logistic Regression model we use another cost function :
$$
Err(w) = \sum[-y\log(\phi(z)) - (1-y)\log(1-\phi(z))]
$$

I just want to know what does the cost funciton really mean, it's a probability ? Or just something arbitrary and convex ?
Is there a way to find "the best" cost function for a specific activation function ?

The cost function is the judge for your model. It judges how well your model perfoms. By choosing a loss function you choose which properties of your model outputs the loss function will judge. Mathematical convenience usually is desired for the loss function to be applicable

The MSE will punish outputs that are further away from the desired value more severely than those that are closer because of the quadratic. Therefore, the outputs that are the furthest away from the desired value impact the cost function and thus the optimization greatly. If your dataset has many outliers, then these can influence your cost function a lot. In this case you could use techniques like dimensionality reduction or just choose another loss function like the $L_1$ loss, which is more robust to outliers. Furthermore, if you choose MSE you implicitly assume that the noise of your data is guassianly distributed, since the MSE loss will minimize the crossentropy between the empirical distribution of your outputs and the gaussian distribution. See https://stats.stackexchange.com/questions/288451/why-is-mean-squared-error-the-cross-entropy-between-the-empirical-distribution-a

The logistic regression cost function will judge the outputs of your model by other characteristics. The logistic regression intends to classify your outputs correctly into two classes. The output of your model lies between $[0, 1]$ the respective classes. Once outputs lie on the correct side of the decision boundary their impact on the lossfunction decrease rapidly with the distance to the decision boundary. Look at the logistic curve and observe where points are closest to the classes ($0$ or $1$) and then check out where the loss function is the steepest for each class. So the cost function of the logistic regression will punish those outputs severly that are missclassified, and those, although a bit less, that are correctly classified but lie very close to the decision boundary. As a result it is a decent loss function for binary classification. There are probably a few further implicit assumptions with logistic regression, however I have not dug into that in the past to be able to say what they are.

Since one usually assumes that the data is i.i.d (identically, idependently distributed: each datapoint is to be treated equally), the loss for each output is summed up.

In summary: Loss functions are chosen to judge the desired properties of the model outputs, and usually exhibit favorable mathematical qualities for the optimization. By choosing the loss function you can alter the criteria your optimization should follow and often times thus introduce prior knowledge.


Is it possible to implement a vectorized version of a Maxout activation function? I want to implement an efficient and vectorized Maxout activation function using python numpy.
Here is the paper in which "Maxout Network" was introduced (by Goodfellow et al).
For example, if k = 2:
def maxout(x, W1, b1, W2, b2):
    return np.maximum(np.dot(W1.T,x) + b1, np.dot(W2.T, x) + b2)

Where x is a N*D matrix.
Suppose k is an arbitrary value(say 5). Is it possible to avoid for loops when calculating each wx + b? I couldn't come up with any vectorized solutions.

If you could combine all the weight vectors into a matrix W and all b's into a vector b, then you could do
np.maximum(np.dot(W.T,x)  + b)


Naming the Axes of my Graph on iPython Using Matplotlib.pyplot I am new to data science. I use Anaconda on windows 7.
I plotted a sine curve by doing the following on iPython:

import numpy as np
import matplotlib.pyplot as plt
x = np.linspace(0, 10, 1000)
y = np.sin(x)
plt.plot(x, y)

And I got this:

But when I got ready to name the axes, the curve disappeared. I wrote this code:

plt.xlabel("Time")

And I got this:

I also wrote this:

plt.ylabel("Speed")

And got this:

So my question is, how can I plot a curve with labelled axes? (In other words, I will like the x-axis to be Time, the y-axis to be Speed and the curve intact)

When you use matplotlib's plot function, it holds an object behind the scenes for you. You can change this object with more calls to plt and then only once everything has been done should you plt.show() the graph.

Here is a simply example that does what you want:

In [1]: import matplotlib.pyplot as plt
In [2]: import numpy as np

In [3]: x = np.linspace(0, 10, 100)
In [4]: y = x ** 2

The following lines change the plot object, adding the axes labels - but we don't show it until all are complete...

In [5]: plt.plot(x, y)
Out[5]: [<matplotlib.lines.Line2D at 0x7f3c896bb9b0>]

In [6]: plt.xlabel("Time")
Out[6]: Text(0.5, 0, 'Time')

In [7]: plt.ylabel("Speed")
Out[7]: Text(0, 0.5, 'Speed')

Now we are done, so show it:

In [8]: plt.show()

Have a look here for a more thorough demo, which also shows the object explicitly and lets you better understand what is going on.


GloVe vector representation homomorphism question In the paper GloVe: Global Vectors for Word Representation, there is this part (bottom of third page) I don't understand:

I understand what groups and homomorphisms are. What I don't understand is what requiring $ F $ to be a homomorphism between $ (\mathbb{R},+) $ and $ (\mathbb{R}_{>0},\times) $ has to do with making $ F $ symmetrical in $ w $ and $ \tilde{w}_k $.

Am I misunderstanding something? We want $ F $ to be unchanged if we either interchange $ w_i $ and $ \tilde{w}_k $ OR interchange $ w_j $ and $ \tilde{w}_k $, right? Is this the only way to achieve the symmetry between  $ w $ and $ \tilde{w}_k $?

If you're asking if the group homomorphism makes the the process symmetric then no it doesn't directly.  However, they use the fact that they require a group homomorphism to show that $w_{i}^{T} \tilde{w}_k = log(P_{ik})=log(X_{ik}) - log(X_{i})$  This nearly gives us symmetry.  Finally by adding $\tilde{b}_{k}$ into the equation you restore symmetry.

So in short $w_{i}^{T} \tilde{w}_k + b_{i} + \tilde{b}_{k} = log(X_{ik})$ is what ensures symmetry, and the group homomorphism is a tool to get there.

Update:

Some more details
Essentially, what we want is the ability to peform a label switch.  Group homomorphism helps with this process because it perseves a mapping between the $(R, +)$ and $(R, x)$.

$F((w_{i}^{T} - w_{j}^{T})w_{k}^{'})=F(w_{i}^{T}w_{k}^{'}+( - w_{j}^{T}w_{k}^{'})) = F(w_{i}^{T}w_{k}^{'}) \times F(-w_{j}^{T}w_{k}^{'} )= F(w_{i}^{T}) \times F(w_{j}^{T}w_{k}^{'})^{-1} = \frac{F(w_{i}^{T}w_{k}^{'})}{F(w_{j}^{T}w_{k}^{'})}$

The group homomorphism here allows for that to occur.   Therefore we can see that by setting $F(w_{i}^{T}w_{k}^{i}) = \frac{X_{ik}}{X_{i}}$

Now finally we can say that $w_{i}^{T} {w}_k^{'} = log(P_{ik})=log(X_{ik}) - log(X_{i}).$

So as far as your comment, it is the most sensible chocie for their method and of which they buld the core mathematicals to GloVE.  Changing it, I imagine wouldn't be a trivial thing.  I imagine if you did, much of what is derived, including the loss function would change.  But with that said, I imagine there are otherwise to achieve label switching.


What is the position embedding code? https://github.com/google-research/bert/blob/master/modeling.py#L491-L520

The code of BERT is one of the implementation. But it is not what I need.

I search a lot but can not judge.

But where is the code which seems uses cos and sin?

Here is the code implemented by TensorFlow:

Implementation 1:

Implementation 2:


The OLAP (On-Line Analytical Processing) cube with 4 dimensions A typical OLAP cube looks like this:

As I can see, this cube can work with 2 or 3 dimensions, but what if I have 4 dimensions to produce facts?
Should I use star schema instead when having more than 3 dims?

Your example is a star schema, it's just a star with three points (dimensions).
It's OK to have star schemas with more dimensions. Some large OLAP schemas can have tens of dimensions.
The star schema holds the underlying data. The cube is a convenient set of pre-aggregated values that make our run-time faster. The "cube" name is a handy visual and linguistic analogy but not a fundamental part of it's definition. Specifically, it is not limited to 3D. As in maths were a hypercube can be reasoned about in many dimension so an OLAP cube can be defined over many star schema dimensions. It just gets harder to draw in marketing literature, is all.


Python : data type handling by sklearn and impact on memory usage and performance I am currently working on reducing the memory usage of my data (Pandas DataFrame). This is going quite well : downcasting floats into smaller floats, integers into smaller ones and transforming string objects into categories, then using these new defined types directly when reading the data. I would even be interested in going further (typically rounding some floats to downcast them into int).
However I suspect that it may not have so much impact on the rest of my sklearn Pipeline (pre-processing of my data types ints, floats and categories, and calibration of a small ANN). More specifically :

I can't find anything in the documentation, but there seems to be a tremendous amount of discussion on GitHub on which transformer can handle float32. This might limit the usefullness of downcasting my float, but also for some transformation (min/max scaler, logarithmic transformation) might 'upcast' my ints into floats.

Category seems handled like objects in the pre-processing step. Those features mainly go trough categorical encoding. So as far as the labels are handled correctly, I don't think it should have much impact.

It's unclear to me how those preprocessed data are handled trough the Main model fitting sequence. From litterature, it seems that the float precision may have a significant impact overall (reducing float precision, without getting rid of too much information, can improve speed of calibration and overall statistical performance of the model). But it's unclear how to ensure that enough, but not too much, float precision is used trough the whole Sklearn process.

So, quite generally speaking, what are the steps to ensure that Sklearn is efficient in terms of memory usage and precisions of operations ?

Overall, scikit-learn is designed to be not inefficient but it's goal is not hyperefficiency. If your goal is hyperefficiency then you should switch to a different machine learning package.
The goal of scikit-learn is to provide a high level interface with machine learning and handle the implementation details "under the hood". That is why there is little documentation about how different datatypes are handled. If you are interested in how different algorithms handle different datatypes, you have to look directly at the source code. All of the scikit-learn source code is on GitHub.


Matching similar strings I have a list of conferences on different topics, e.g.

Conference on genomics and neurosciences
Advances in string theory and astrophysics
Genomics and neuroscience: 20 years of research
Swiss Physics society meeting on string theory and astrophysics
...

They fall into different classes, like 1 and 3, 2 and 4 together. What is the right tool to group those titles?

I assume you have some training data with labels, i.e. data where the titles are already linked to a given class? This is then supervised learning (as opposed to unsupervised learning), and so you could folow the following steps:

Step 1: you have words as input, so you will need a method to create numerical representation (vectors). For that you could look into algorithms such as Word2Vec, Doc2Vec, GLoVE or something like TF-IDF. If you go for the first, you might consider trying the spaCy library in python. Here is a tutorial on Word2Vec using spaCy.

Step 2: once you have your numerical representations for each of your titles, you need to somehow classify them. You could do this a few ways. Perhaps the simplest would be something like a clustering algorithm, e.g. the DB-Scan algorithm in SciKit Learn - here is a demo.
You could try more complicated methods, such as Support Vector Machines or Neural Networks, but probably best to start with a method that will get you to some results more quickly. You are classififying titles, so be sure to form your problem as a classification as opposed to a regression problem.

Step 3: assess your results and try changing a part of the loop above.

In the above, I assumed you are talking about the semantic meaning of the conference titles, and not similarity between literal word/letter combinations. That could of course be computed analytically, without the use of a model that learns.

In response to OP's comment:
From my experience, using TF-IDF or something called minimal new sets might be a good way to get your titles into representations that allow clustering. Once clusters are formed, it would be up to you to then interpret them and assign labels. If you know that there are e.g. only 10 conference, it shouldn't be too difficult to reach results. Have a look at this master thesis that does a similar thing - instead of conferences, they want to detect topics. Disclaimer: I supervised that thesis.


How to find AUC metric value for keras model? I have wanted to find AUC metric for my Keras model. Keras doesn't have any inbuilt function to measure AUC metric.  So I found that write a function which calculates AUC metric and call this function while compiling Keras model like:

from sklearn import metrics
from keras import backend as K

def auc(y_true, y_pred):
    return metrics.roc_auc_score(K.eval(y_true), K.eval(y_pred))
model.compile(loss="binary_crossentropy", optimizer='adam',metrics=['auc'])

But this doesn't work in my case. Please help me to figure out this query.

thanks

There is now a built-in function to compute (an approximation of) the AUC. See tf.keras.metrics.AUC. Apparently, you just need to do the following

...
model = tf.keras.Model(inputs, outputs)
model.compile('sgd', loss='mse', metrics=[tf.keras.metrics.AUC()])


Where can I find freely available multi-label datasets online? I'm trying to find multi-label classfication datasets, which are available for free online.

By "multi-label" I mean that each instance can be labeled with anywhere from a single to $k$ labels, where $k$ is the total number of different labels in the dataset. Typically all information about the labels would be represented in a binary matrix $\mathbf{M}$, where $\mathbf{M}_{ij}=1$ if instance $i$ has label $j$, and $0$ otherwise.

I've found the following two datasets so far:

iMaterialist Challenge (Fashion) at FGVC5 from Kaggle.com
DeliciousMIL: A Data Set for Multi-Label Multi-Instance Learning with Instance Labels Data Set

I've also looked at the Mulan multi-label datasets page, but they are pretty opaquely (and sometimes erroneously) described.

Where can I find more multi-label datasets (preferably with 20-200 different labels in total)?

Try, Kaggle Toxic Comments Challenge. You have to classify the answer to multiple classes at the same time. It is a multi-label classification problem.


Benchmarks based on neural networks libraries to compare the performance between different GPUs I am looking for benchmarks based on neural networks libraries (Theano/TensorFlow/Torch/Caffe/…) to compare the performance between different GPUs.

I am aware of:

https://github.com/jcjohnson/cnn-benchmarks (CNN in Torch)
https://github.com/jcjohnson/neural-style#speed (CNN in Torch)
https://github.com/glample/rnn-benchmarks (vanilla RNNs and LSTM in Theano, TensorFlow, and Torch)

What are some other benchmark codes?

Here is my list with collection of benchmarks: https://github.com/mrgloom/Deep-Benchmark-List

https://github.com/dividiti/ck-caffe looks more promissing, but complicated.


Random Forest prediction fails due to unseen Features I have trained a Random Forest Model on some dataset and like to predict outcomes on other data which were not seen in training. When doing this, I get

ValueError: Number of features of the model must match the input. Model n_features is 12 and input n_features is 13

The problem is that there are some variables from the training data not existent in my prediction set. E.g. I capture the count of some feature via dummy variables D_0, D_1, D_2, D_3 indicating the number of occurences of D. I might have no D_2 in my training data but D_2 in my prediction data set.

What's best practice in such a case? I am planning to use this estimator repeatedly on future data and I can't know which features will be existent. Should I rather check for inconsistencies between both feature lists and manually correct those which do not overlap? In the above example, I'd code all occurences of D_2 to D_3 in order to align feature lists.

Problem is the way you're onehot encoding.

Best practice for any type of encoding :

You should train an estimator for  Onehot encoding on the training data only, and when encoding test data, you should use the same estimator used on training data.

Eg : sklearn.preprocessing.OneHotEncoder does this, and it has a parameter called : handle_unknown.

  handle_unknown{‘error’, ‘ignore’}, default=’error’
  Whether to raise an error or ignore if an unknown categorical feature is present during transform (default is to raise). When this parameter is set to ‘ignore’ and an unknown category is encountered during transform, the resulting one-hot encoded columns for this feature will be all zeros. In the inverse transform, an unknown category will be denoted as None.

Optimal option is : You could use this parameter and set it to ignore, in order to ignore the unknown feature value and avoid an error, until you retrain your model eventually and add the new feature values to your model.

from sklearn.preprocessing import OneHotEncoder
ohe=OneHotEncoder(handle_unknown='ignore')
train=ohe.fit_transform(train)
test=ohe.transform(test)

Or you could , as you said , manually correct differences in the feature space, but would be time-consuming at each update of your model, without excluding the possibility of your code raising an error if you're sloppy in your manual correction.


Strange Neural Network overfitting I'm experiencing a very strange behavior in training the following NN model for multiclass classification:

METRICS = [
    keras.metrics.AUC(name='auc')
]

model = keras.Sequential()
model.add(layers.Dense(hidden_units, activation='relu', kernel_regularizer=l2(0.1), input_shape=(input_len,)))
model.add(keras.layers.BatchNormalization())
model.add(keras.layers.Dropout(dropout_rate))

for i in range(hidden_layers-1):
    model.add(layers.Dense(hidden_units, activation='relu', kernel_regularizer=l2(0.1)))
    model.add(keras.layers.BatchNormalization())
    model.add(keras.layers.Dropout(dropout_rate))

model.add(keras.layers.Dense(output_len, activation="softmax"))

model.compile(
    optimizer=tf.keras.optimizers.Adamax(learning_rate=lr),
    loss='categorical_crossentropy',
    metrics=METRICS)

Testing different parameters combinations, my model always overfits, but the test accuracy is always lower than the train accuracy of 0.001, in terms of AUC.
The following is an example of output obtained with 3 different combinations (learning rate, epochs, batch-size, hidden layers, and hidden units per each layer).

lr: 0.001, e: 10, b: 128, l: 1, u: 200
Train : 0.992
Test  : 0.991

lr: 0.001, e: 10, b: 128, l: 2, u: 200
Train : 0.984
Test  : 0.983

lr: 0.001, e: 10, b: 500, l: 1, u: 200
Train : 0.988
Test  : 0.987

lr: 0.001, e: 10, b: 500, l: 2, u: 200
Train : 0.974
Test  : 0.973

This is how I evaluate the model:

train_auc = roc_auc_score(y_train, model.predict(X_train), average='weighted')
test_auc = roc_auc_score(y_test, model.predict(X_test), average='weighted')

Please note that I'm using average='weighted' because I'm dealing with an imbalanced dataset.

I've tried training the model with and without Dropout and regularizers, but I've obtained the same strange behavior. What am I doing wrong?

I wouldn't say your model is overfitting, for two reasons:

The validation AUC is extremely high, I don't think many models end up with ~0.98 AUC in validation. This essentially means that the model is perfectly learning the task given, with the possibility of some exceptions. These exceptions might be always a thing the same, for each training experiment you've done, and for this reason you see a consistent drop of performance.
Whenever you train a model, it is normal for the validation performance to be slightly lower than the training performance, and that doesn't necessarily mean the model is overfitting, especially when the difference is around 1/1000 of AUC. I think this question might help you understand, for instance. I would say a model overfits if there is a significant drop, which I don't see here.


Evaluating clusters (e.g. built by kmean) using Random Forest I have made clusters for my data set (1.5 million samples and 800 features) using k-mean. I am aware of internal indices for evaluating clusters. However, I was thinking about training a supervised classification model (e.g. random forest classifier) in which cluster numbers are dependent variable and all other features are independent variable. Is using the accuracy of this model a good evaluation of the kmean cluster and why?

If you were to have fully formed trees in your random forest, then they should be able to recreate the rules you ended up clustering by, and accuracy would be 100%. Now, if you are cutting off the depth of the trees in the random forest (which is pretty standard, let's say you set max_depth = 8), then you will have an accuracy that may be less than 100%. This means that there are salient features that you are not consistently getting information from because they are not part of the 8 levels that one or more decision trees are using to classify.

My question would then be: why do you want to do this? Once you have the centroids for each cluster using k-means, it is pretty quick to be able to determine which cluster a new record would belong to. I'm not sure I understand why you would want to use a random forest in this fashion.


Cash-flow prediction with machine learning In our company we have huge incoming cash flows every day. Having a very good prediction on the incoming cash flows for the next working day would be for various reasons very useful.

As for the data: I have every incoming payment of the last 2 years. Every transaction has the actual day of the payment, the day on which we expected the payment to be received, the amount of money paid and an unique identifier for the user.

Of course we are only interested in the aggregated payments for one day. Due to our business we have a strong seasonality around the first working day of the month and an even bigger peak for the first working day of the year.

How would you try to model something like that? I know that LSTM is often used for time series forecasting, but would that also be a good idea to try that out in our case? Or from experience do you have an idea which other model might be suitable for this prediction task?

There are number of different time series forecasting algorithems other than LSTM's and other types of RNN's. In general the other methods that use statistical approaches take less time to run and are easier to understand.

From my experience some of the most important things is the data preprocessing rather than the algorithms (eg finding the seasonal period, finding how many seasonal periods there are identifying the trend to make a stationary series, removing outliers).
Once you have done this then you can evaluate different models with a test train split to see how they perform and choose the best model rather than starting off by choosing an algorithm.

You will probably find some interesting things will trying to account for the monthly seasonality due to the lengths of the months changing over time. One way to adjust for that is to use a dummy variable for the start of the month.

Depending on the problem there are a number of options.
Forecasting using statistical methods:

Naive (uses the mean as the prediction)
Seasonal Naive
Arima (a Class of algorithms)
Tbats

If there is a feature that is known in the forecasting window:

Regressing

If there are multiple Features:

Vector auto regression

I found this site very useful for learning what can be done with time series forecasting. (Its the html version of Forecasting: Principles and Practice Rob J Hyndman and George Athanasopoulos)


Creating an "unclassified" class in Random Forest I am trying to classify satellite based images by creating a region of interest and then classifying according to it.
I am using a Jupyter notebook using python to do that.
I used a Random forest classifier and got a nice model and result, but the problem is that the image is "over classified" meaning that all the pixels et value and force to be classified.
I would like to define level of similarity that a pixel has to have in order to be classified, otherwise, it will not get any class.
For example, the black suppose to be asphalt:

However, in the RGB, you can see it's not asphalt:

Is there any way to define in random forest or any other algorithm "level os similarity"? (For example something similar to n-D angle to match pixels to reference like ised in SAM, but under random forest, or another algorithm that allows define that)
SAM- https://www.harrisgeospatial.com/docs/SpectralAngleMapper.html
My end goal: to get "unclassified" values based on similarity level to the calibration data

It seems you can use RandomForest to get probabilities of being in both class by using predict_proba(X).
You could just get thoses probs, get the higher one (which is currently the class assigned to your data), and manually set a threshold, for example setting that if the most probable class is at less than 75%, then you manually classifies it as "Unknown"


Activation method and Loss function for multilabel multiclass classification I am using CNN for Sentence Classification code by Yoonkim. This is used for text classification. I noticed that he uses softmax layer and negative log likelihood error. This is optimal for single label multiclass classification. I now have a dataset which is for multilabel-multiclass classification.

What are the ideal activation methods and error functions to use?
I felt that, using softmax layer will give the sum of probabilities as 1 but in this case since it belongs to different labels, at each instance we might have high probability for each class. So I believe that Softmax is not an ideal option. Am I right?
I find data for images but nothing prominent for text. Can anyone refer me to some source?

Assuming all the labels have the same importance you can have a sigmoid for every class at the output. For each of the classes it will ask, is this class part of this sentence or not? The loss is just the sum of the individual log losses for the outputs. If some labels are more important you can scale them accordingly in your loss function.


Algorithm for purely categorical data Looking for an algorithm to deal with purely categorical data.  It was suggested to me to look into the K-medoids algorithm.  Anyone know if there is a K-medoids algorithm R library(package)?

K-medoids of course works for purely categorical data.

But there also is k-modes, easy to find with a little bit of research... It's much cheaper to run, it is O(n), not O(n²)

It doesn't solve the problem that categorical data usually will not cluster in a meaningful way...


Interpretation of the loss function for word2vec I am trying to understand the loss function which is used for the word2vec model, but I don't really follow the argumentation behind this video https://www.youtube.com/watch?v=ERibwqs9p38&t=5s, at 29:30.

The formula which is unclear is the following:

$J(\theta) = \displaystyle-\dfrac{1}{T} {\sum_{t=1}^{T} \sum_{-m <= j <=m, \\j\ne0} log(p(w_{t+j}|w_t))}.$

$T$ is the number of words in the vocabulary
$w_t$ is a given word and we try to calculate the probablity that another word $w_{t+j}$ occurs within a window of +/- $m$ words ahead.
$\theta$ is the solution we're after. It is essentially a $2*d*Tx1$ dimensional vector which contains all the columns of the matrices V and U.

At a first sight it looks all pretty clear: we're iterating though the whole vocabulary and for each (fixed word then), we add up all probabilities that another word occurs within a window around that fixed word.

However, it fails apart for me when I consider that a word $w_t$ occurs in many positions in the corpus and might occur multiple times with a different word. E.g, the words 'deep learning' often occur together, which indicates that there's a contextual relation between them. Why would we only count them twice? It seems like the formula above is counting each pair $p(w_{t+j}|w_t)$ just twice (e.g. once for $p(deep|learning)$ and once for $ p(learning|deep)$). IMO we should need a correcting term that adjusts for the missing 'frequency', e.g.

$J(\theta) = \displaystyle-\dfrac{1}{T} {\sum_{t=1}^{T} \sum_{-m <= j <=m, \\j\ne0} log(\lambda(w_t,w_{t+j})p(w_{t+j}|w_t))}.$

In the case where $\lambda(x,y) = 1$, we get the formula above, but we could also be free to chose a function that boosts frequent occurrences (pairwise). The formula above could then be seen as a special case when you don't care that words that occur more often together get a boost.

On the other hand, when the formula above already accounts for multiple occurrences, then where is this visible?
The author then continues and defines $p(o|w)$ as $exp(o^T*w)/\sum(exp(u^T*w))$. In this particular case, I don't see that we're counting the dot-product as many times as the word $o$ is in the neighbourhood of the word $w$. Maybe the choice of $p$ is just very simplistic and represents a model where the fact that 2 words are in the neighbourhood (just somewhere in the corpus) is enough (bag of words model???). It's hard to see then that such models deliver good performance in NLP though.

Not sure what the video said, but $T$ should not be the vocabulary size, but the training corpus size (number of all words).

For example, if your training corpus is

deep learning is popular . i love deep learning . i want to learn more about it.

Then when you sume up over $T$, you will sum up all the word pairs in the corpus including duplicates. The word pair (deep learning) is indeed calculated twice.

For details please refer to the original skip-gram paper, and notice the definitoin (1) on page 2.


What is the input space of a neural network (or other supervised learning algorithms)? While training the neural network (or any other supervised learning algorithms), we supply input variables and corresponding outputs. The input variables can be continuous or discrete (binary in many cases).

What happens if after training with a binary input data, we supply a continuous value for the same input at the time of evaluation? Does the algorithm internally treat all variables as continuous variables?

For example, suppose one of the inputs is Young/Old encoded in the form of 0/1 in the training dataset. What happens if we supply a value of, say, 0.2 at the prediction stage? Does it/should it make any sense to the network?

I guess it depends on the algorithm, but linear models, as well as neural networks, treat all variables as continuous. The algorithm will not explode or anything if you supply 0.2 at prediction stage. However, your algorithm is trained on data. The algorithm can at best do what it has learnt from the training data. For this reason, do not expect anything meaningful when you feed an example with a value that has not been seen in the whole training set, or that it does not follow the training set distribution.


Concept of xml files for haar cascade in object detection with opencv? I've just started ML and since I entered a project on object detection, their haar cascade is used and XML files are loaded via OpenCV for detection. So I would like to understand why XML files are used. Also can I get such files for detection of any other features?

XML files are generated when you train the Haar Cascade Classifier. From the OpenCV docs:

Object Detection using Haar feature-based cascade classifiers is an effective object detection method proposed by Paul Viola and Michael Jones in their paper, "Rapid Object Detection using a Boosted Cascade of Simple Features" in 2001. It is a machine learning based approach where a cascade function is trained from a lot of positive and negative images. It is then used to detect objects in other images

More information on Haar cascades can be found here.
Some tutorials for training Haar Cascades are available.

Link 1
Link 2
Link 3


Negative Latent Factors in Factorized Machines I'm studing a specific implementation of a recommendation system leveraging on a factorization machine algorithm.
For each person_id and item_id combination, I have an implicit rating of 1 or 0 depending on if the user downloaded the content or not.
In the base model, I have just utilized as input variables the person_id and the item_id.
I selected a latent factor number equal to 5.
In the model output, some of the 5 the latent factors associated to some person_id and item_id are negative, and some predictions of the rating for the combination person_id/item_id are negative too.
I have searched for some theoretical explanations but not found much material, so here I am.

How a negative latent factor can be explained in this setting?
Being the training dataset provided with the target variable equal to 1 or 0, how the model end up with negative predictions for the implicit rating?

Many thanks for any hints or material
bests

Found after some digging many concepts related to Non-Negative Matrix Factorization, which if properly setup constrain a FM algorith to come up with non negative factors (and therefore predictions.). Many useful material here:

https://en.wikipedia.org/wiki/Non-negative_matrix_factorization
https://towardsdatascience.com/parts-based-learning-by-non-negative-matrix-factorisation-1279ffd29618


Finding the usage percent to perform predictive analysis for new users Problem Statement-
I have to find the average feature usage all the users and the usage of user X to suggest if he should use the feature.

Example - On google home page out of all the user's avg 85% of times uses the search button.
If a user X comes on the home page and based on his activity we calculate that if only 35% of the time he click on the search button. We want to notify him about the benefits of the search button.

Data We have-

User | Landed on home page | used search button
 1        1000                  100
 2        100                   10
 3        1                     1
 4        10                    10
 5        10000                 1

Issues-

How to eliminate user 5 as  this is exceptionally making the data skewed. Median might be a solution for the use case. Is there any better suggestion?
How to find the Average usage, I mean 1/1 (User 3) and 10/10 (User 4) are not the same i.e. 10/10 (User 4) should have more value than 1/1 (User 3)
If these users(USER 1-5) stopped coming on the home page from 1 month, still the usage average would be the same- which is wrong. Since it was getting used earlier but not recently so the usage average should get decayed.

So my question in addition to above is that, am I approaching in the right direction? Is there any build-in algorithm or tool available for the problem statement? Any new approach is most welcome.

Create a new column containing the % of times each user used the search bar (i'll call it chance)

You could eliminate all users outside of 'n' standard deviations of the new column eliminate both high and low values or you could manually set a threshold
calculate the logarithm of each landed on home page ($landed_i$) count with the base as the max landed on home page value ($landed_{max}$) $log_{landed_{max}}(landed_i) = weight$, then multiply every weight by it's chance to create chanceWeighted. This weighted chance ensures that more frequent users have more weight attributed to them. You can now find whatever average you wish off this data
if you wanted to decay users depending on last visit you would need to record the users last visited date and them perform another function on them similar to the weighted average


I have data with customer personal information and customer transaction. I cannot figure out how to use the data for training my model? Customer information attributes:
ID
Age
Gender
State etc

Customer transaction
ID
Store ID
No of items bought
State etc

Store info
Store ID
State
Daily revenue
Store size etc

I want to predict if customer will buy at a particular store or not/
So can I have the train data with suppose 5 different stores for every customer where the customer shops and then predict in other store?

Yes you can.
On the files you have mentioned above i guess, you have merge the data. But keep in mind to remove redundant columns.
After that you have to :
* check if there are any missing values or null values
* find the relation between one independent variable to other independent variable
* find the relation between independent variable and the dependent variable

Then split the data into train set and test data and apply your model


How to make a Power Spectral Density Plot in R I have a time series point process representing neuron spikes. I have computed and plotted autocovariance using acf but now I need to plot the Power Spectral Density.

Power Spectral Density is defined as the Fourier Transform of the autocovariance, so I have calculated this from my data, but I do not understand how to turn it into a frequency vs amplitude plot.

I have used the following code

# X is some set of Wait times between spikes, below is just an example
X <- c(56, 3, 4, 119, 3, 4, 121, 3, 3, 121, 3, 4, 120, 3, 4, 4, 115)
acf <- acf(X,type="covariance")
psd <- fft(acf$acf)

Now psd is a complex valued array across the default 24 lags from the acf function.

How do I turn this array into a PSD plot?

From further research I've discovered that the frequency is given by the index of the FFT multiplied by the sampling rate and divided by the size of the array. And the amplitude is the magnitude of the complex number.

So the full code for such a plot would be as follows

# X is some set of Wait times between spikes, below is just an example
X <- c(56, 3, 4, 119, 3, 4, 121, 3, 3, 121, 3, 4, 120, 3, 4, 4, 115)
acf <- acf(X,type="covariance")
ft <- fft(acf$acf)
freq <- (1:nrow(ft))*1000/nrow(ft) #In my case we sample by ms so 1000 hz
A <- (Re(ft)^2 + Im(ft)^2)^.5 #amplitude is magnitude
PSD <- cbind.data.frame(freq,A)
ggplot(PSD, aes(x=freq,y=A)) + geom_line()


Learning sum of a function using Keras I am fitting a function $y:R^3\to R$ using the following model in Keras.

from keras.models import Sequential
from keras.layers import Dense

model = Sequential()
model.add(Dense(50, input_dim=3, activation='tanh'))
model.add(Dense(1, activation='linear', kernel_initializer='normal'))

model.compile(loss='mean_squared_error', optimizer='adam', metrics=['accuracy'])
model.fit(x_train, y_train, epochs=10, batch_size=8)
predictions = model.predict(x_test)

To train, I used a data set consisting of $\{\mathbf{x}_i\}_{i=1}^N$ and $\{y_i\}_{i=1}^N$. This worked as expected.

Now imagine a slightly different situation where for each sample $i$ I have a pair of $\mathbf{x}_{i,1}$ and $\mathbf{x}_{i,2}$ ($\mathbf{x}_{i,\alpha} \in R^3$) and my target data point corresponding to $i$ is modelled as $y_i=z(\mathbf{x}_{i,1})+z(\mathbf{x}_{i,2})$, with $z:R^3 \to R$. Is it possible to define such a model in Keras, and how to do it? If not Keras, any other framework?

If you had two different functions $z_1$ and $z_2$, I would agree with Piotr: you could have two parallel networks, one for each part of the input, that join together at the end.  This would minimize (though not eliminate) information from $x_{i,1}$ bleeding into the fitting of $z_2$.

But with $z_1=z_2=z$, we'd like to enforce that those two parallel networks are the same.  The idea of using the same weights for different parts of the input remind of convolutions, and I think that will work here: let $x_{i,1},x_{i,2}$ be the columns of a $3\times 2$ input, apply 1-dimensional width-1 convolutional layers instead of dense ones, and add a "Sum Pooling" layer at the end.  (Without writing something custom, you could manage that in Keras with average pooling and manually deal with the factor of 2.)


Pre-processing mixed data prior to clustering I am new to hierarchical clustering, and wish to perform clustering on mixed data. I am slightly confused on the necessary pre-processing steps. I understand how to pre-process purely continuous data, what I haven't been able to identify is what pre-processing steps are necessary for mixed data? Do I just scale my continuous variables, impute missing data, and leave the categorical variables alone? Or do I need to perform transformations across all of my variable types?

If it's categorical you can use the presence/absence of each category as a feature.

If it's nominal (ordered) you can normalize their values to be between 0 and 1 (or whatever range you normalized the rest of your data to).


How to analyze which site has most numbers I am trying to determine which site in our organization is in greater need of upgrades to SEP 12, so when I run a query to count, I get these type of numbers

Group       Windows_SEP_11  Mac_SEP_11  Windows_SEP_12  Mac_SEP_12
Arizona\A   417                  29              219         6
Arizona\B   380                  20              282        15
Arizona\C   340                  30              383        507
Arizona\D   310                  104             186        857
Arizona\E   307                  74              403        243
Arizona\F   285                  171             522        14
Arizona\G   269                  1               559        41

However, when I find percentages, I get these numbers

Group          Win_Sep_11_%   Mac_SEP_11_%  Windows_SEP_12_%    Mac_SEP_12_%
Boston/Site 1   100               0                0               0
Boston/Site 2   100               0                0               0
Boston/Site 3   94                0                0               5

And obviously, percentage isn't good indicator because Boston/Site 1 has only 3 computers, Boston/Site 2 only has 4 computers, etc.

What is the best way to analyze data? I ultimately need a visual of sites that have

many computers, and
a great need for upgrades to SEP 12, i.e. if there are more computers with SEP 11 than SEP 12.

Please point me in the right direction.

The most obvious way of visualizing this is to have the number of computers on the Y-axis and the size of the dots representing the percentages. The categories (or sites in your case) can be represented on the X-axis. The image below shows an example where the Y-axis represents a continuous value (can be mapped to number of computers in your case), the X-axis represents a discrete value (can be mapped to sites in your case), and the size of the dots represents another attribute (like percentage in your case). I have used the R package ggplot2 for this.


How to treat undefined input space in supervised learning algorithms? In a supervised learning problem (like a neural network) with binary output, we supply input data for which the output values are defined 0 and 1 (indicating probabilities of occurrence of an event).  But the input data not necessarily cover the entire input space. How does the algorithm treat the rest of the input space?

For example, for an annular dataset in the figure, there are two sets of data in the 2-D input space annotated as 0s and 1s. But there are a lot of input space which is not defined (neither 0 nor 1). What output can be expected at the prediction stage for an input that does not lie around the input data clouds?

It depends on the machine learning technique. The following link gives a really good example comparing many classifiers and their segregation area. The same happens with clustering methods.

Classifier comparison


ValueError: No gradients provided for any variable I have this error when running training on my model.
I found this issue on different sites, but could not find a solution to my problem.
Here is my model :
import keras
import tensorflow as tf
import tensorflow.keras.layers as L
import tensorflow.keras.models as M
import tensorflow.keras.callbacks as C
import tensorflow.keras.utils as U

def make_model_lstm_pooling(inshape=50000):
    z = L.Input(shape=(inshape, 10))
    x = L.AveragePooling1D(pool_size=1, strides=100)(z)

    x = L.Bidirectional(
        L.LSTM(10,
            dropout=0.1,
            return_sequences=False,
            kernel_initializer='ones',
            bias_initializer='zeros')
    )(x)

    x = L.Dense(10, activation='linear')(x)
    x = L.Dense(1, activation='linear')(x)

    model = tf.keras.Model(z, x)
    model.compile(optimizer='adam')
    return model

I run the training then :
callback_lr = C.ReduceLROnPlateau(
                monitor='val_loss',
                patience=3,
                verbose=0,
                mode='min')

checkpoint = C.ModelCheckpoint(
                filepath='best_pool.h5',
                save_best_only=True,
                monitor='val_loss',
                mode='min')

model = make_model_lstm_pooling()
model.summary()
history = model.fit(
            X_train, Y_train,
            validation_data=(X_dev, Y_dev),
            epochs=100,
            callbacks=[checkpoint, callback_lr]
                   )

The whole error is this one :

ValueError: No gradients provided for any variable:
['bidirectional_16/forward_lstm_50/lstm_cell_83/kernel:0',
'bidirectional_16/forward_lstm_50/lstm_cell_83/recurrent_kernel:0',
'bidirectional_16/forward_lstm_50/lstm_cell_83/bias:0',
'bidirectional_16/backward_lstm_50/lstm_cell_84/kernel:0',
'bidirectional_16/backward_lstm_50/lstm_cell_84/recurrent_kernel:0',
'bidirectional_16/backward_lstm_50/lstm_cell_84/bias:0',
'dense_90/kernel:0', 'dense_90/bias:0',  'dense_91/kernel:0',
'dense_91/bias:0'].

I get the problem when running fit.
I saw that the problem can appear when using bad types : I have float type in the input and int in the labels. I have NO nan in the input.
I see in the error there is a problem related to the kernel initializer, the default is glorot_uniform, it appears to me it is not zeros, if I am not mistaken.
I tried to change kernel_initializer but did not improve.
Something else : I made a test on several samples, and in my test I have less samples then features. Have anyone any idea if the problem is related to this ?
Any help will be appreciated.

I found the solution to my problem.
First of all, I had to declare a loss when compiling the model :
model.compile(
    optimizer='adam',
    loss='mean_absolute_error',
    metrics=['mean_absolute_error']
)

In addition to that, I changed the monitor for the checkpoint:
import tensorflow.keras.callbacks as C

checkpoint = C.ModelCheckpoint(
                filepath='best_pool.h5',
                save_best_only=True,
                monitor='val_mean_absolute_error',
                mode='min')

Changing the monitor gave a better result in this case.


How would you build a big production ready image training dataset from scratch? How would you most likely create a large production ready image training dataset from scratch including annotations for a image classification task?
We will take a large amount of images (~1 million) with industrial cameras and save them in a S3 bucket. Do you think a data lake infrastructure is necessary?
In your opinion, what are the most suitable methods for annotating the images in the shortest possible time (bounding boxes not needed).
Solutions that I have been able to find so far are the following:

Use an open source web based image annotation tool like make-sense or LOST (Problem: who will annotate the images? These tools doesn't seem to be perfect for big amount of image data). See also awesome-data-labeling
Build a gamificated web application and let users annotate images and earn discount codes to motivate them
Use third party tools with annotation workforces like Playment, Labelbox, Amazon Mechanical Turk

Are there any options I missed? In principle, it would be possible to pay for the annotation, but should be avoided or kept as small as possible.
Are there things that should be considered architecturally with such a large database?

I'm not expert in images classification, I'm just going to give some general advice here.
The strategy should be progressive, for instance:

Proof of concept: devise a first draft of annotation process with some general guidelines, then take a random subset of a few hundreds images and try annotating them following the guidelines. Observe carefully the problems found at the annotation stage: any ambiguous cases, any problems with the granularity of the classes (for instance a class which is too general), potential error cases. Devise a training/testing setup to train a mockup system, test it with the small dataset, fix any bugs in the process, possibly try different methods. The goal of this stage is to iron out the different stages, possibly rule out some options which turn out to be unworkable.
Prototype: have a team of a few laypeople annotate a few thousands images following the updated annotation guidelines. Again pay attention to any problem, especially human errors in the process. At least a subset of the images should be annotated by several annotators, in order to detect differences. Ask annotators for feedback: were there any difficulties, was it fun, would they do it as a game or not, etc. At this stage you can start having a real ML system, albeit not optimal yet. Analyze the performance of the system on the different classes, possibly try different methods, estimate the minimum number of images per class to obtain decent performance. At this stage a lot of other tests could be done, and it should become clear how accurate the manual annotation is and whether annotators need to be compensated or not.

It's only around this stage that the final annotation process can be fully designed. Depending on the strategy, you could consider devising a strategy for iterative manual annotation: some classes are going to be quickly learned by the model, so it could make sense to use the model in order to propose for annotation at the next round the images for which the model fails. Be careful to avoid bias, keep evaluating and refining the model at every round of annotations.


Looking for a 'CITY, STATE' within a body of text (from a CITY-STATE database) I'm looking for an optimal way to search a large body of text for a combination of words that resemble any CITY, STATE combination I have in a separate CITY-STATE database.

My only idea would be to do a separate search against the body of text for each CITY, STATE in the database, but that would require a lot of time considering the amount of CITY, STATE combinations the database has in it. The desired result from this query would be to pull a single CITY, STATE for each body of text I am analyzing to tell the geographical side of the story for this data subset.

Anyone know of an optimal way/process to do such a query?

The only thing I can see would be to separate both city and state lists and treat the problem as an automaton: parse your text, run through the n-grams, whenever you detect a CITY token (meaning a n-gram present in your list of cities or close to it in a similarity sense, as there might be misspellings) then look for a STATE token in its neighbourhood (similarly by looking into a list of states, using an edit distance metric to allow for misspellings). If you find one, then you can tag your text with that geographical location.

Of course, allowing for misspellings will bring some false positives but you could easily bypass that by doing a quick lookup through your corpus to see that "SALAMI, OREGANO" is different from "SALEM, OREGON" (because the frequency of the latter will hopefully be higher than the former)


How to improve tensorflow 2.0 code for policy gradient? I recreated some code I found online for solving the bandits problem using policy gradient. The example was in tensorflow 1.0 so I recreated it with tensorflow 2.0 using eager execution and gradient tape, however, when training the model, I have to convert the weights Tensor to a numpy array, update the weights then reassign the tf.Variable back from the numpy array. I feel this is not performant and could find a better way. Full code is here https://github.com/entrpn/reinforcement-learning/blob/master/tf2_rl/bandits.py

The main code I'm looking to improve as follows:

def train(agent,action,reward, learning_rate=0.001):
    with tf.GradientTape() as t:
        current_loss = loss(agent(action),reward)
    dW = t.gradient(current_loss,[agent.weights])
    weights_as_np = agent.weights.numpy()
    responsible_weight = agent.weights[action]
    responsible_weight_dw = np.array(dW)[0][action]

    weights_as_np[action] = weights_as_np[action] - learning_rate*responsible_weight_dw

    agent.weights.assign(tf.Variable(weights_as_np))

If possible, try to use just tensorflow functions (put the .numpy() part out, for example), and put a @tf.function decorator on top of train() function.

The role of @tf.function is that it transforms a whole function into a tensorflow op. The whole function will be execute an order of magnitude faster than a normal Python function.


Class distribution discrepancy training/validation. Loss now uninterpretable? I have a 3-class image classification problem. The classes are highly unbalanced (with about 98% of the images beloning to one class). To counteract this unbalanced data, I weight the losses by using the inverse of the class probabilities (1/(Proportion of class)). This means that the loss gets multiplied a great lot for the minority classes. This does help in counteracting the class imbalance.

However, for this weighted loss I use the proportions in the training dataset. In the validation dataset, the distribution is somewhat different (96.66%, 2.59%, 0.75% in val set as opposed to 98.28%, 0.98%, 0.73% in the training set). This means that if I use the weights based on the training set, mainly misclassified class 1 images gets punished a lot more than they should according to the distribution in the validation set.

The reason for this distribution discrepancy is that I can't just do a random split on my images, since they are taken by some 10 cameras all taking pictures of the same respective static scenery (highways). This means that I have to hold out all images of about 2/3 cameras as validation and use the other cameras for training, otherwise the model will be too optimistic, since validation images come from the same camera as the train images.

This makes the weighted loss uninterpretable for the validation data. Is there a way to counteract this? For instance, would giving separate weights for the train/validation sets (based on their respective distributions) help?

The validation set is used to estimate how well the model is fitting to your solution space.  For instance, if your training loss continues to decrease but your validation loss remains constant or increases, then you should employ early stopping as to avoid overfitting.  This does not require a weighted loss.

If you wanted to compare the training loss to the validation loss, you cannot use the same weights if the distributions differ.  Instead, you can simply compare the average unweighted loss per set.  The weighted loss is for penalization.


What is the difference between Linear SVM and SVM with linear kernel? I'm wondering whether there is a difference between Linear SVM and SVM with a linear kernel. Or is a linear SVM just a SVM with a linear kernel?

If so, what is the difference between the two variables linear_svm and linear_kernel in the following code.

from sklearn import svm
linear_svm = svm.LinearSVC(C=1).fit(X_train, y_train)
linear_kernel_svm=svm.SVC(kernel='linear', C=1).fit(X_train, y_train)

As you can read in the documentation (http://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html) it's conceptually similar, but implemented in another way to add flexibility:

  Similar to SVC with parameter kernel=’linear’, but implemented in terms of liblinear rather than libsvm, so it has more flexibility in the choice of penalties and loss functions and should scale better to large numbers of samples.

So if you only want to use linear support vectors use this one because it scales better and you get more freedom with your loss functions, but if you want to more easily try different kernels in a grid search use the more generic one.


XGBoost vs ARIMA for Time Series analysis Doing time series analysis, I have doubts on choosing the right model. I want to predict the next 30 mins window, from the input dataset which contains the no. of error count for that particular 1 min interval.

Should I use XGBoost or ARIMA regressions?

Most of article or tutorial I found online use ARIMA for time series, while XGBoost is used more in Kaggle competitions.

Usually, ARIMA regressions are used in classical statistical approaches, when the goalis not just prediction, but also understanding on how different explanatory variables relate with the dependent variable and with each other. ARIMA are thought specifically for time series data.

On the contrary, XGBoost models are used in pure Machine Learning approaches, where we exclusively care about quality of prediction.
XGBoost regressors can be used for time series forecast (an example is this Kaggle kernel), even though they are not specifically meant for long term forecasts. But they can work.

In conclusion, I don't know your data, but if I had to bet I'd go for ARIMA. However both could work great, and I suggest you to try both and pick the best for your particular need. They are comparatively quick to implement, it shouldn't take too much time.


Creating pronunciation dictionary for ASR I am working on ASR(automatic speech recoginition) on Somali data as master thesis and now I am stuck with how to create a phonetics or pronunciation dictionary for it. I searched over net and could not find one.

I'm not sure how to tackle this. Can someone guide me ?

I had posted a similar question in reddit as well and I got a response from Nikolay Shmyrev :

"If you want to convert latin script, you can write simple rules yourself. Something like this.  Or you can use epitran as is. "

Thanks to Nikolay Shmyrev who originally answered in reddit.


In smoothing of n-gram model in NLP, why don't we consider start and end of sentence tokens? When learning Add-1 smoothing, I found that somehow we're adding 1 to each word in our vocabulary but not considering start-of-sentence and end-of-sentence as two words in the vocabulary. Let me throw an example to explain.
Example:
Assume we've a corpus of three sentences: "John read Moby Dick", "Mary read a different book", and "She read a book by Cher".
After training our bi-gram model on this corpus of three sentences, we need to evaluate the probability of a sentence "John read a book", i.e. to find $P(John\; read\; a\; book)$
To differentiate John appearing anywhere in a sentence from it's appearance at the beginning, and likewise for book appearing at the end, we rather try to find $P(<s>John\; read\; a\; book<\backslash s>)$ after introducing two more words $<s>$ and $<\backslash s>$,  indicating start of a sentence, and end of a sentence respectively.
Finally, we arrive at the

$P(<s>John\; read\; a\; book<\backslash s>)$ as
$P(John|<s>)P(read|John)P(a|read)P(book|a)P(<\backslash s>|book)=\frac{1}{3}\frac{1}{1}\frac{2}{3}\frac{1}{2}\frac{1}{2}$

My Question:
Now, to find $P(Cher\; read\; a\; book)$, using Add-1 smoothing (Laplace smoothing) shouldn't we add the word 'Cher' that appears first in a sentence? And to that, we must add $<s>$  and $<\backslash s>$ in our vocabulary. With this, our calculation becomes

$P(Cher|<s>)P(read|Cher)P(a|read)P(book|a)P(<\backslash s>|book)=\frac{0+1}{3+13}\frac{0+1}{1+13}\frac{2+1}{3+13}\frac{1+1}{2+13}\frac{1+1}{2+13}$

The 13 added to each numerator is due to the unique word count of the vocabulary which has 11 English words from our 3-sentence corpus plus 2 tokens - start and end of a sentence. In few places, I see 11 is added instead of 13 to the numerator, wondering what I'm missing here!

It depends on the definition of vocabulary (V).
Most teaching examples only include words in the vocabulary for simplicity. Start and stop of sentences tags can also included in the vocabulary.
Vocabulary can also include punctuation, or stop words can be removed from the vocabulary.


In orange, how do you split a given table at a certain row id? I was thinking of adding another column called "Test / Train" in which a 0 would indicate test and a 1 would indicate train data, and then as I'm collecting data end the train data collection phase and start entering each row with 1 in that column.  Then using select rows widget to select all rows with "Test / Train" = 0 or 1.  However, under PCA, the original columns go away so this wouldn't work!

How then, can I just select all rows with row id > k?  I need this because the past data is supposed to predict a smaller set of future data, so I want to be sure that it works and not just select the test set randomly from the training data (which is an option in Test & Score).

Actually, I can add in a string column and should still be able to do the row selection trick even with PCA.  In the file widget you set that column as text / meta and it shows up in the forward row selection widget.

I think this will work.


How do I know what the best number of layers is required to achieve the highest accuracy I'm learning from Udacity using this video.
I saw this piece of code:
model = nn.Sequential(nn.Linear(784, 128),
                      nn.ReLU(),
                      nn.Linear(128, 64),
                      nn.ReLU(),
                    #   nn.Linear(64, 32),
                    #   nn.ReLU(),
                      nn.Linear(64, 10),
                      nn.LogSoftmax(dim=1))

criterion = nn.NLLLoss()
optimizer = optim.SGD(model.parameters(), lr=0.003)

epochs = 5
for e in range(epochs):
    running_loss = 0
    for images, labels in trainloader:
        # Flatten MNIST images into a 784 long vector
        images = images.view(images.shape[0], -1)

        # TODO: Training pass
        optimizer.zero_grad()

        output = model(images)
        loss = criterion(output, labels)
        loss.backward()
        optimizer.step()

        if e == 1 and running_loss == 0:
            print(loss)

        running_loss += loss.item()
    else:
        print(f"Training loss: {running_loss/len(trainloader)}")

Then, I was playing around with layers and neurons.
I have tried:

Adding 1 more layer (64, 32) between (128, 64) and (64, 10), and changing the (64, 10) to (32, 10), increasing the epochs.
Replacing the (128, 64) with (128, 32), increasing the epochs
Adding 1 layer (784, 256), (256, 128), increasing the epochs

Etc.
From what I saw here, they can all achieve good accuracy with enough epochs of training.
Then, my question is how can I find the best number of layers and neurons in each layer (architecture) which can:

Achieve the highest accuracy
Still be the simplest model (least number of parameters)

The number of layers and the number of nodes in each layer are model hyperparameters that you must specify. So, we cannot directly say this no of hidden layers / this no of hidden units / this learning rate .. are better.  But we can use some techniques :

Intuition:
So based on the complexity of your model, you need to choose a value that you feel suits it the most.

Experiment:
Now, if your intuition fails, then you should experiment and see the nature of the model. So, try to plot the cost function for different values of the hyperparameter and understand the behavior of the model, with which you can probably get the best value for your hyperparameter.

Generally, if your model is expected to learn complex patterns, then you should go for depth.
For no of neurons in a layer, there are some rule-of-thumb methods:

The number of neurons should be between the size of the input layer and the size of the output layer. (or mean of both)
The number of neurons should be : (2/3 the size of the input layer + the size of the output layer)

There are some other rules and formulas, but I think these rules should give you a good start. But still, the key is your intuition and experimentation.


How does backpropagation work with averaging layers? I'm studying Word2Vec algorithm, and so far i understood that, in the case of input context bigger than 1 (so multiple words) we have our hidden layer that performs averaging between the inputs (as explained here: Word2Vec CBOW)

I didn't understand how does backpropagation work in this case.

When i start backpropagating from the softmax output layer and i reach for the hidden layer in my process, am i coping the weight over all the "same position" inputs?

To make my question clearer, if i have 2 context words coded as one hot vectors and i call the first weight of the first word as W11 and the first weight of the second word as W21, when i'm backpropagating, those two weights will be updated with se same value, right?

Highly unlikely.

Simple thought experiment. If the vertices (graphically == weights) are the same for these two nodes (first cell of the vector for the first word and the first cell of the vector for the second word in context) (what I am trying to say they are fully connected) BUT input values when you feed-fowarded these values were different, not only that this difference spaned accros other layers, hence you can not expect that weight updates via backprop will be same.


How do I know if my training data set is sufficient? If the number of classes is $C = 5$, training set contains $N = 3000$ samples, and $D = 4$ dimensions.

Is my data set is sufficient?

We can not judge whether our training data set is sufficient or not directly from the shape description for your data.

At the most of the time, we sample some instances, hopefully they are distributed as real-world data, so that the final model constructed through these data can reveal the truth of the real-world. Theoretically, the size of training data is not the problem, but we shall collect training samples as many as possible to ensure our model will have good performance on predicted instances.

Anyway, we can get the answer by testing whether the distribution of our predicted value is consistent with that of truth.


keras UserWarning: The input 1303 could not be retrieved. It could be because a worker has died I am trying to train an autoencoder CNN on Google Colab using keras. I have mounted my Google Drive which contains all training data. The training uses six workers and the data is loaded by the following custom generator class:

class ImageDataGenerator(keras.utils.Sequence):

    def __init__(self, filepaths, batch_size, shape, shuffle=True):
        self.shape = shape
        self.batch_size = batch_size
        self.filepaths = filepaths
        self.shuffle = shuffle
        self.on_epoch_end()

    def __len__(self):
        return int(np.ceil(len(self.filepaths) / self.batch_size))

    def __getitem__(self, index):
        from_index = index * self.batch_size
        to_index = min(len(self.filepaths), (index + 1) * self.batch_size)

        filepaths_temp = self.filepaths[from_index : to_index]

        images = load_images(filepaths_temp) / 255.0

        return images, images

    def on_epoch_end(self):
        if self.shuffle:
            np.random.shuffle(self.filepaths)

My problem: Sometimes when I start training I get this:

Epoch 1/100
/usr/local/lib/python3.6/dist-packages/keras/utils/data_utils.py:610: UserWarning: The input 1303 could not be retrieved. It could be because a worker has died.
  UserWarning)
1599/1599 [==============================] - 2461s 2s/step - loss: 0.0239 - val_loss: 0.0251

Epoch 00001: val_loss improved from inf to 0.02512, saving model to /content/drive/...
Epoch 2/100

Apart from the warning epoch 1 seems to finish successfully. But after the last line is printed, nothing happens anymore. Epoch 2 never starts.

I suspect the mounted Google Drive filesystem is the problem, because I had problems with Google Drive timeouts in the past. My Question is: Is there something I did wrong? Did one of the six workers die? Is there something I can do to restore the workers after the first epoch? After all, it seemed to work fine for the first epoch.

The solution for me was to copy all training data from the mounted Google Drive file system to the Google Colab file system. I didn't realize before that the mounted Drive file systems has a comparatively bad performance that leads to this error.


Comparing AUC, logloss and accuracy scores between models I have the following evaluation metrics on the test set, after running 6 models for a binary classification problem:

  accuracy logloss   AUC
1   19%      0.45   0.54
2   67%      0.62   0.67
3   66%      0.63   0.68
4   67%      0.62   0.66
5   63%      0.61   0.66
6   65%      0.68   0.42

I have the following questions:

How can model 1 be the best in terms of logloss (the logloss is the closest to 0) since it performs the worst (in terms of accuracy). What does that mean ?
How come does model 6 have lower AUC score than e.g. model 5, when model 6 has better accuracy. What does that mean ?
Is there a way to say which of these 6 models is the best ?

I will start with your last question as the answer explains the other questions. In general, this is a hard question. That is exactly why there are different metrics available. Depending on your problem and what you want to achieve, you should choose the metric that measures this best.

The differences in metrics can be found all over the internet. Others have explained it much better than I can.


Intro to Machine Learning I do not know if this is the right place to ask such a generic question, if not please let me know and I will remove it asap.

Well, I am a Software Engineering student (BEng hons) and this September my 3rd year starts and therefore I have to start my final year project. I want to do something in Machine Learning and more specifically in Prediction Algorithms.
I would like to do predictions for stock prices etc.

I have been reading a book provided by my university online the last days, but it doesnt make that much sense as there are a lot of algorithms and different approaches.

What algorithms would work for the project I want to do????
Do you know any specific books or online courses that you would recommend?

Any guidance would be really much appreciated!
Thank you!

Andrew Ng's course on Machine Learning on Coursera is terrific. I'm doing it right now, and he has wonderfully gauged student understanding. He helps you along and doesn't skip lots of steps. It apparently is not his goal to convince you of how smart he is - he's more concerned with explaining everything clearly. Highly recommended!


Orange Linear Regression and scikit-learn linear regression gives different results I am trying to cross-validate my scikit-learn script by mean of Orange3, thus obtaining a nice visual representation.

Doing step by step, to keep it simple I just tried to cross-validate a simple linear regression.

Following is the simple script for linear regression in python/scikit-learn

lr=LinearRegression(fit_intercept=True,normalize=False,copy_X=True,n_jobs=1)
lm=lr.fit(X_train, y_train)
lmscore_train=lm.score(X_train,y_train)  ## R2 = 0.6264021467338086
lmscore_test=lm.score(X_test,y_test)     ## R2 = -12.344747578839215

Whilst I was expecting to get the same result, this is not the case.

In Orange, I get R2=-10.792 whilst in Python, I get R2=-12.344747578839215.

Train and test split are the same in both cases.

Do you have a clue why that is?

Orange and Scikit treat outliers in different ways. If you have outliers in your data, remove them and check. The answers then are similar.


Multilabel classification for a learning to rank application I am looking for some suggestions on Learning to Rank method for search engines. I created a dataset with the following data:

query_dependent_score, independent_score, (query_dependent_score*independent_score), classification_label

query_dependent_score is the TF-IDF score i.e. similarity b/w query and a document.

independent_score is the viewing time of the document.

There are going to be 3 classes:

0 (not relevant),
1 (kind of relevant),
2 (most relevant)

I have a total of 750 queries and I collected top 10 results of each, so I have a total of 7500 data points.

I have been thinking of estimating a relevance function like:

w0 + w1*query_dependent_score + w2*independent_score + w3*(query_dependent_score*independent_score)

I can clearly see this is like a classification problem but I wanted some info on whether this is right way to approach this problem.

I referred to Machine learning technique to calculate weighted average weights? for some ideas.

Following is the code that I have written:

from sklearn.linear_model import LogisticRegression
import numpy as np

DATASET_PATH = "..."

search_data = np.genfromtxt(DATASET_PATH, delimiter=',', skip_header=1, usecols=(1, 2, 3, 4))
document_grades = search_data[:, 3:4]
document_signals = search_data[:, :3]  # This has 3 features.

total_rows = np.shape(search_data)[0]
split_point = int(total_rows * 0.8)

training_data_X, test_data_X = document_signals[:split_point, :], document_signals[split_point:, :]
training_data_y, test_data_y = document_grades[:split_point, :], document_grades[split_point:, :]

clf = LogisticRegression(multi_class="multinomial", solver="lbfgs")

clf.fit(X=training_data_X, y=training_data_y.ravel())

print(clf.classes_)  # [0, 1, 2]
print(clf.coef_)  # This is a 3 x 3 matrix?
print(clf.intercept_)  # An array of 3 elements?

Based on the sklearn's documentation coef_ should give me the values of w1, w2 and w3, and intercept_ should give me the value of w0.

But I have a matrix and an array for those weights. I am not sure how to get the values of the weights for the relevance function?

I am looking into learning to rank for the first time, so any suggestions are welcome.

In the multinomial mode, the docs specify that the outputs of coef_ and intercept_ are as you are seeing them: one output for each target class. The underlying model is three logistic regressions, whose outputs are softmax'ed (or with mode ovr, simply normalized).

As to the broader question, since your three output classes are ordered, you might benefit from using that information. Either just perform regression (assumes that the numeric 0,1,2 are meaningful) or use "ordinal regression."


Reference about social network data-mining I am not in the data science field, but I would like to examine in depth this field and, particularly, I would like to start from the analysis of the social networks data.

I am trying to find some good references, both paper, websites and books, in order to start learning about the topic. Browsing on the internet, one can find a lot of sites, forum, papers about the topic, but I'm not able to discriminate among good and bad readings.

I am an R, Matlab, SAS user and I know a little bit of python language.

Could you suggest any references from which I could start studying and deepen the industry?

I'm going to pursue the following series of online courses on the Coursera:
Become a Social Scientist: Methods and Statistics by University of Amsterdam. The good news - it is free, or you can get a nice-looking certificate for $49 or so. The bad news - the nearest enrollment is Aug 31st 2015. You will have opportunity to get a lot of information in condensed way during a short time frame and you will be enforced to actually apply the knowledge in exercises, quizzes and project assignments. You will also have opportunity to discuss lessons/projects on the forum with many other students and lecturers.

[update] I apologize, I just remembered there is another course Introduction to Statistics for the Social Sciences by University of Zurich - just started April 28th, 2015. If you want to pursue it - do not forget about deadlines for quizzes and homeworks. Good luck!


Which algorithm would be suitable for clustering a billion datapoints? I am running a K-means algorithm (using the sklearn implementation) on an aggregated dataset of ~350k datapoints on a 6 dimension hyper-plane (using 6 features).
I would like to do the same but in the "non-aggregated" version of my dataset, which is ~1b datapoints using the same 6 features
I know this is a very heavy task for K-means, the number of datapoints is just too big, even though the dimensions' size is pretty small.
Are there any suggestions of other algorithms that would help me on this task, apart from mini batch K-means ?

I don't know if there are other clustering methods which would work with this amount of data, but with K-means I would suggest this:

Run K-means with a varying number of instances picked randomly and study how much variation there is between the centroids depending on the data size (you can also study the variation across different random samples). I would expect the centroids to stabilize quite quickly with respect to data size: assuming the centroids become stable with data size N, there's no point running the full K-means process with more data.
Having obtained a model with N data points (the centroids), the model can be applied to all the remaining data points in order to find which cluster they belong to.

This is much more efficient than running the K-means process over the whole data since in the second stage the centroids are fixed, the algorithm doesn't have to iteratively update them.