What is the relationship between "landmark Detection" and "landmark localization" I am reading this paper "Grand Challenge of 106-Point Facial Landmark Localization"

In the context of face recognition "Landmark Detection" is to detect a face by matching landmarks on a face.

"Landmark Localization" is to predict the coordinates of a set of pre-defined key points on human face.

What is the relationship between "Landmark Detection" and "Landmark Localization"?

Landmark detection and Landmark localization are the terms that have not been defined well. It seems like they are just "created" because someone (scientists) needed them. They are similar to object localization and object detection. After seeing the question I have been searching for finding the contexts that they are used so they can be compared. However, I can say either people confuse them or they are used for the same meaning interchange. Before, proving that I will note one another thing. Normally, localization means to simply localize an object that is, finding the position of an object, or specifically drawing a bounding box around an object. Detection uses localization as a function, for example, in object detection all the objects in the image are localized and labeled accordingly. Thus, it might be expected that if a single set of landmarks (human face) are identified then it would be a landmark localization, and if multiple sets of landmarks (human face, human hands) are identified then it would be landmark detection as in object localization and detection.
So now, check this link where Nvidia names identification of different sets of "objects" as landmark localization.
Secondly, here Andrew NG explains exactly the same thing in the introduction of the paper that you provided, but he names it landmark detection. If you search for it, you will see there are a lot of papers that use landmark detection and landmark localization for explaining the same thing.
I don't claim they are the same thing, de jure they might be different concepts but de facto they are used to define the same thing, maybe because they are extremely confused. Thus, it does not "give chance" to understand their meanings.
Update: I asked the question to my Prof. from the university. I will briefly paraphrase the respond: When it comes to localization, it seems to give more importance to location. For example, in a robot application, the robot does object localization or landmark localization to determine the exact coordinates of the object to maybe hold it or touch it. In detection, we actually mean there is something here. For example, I took a photo. There is a tower in this photo. As a tourist I want to know which tower this building is. Finding the tower in the picture can be described more as a detection problem. Another example is a tumor. For example, there is a tumor in a medical image. I might want to localize this tumor in the real body using the image. Or I can detect the tumor in a biopsy image and say that this person has cancer. Although they are usually used for the same thing, you can see one is used more than another in different application areas.


In random forest, can features with high Gini importance show insignificant p-values in t-tests? In have two classes in my data, and one of my features with Gini importance 0.023 (8th highest out of 95 features) shows a p-value of 0.44 when doing a t-test on the values from the two classes. I'm wondering if this makes sense. Intuitively, wouldn't features with high Gini importances have very different values between classes? I'm wondering how to reconcile these two facts.

EDIT: If it's helpful, all of the Gini importances range between 0.0011 to 0.0295, and the p-values ranged from 5.13e-109 to 1. These p-values were seen for both t-tests and rank-sum tests.

Intuitively, wouldn't features with high Gini importances have very different values between classes? I'm wondering how to reconcile these two facts.

If I understood you correctly, these two facts perfectly reconcile in case your classes are not separable in one-dimensional feature space.

The following exaggerated toy example should illustrate such situation (and power of RF).

#uncomment the following line to remove all variables
# rm(list=ls())

library(ggplot2)
library(caret)
library(MASS)
library(GGally)
library(dplyr)

SEED <- 123456
SIMS <- 1000

set.seed(SEED)

x.signal <- rbind(
    mvrnorm(n = SIMS/2, c(0,0), matrix(c(1,0.9,0.9,1),2,2)),
    mvrnorm(n = SIMS/2, c(0,0), matrix(c(1,-0.9,-0.9,1),2,2)))

x.noise <- mvrnorm(n = SIMS, c(0,0), matrix(c(1,0.9,0.9,1),2,2))
y <- rep(c(TRUE, FALSE), each=SIMS/2)
df <- data.frame(
    x1.signal=x.signal[,1],
    x2.signal=x.signal[,2],
    x1.noise=x.noise[,1],
    x2.noise=x.noise[,2],
    y=factor(y))

yName <- "y"
xName <- setdiff(colnames(df), yName)

ggpairs(df, aes(colour=y))

ctrl <- trainControl("cv", number = 5, verboseIter = TRUE)
rffit <- train(df[,xName], df[,yName],
    method = "rf", trControl = ctrl, tuneLength = 5)

varImpPlot(rffit$finalModel)

## t = 0.061767, df = 997.98, p-value = 0.9508
t.test(filter(df, y=="TRUE")$x1.signal,
    filter(df, y=="FALSE")$x1.signal,
    var.equal = FALSE, paired = FALSE)

## t = 0.22695, df = 996.04, p-value = 0.8205
t.test(filter(df, y=="TRUE")$x2.signal,
    filter(df, y=="FALSE")$x2.signal,
    var.equal = FALSE, paired = FALSE)

dplyr::group_by(df, y) %>%
    dplyr::summarise(mx1=mean(x1.signal),
    mx2=mean(x2.signal))


Why does cost function on a neural network increase? I'm training a two layer neural network and outputting the cost function during iteration and noticed that the cost function increases dramatically with increasing iteration number. Initially I suspect it's because the learning rate of gradient descent is too high, so I changed from 0.05 to 0.005. However, this doesn't help at all. Any suggestion is highly appreciated!

Below is part of my code:

# construct model
y_pred = multilayer_perceptron(x, weights, biases)

# define cost function(mean squred error) and optimizer(gradient descent)
cost = tf.reduce_mean(tf.square(y - y_pred))
optimizer = tf.train.MomentumOptimizer(learning_rate = learning_rate,
            momentum = momentum).minimize(cost)

EPOCHS = 100
# initialize parameters
init_op = tf.global_variables_initializer()

with tf.Session() as sess:
    sess.run(init_op)
    for epoch_no in range(EPOCHS):
        _, c = sess.run([optimizer, cost], feed_dict={x:X_train , y:Y_train})
       print('Epoch number: {}, cost: {}'.format(epoch_no, c))

# For running test dataset
results, test_cost = sess.run([y_pred, cost], feed_dict={x:X_train, y:Y_train})
print('test cost: {:.3f}'.format(test_cost))
print(y_pred)

Output for the first 3 Epoches:

Epoch number: 0, cost: 509.89886474609375
Epoch number: 1, cost: 287486752.0
Epoch number: 2, cost: 2.262859251393233e+18

It turned out that my activation function is not well chosen. I chose a linear activation function for the output layer where indeed a ReLu should be chosen. The cost function decreases well after I changed the activation function. I guess this solution works case by case though. May it helps.


List of numbers as classifier input I am trying to use my data to predict the classes of the input. My data are as the following:
x1 = [0.2, 0.25, 0.15, 0.22] y = 1
x2 = [0.124, 0.224, 0.215, 0.095] y = 3
...
xn = [...] y = 2

The problem is that my data are just lists of numbers that do not have an order. I mean that x1 can be x1 = [0.2, 0.25, 0.15, 0.22] y = 1 or x1 = [0.25, 0.22, 0.2, 0.15] y = 1 or the numbers in the list to be in any other order.
Is there anything that I can do, so I will be able to build a classifier? Thank you!

The simple option is to design your features so that they represent the distribution of the values: every feature $f_i$ represents a bin and its value for a particular instance is the frequency of the corresponding range for this instance.
Example: let's consider 10 bins between 0 and 1, i.e. $f_1=[0,0.1), f_2=[0.1,0.2),..., f_{10}=[0.9,1]$:

$x_1=[0.2, 0.25, 0.15, 0.22]$ is represented as $[0,1,3,0,0,0,0,0,0,0]$
$x_2 = [0.124, 0.224, 0.215, 0.095]$ is represented as $[1,1,2,0,0,0,0,0,0,0]$
...


What algorithm is best suited to derive the best match between two people in a data set? Say i have a large data set that contains the following data;

username,age,sex,music_genre,sports_genre,art_genre,rating
DanTheMan,25,male,rock,football,cubsim,50
LoopyLucy,23,female,pop,null,popart,76
MadDog,33,null,pop,football,cubsim,57

I would like run through the whole data set and match two users based on ̶t̶a̶s̶t̶e̶s̶
genres and rating that are similar or closer then others like a best match. The gender does not need to be considered, What would be the best algorithm to acquire this result? There is an equal amount of users, Everyone will get a match.

I have taken a look into the stable marriage algorithm (Gale-Shapley) however I would appreciate someone else's suggestion and opinion.

Thanks!

Given your data sample, unless you have a more subtle way to measure similarity between different values for a given genre (e.g. some other resource indicating that football is closer to basketball than to tennis for instance), it seems that the only similarity measure that you can use is to count how many tastes two users have in common.

The similarity score can be only 0, 1, 2 or 3, so I don't think you really need clustering. You can simply build a map where the key is the concatenation of the 3 "genre" columns, and the value is the set of users which have these tastes. Some users might not have an exact match (3 identical genres), so you do the same process but for only 2 genres in common, and then for only one.


Choose points to maximize volume of convex hull Suppose I have N points (labeled 1, 2, ..., k, ..., N) in D dimensions. I'd like to choose the order of points such that, after each point, the volume of the convex hull is maximized.

In other words, the volume of the convex hull is a function of the number of points V(k) (V(1) = V(2) = 0), I'd like to choose the sequence of points {k_1, k_2, ..., k_N} such that V(k) is maximized for each value of k.

The two algorithms I've thought of don't seem very efficient:

Brute force is O(N!)
Divide the space into boxes, bounded by the k points. Pick a point, then look in each of the little boxes, ordered by how far away they are from the box that contains the starting point. If there are multiple points in the box you choose, either randomly pick, or choose the point that's the furthest away. Now compute the centroid of the points you have. Then repeat until all points are consumed. This seems like O(N^p).

Is there a cute, out of the box solution to this problem? Or a paper where someone has solved it already?

Genetic algorithms are often a good option for optimization problems, assuming it's not a requirement for the solution to be the best one but only to be "close enough".


Plot Decision boundary in 3D plot I am running logistic regression on iris dataset.  I computed thetas and this is how I draw a decision boundary line.

x_values = ([min(X_train[:,0]), max(X_train[:,0])])
y_values = - (theta[0] + np.dot(theta[1], x_values)) / theta[2]
plt.scatter(X_train[:, 0], X_train[:, 1], c=y_train, cmap=plt.cm.Set1, edgecolor='k')
plt.plot(x_values, y_values )

I tried this, but the result is odd.

X= np.c_[ X_train, np.zeros(100) ]

theta = theta.reshape(3)

d=0

xx, yy = np.meshgrid(np.arange(np.min(X_reduced[:, 0]), np.max(X_reduced[:, 0])), np.arange(np.min(X_reduced[:, 1]), np.max(X_reduced[:, 1])))

z = (-theta[0] * xx - theta[1] * yy - d) * 1. / theta[2]

fig = plt.figure(1, figsize=(8, 6))
ax = Axes3D(fig, elev=-150, azim=110)

ax.scatter(X[:100, 0], X[:100, 1], X[:100, 2], c=y_train, cmap=plt.cm.Set1, edgecolor='k', s=40)
ax.plot_surface(xx, yy, z, alpha = 0.5)
plt.show()

I guess d in plane equation (ax+by+c*z = d) shouldn't be equal to 0. So I'm completely confused about this.

This answer is borrowed from-

https://stackoverflow.com/questions/36232334/plotting-3d-decision-boundary-from-linear-svm

from sklearn.svm import SVC
import numpy as np
import matplotlib.pyplot as plt
from sklearn import svm, datasets
from mpl_toolkits.mplot3d import Axes3D

iris = datasets.load_iris()
X = iris.data[:, :3]  # we only take the first three features.
Y = iris.target

#make it binary classification problem
X = X[np.logical_or(Y==0,Y==1)]
Y = Y[np.logical_or(Y==0,Y==1)]

model = svm.SVC(kernel='linear')
clf = model.fit(X, Y)

# The equation of the separating plane is given by all x so that np.dot(svc.coef_[0], x) + b = 0.
# Solve for w3 (z)
z = lambda x,y: (-clf.intercept_[0]-clf.coef_[0][0]*x -clf.coef_[0][1]*y) / clf.coef_[0][2]

tmp = np.linspace(-5,5,30)
x,y = np.meshgrid(tmp,tmp)

fig = plt.figure()
ax  = fig.add_subplot(111, projection='3d')
ax.plot3D(X[Y==0,0], X[Y==0,1], X[Y==0,2],'ob')
ax.plot3D(X[Y==1,0], X[Y==1,1], X[Y==1,2],'sr')
ax.plot_surface(x, y, z(x,y))
ax.view_init(30, 60)
plt.show()


How can we apply PCA to reduce dimensionality of a heterogenous dataset? I have a dataset containing insurance Claims with quantitative and qualitative variables but PCA refuses to convert or work with "string" type variables.

This is the code I used :

from sklearn.decomposition import PCA
claims=pd.read_csv('./insurance_claims.csv',sep=',',header=0)
X=claims.ix[:,1:].values
pca=PCA(n_components=12)
pca.fit(X)

I'm trying to reduce dimensionality to cluster the dataset and detect fraudulous claims. If there is any alternative to PCA for heterogenous data much appreciated.

Yes, Since machines cannot quantify qualitative (or) sometimes called categorical data. We manually have to quantify data when processed along with quantitative data. One way of converting these categorical data is One hot encoding, By performing one hot encoding, you convert categorical data into numerical data, something that a machine can understand
There are some cases where one hot encoding fails. When the number of categories are high, you will get highly sparse features which makes no sense Ex: Countries as categorical labels. In these types of situations you can calculate distance of all countries from a certain point on the earth Ex: Calculating the distance of each country from equator. I'm just a beginner in DS.


Prediction approach on unique data or progressive data In a employee attrition analysis with a table having rowwise data for a (employee like Id, name, Date_Join Date_Relieving Dept Role etc)

eID eName    Joining Releiving Dept Married Experience
123 John Doe 10Oct15 12Oct16   HR   No      12
234 Jen Doee 01jan16 -NA-      HR   No      11         (ie she is available)

I can run regression on this data to find the beta coefficients

eID eName    Joining Releiving Dept Married Experience
123 John Doe 10Oct15 12Oct16   HR   No      12
234 Jen Doee 01jan16 -NA-      HR   No      11

But I've seen other approach too..
where employee have multiple entries depending on their difference between joining date and current month or relieving month(say Employee A joined in Jan and Left in Dec so he'll have 12 entries updating corresponding columns like experience and marriage etc)

eID eName    Dept Married Experience
123 John Doe HR   No      0
123 John Doe HR   No      1
123 John Doe HR   Yes     2
123 John Doe HR   Yes     3

can someone tell what differentiate two approaches.. and what are the benefits of this second approach.

Creating model on second scenario is difficult but more information driven and give you more prediction capibility.

What you can do is use second scenario and create some featured variable. These variable will give information like after marrying in how many months attrition happen and so on.

So think what other info you can take out of this trend.

First scenario is not of much use, because it's just giving you last information of employee which hides many important relevant information which are actually driving attrition.


Predict if user will come the next day or not im novice in data science field,
i have a problem statement: to predict which users will come online the next day.

i have two datasets of the user:

1st dataset of the user ( master table ) which has the overal information of the user:

uid totlike totview totshare last_social_activity   last_service_activity   last_active_date    most_active_day   is_active( True if(last_active_day - present date )< 30 )
1   200     500     115      like                   see_news_feed           11/12/2017           Monday             TRUE
2   300     600     2223     share                  see_dashboard           11/13/2017           Monday             TRUE
4   500     237     311      comment                see_news_feed           11/14/2016           Wednesday          FALSE

totlike - total like
totview - total view
totshare- total share

2nd data set of the user ( everyday activity ): Historical Data of user activity with time stamp.

date       uid  total_like  total_view  total_share last_social_activity    last_service_activity
11/4/2016   1     10        20         1            like                        see_news_feed
11/4/2016   2      9        25         2            share                       see_dashboard
11/4/2016   4      1        23         3            comment                     see_news_feed
11/4/2016   3      3        14         4            like                        see_dashboard
11/8/2016   1      4        14         5            comment                     see_news_feed

what is the right way to predict if the user will come tomrrow?

I think that you should stop thinking about modeling and instead start thinking about the meta datasets that you can draw from what you already have. I have also downvoted the previous answer you received because it's making the same mistake.

One of the most consistent mistakes I see in data science (by all levels of scientists) is that they skip the data transform/wrangling stage and just go straight into modeling. That's a MASSIVE mistake and the answer you seek starts well within this stage.

So, I want you to take a step back and think about your data. What do you need to know about a user? What are some of the features of each user? What are the features of when they visited? What day? What time? Was it the morning or the afternoon? How many days in a row was it? What is the gender of the user? And so on and so on.

Your goal is to create a dataset that has one line per user and the same features for all users. From there, you'll be in a better position to make better decisions about modeling and anything else you want to know about your data.


How do I choose a laptop if I'm interested in learning and applying data science? I just got started with learning data science, and I was wondering what type of laptop I'd need to buy. I understand that this might come probably a little too early -- but my current laptop absolutely needs to be replaced and I don't want to buy a new laptop only to find out about a year later that I can't apply what I've learnt.

So, what are the minimum specs I should be looking out for? I've read about Amazon's AWS that allows us to create and access virtual machines using the cloud -- is that a good idea, especially considering that I'm only starting out?

I won't be using a laptop for any other intensive activities such as gaming or video-editing. Should I get a high-end laptop or stick to a basic model and just use cloud based services for my data science needs? If the latter (buying a basic model) is a better choice, does that also need certain basic hardware requirements?

This is a bit off-topic and very subjective. It is subjective as it depends on what you want to achieve. I am answering because I have been in this journey for the last 2 years and I can confidently comment

  You certainly DO NOT need a high-performing laptop to start off with data science/machine learning.

based on the following reasons (at least not for the first year or two):

Building the foundations of machine learning, statistics and programming takes longer time that you may think (your objective is solely learning), till you get to do heavy deep learning optimization on images etc.
Even if you get to heavy tasks earlier, there is increasing availability of free clouds like Kaggle, Google Colab etc. that you can run heavy jobs. Yes, runtime on those clouds are at present limited (e.g. as of now it is 1-hr GPU computing at Kaggle), but this will change really soon.

Thus, you are good with a laptop with 1TB hard-disk, 16GB RAM and Core i7, surely depending on your budget lower specification like mine (MacBook Pro 256GB, 8GB RAM, Core i5) also works just fine, unless you have more money to spare.

P.S.: I admit not being able to work on some image-based ML competitions in Kaggle due to limited computing power, but that time (~ a year ago) free clouds were less available.

By the way, Welcome Onboard!


Non-brute force approach to finding permissible English word anagrams I am interested in proposed base R/Python solutions (i.e. without relying on external packages/libraries) to the following problem:

  You have a large dictionary containing millions of valid English
  words. You are given an input “word”, which may or may not be an
  anagram of one or more words in the dictionary. Find all possible
  valid words in the dictionary of which the input may be an anagram.
  Note, the input “word” may be meaningless. For example,

  input: lloeh output: hello

  input: cksli output: slick, licks

  Rather than use a brute force technique that computes all possible
  character combinations for each input target and compares them to a
  dictionary, you want to use a more efficient technique.

The approach I was considering was to pre-compute impossible string onsets in the target language. For example, in English, the onset sequences ^ck and ^ng are not permissable because they violate the orthographic and phonotactic constraints of the language. Given an off-line list of impossible_sequences at run time you would incrementally build possible "words" from the target. During each iteration the possible word is checked against the impossible_sequences list and if it matches the search
down that search branch is terminated. The following pseudo-code in R uses for loops to illustrate my idea.

target <- "cksli"
impossible_list <- c("ck", "ng", "rt", "zz") # pre-computed not-possible English onsets

for (i in 1:nchar(target)) {
  word <- substr(target, i, i)
  if (word %in% impossible_list) break
  for (j in 2:nchar(target)) {
    word <- paste0(word, substr(target, j, j))
    if (word %in% impossible_list) break
    for (k in 3:nchar(target)) {
      ### same logic
      ### goes deeper and deeper
    }
  }
}

(1) Are there any issues with the underlying logic of my proposal?
(2) Obviously nested for loops are inelegant - Could one do this recursively and if so, how?
(3) What other approaches should be considered?

A simple solution is store the words in a dictionary (since you have to store them in some data structure) whose key is the character distribution; e.g., Counter("hello") = {h: 1, e: 1, l: 2, o: 1}. Querying the dictionary would give you the anagrams.

For storing the immutable key (character distribution) can either use a tuple list, incurring a sort, or you can use a vector the length of your alphabet (26). So you can make a speed-space trade-off in the preparation stage; the look-ups are constant time, not counting the time it takes to calculate the character distribution of the query word. If you go the latter, fixed-width route, you can make another trade-off by hashing the key, since you know the cardinality of the input (number of unique words).


How to optimising caret for sensitivity I am currently using caret which optimises for accuracy. Is it possible to optimise for sensitivity. I see documentation mentioning that metric = 'roc' has been used, but sensitivity does not appear to be one of the options.

Probably.  It depends on what you are doing exactly, and in particular which summaryFunction you are using.  If it's twoClassSummary for common binary classification then you should be able to specify metric='Sensitivity'.  If that's not the case you also have the option of overriding the default performance summaryFunction with your own implementation that calculates the metrics you desire.

I highly recommend reading through the documentation at http://topepo.github.io/caret/training.html#control


Dataset recommendation, ordinal target variable Does anyone have any recommendations for datasets with an ordinal target?
I'm currently using a rounded target from the Boston dataset.
from sklearn import  datasets
boston = datasets.load_boston()
X, y = boston.data, boston.target

y -= y.min()
y_ord = np.round(y)

Not sure what you are after, but Kaggle has many datasets you can use. You can use this search to get started:
https://www.kaggle.com/search?q=ordinal+datasetSize%3Asmall+datasetSize%3Amedium
another source:
https://www.gagolewski.com/resources/data/ordinal-regression/


"Pip install tensorflow" only installs version 1.14 – how to install version 2 (with eager mode enabled?) The question is already in the title: when installing tensorflow in command line (in a virtual environment or not...), it is always tensorflow version 1.14.0 which is going to be installed. Consequently, eager execution does not work, I get error messages ("Tensor Object has no attribute numpy..." etc.). However, the tensorflow documentation states that it should the recent tensorflow 2 version which is in fact the one I need. I would be glad to read your advice on this issue.

Kehrwert

You install the version you want with:

pip install tensorflow=={version you want}

for example:

pip install tensorflow==2.0.0-beta1

If you are working in Google Colab it's even simpler, just type:

%tensorflow_version 2.x
import tensorflow as tf

and it will automatically import the latest version of TensorFlow 2.

Eager mode is default in TensorFlow 2.x, therefore you don't need to manually activate it. As soon as you import it it's immediately available.


Set value of a column based on values of other columns (Pandas) The last time I used Python is some time ago so some things have faded away.

I have a given dataset, with multiple columns. I want to create a new column and set the values based on multiple values (text or value) of other columns.

So in the example below, c1 consists of [a,a,b,b] and c2 of [a,b,a,b]. Now I want the new column c3 to be [1,2,3,4]

All help is appreciated!

import pandas as pd
data = pd.read_excel('data')

data['c3'] = 0

for i in range(len(data.c3)):
    if data.c1[i] == a and data.c2[i] == a:
        data.c3[i] == 1
    elif data.c1[i] == a and data.c2[i] != a:
        data.c3[i] == 2
    elif data.c1[i] != a and data.c2[i] == a:
        data.c3[i] == 3
    elif data.c1[i] != a and data.c2[i] != a:
        data.c3[i] == 4```

I would use apply. It allows you to perform a function row-wise or column-wise. Check this post on StackOverflow, it provides good examples of what this looks like.

The gist is:

create a function that creates the value you want based on a row
use apply to create a dataframe containing the results
if you're happy with the results, merge that dataframe into your existing one


In Keras library, what is the meaning of "nb_steps_warmup" in the DQNAgent Object initialization? I can't understand the meaning of "nb_steps_warmup", a parameter of the __init__ function of DQNAgent class of the Keras_RL module.

I just know that when I set small value for "nb_steps_warmup" the command line prints: UserWarning: Not enough entries to sample without replacement. Consider increasing your warm-up phase to avoid oversampling!

This is my code:

import numpy as np
import gym
import gym_briscola
import argparse
import os

from keras.models import Sequential
from keras.layers import Dense, Activation, Flatten
from keras.optimizers import Adam

from rl.agents.dqn import DQNAgent
from rl.policy import BoltzmannQPolicy
from rl.memory import SequentialMemory
import logging

def getModel(input_shape, nb_actions):

    # Next, we build a very simple model.
    model = Sequential()
    model.add(Flatten(input_shape = input_shape))
    model.add(Dense(nb_actions, activation = 'relu'))
    for i in range(2):
        model.add(Dense(2, activation = 'relu'))

    model.add(Dense(nb_actions, activation = 'relu'))
    # print(model.summary())
    return model

def init():

    ENV_NAME = 'Briscola-v0'

    # Get the environment and extract the number of actions.
    env = gym.make(ENV_NAME)
    env.setName("Inteligence")
    env.cicle = True
    nb_actions = env.action_space.n
    window_length = 10

    input_shape = (window_length, 5)

    # Finally, we configure and compile our agent. You can use every built-in Keras optimizer and
    # even the metrics!
    memory = SequentialMemory(limit=50000, window_length=window_length)

    #Maxwell-boltzmann distribution
    policy = BoltzmannQPolicy()
    dqn = DQNAgent(model=getModel(input_shape, nb_actions), nb_actions=nb_actions, memory=memory, nb_steps_warmup=11,
                   target_model_update=1e-5, policy=policy, dueling_type='avg', enable_dueling_network=True)

    print("Compila")
    dqn.compile(Adam(lr=1e-5), metrics=['mae'])

    try:
        dqn.load_weights('dqn_{}_weights.h5f'.format(ENV_NAME))
    except OSError:
        print("File non trovato")

    return dqn, env, ENV_NAME

def startTraining():

    dqn, env, ENV_NAME = init()

    print("Fit")
    dqn.fit(env, nb_steps=5E6, visualize=False, verbose=1, log_interval=1000)

    # After training is done, we save the final weights.
    dqn.save_weights('dqn_{}_weights.h5f'.format(ENV_NAME), overwrite=True)

def startTest():

    dqn, env, _ = init()

    print("Test")
    # Finally, evaluate our algorithm for 5 episodes.
    dqn.test(env, nb_episodes=10, visualize=False)

#Log config
def setLogging(show = True):
    logging.getLogger("Briscola").propagate = show
    logging.getLogger("IA").propagate = True
    logging.getLogger("Client").propagate = show
    logging.getLogger("Vincitore").propagate = show

    logging.basicConfig(level=logging.INFO)

if __name__ == "__main__":

    #Parameter settings
    parser = argparse.ArgumentParser()
    parser.add_argument("-m", "--modality", help="The modality of the program", const="train", nargs='?')
    parser.add_argument("-l", "--logging", help="Enable logging", type=bool, const=False, nargs='?')
    args = parser.parse_args()

    setLogging(True)
    print("Avvio modalita' ", args.modality)

    if args.modality == "test":
        startTest()
    else:
        startTraining()

    print("Fine")

This is the doc of the module: https://keras-rl.readthedocs.io/en/latest/agents/dqn/

I hope I was clear with my English.

Often times in reinforcement learning the error rate of the first few steps will be very large and may cause your parameters to oscillate. This is usually attributed to the lack of specificity of the deeper layers in your network. Thus we can come up with some schemes where the learning rate changes in a pre-determined way. For example we can use constant warm-up or gradual warm-up.

The convergence of stochastic gradient descent is a function of the learning rate and the batch size. When the batch size is increased too much then the needed increase in the learning rate can be such that it is beyond the possible curvature of the loss function. We thus introduce warm up as a means by which we can introduce large learning rates without the instability.


Distributed Scalable Decision Trees Are there any good sources that explain how decision trees can be implemented in a scalable way on a distributed computing system.  Where in a given source is this explained?

Apache Spark can do it, using the new MLLib library. Here's a presentation, and here are some benchmarks. Bindings are available for python, scala, and java.


Use MinMaxScaler , label encoder, one hot encoder , keras prediction file for later prediction I'm new to neural networks and data science field. I have a dataset with over 90,000 rows. which Include 9 text columns & 29 Number Columns. after encoding with label encoder and one hot encoder It has over 10,000 columns.  Now I like to save those scalers, encoders and prediction files for later use.  But I have no Idea how to save and use them later for single prediction. Any help is appreciate.  Thank You

You could use pickle to store your encoders/scalers/etc. It is a common way of storing Python objects

from sklearn.preprocessing import LabelEncoder
import pickle

# Fit a label encoder
le = LabelEncoder()
X = le.fit_transform(X)

# Pickle the encoder for later use
with open('path/and/name', 'wb') as f:
    pickle.dump(le, f)

Then when you have it stored it can be used again by loading the pickle

# Read the pickle from file
with open('path/and/name', 'rb') as f:
    le = pickle.load(f)

# Use the already fitted encoder to transform new data
X = le.transform(X)

Then repeat the process for your all your preprocessing objects


collaborative filtering using graph and machine learning What are the advantages and disadvantages of using Collaborative filtering based recommendation using machine learning approach and graph based approach ?

Say I have user purchase data (user_name, user_location, user_company_name, product_name, product_price, product_ingredients) and would like to recommend product for user based on what other user from the same location, company are buying, based on product price, ingredients etc.

How to decide on which of them is suitable for a given use case?
I would like to evaluate Neo4j (Graph database) and Mahout (Machine learning) for Collaborative filtering.

I can only speak about Graphs:

Advantages:

Using graphs, you can easily find products bought/rated by users that bought or liked an item, or users that have similar "taste" to another user. From my experience the traversal process is fast enough.
Closely matched products are easy to find, depending on how you model your graph: e.g. (userA)-[]->(basketA)-[with]->(productA)<-(basketB)-[]-(userB). Finding similar products by basket is computationally cheap here

Disadvantages:

Depending on how you rank your results, you can easily run into the trap of constantly recommending the same products. Users that bought A, bought B. So you suggest B. People buy/like B. Next time you run the query, B will keep coming on top.
Your results will rarely uncover new products. You'd have to find smart approaches around this, e.g. find other products, within this price range, with this category
If you have very large data sets, you'll have to use a distributed graph database that performs well (this is not as easy as it sounds, unless you're willing to pay large sums of money)

My little experience with ML for collaborative filtering, is that when your data grows large (50GB+), building a model takes a considerable amount of time (hours, days), and you're not likely to get good recommendations on new products. Having to update your model becomes a huge problem too. From my experience, I lean towards graphs for small use cases.

Note that for both cases, you never recommend newly added products that have not been bought or liked by someone. I say this because the goal of recommendations is to help users uncover new products. There is a blog post I wrote, where I talk about modelling graphs, and I discuss recommendations in some examples here, so you can skip to that section to get an idea on how to model this kind of problem.

I advise you to read up on Amazon's paper on their approach to collaborative filtering, which is pretty simple, in theory, and yields good results for them. I think you'd always want to implement a few different approaches combined, to tackle different parts of the problem.

Good luck.


Detection of anomal data in the text I work with texts where there is a dialogue between two people (a client and a call center employee, the beginning and end of each person’s phrase is not defined). My goal is to classify texts in which a call center employee names words from my list.
If the texts are manually marked up, can such a classification problem be solved?
Are there any tricks to solve this type of problem?

Sample data:
"hello hello my name is Sam Chin I'm calling for pizza delivery Okay now check your order wait a minute Sam"

There's a lot of ways to do this, one approach is to use token-based matching. You can use this to easily find any "tokens" in the text, like names, places, or just plain words.

Methodology

I'd recommend using Rule-based Entity Recognition in spaCy. You'll define the "rules" of what the entity looks like, here's the example from the docs where we define the following patterns to find:

An entity type of Organization and the word Apple
An entity type of Location and the words san and francisco

Here's that in code (live example):

from spacy.lang.en import English
from spacy.pipeline import EntityRuler

nlp = English()
ruler = EntityRuler(nlp)

# These are the rules you define, look at the docs to see what your options are.
# You don't have to use the "label", you can just look for a "pattern" if you want.
patterns = [{"label": "ORG", "pattern": "Apple"},
            {"label": "GPE", "pattern": [{"LOWER": "san"}, {"LOWER": "francisco"}]}]

ruler.add_patterns(patterns)
nlp.add_pipe(ruler)

# The text you're searching through to find your patterns
doc = nlp("Apple is opening its first big office in San Francisco.")

# This prints out the matches
print([(ent.text, ent.label_) for ent in doc.ents])

The output of this code is: [('Apple', 'ORG'), ('San Francisco', 'GPE')]

Usage

Thankfully spaCy has some fantastic online tools for helping you write your patterns, I highly recommend you check these links out.

Install spaCy
Evaluate if you should use rules or a model (I suggest rules but I could be wrong), and if you should use token matcher or phrase matcher
Read the documentation to determine how to write your patterns
Test your patterns using the Live Rule-based Matcher Explorer
Use the code snippet above or the code samples in the docs to see how to use your new patterns


Why does an attention layer in a transformer learn context? I understand the transformer architecture (from "Attention is All You Need"), as well as how the attention is computed in the multi-headed attention layers.
What I'm confused on is why the output of an attention layer is a context vector. That is to say: what is it about the way that a transformer is trained causes the attention layers to learn context? What I would expect to see in the paper is a justification along the lines of "when you train a transformer using attention on sequence-to-sequence tasks, the attention layers learn context and here's why...". I believe it because I've seen the heatmaps that show that attention between related words, but I want to understand why that is necessarily the result of training a transformer.
Why couldn't it be the case that the attention layers learn some other features that happen to also be beneficial in sequence to sequence tasks? How do we know that they learn context, other than that's what we observe?
Again, I get the math and I know there are several posts about it. What I want to know is what about the math or the training process implies that the attention layers learn context.

To provide a simplistic and less mathematical reasons. You can assume like this:
In a simple feed-forward neural network (a black-box of course), you shall learn the set of weights, learning a function to map inputs to outputs.
But, in the transformers based architecture, you have Attentions. Here, the weights are structured into Query, Key and Value (Q,K,V). These 3 set of weights drivers of attention and are responsible to learn the context. How precisely it works still remains a black-box like feed forward networks. But yeah, it works something like this, every token's embedding is transformed its query, key and value vectors using their respective weight matrices. For a given token, its query vector is multiplied with all the other tokens' key vector to obtain a value vector. These values determines the importance of every token with respect to the query token. Thus, with back-propagation, you try to optimize these Q, K, V weights, and thus learn it to better map the relationship between tokens.


If I have negative and positive numbers for a feature, should MinMaxScaler be -1 to 1? I have a variable X with values ranging from -150 to 400. All the other variables in my training set are positive so I normalized them to be from 0 to 1, or they’re already binary, or they had a Gaussian distribution so I used StandardScaler.

For this variable X with some negative values, is it important, generally, that I normalize them to -1 to 1 (due to the negative values) rather than 0 to 1?

In my opinion, it depends on the importance of the meaning of negative sign.

If the sign of the value indicates the direction of the value, then it will be better to retain the negative sign. For example, +1 indicates moving rightward and -1 indicates moving leftward.

In the other hand, if the the negative sign only means it is smaller than 0, it will be fine to normalize them to 0 to 1. For example, the variable recording temperature, negative values only means it is lower than positive values.


What is an interpretation of the $\,f'\!\left(\sum_i w_{ij}y_i\right)$ factor in the in the $\delta$-rule in back propagation? In the $\delta$-rule which is used for error back propagation in neural networks, there is a factor $f'\!\left(\sum_i w_{ij}y_i\right)$, often written as $f'(\text{net}_i)$, which is just the derivative of the activation or transfer function evaluated at the weighted sum of the input. The derivative of $\tanh$ is bell-shaped so that values $\left|x\right|\geqslant3$ it is almost zero:

This means that units with a strong activation in either way have a more limited scope of action, where as values close to zero have more leeway. If I'm understanding it correctly, it keeps neurons that have already strongly decided one way or the other in their configuration. What is the purpose of this for weight updating? How would the networks behave differently if one would remove or change this factor?

As a reminder, the full $\delta$-rule looks like this for hidden neurons and output neurons:

$$\Delta w_{ij} = \eta\;\delta_j\;y_i$$

$$\delta_j = (\hat y_j - y_j)\;f'(\text{net}_j) \quad \text{or} \quad \delta_j = \sum\limits_k^K\left(w_{jk}\delta_k\right)\;f'(\text{net}_j)$$

where $\Delta w_{ij}$ is the change of the weight that connects the neuron $i^{\text{th}}$ neuron in the preceding layer with the $j^{\text{th}}$ neuron in the layer we are currently considering; $\eta$ is the learning rate, $y_j$ is the output of the current neuron, $\hat y_j$ is the teacher value for the $j^{\text{th}}$ output, $y_i$ is the output of the $i^{\text{th}}$, and $k$ is an index in the subsequent layer.

The equation that you show calculates (the negative of) a gradient to the objective function. The value $\delta_j$ is the rate of change of the objective function for an isolated change of output of a neuron before applying the activation function for the neuron. The factor of $f′(net_i)$ is required because it is correct - it is due to the chain rule applied twice - once due to the linear relationship linking weights between layers, and again to get the gradient pre-activation function in the next layer down.

An interpretation of $\delta_i$ is "the (negative) gradient of objective function with respect to a pre-activation value of a neuron the network". If, somehow, the pre-activation value could be changed - ignoring how it could be changed - then $\delta_i$ tells you the linear scale of the impact on the objective function, at least for very small changes.

This value is not normally used directly in weight updates, but is fed deeper backwards into the networks layers if there are yet lower layers to update, because it can be used for weight updates in those layers using the same rule.

The property of low gradients associated with strong activations is because you previously chose tanh as an activation function, and is not because of the way you calculate the gradient. Once you chose tanh, you were not really free to choose a different backpropagation gradient calculation. Or rather, there is not much to gain - and potentially a lot to lose - by doing so.

  What is the purpose of this for weight updating?

The value shown is not used directly, but used to derive $\frac{\partial J}{\partial W_{ij}}$ gradients for weights in deeper layers (where $J$ is value of your objective function).

In terms of your equations

$$\frac{\partial J}{\partial W_{ij}} = -\delta_j y_i$$

This is very simple, so it has been folded in to your delta update rule. However, doing that is hiding something important. Out of all the intermediate values you calculate during back propagation, $\frac{\partial J}{\partial W_{ij}}$ is the key one. It is the gradient of the objective function with respect to parameters that you can change.

Once you have a set of gradients for  $\frac{\partial J}{\partial W_{ij}}$, this can be used in various ways.  Using this gradient directly for weight updates using the delta rule (as in your equation $\Delta W_{ij} = \eta \delta_j y_i$) is a basic approach. The purpose of this rule is to reduce the value of the objective function by taking a small step in the direction that you have just calculated will reduce the value of objective function. The step size needs to be small (multiplied by a factor $\eta$) because you have only measured the gradient at one point, and don't know for what step size the relationship still holds - complicated by the fact that you are making a similar step in multiple dimensions at once.

The delta update rule is a relatively weak update method compared to e.g. Nesterov momentum, Adagrad, Rmsprop optimisers which will use the calculated gradient as input, but adjust the update weights based on history of previous gradients. These optimisers can deal with low but consistent gradient values and make large updates.

  How would the networks behave differently
  if one would remove or change this factor?

This is not advisable as effectively you would be optimising against a different network architecture than the feed-forward network you had constructed. The most likely way they would behave is to either diverge or settle on a larger value of your objective function than you would otherwise see. Making small changes (such as a minimum absolute gradient) can work, but usually such a tweak is best done consistently, by changing the activation function as well.

In a few edge cases, you might consider "lying" in the gradient calculation. A trivial case could be considered the ReLU activation function ($y = 0 (if x \lt 0), y = x (otherwise)$) where the gradient is technically undefined at $x=0$, but can safely be treated as either 0 or 1.

An alteration to updates that is more usually done (by e.g. momentum variants, Adagrad, RMSProp et al) is to use the gradient as-is, but then multiply update amounts by various factors based on history or estimates of higher-order derivatives, because you can guess that the gradient will not switch sign if you update a weight by a smaller or larger amount. These approaches replace the delta update rule with update mechanisms that are more robust when gradients become small.


Understanding dropout and gradient descent I am looking at how to implement dropout on deep neural networks and found something counter intuitive. In the forward phase dropout mask activations with a random tensor of 1s and 0s to force net to learn the average of the weights. This help the net to generalize better. But during the update phase of the gradient descent the activations are not masked. This to me seems counter intuitive. If I mask connections activations with dropout, why I should not mask the gradient descent phase?

In dropout as described in here, weights are not masked. Instead, the neuron activations are masked, per example as it is presented for training (i.e. the mask is randomised for each run forward and gradient backprop, not ever repeated).
The activations are masked during forward pass, and gradient calculations use the same mask during back-propagation of that example. This can be implemented as a modifier within a layer description, or as a separate dropout layer.
During weight update phase, typically applied on a mini-batch (where each example would have had different mask applied) there is no further use of dropout masks. The gradient values used for update have already been affected by masks applied during back propagation.
I found a useful reference for learning how dropout works, for maybe implementing yourself, is the Deep Learn Toolbox for Matlab/Octave.


Is it possible to use Word2vec for text paraphrasing? After reading several papers I am not sure if it is possible to some how generate text with the same meaning (paraphrase it) using only Word2vec.

I found out other approaches that use sequences of sentence pairs, and they train Neural nets to find the most similar, but this is hard to maintain and it will be hard to generate relevant content like this.

I would like to give raw text to Word2vec powered algorithm that gives paraphrased text.

Someone has defined a problem for paraphrasing on Tensor2Tensor:  https://github.com/tensorflow/tensor2tensor/releases

You can also define your own problem but you might have to supply the corpus:  https://tensorflow.github.io/tensor2tensor/new_problem.html

If you are just looking for a vectorized representation of your sentences, Google BERT might be worth looking at.  Bert-as-service is quite convenient:  https://github.com/hanxiao/bert-as-service

Perhaps you could reframe your problem slightly as a question-answer type problem for which I think both T2T and Bert are well equipped to handle. The questions could be of type, "The three little pigs each had their own home?"  For which an answer paraphrasing the question could be, "Each of the three little pigs had their own house."  (But I'm taking a guess here).  Maybe there is a way to tweak the problem so you don't have to include a question mark?  Then it would be a paraphrasing task...

I'm sorry I can't provide you with a more professional answer but I think T2T is worth looking at for sure.


Categorical vs continuous feature selection/engineering I'm working with a dataset with a number of potential predictors like :

Age : continuous

Number of children : discrete and numerical

Marital Situation : Categorical ( Married/Single/Divorced.. )

Id_User : Categorical ( an id of the user who conducted the first interview with this person )

I'm stopping at four potential predictors, there are more, but for the sake of shortness, these would be enough to ask my question.

Question : Continuous features are easy to deal with, normalize, and feed it to the model, what about categorical and independant ?

Note : I get that categorical features that follow a certain pattern can be encoded as integers and fed to the model, but what if those categorical features have no meaning as integers ( 1 for single, 2 for married , 3 for divorced ; for the model that treats it as a quantitative predictor it doesn't make sense to feed it to it like that)

Any ways to deal with these different types of features?

For encoding categorical features, there is two common ways:

  Ordinal encoder

This is the way you mentioned as 'encoded as integers'. In this method, an integer starting from 0 is assigned to each category. The problem of this method is that it randomly prioritize categories. So in cases when there is no priority among categories, this encoding is meaningless as you mentioned. The only case it work is when assigning larger integer to some categories is meaningful.

  One-hot encoder

This method makes a feature vector (one-hot vector) for each categorical feature which is the same size as the number of categories. The method assigns each component of the vector to one of the categories. For each data sample, it assigns 1 to component which its corresponding category is present at the sample and assigns 0 to other components. The benefit of this method is that unlike ordinal encoder it does not prioritize any category.

So in your case, I highly recommend that you use one-hot encoder.


Can macro F1 score be greater than micro F1 score? I am reading about evaluation metrics, and it seems that micro scores are more useful. But I was wondering about scenarios where macro F1 score is greater than micro F1 score, and if this is at all possible.

Yes, it can.

Example with Precision

Class A: 1 TP and 1 FP
Class B: 10 TP and 90 FP
Class C: 1 TP and 1 FP
Class D: 1 TP and 1 FP

Here, $P_A = P_C = P_D = 0.5$, $P_B = 0.1$

Macro-F1 is: $P_M = \frac{P_A + P_B + P_C + P_D}{4} = 0.4 $,
while micro-F1 is: $P_m=\frac{TP_A+TP_B+TP_C+TP_D}{TP_A+TP_B+TP_C+TP_D+FP_A+FP_B+FP_D+FP_D} = 0.123$, hence $P_M > P_m$.


What is the use of applying img_to_array() after cv2.imread() In a book, I saw the following code to load images from a directory:
1.image = cv2.imread(imagePath)
2.image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
3.image = cv2.resize(image, (28,28))
4.image = img_to_array(image)
5.data.append(image)

cv2.imread()

It converts image to a numpys array in BGR format

img_to_array()

It converts PIL instance to numpy array, but in this case image will stay in BGR format as the image is loaded by cv2.

What is the use of img_to_array() (from keras.preprocessing.image module) after cv2.imread(), as it is already in numpy array.
Also:
To double check, I displayed images before and after applying img_to_array() using cv2.imshow().
BEFORE applying img_to_array()

AFTER applying img_to_array()###

Also Also:
But, if I try to save the images using cv2.imwrite into a file, they both are saved as normal (ie. like the first image). Why is this happening?

This is because of the data type you provide to the imshow() function.
Check the documentation:

The function may scale the image, depending on its depth:
If the image is 8-bit unsigned, it is displayed as is. If the image is
16-bit unsigned or 32-bit integer, the pixels are divided by 256. That
is, the value range [0,255*256] is mapped to [0,255]. If the image is
32-bit or 64-bit floating-point, the pixel values are multiplied by
255. That is, the value range [0,1] is mapped to [0,255].

So when you use cv2.imread() it reads data uint8, that is 8-bit unsigned format thus imshow() displays the original data.
But when you use img_to_array() it converts data to the float. Thus imshow() multiplies it with 256. It expects the float values in normalized form, that is all the pixels are divided by 256. That is the reason for such a display.
This is how cv2.imread() data looks like...

and its data type...

This is how img_to_array() data looks like...

and its data type...

If you divide your image by 1.0 to convert it to the float...
import cv2
# from keras.preprocessing.image import img_to_array
image = cv2.imread("car.jpg")
image = image/1.0
cv2.imshow("Divided by 1.0", image)
cv2.waitKey(0)

...this is what you will get:

Same as img_to_array.
But it you divide image with 256.0 as follows:
import cv2
# from keras.preprocessing.image import img_to_array
image = cv2.imread("car.jpg")
image = image/256.0
cv2.imshow("Divided by 256.0", image)
cv2.waitKey(0)

You get the original image since imshow() multiplies the float with 256.

So what you need is to divide your img_to_array() output by 256 or convert it to the uint8.


Machine learning problem with only original data without test and validation data I am new to machine learning and I am trying to solve a problem where I have to predict if a customer will buy a home insurance product or not.

I have got a dataset which tells me that which of the bank's customer bought a mortgage from the bank.
I have got another data of the customers who bought the mortgage first, then the bank ran a campaign to provide them with home insurance randomly and this dataset tells me that which of the mortgage customers actually bought home insurance from the bank.

Now my job is to predict which customers should I pick for the bank that will have the highest possibility to subscribe to the home insurance product.
I do not have a separate train/test/validation dataset, but just one dataset. How do I  approach this problem? Should I create the validation and test data from my original dataset that I have been given? how should I approach this problem to predict correctly?

Usually, with a predictive model, you generate a train and test sample from the original dataset, e.g. by using sklearn.
import numpy as np
from sklearn.model_selection import train_test_split
X, y = np.arange(10).reshape((5, 2)), range(5)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)

https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html
Train some model on one set and test on the other by comparing predictions to actual (true) outcomes.
You could also use cross validation, in which case a model is trained on one part of the data (e.g. 4/5 of the data with 5-fold cv) and tested on the remaining 1/5 of the data. This is done for all of the „folds“. See Ch. 5.1. of „Introduction to Statistical Learning“ for more details on that. https://www.statlearning.com/
Simply make sure you obtain a test score based on data not used for training.


Incorrect Text Classification, But Accurate Model. Do I Perform Manual Text Classification For A Data Set? I'm currently using Google's BERT pre-trained sentiment analysis model that is trained on an IMDb pos/neg review dataset. I'm using this model to predict whether tweets are positive (bullish) or negative (bearish). While the model is accurate when plugging in my own test data (F1 Score ~86%), the classification itself is not accurate. Tweets that are undoubtedly positive/bullish, and not classified as so. Perhaps this is because the language in the investment world is different than a movie review - which uses universally recognized positive/negative words and/or sentences.

The same is true when I take my tweet dataset and use Vader SentimentIntensityAnalyser to parse pos/neg tweets into separate folders.

So my question is... since the language that is used for telling whether a stock is bullish/bearish is uniquely different from that of an Amazon review, or movie review, would it be optimal for me to manually classify my dataset into positive (bullish) and negative (bearish) datasets?

There can be two distinct reasons to use instances annotated with the gold-standard class, i.e. the true answer for the target application:

In order to perform proper evaluation your test set must contain the gold-standard labels. The principle of evaluation is to measure by how much the predictions deviate from the truth, but without the truth the performance that you obtain on the test set is meaningless for the task that you are doing.
In order to train a supervised or semi-supervised model, the training set must contain the gold-standard labels.  Semi-supervised methods offer some options to adapt a training set to a different task.

You can't rely on a model if you can't evaluate it at least on a small sample, so yes you probably need to manually annotate a subset of the data. It's only after that you can start thinking about how to improve performance.


How to encode positional information of a string in a NN effectively So I want to compare strings and find out there relation to one another. For this I figured out I also have to encode the positional info. For eg:
aaaaassssssbbbbbb should be more relevant to aaaasssssbbbbrather than bbbbaaasssss. The output will be a relative measure and 1 and 2 should match more compared to 1 and 3. Also I will represent each character by a number.

So how can I re-design my feature vector (feature transformation) to be input in a NN to capture this information effectively?

If you just want to compare the letter and position, you could just use the strings as lists of characters and either compare up to the index of the shortest list or by sliding the shorter list over the longer list and grabbing the average number of matches.

Depending on your needs though I would recommend using more than just the positional data in the string. Two approaches I would probably try in addition to the above.

You could try converting the string to m length subsequences and then compare or aggregate in some way the number of matching or similar subsequences.
You could try converting the string to a regex and then compare the regex similarity.


Fluctuating accuracy of Autoencoder I am working on an Autoencoder in keras with the following setting: 185-86-32-2-32-86-185. The problem is that its accuracy is fluctuating. Also, it is giving a new accuracy at every run.
If this is due to the random uniform kernel initialization, how can I save the output values of the initializes in order to use them later?
My code:

Accuracy makes no sense for an autoencoder, because you're essentially doing regression (from X to X), you should use a different metric instead, like MSE, MAE, RMSE.


Missing features for classifier If I am given 60 features along with test label and I was to find values of other features what is the best way to do it ?

A good answer to this question has to rely on the specific dataset / domain.

The questions I would ask myself are (in this order):

Can I solve my classification task without those features? → Just remove those features
Do I know of a relationship between features I know and the (partially) missing feature? → Find hard rules to fill those missing ones
Otherwise: Train a classifier / regression model to fill those missing features


Many questions training unbalanced and duplicated data I'm a DS student. I have like 30.000 of bank statements, all labeled with a specific category(cat1, cat2, ...). With that data I'm trying to train a classification model but I found several problems:

The category that has more statements has 10k, the one with less 100 (unbalanced dataset)
Many rows are repeated with the same text ( for example "Buy for a total of X$ in the supermarket Y", but the repeated rows appears in each label)

Should I train with repeated statements? if I do so the model will be trained for some specific cases and the predicted results will not be real because for example if I have repeated statements with the same label:

A, B, B, B, B, B (labels)
and I take only one row with label A and B, since the statement will be repeated the accuracy predicting the others B's will be 100%.

If I do not train with repeated statements am I not deleting important information about my data and can affect the prediction later?
Should I undersampling or oversampling? It's correct for this kind of data? I will have more repeated rows in the case of oversampling, and undersampling I'll lose a lot of information.

I do not know if what I'm thinking has some sense. I really need some tips in order to train my data.

Repeated entry is as good as you are adding weighted samples i.e. those samples will have extra weights in the learning processes.
 So, you should look into the data and decide if that weight is valid for those samples. Otherwise, it might overfit the Model
On Oversample, Undersample
 This is a very broad question. Please do read some relevant blogs/SE posts.
 You should also try
- Both, Over-sampling and Under-sampling together i.e. Hybrid approach
- Adding appropriate class-weight to the learning i.e. Cost-sensitive learning
You may start here -
Python Library
Blog
Blog
Book


why do we have to calculate the entropy of parent node in Information Gain? Why do we need the entropy of parent node in the Information Gain.
Information Gain = entropy(parent) - w * entropy(children)

We can compare the entropy of the children without the need for the parent entropy.

It's essential; you're computing gain from the parent to the same data split in the children! Not comparing children. A good split takes a high-entropy data set (lots of several different labels) and turns it into lower-entropy data sets (some labels on one child mostly, other labels on the other).


Bag of Words creation in image In SIFT feature extraction how the key points will be generated and how the features will be stored in the database. In image will the bag of visual words be images or text words?

How to calculate key points:

Take your image $I(x,y)$ and convolve it using a Gaussian $G(x,y,k\sigma$) at different scales $k \sigma$ to obtain $L(x,y,k\sigma)$. This produces several versions (one for each scale $k\sigma$) of your image $I(x,y)$ with different degrees of blurring.
Separate your blurred images according to octaves (an octave is usually taken as $2\sigma$). Within a given octave, take the difference between adjacent blurred images $L(x,y,k_{i}\sigma)$ and  $L(x,y,k_{j}\sigma)$. This difference is called a Difference of Gaussians (DoG) $D(x,y,\sigma) = L(x,y,k_{i}\sigma) - L(x,y,k_{j}\sigma)$. At this point, an image will be helpful (source: opencv):

As you can see, you have the blurred/convolved/filtered images on the left and the differences between adjacent images on the right.

Then, take one pixel on a DoG and compare it to its 8 neighbors in the same DoG, as well as to the 9 equivalent pixels of the DoG located in the next scale  and to the other 9 pixels in the previous DoG. In the drawing, this corresponds to the DoG's located above and below. If this pixel is a local extrema, then you have a candidate keypoint. Keep in mind that candidates can be discarded based on other criteria.

Once you have these keypoints, you proceed to generate descriptors. For each keypoint, calculate an image gradient. As you should know, gradients tell you the direction of maximum rate of change. Therefore, you can construct a grid around each keypoint that is oriented according to the dominant gradient around that point. This grid has subregions (usually 16 subregions) and for each subregion, calculate an 8-bin histogram. Finally, concatenate every histogram you obtained from every subregion in your grid and that is the feature vector for that keypoint (actually, the full feature vector also includes the location and rotation angle.) A helpful illustration (from Solem's book Programming Computer Vision with Python):

At this point, you have a feature vector with 132 values for each keypoint. As for your second question, I'm not sure about the best approach to store an array in a database. Maybe others can expand on this point. Of course, there are several options:

Create an array datatype to store your data
Simply use a VARCHAR field
Store it as a binary file
Use a database specifically designed to handle arrays.


How do I make an interactive PCA scatterplot in Python? The matplotlib library is very capable but lacks interactiveness, especially inside Jupyter Notebook.  I would like a good offline plotting tool like plot.ly.

There is an awesome library called MPLD3 that generates interactive D3 plots.

This code produces an HTML interactive plot of the popular iris dataset that is compatible with Jupyter Notebook. When the paintbrush is selected, it allows you to select a subset of data to be highlighted among all of the plots. When the cross-arrow is selected, it allows you to to mouseover the data point and see information about the original data. This functionality is very useful when doing exploratory data analysis.

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sb
import mpld3
from mpld3 import plugins
%matplotlib inline

iris = sb.load_dataset('iris')
from sklearn.preprocessing import StandardScaler
X = pd.get_dummies(iris)
X_scal = StandardScaler().fit_transform(X)

dim = 3
from sklearn.decomposition import PCA
pca = PCA(n_components = dim)
Y_sklearn = pca.fit_transform(X_scal)

# Define some CSS to control our custom labels
css = """
table
{
  border-collapse: collapse;
}
th
{
  color: #ffffff;
  background-color: #000000;
}
td
{
  background-color: #cccccc;
}
table, th, td
{
  font-family:Arial, Helvetica, sans-serif;
  border: 1px solid black;
  text-align: right;
}
"""

fig, ax = plt.subplots(dim,dim, figsize=(6,6))
fig.subplots_adjust(hspace=.4, wspace=.4)
tooltip = [None]*dim

N = 200
index = np.random.choice(range(Y_sklearn.shape[0]),size=N)

for m in range(dim):
    for n in range(m+1):
        ax[m,n].grid(True, alpha=0.3)
        scatter = ax[m,n].scatter(Y_sklearn[index,m],Y_sklearn[index,n],alpha=.05)

        labels = []
        for i in index:
            label = X.ix[[i], :].T.astype(int)
            label.columns = ['Row {0}'.format(X.index[i])]
            labels.append(str(label.to_html()))

        ax[m,n].set_xlabel('Component ' + str(m) )
        ax[m,n].set_ylabel('Component ' + str(n) )
        #ax[m,n].set_title('HTML tooltips', size=20)

        tooltip[m] = plugins.PointHTMLTooltip(scatter, labels,
                                           voffset=20, hoffset=20, css=css)
        plugins.connect(fig, tooltip[m])

plugins.connect(fig, plugins.LinkedBrush(scatter))
test = mpld3.fig_to_html(fig=fig)

with open("Output.html", "w") as text_file:
    text_file.write(test)

See it in action on my blog.

Update [July 9, 2016]:
I just found out that Plot.ly has an offline mode and is now open source. It has a lot of the bells and whistles prepackaged, but MPLD3 may still be appropriate in some cases.


Correct number of biases in CNN What is the correct number of biases in a simple convolutional layer? The question is well enough discussed, but I'm still not quite sure about that.

Say, we have (3, 32, 32)-image and apply a (32, 5, 5)-filter just like in Question about bias in Convolutional Networks

Total number of weights in the layer kernel trivially equals to $3x5x5x32$.
Now let us count biases. The link above states that total count of biases is $1x32$, which makes sense because weights are shared among all output cells, so it is natural to have only one bias for each output feature map as a whole.

But from the other side: we apply activation function to each cell of output feature map separately, so if we will have different bias for each cell, they do not sum together, so the number $0x0x32$ instead of $1x32$ makes sense too (here 0 is the output feature map  height or width).

As I can see, first approach is widely used, but I also saw the second approach in some papers.

So, ($3x5x5 + 1)x32$ or  $(3x5x5 + 0x0)x32$?

As you say, both approaches are used.  It's called tied biases  if you use one bias per convolutional filter/kernel ((3x5x5 + 1)x32 overall parameters in your example) and untied biases if you use one bias per kernel and output location ((3x5x5 + OxO)x32 overall parameters in your example).

Untied biases increase the capacity of your model, so they can be a good idea if you are underfitting. But in this case using tied biases and more filters and/or layers might also help, see https://harmdevries89.wordpress.com/2015/03/27/tied-biases-vs-untied-biases/.


Computing SVD through Eigendecomposition of correlation matrix I am following the excellent series on SVD by Steve Brunton from the University of Washington, on YouTube, but I have trouble interpreting his 4th video on the subject.
If I understand correctly, he mentions that one can compute the economy SVD decomposition $X = \hat{U}\hat{\Sigma}V^T$ with the following :
$$X^TX= V\hat{\Sigma}\hat{U}^T\hat{U}\hat{\Sigma}V^T = V\hat{\Sigma}^2V^T \implies X^TXV = V\hat{\Sigma}^2 $$
$$XX^T= \hat{U}\hat{\Sigma}\hat{V}^T\hat{V}\hat{\Sigma}\hat{U}^T = \hat{U}\hat{\Sigma}^2\hat{U}^T \implies XX^T\hat{U} = \hat{U}\hat{\Sigma}^2 $$
Instead of using U,S,VT = svd(X) in Python, I want to try decomposing the image in its SVD, and reconstructing the original using only $r$ values with this technique. I am trying to apply this on a picture of The Starry Night, loaded in a grayscale numpy array X, which I do this way :
r = 10
XT = X.transpose()
C_1 = XT @ X
C_2 = X @ XT
[Lambda_1, V_hat] = np.linalg.eig(C_1)
[Lambda_2, U_hat] = np.linalg.eig(C_2)
V_hat_T = V_hat.transpose()
X_tilde = U_hat[:,:r] @ np.diag(Lambda_1)[0:r,:r] @ V_hat_T[:r,:]

But I can't reproduce the image, whatever the value I attribute to $r$, whereas the product of U @ S @ VT , computed with the svd(X) function, perfectly reconstructs the original picture.
What am I doing wrong? It is certainly superfluous to mention that I am a beginner, and I probably made some big mistake.

The problem is occurring for a few reasons:

You must ensure the correct ordering of eigenvalues/eigenvectors.
You should be taking the square root of the eigenvalues (Lambda_1) to get the singular values.
You need to take special care to ensure you are using the right signs for the eigenvectors as these won't necessarily be correct by default. This is because we assume that the singular values are the positive roots of the eigenvalues, but the two sets of eigenvectors we retrieve don't necessarily respect this assumption when joined in the singular value decomposition.

Here is the working example:
r = 10
XT = X.transpose()
C_1 = XT @ X
C_2 = X @ XT
[Lambda_1, V_hat] = np.linalg.eig(C_1)
[Lambda_2, U_hat] = np.linalg.eig(C_2)

# The eigenvalues/eigenvectors returned from np.linalg.eig are not sorted so may differ between Lambda_1 and Lambda_2
# When performing economy SVD we also want to ensure the largest eigenvalues come first
i_1 = np.argsort(Lambda_1)[::-1]
i_2 = np.argsort(Lambda_2)[::-1]
V_hat = V_hat[:, i_1]
U_hat = U_hat[:, i_2]

# We must take the square root of eigenvalues to get the singular values
Lambda = np.sqrt(np.sort(Lambda_1)[::-1])

# X @ V_hat and U_hat @ np.diag(Lambda) should be equal. But the eigenvectors retrieved may not have the correct signs
# This code checks if  X @ V_hat and U_hat @ np.diag(Lambda) have different signs and then updates V_hat wherever the sign is incorrect
same_sign = np.sign((X @ V_hat)[0] * (U_hat @ np.diag(Lambda))[0])
V_hat = V_hat * same_sign.reshape(1, -1)

V_hat_T = V_hat.transpose()
X_tilde = U_hat[:,:r] @ np.diag(Lambda)[0:r,:r] @ V_hat_T[:r,:]


Convert non-binary value into binary value? I'm kind of a newbie in ML

I have a dataset like:

282,4,1,1,0,0,1,1,41,0,1,1,1,0,0,0,0,0,0,0,0,1,0,8,4
282,5,1,1,0,1,1,2,41,0,1,1,1,0,0,0,0,0,0,3,1,2,0,12,4
161,5,2,1,0,0,1,1,41,0,1,0,0,0,0,0,0,0,0,3,1,4,0,3,5
282,4,2,1,0,32,1,3,41,0,1,0,0,0,0,0,0,0,0,0,0,1,0,4,1
0,3,1,1,0,32,1,1,32,1,1,1,1,0,0,0,0,0,0,0,0,4,0,8,4
0,4,2,5,0,0,1,1,32,0,1,1,1,0,0,0,0,0,0,0,0,1,0,2,5
....

With output is the last column, it has 6 outputs {A,B,C,D,E,F} reference to {0,1,2,3,4,5}. I'm using NN and SVM(C-SVC, NuSVC) for training this dataset, but the accuracy still low (>60%).

I want to try something else to improve the performance of current results so I plan to train 6 binary models like {A, not_A}, {B, not_B}...

Should I only change the output into binary and keep all features for training?

Or I need to change all features into binary too? If yes, can you give a short explanation for me?

Thanks for all your help!

What I understood from your question is, for your OneVsAll classifiers, is it ok to convert feature set in to binary or not?

  If this is correct, you should not convert your features into binary.
  From the shared data set, it seems you have continuous data set and if
  you convert into binary one, it will become categorical data, which
  will have different model representation. So according to me you
  should consume your data as it's. Though you can perform data
  normalization.

  Note: Multinominal Logistic Regression works on the similar principal
  i.e. OneVsAll. So you can try this one as well.


Data transposition code in R I've been working in SAS for a few years but as my time as a student with a no-cost-to-me license comes to an end, I want to learn R.

Is it possible to transpose a data set so that all the observations for a single ID are on the same line?  (I have 2-8 observations per unique individual but they are currently arranged vertically rather than horizontally.)  In SAS, I had been using PROC SQL and PROC TRANSPOSE depending on my analysis aims.

Example:

ID    date        timeframe  fruit_amt   veg_amt

4352  05/23/2013  before     0.25        0.75

5002  05/24/2014  after      0.06        0.25

4352  04/16/2014  after      0           0

4352  05/23/2013  after      0.06        0.25

5002  05/24/2014  before     0.75        0.25

Desired:

ID    B_fr05/23/2013   B_veg05/23/2013  A_fr05/23/2013  A_veg05/23/2013   B_fr05/24/2014   B_veg05/24/2014   (etc)

4352  0.25             0.75             0.06            0.25              .                .

5002  .                .                .               .                 0.75             0.25

Try 'arrange(Data.frame.name, ID)' function from package 'dplyr'


Difference between advantages of Experience Replay in DQN2013 paper I've been re-reading the Playing Atari with Deep Reinforcement Learning (2013) paper. It lists three advantages of experience replay:

  This approach has several advantages over standard online Q-learning
  [23]. First, each step of experience is potentially used in many
  weight updates, which allows for greater data efficiency. Second,
  learning directly from consecutive samples is inefficient, due to the
  strong correlations between the samples; randomizing the samples
  breaks these correlations and therefore reduces the variance of the
  updates. Third, when learning on-policy the current parameters
  determine the next data sample that the parameters are trained on. For
  example, if the maximizing action is to move left then the training
  samples will be dominated by samples from the left-hand side; if the
  maximizing action then switches to the right then the training
  distribution will also switch. It is easy to see how unwanted feedback
  loops may arise and the parameters could get stuck in a poor local
  minimum, or even diverge catastrophically [25]. By using experience
  replay the behavior distribution is averaged over many of its previous
  states, smoothing out learning and avoiding oscillations or divergence
  in the parameters. Note that when learning by experience replay, it is
  necessary to learn off-policy (because our current parameters are
  different to those used to generate the sample), which motivates the
  choice of Q-learning.

I am confused on how the second and third advantages differ. Isn't the third advantage just another case of breaking correlation?

Thank you in advance for your help!

Isn't the third advantage just another case of breaking correlation?

It could be viewed that way, but it is a different kind of correlation.

The second advantage is about breaking correlation due to samples being from adjacent time steps on the same trajectory. This is a more important problem when state vectors evolve slowly/incrementally per time step.

The third advantage is about breaking correlation due to samples being taken from the same policy. This is a more important problem when some actions have very similar reward and state progression independently of state, or may result in no change to state (e.g. an agent tries to move into a wall - agents getting stuck in corners due to runaway feedback of the direction being "best" is a thing you can sometimes observe whilst DQN is learning).

Both can also be a factor when an early event puts the remainder of a trajectory into a single part of the overall space, which can occur in environments where state history is important - in those cases, almost all of the time steps in a single episode could be correlated. Think of a resource management game, where expending or keeping a key resource early on in the game has a large influence on eventual success at a task. This is impacted by elements of second and third advantages - i.e. the states in the trajectory are correlated due to state of the resource throughout, and across multiple episodes the current policy might prefer expending or keeping that resource at a specific stage.

The correlations avoided by the second and third advantages typically occur over different scales of time, although that does depend strongly on the specific problem. The second advantage might be gained with a relatively small replay memory (depending on how the state evolves and lengths of episodes). The third advantage will usually require a larger memory, in order that it captures episodes with varying policies. In addition the replay memory should include exploratory actions which also help with this third issue.


Extracting model equation and other data from 'glm' function in R I've made a logistic regression to combine two independent variables in R, using pROC package and I obtain this:

 summary(fit)

Call: glm(formula = Case ~ X + Y, family = "binomial", data = data)

Deviance Residuals:
  Min       1Q     Median     3Q      Max
-1.5751  -0.8277  -0.6095   1.0701   2.3080

Coefficients:
             Estimate  Std. Error z value Pr(>|z|)
(Intercept) -0.153731   0.538511  -0.285 0.775281
X           -0.048843   0.012856  -3.799 0.000145 ***
Y            0.028364   0.009077   3.125 0.001780 **
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

Null deviance: 287.44  on 241  degrees of freedom
Residual deviance: 260.34  on 239  degrees of freedom
AIC: 266.34

Number of Fisher Scoring iterations: 4

>     fit

Call:  glm(formula = Case ~ X + Y, family = "binomial", data = data)

Coefficients:
  (Intercept)       X            Y
   -0.15373     -0.04884      0.02836

Degrees of Freedom: 241 Total (i.e. Null);  239 Residual
Null Deviance:      287.4
Residual Deviance:  260.3        AIC: 266.3

Now I need to extract some information from this data and I'm not sure about how to do it. First, I need the model equation: suppose that fit is a combined predictor called CP; could it be CP=-0.15-0.05X+0.03Y?

Then, the resulting combined predictor from the regression should present a median value, so that I can compare median from the two groups Case and Controls which I used to make the regression (in other words, my X and Y variables are N-dimensional with N = N1+N2, where N1 = Number of Controls, for which Case=0, and N2 = Number of Cases, for which Case=1).

In order to extract some data from the fitted glm model object, you need to figure out where that data resides (use documentation and str() for that). Some data might be available from the summary.glm object, while more detailed data is available from the glm object itself. For extracting model parameters, you can use coef() function or direct access to the structure.

UPDATE:

From Princeton's* introduction to R course's website, GLM section - see for details & examples:

  The functions that can be used to extract results from the fit
  include

- 'residuals' or 'resid', for the deviance residuals
- 'fitted' or 'fitted.values', for the fitted values (estimated probabilities)
- 'predict', for the linear predictor (estimated logits)
- 'coef' or 'coefficients', for the coefficients, and
- 'deviance', for the deviance.

  Some of these functions have optional arguments; for example, you can
  extract five different types of residuals, called "deviance",
  "pearson", "response" (response - fitted value), "working" (the
  working dependent variable in the IRLS algorithm - linear predictor),
  and "partial" (a matrix of working residuals formed by omitting each
  term in the model). You specify the one you want using the type
  argument, for example residuals(lrfit,type="pearson").

*) More accurately, this website is by Germán Rodríguez from Princeton University.


What is NLP technique to generalize manually created rules in text? Let's say we have a free text containing key-value entities.

Example: "... patient's tumour has width 6 cm and height 5 cm"

Then an expert comes, marks it as important, thus we do have the rule for finding the same entity in new, different texts.

What is the name of NLP technique that produces generalized rule, so that we are able to recognize also synonyms/variations with the same meaning?

Example: "... patient with cancerous growth, 60 x 50 mm"

I need to to learn terminology to be able to search for papers. If you could add also some references to state-of-art approaches, would be great.

You have many options depending on the level of complexity and creativity you may have. Among all, I would suggest going through "Natural Language Understanding" techniques.

Introduction

Not so precise but practically so to speak, NLP is about processing text and taking each token/word/phrase as an object and learning different models based on appearance of these objects and answering different questions. Here you do not really work with what they mean. Even when you do sentiment analysis (which seems to be a semantic concept) what you really do is counting tokens and seeing their labels in training data and predict the sentiment of the new one.

In NLU, you mainly deal with semantics, meanings and relation between words. For example imagine a Google search query. When I search "who was the president of US when Italy won the world cup last time?". If Google wants to rank pages for me based on NLP then many things might come up including these words and their close words (which of course helps me find the answer in related webpages). But if google needs to deliver the precise answer (which is still not easy!), it needs to understand what I say as you see an example here. For this, they use a Knowledge Base (for Google, it's called Knowledge Graph) which is the answer I have to your question.

Suggested Idea

You can make a knowledge base for your corpus if there is enough text. For this, you can extract patterns of desired phrases manually and through this patterns search all occurrences of your desired sentences (e.g. "|MEDICINE| is used for |DISEASE|") and all occurrences of your desired objects (e.g. "Tumor"). Hearst Patterns can be used for such purpose as you see in this paper. Then connecting things though a graph gives you an option for understanding that the similar pattern is happening. For synonyms and related words you can easily use existing Knowledge Bases which are a lot.

Hope it helps :)


Bagging vs Dropout in Deep Neural Networks Bagging is the generation of multiple predictors that works as ensamble as a single predictor. Dropout is a technique that teach to a neural networks to average all possible subnetworks. Looking at the most important Kaggle's competitions seem that this two techniques are used together very often. I can't see any theoretical difference besides the actual implementation. Who can explain me why we should use both of them in any real application? and why performance improve when we use both of them?

Bagging and dropout do not achieve quite the same thing, though both are types of model averaging.

Bagging is an operation across your entire dataset which trains models on a subset of the training data.  Thus some training examples are not shown to a given model.

Dropout, by contrast, is applied to features within each training example.  It is true that the result is functionally equivalent to training exponentially many networks (with shared weights!) and then equally weighting their outputs.  But dropout works on the feature space, causing certain features to be unavailable to the network, not full examples.  Because each neuron cannot completely rely on one input, representations in these networks tend to be more distributed and the network is less likely to overfit.


Fastest batch perspective transform for image augmentation I need to do some augmentation for my training images for a neural networks.
The problem is that even when loading batches in parallel, the augmentation is taking longer to perform than the network training.

The problem is in the perspective transforms I'm applying.
Both cv2.warpPerspective and torchvision.transforms.functional.perspective (with PIL images) are taking roughly the same time.

But both of them do it image by images (which seems very inefficient to me).

So, what are the best (fastest and batch-wise) ways of performing perspective transformation?

I think of doing it directly with PyTorch functions, but I don't know the algorithm. (Accepting a good algorithm as answer)
Any other package that can do it faster?

Image augmentations heavily relies on your DataGenerator and DataLoader design, mostly along with the hardware resources that you are using. Apart from that, here is a quick comparison chart to help you with the transformations along with the libraries links.
The numbers represent the number of images it processes per second.

Here are the links to all the required libraries:

Albumentations
Imgaug
Augmentor
SOLT

Ref: Albumentations benchmark results


How should I expose data from my app for data scientists? I'm the product manager for an online app.  I'm currently researching a new feature where our users will be able to access "all their raw application data".  This data is likely to be used by data scientists, it's also likely to be loaded into BI tools.  The datasets will contain up to a few million rows.

How should I actually expose the data in a practical sense?

For example:

Online datasource like Amazon Redshift
Some other RDBMS available online (e.g. a dedicated postgres installation)
CSV files available on S3
CSV files available for download in a web interface
Dumps into Google sheets
Doesn't matter as decent data scientists can easily handle and automate anything

Usually, people are indeed quite indifferent to the format, as long as there is an easy way in which they can transfer it to their favorite format.

CSV is a very common format, since most tools can load it.
Note that there are some dataset whose CSV representation is inconvenient (e.g., the text has many commas, data set in which the type is importnat yet hard to deduce).

As for the storage of the format most people are even more indifferent to the source too. Web interface is the common option.

If you use a relational database you gain few advantages:

The dataset is structured so you are protected from wrong structuring problems.
The dataset can be naturally updated (e.g., you can keep appending records to it on a daily basis).
You can allow the users load only part of the dataset, which is convenient with large dataset (e.g., give me just America users from the last month).
In case that you are willing to enable that, you can let the users work on your database directly (e.g., as Google Big Query host github data)

To summarize, the location of the information is not important.
Unless you need one of the advantages provided by a database, use a CSV.

Need a database:

Online datasource like Amazon Redshift
Some other RDBMS available online (e.g. a dedicated postgres installation)
Big query

The raw data is enough

CSV files available on S3
CSV files available for download in a web interface
Dumps into Google sheets

General true

Doesn't matter as decent data scientists can easily handle and automate anything (anything reasonable but yet, it is nice that you are looking for the most convenient format).


how to create input file for collaborative filtering http://spark.apache.org/docs/latest/mllib-collaborative-filtering.html
https://github.com/apache/spark/blob/master/data/mllib/als/test.data

I want to find missing value in a matrix based on past matrices

But I find that the test.data are not matrices, instead a long list of 3 columns

How do I understand test.data and create my own data?

Which value represent missing value?

and how to understand the result, which value represent result?

This data set is in a sparse format. Most ratings are unknown, which is a common scenario in recommender systems. This data format is much smaller due to leaving out all the unknowns. You can interpret this data as follows:

user, item, rating

What this means is that user U has rated item I with rating R. Every UxI combination not in your dataset is unknown and has to be predicted using collaborative filtering.


Can colored images have more than 3 channel values? I was reading this well-known paper and noticed something in figure 1 below:

It says in the caption (The number of channels is denoted on top of the box). You can see that the number of channels is ranging from 1 up to 1024. I am confused here because it is known that the number of channels in colored images are 3 (R,G,B). Did I misunderstand something here?

Thank you.

Yes, Images can have more than 3 channels. Satellites routinely record multiple frequencies at once (for instance infrared). Normal monitors can't render that outright and you'll have to project those channels back to RGB. A simple way to do that is false colors.

These three false-color images demonstrate the application of remote sensing in precision agriculture: The left image shows vegetation density and the middle image presence of water (greens / blue for wet soil and red for dry soil). The right image shows where crops are under stress, as is particularly the case in fields 120 and 119 (indicated by red and yellow pixels). These fields were due to be irrigated the following day.

But convolutional nets take inputs (for instance RGB) and transform them to higher dimensional representations by applying several convolutions over the data. That is what you are referring to now. You can think of those bands as different filters that highlight features that we originally detected in the RGB-channels. They live inside the pipeline, and are then projected to a (typically lower dimensional) target (in this case 2)

So You can also input multi-band pictures, convolve those to even more channels.


Relationship Between Independent and Dependent variables I calculated the distance correlation among the independent variables and the dependent variables to verify the nonlinear relationship among the variables and the values I am getting for each independent and dependent variable pair is not more than 0.1, which means that the relationship among the variables is not strong.
So is there any way of, still, applying the neural network model to get the better the results.

Yes you can use neural network for any non linear function. Maybe your variables don't have a strong linear correlation but you can train it using neural network to find the best pattern possible . Although it is tough to say whether it will generalize or not


Dynamic pricing with aggregate constraints so I have this situation. I am trying to understand whether my customers will buy my product at a certain price based on a previous offer made to them. Specifically, I have a lot of data on my clients and I have rolled a previous "campaign" round and I saw who bought it and who did not. Now I am trying to calibrate new prices for those who did not and I wanted to find prices that maximize the chances of them buying it but at the same time I have an aggregate constraint that I have to uphold (a weighted aggregate profit margin).

I am in the early stages but I wanted to understand the models of reference for this kind of problems. Any suggestion to which area/material is most relevant is very appreciated.

p.s.: I have tried to scour the forum for similar questions without success but, in case I missed something, do close the question and kindly refer me to that content.

Hi Asher11 and welcome to the community. I have worked in pricing in the past and I will try to give you some directions, although your question is quite general. It all comes down to how you want to formulate your problem. I suggest you to not look for models first but to figure out the problem you are facing and translate into math.

For example:

Do you want to know if a customer will buy a specific product according to some characteristics (features) of them? Then you have a prediction problem. You could possibly represent each customer with a set of features and try to predict if they will buy a product by training a Supervised Learning model. Start with linear regression in order to have a performance baseline.
Do you want to optimize profit/revenue? then you have an optimization problem and potentially you can use Reinforcement Learning methods to solve it. If the sequence of the events (offering and buying) does not play an important role in the decision then you can try algorithms from Bandit theory.

Another question you need to answer is why you want to use ML methods? You might be surprised but you will have a way easier solution by looking at Pricing Optimization with Linear Programming. You will need to make some assumptions about the supply and demand curve of each product (or a group of products), add appropriate constraints and optimize in order to get the best price. The product grouping might be done with ML methods or according to your intuition. By doing this exercise you will start getting an idea of the best way to formulate your problem.

Be aware in order to train an RL agent for optimizing profit/revenue it means that the agent will try random prices LIVE (RL training is based on trial and error). For this reason you need to look up for techniques on how you can pre-train a network to suggest prices within a specific range and then use RL live without the fear that the algorithm will output unreasonable values. Another way is to build a "market" simulator with the data that you have and train the RL agent directly there.

By writing the above, I tried to give you few keywords that you could look up in order to specify your problem better and possibly find a solution.