Removing outliers with orders of magnitude differences I have a dataset of virtual currency earn and spend events from a mobile game app. Unfortunately, people cheat in the game to get more currencies. These cheaters use different techniques so its quite hard to detect them all in game. The dataset is also about 50 TB so the only option I have is to use SQL (on Google Bigquery).

It tried to make a standard outlier detection system in which I find the average and standard deviation of the currency earn and spend in each level. This works for the biggest outliers. However, some people cheat to earn, for example, 1e15 gold, while other "only" cheat to get 10000 gold. The normal gold earn rate should not be higher than about 1000. The standard outlier detection system works for the 1e15 gold earn person, but as the avg and std are so high due to that person, the 10000 gold is not found to be an outlier.

Does anyone have an idea how to find these outliers successfully?

My dataset looks something like this. (It should be noted that cheaters don't show up this often and that the dataset has in the order of billions of these rows):

user_id, currency, earn_or_spend, source_or_sink, amount
'user_1', 'gold', 'earn', 'quest', 3
'user_1', 'cash', 'earn', 'building_collect', 10000
'user_3', 'gold', 'spend', 'quest', 1
'user_2', 'gold', 'earn', 'quest', 4
'user_1', 'cash', 'earn', 'building_collect', 50000
'user_1', 'gold', 'earn', 'quest', 5
'user_4', 'gold', 'earn', 'quest', 99999 # cheater
'user_3', 'gold', 'spend', 'quest', 3
'user_5', 'gold', 'earn', 'quest', 1E15 # cheater
'user_3', 'cash', 'earn', 'level_up', 100000
'user_1', 'gold', 'earn', 'quest', 3
'user_1', 'cash', 'spend', 'build_building', 50000

It can be generated in Google BigQuery with this code:

WITH data as (
  SELECT
    *
  FROM UNNEST
  (
    ARRAY<STRUCT<user_id STRING, currency STRING, earn_or_spend STRING, source_or_sink STRING, amount FLOAT64>>
    [
      ('user_1', 'gold', 'earn', 'quest', 3),
      ('user_1', 'cash', 'earn', 'building_collect', 10000),
      ('user_3', 'gold', 'spend', 'quest', 1),
      ('user_2', 'gold', 'earn', 'quest', 4),
      ('user_1', 'cash', 'earn', 'building_collect', 50000),
      ('user_1', 'gold', 'earn', 'quest', 5),
      ('user_4', 'gold', 'earn', 'quest', 99999), # cheater
      ('user_3', 'gold', 'spend', 'quest', 3),
      ('user_5', 'gold', 'earn', 'quest', 1E15), # cheater
      ('user_3', 'cash', 'earn', 'level_up', 100000),
      ('user_1', 'gold', 'earn', 'quest', 3),
      ('user_1', 'cash', 'spend', 'build_building', 50000)
    ]
  )
)

SELECT * FROM data

I managed to do it. I finally used three outlier filters.

Filter extreme cheaters: Per [level, currency source or sink, type (earn/spend), payer/non-payer, month of start (if game changes over time, new people might earn or spend in different ways than before)] I calculated the median and median absolute deviation (MAD) of the amount of currency earned/spend. Filter user if: amount earned/spend > 100 * median + 100 * MAD
Filter users that spend more of a currency than they have earned.
Filter users that earn a currency by means of an in app purchase (IAP) but didn't generate any real money revenue in that level (i.e. they did an IAP but didn't pay for it).


Efficient Dataframe Transformation So I'm trying to transform a pandas dataframe where I have multiple duplicates of a "user" value (my index being a user ID number), demographic info (which shouldn't change) and various purchase  variables which the goal is to sum (in a timely fashion)

Right now I'm using a couple for loops which I now is bad practice when using pandas:

for n in users:

     #Creating an empty array to use to shuffle the new values into
     #n being the user ID number
     uservalue = [n]
     #Add independent variables
     prods = df[df["User_ID"]==n]["Product_ID"].tolist()
     uservalue.append(prods)
     for v in range (0,len(xvars)):
         uservalue.append(df[df["User_ID"]==n][xvars[v]].mean())
     #Adding total spending variables
     for v in range (0, len(yvars)):
         uservalue.append(df[df["User_ID"]==n][yvars[v]].sum())
    user = pd.Series(uservalue, userdfcols)
    userdf = userdf.append(user, ignore_index = True)

So, rather than iterating over the entire 500k entry dataframe, putting everything in a container list and then appending that list into a new row, is there a more efficient way to transform by user ID? My feeling is that I should be appending values to the row individually using pandas methods and skip the list but would that even help?

Thanks

So the question is count the number of duplications by user ID and other variables.
For this you can use the df.groupby().size() method. This will return the counts of unique entries in with a particular value in a column.

Here is the documentation for this function: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.groupby.html


Class token in ViT and BERT I'm trying to understand the architecture of the ViT Paper, and noticed they use a CLASS token like in BERT.
To the best of my understanding this token is used to gather knowledge of the entire class, and is then solely used to predict the class of the image. My question is — why does this token exist as input in all the transformer blocks and is treated the same as the word / patches tokens?
Treating the class token like the rest of the tokens means other tokens can attend to it. I'd expect that the class token will be able to attend other tokens while they could not attend it.
Also, specifically in ViT, why does the class token receive positional encodings? It represents the entire class and thus doesn't have any specific location.
Thanks!

My question is — why does this token exist as input in all the
transformer blocks and is treated the same as the word / patches
tokens?

The CLASS token exists as input with a learnable embedding, prepended with the input patch embeddings and all of these are given as input to the first transformer layer. The CLASS token gathers information from all the patches using Multihead Self Attention (MSA). It is basically treated the same as patch tokens but at the end when doing classification, only the hidden output from the CLASS token are used as input to the classification layer.

Treating the class token like the rest of the tokens means other
tokens can attend to it. I'd expect that the class token will be able
to attend other tokens while they could not attend it.

I think that would help training all the weights during back-prop, including the weights of the embedding layer of the input patches.

Also, specifically in ViT, why does the class token receive positional
encodings? It represents the entire class and thus doesn't have any
specific location.

The positional encoding would tell that this is the first element of the sequence. Would help if you have a bi-directional transformer model like BERT.


What GPU specifications matter when training and using neural networks? I need to purchase some GPUs, which I plan to use for training and using some neural networks (most likely with Theano and Torch). Which GPU specifications should I pay attention to?

E.g.:

one should make sure that the VRAM is large enough for one's application
the more teraflops, the faster programs running exclusively on the GPU will run
the higher the thermal design power (TDP), the higher the electricity bill
the compute capability, which determines which version of CUDA and cuDNN one can use.

What else matter, and what else doesn't matter?

E.g., is it reasonable to assume that the number of sockets, clock speed and the number of cores do not bring any additional useful information (since we already consider the number of Tflops)?

Besides the criteria listed look for

The number of cores
The ability to do low precision arithmetic

In practice you are probably best off just getting the latest generation NVIDIA card (currently 10xx).


Expected 2D array, got scalar array instead Can anyone help me with this error. I did the following code but it does not work and I am getting the following error:

ValueError: Expected 2D array, got scalar array instead:
array=6.5. Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.

My code:

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

import pandas
dataset = pandas.read_excel('PEG RATIOS.xlsx')

X = dataset.iloc[:, 2].values
X =X.reshape(-1,1)
y = dataset.iloc[:, 3].values
y = y.reshape (-1,1)

from sklearn.linear_model import LinearRegression
lin_reg = LinearRegression()
lin_reg.fit(X, y)

from sklearn.preprocessing import PolynomialFeatures
poly_reg = PolynomialFeatures(degree = 4)
X_poly = poly_reg.fit_transform(X)
poly_reg.fit(X_poly, y)
lin_reg_2 = LinearRegression()
lin_reg_2.fit(X_poly, y)

X_grid = np.arange(min(X), max(X), 0.1)
X_grid = X_grid.reshape((len(X_grid), 1))
plt.scatter(X, y, color = 'red')
plt.plot(X_grid, lin_reg_2.predict(poly_reg.fit_transform(X_grid)), color = 'blue')
plt.title('PEG Ratios verrus Exoected Growth: Semiconductor Firms')
plt.xlabel('Expected Growth rate')
plt.ylabel('PEGH Ratio')
plt.show()
lin_reg_2.predict(poly_reg.fit_transform(6.5))

The error itself solves your problem. Just follow what it says. The predict() method takes a 2d array of values you want to predict on. Each item in the array is a "point" you want your model to predict on. So try,

lin_reg_2.predict(poly_reg.fit_transform([[6.5]]))

Here the input is a 2D array of shape (1,1).

Or as the error suggests:

lin_reg_2.predict(np.array([6.5]).reshape(1, 1))


how to create a searchable tree on Persian text? I wanna clean a Persian text from stop-words. I already have stop-word data that is provided on the below link. It seems to me, if I have a pre-built tree on stop-words, I could save lots of time. I want to search each word of text in this pre-built tree, if the word is in the tree I delete it from the text, if not I hold it.

O(n * l) to O(n*log(l)).

This is my stop-words

If you have better suggestions than the pre-built tree search, I would be grateful to share it with me.

This is an answer with tire tree:

readindg data:

#readindg stopword data
stopwords = pd.read_csv('STOPWORDS',header=None)

tire tree:

#creating tire tree
class TrieNode:

    # Trie node class
    def __init__(self):
        self.children = [None]*15000

        # isEndOfWord is True if node represent the end of the word
        self.isEndOfWord = False

class Trie:

    # Trie data structure class
    def __init__(self):
        self.root = self.getNode()

    def getNode(self):

        # Returns new trie node (initialized to NULLs)
        return TrieNode()

    def _charToIndex(self,ch):

        # private helper function
        # Converts key current character into index
        # use only 'a' through 'z' and lower case

        return ord(ch)-ord('!')

    def insert(self,key):

        # If not present, inserts key into trie
        # If the key is prefix of trie node,
        # just marks leaf node
        pCrawl = self.root
        length = len(key)
        for level in range(length):
            index = self._charToIndex(key[level])

            # if current character is not present
            if not pCrawl.children[index]:
                pCrawl.children[index] = self.getNode()
            pCrawl = pCrawl.children[index]

        # mark last node as leaf
        pCrawl.isEndOfWord = True

    def search(self, key):

        # Search key in the trie
        # Returns true if key presents
        # in trie, else false
        pCrawl = self.root
        length = len(key)
        for level in range(length):
            index = self._charToIndex(key[level])
            if not pCrawl.children[index]:
                return False
            pCrawl = pCrawl.children[index]

        return pCrawl != None and pCrawl.isEndOfWord

Example of use:

# Input keys (use only 'a' through 'z' and lower case)
keys = list(stopwords.loc[:,0])

output = ["Not present in trie",
        "Present in trie"]

# Trie object
t = Trie()

# Construct trie
for key in keys:
    t.insert(key)

print("{} ---- {}".format("از",output[t.search("از")]))

Output:

از ---- Present in trie


Linear transformation from one sample to another Generate a Sample $\underline{Z_1}$ $\underline{Z_2}\dots \underline{Z_{5000}}$ , while $\underline{Z_i} \sim N_2[(0,0)^T,I_2]$
generate new sample with size of $ n = 5000$ by applying linear transformation on $\underline{Z_i}$
$\underline{X_1}$ $\underline{X_2}\dots \underline{X_{5000}}$ , while $\underline{X_i} \sim N_2[(1,2)^T,\begin{pmatrix}2&1.5\\ 1.5&2\end{pmatrix}]$.
My attempt:
n1 <- 5000
mu <- c(0,0)
sigma <- diag(2)
y2 <- mvrnorm(n1,mu,sigma)

I have generated the first sample but aside from that , I have no idea how to continue..
Update
$X = A*Z + \mu$ while $\mu = (1,2)^T.$
To find $A$ we use this equation $\Sigma = A*A^T$.
ed = eigen(Sigma)
A = ed$vectors %*% diag(sqrt(ed$values))

but still I'm not getting the right value for $A$.
because $A*A^T\neq \Sigma$.

Say we want to sample $x$ from $N(a,b)$. We could certainly do this in R using rnorm(a,b). However, we could also sample $z$ from $N(0,1)$ and apply the linear transformation $a + zb$. This transformation would then give us samples from $N(a,b)$.
What I've described is for the univariate case. You can easily extend this to the multivariate case.


Time series regression I m new in the domain of machine learning. I m here to ask for some elucidation.
I have a data set presented as a time series( from a strain sensor coming from a wind turbine). In this time series, we can capture stop and start as u can see in the attached figure . I would like to know. which type of layer should I use? is it possible to make a model that counts up the start and stop of the signal(output dimension = 2)? Until now I just did classification using conv1D and the classes are (there is a stop, there is a start, there is nothing) but this way proceed is not convenient since sometimes we can have a rotor stop and start.
I already thanks you a lot for your help,

Counting of events is probably best done as a separate software component that takes the output of your event classifier. The event classifier model would output for each time-step two probabilities: one for started and one for stopped.

The easiest way to count the events to have a state machine that compares the probabilities against a fixed threshold to create discrete events. And once you have discrete events, counting is trivial. Pseudo-code below:

# returns a tuple with (new_state, event_name)
# event_name can be None if no event occurred
def next_event(state, start_prob, stop_prob):
    start_threshold = 0.5
    stop_threshold = 0.5

    if state == 'running':
        if stop_prob > stop_threshold:
           return ("parked", "stop")

    if state == 'stopped' or state == 'unknown':
        if start_prob > start_threshold:
           return ("running", "start")

    return (state, None) # no change

...
# initialization
start_counts = 0
state = 'unknown'

...
# for every new time window X
   event_prob = model.predict(X)

   state, event = next_event(state, event_prob[0], event_prob[1])
   if event is not None and event == 'start':
        start_counts += 1
   ....


Policy Gradient not "learning" I'm attempting to implement the policy gradient taken from the "Hands-On Machine Learning" book by Geron, which can be found here. The notebook uses Tensorflow and I'm attempting to do it with PyTorch.

My models look as follows:

model = nn.Sequential(
    nn.Linear(4, 128),
    nn.ELU(),
    nn.Linear(128, 2),
)

Criterion and optimisers:

criterion = nn.BCEWithLogitsLoss()
optim = torch.optim.Adam(model.parameters(), lr=0.01)

Training:

env = gym.make("CartPole-v0")

n_games_per_update = 10
n_max_steps = 1000
n_iterations = 250
save_iterations = 10
discount_rate = 0.95

for iteration in range(n_iterations): # Run the game 250 times
    all_rewards = []
    all_gradients = []
    n_steps = []
    optim.zero_grad()
    for game in range(n_games_per_update): # Run the game 10 times to accumulate gradients
        current_rewards = []
        current_gradients = []
        obs = env.reset()
        for step in range(n_max_steps): # Run a single game a maximum of 1000 steps

            logit = model(torch.tensor(obs, dtype=torch.float))
            output = F.softmax(logit, dim=0)
            c = Categorical(output)
            action = c.sample()

            y = torch.tensor([1.0 - action, action], dtype=torch.float)
            loss = criterion(logit, y)
            loss.backward()

            obs, reward, done, info = env.step(int(action))
            current_rewards.append(reward)
            current_gradients.append([p.grad for p in model.parameters()])
            if done:
                break
        n_steps.append(step)

        all_rewards.append(current_rewards)
        all_gradients.append(current_gradients)

    # Performs the discount and normalises
    all_rewards = discount_and_normalize_rewards(all_rewards, discount_rate=discount_rate)

    # For each batch of 10 games multiply the discounted rewards against the gradients of the
    # network. Then take the mean for each layer
    new_gradients = []
    for var_index, gradient_placeholder in enumerate(gradient_placeholders):
        means = []
        for game_index, rewards in enumerate(all_rewards):
            for step, reward in enumerate(rewards):
                means.append(reward * all_gradients[game_index][step][var_index])
        new_gradients.append(torch.mean(torch.stack(means), 0, True).squeeze(0))

    # Apply the new gradients to the network
    for p, g in zip(model.parameters(), new_gradients):
        p.grad = g.clone()
    optim.step()

When I run the code for 250 interactions I print the average game length I get:

Iteration: 50, Average Length: 18.2
Iteration: 100, Average Length: 23.4
Iteration: 150, Average Length: 29.9
Iteration: 200, Average Length: 11.2
Iteration: 250, Average Length: 38.6

The network isn't really improving and training for longer doesn't help. My two questions are:
1. Is there anything obviously wrong that I'm doing?
2. I've notices the log of the probability is used in the tensorflow implementation, but I'm not sure how to integrate it here

I can't say for sure but I think the issue here is you're not subtracting the mean of the rewards.

The idea is that actions with above average reward are positive after mean normalization, while actions with below average reward are negative after mean normalization.

Your update step is -log(P(action))*reward, which you then minimize with your optimizer.

P(action)<1 therefore log(P(action))<0, -log(P(action))>0

If reward>0, -log(P(action))*reward>0. Minimizing this value is the same maximizing log(P(action))*reward<0, which is maximized when P(action)=1.

Conversely, if reward<0, -log(P(action))*reward<0. This has the opposite effect, where P(action) is driven to 0.

The important part is the different sign on above average/below average rewards causes actions associated with good rewards to have their probability increased, while actions associated with bad rewards have their probability decreased.


Is there a way to; easily, merge data into a CSV (or Excel) file without having to edit each file individually? I have several marketing files that I need to download on a weekly basis.
Let’s say 15 different files. Each file contains the same field names: field 1, field 2, field 3, field 4, field 5, field 6, field 7, field 8, field 9. The data contained in the fields are different and specific to each file.
Each file can contain 10 to 1000s records.
What I need to do, is add 2 additional fields to each record in each file. These fields; field A and field B will contain the same info for every record in each specific file. Each file will contain different info to add.
File 1      Need to add Field A and Field B to every record.
File 2      Need to add Field C and Field D to every record.
File 3      Need to add Field E and Field F to every record.
I can edit each file in Excel, add the columns, name the columns, add the 2 fields to the first row and
then copy and paste to all the other rows.
Is there a way to specify the file, field 1 and field 2 in order to add the info more efficiently and quickly?
Thank you!

You can either load the excel files directly into Python, e.g. using the pandas package, or you can use a package that hits the data where it lives i.e. a package that directly modifies the Excel files.

Python and Pandas

import pandas as pd

data = pd.read_Excel('/path/to/your/file)

Data is basically a table that will look like the Excel itself. You can add fields (usually referred to as columns) like so:

data['new field'] = new_data

new_data will contain all values for all rows in the table, e.g. as a list:

new_data = [12, 24, 13, 66, 72, 45]    # assuming there are 6 rows (a.k.a. samples)

You will have to alter the code for each of your files. You will also need to think about how to get your new data into that list (or a similar construction).

Caveat: this way might have problems if you have some very specific formatting or special Excel features in use (more than just conditional formatting - think dropdown lists and hidden/folded rows and columns).

Excel interface

There are a few good packages that will allow you to speak directly with the excel file and make use of things such as sheets/tabs and references to cells e.g. A1 is the top left cell in Excel.

Have a look at the XlsWriter package or the python-excel package. Both let you access the fill and add/remove data as you please. I have used both for simple tasks and they both got the job done quite nicely.

Follow those links to see some nice examples of how to manipulate Excel programmatically.

It is worth mentioning that the first option above, using something like Pandas, will offer a lot more functionality and would likely not present a steeper learning curve (in my opinion!).


Book about distances for data science (can't remember the name) I saw a book somewhere that listed a big amount of mathematical distance functions (the usual euclidean norms, the discrete distance, the hamming distance, etc) used for data science.

The name of the book might be similar to "101 metrics for data science". I cannot find it. Does anyone know about it (or a similar book)? Thanks!

Not a book, but a webpage with Open Source code is the ELKI Data Mining library of distance functions.


Implementing training in PyTorch I wish to accomplish the following task in PyTorch-

I have the COCO dataset, wherein each data sample is used in training YOLO v3. After being processed by the model, the sample is to be deleted if it satisfies a certain condition. The data sample is thus no longer used for training in further epochs.

I now have two questions regarding implementation -

1) How do I process each sample individually? Do I go about this by setting batch size = 1? Or is there any advantage to disabling automatic batching. If so, how do I go about this.

2) How exactly do I delete the sample from the dataset for further epochs. Is there any way to skip this sample in the DataLoader?

1) How do I process each sample individually? Do I go about this by setting batch size = 1? Or is there any advantage to disabling automatic batching. If so, how do I go about this.

If you set batch size to 1, then you are effectively disabling batching.  Samples will be processed one-at-a-time and gradients will be computed for single samples.  This is not necessarily a problem - just keep in mind that you'll lose the advantages of mini-batching.

If you are using PyTorch's DataLoader, then just set batch_size and batch_sampler to None.  According to the docs, this will disable automatic batching.

2) How exactly do I delete the sample from the dataset for further epochs. Is there any way to skip this sample in the DataLoader?

You can implement a custom PyTorch dataset.  Internally, keep track of which samples have been deleted.  If you're using an iterable-style dataset, then the next step is pretty easy.  Just skip over the deleted samples in __iter__().

Map-style datasets will be a little more tricky.  Your len() function will need to return len(dataset) - # of deleted samples.  And you will have to find some indexing scheme that prevents deleted samples from being accessed.  One naive idea is simply to reindex after every deletion.  The obvious problem is that the DataLoader's random sampler won't have any way to know this, so a single sample could be processed multiple times in one epoch.  Another idea is to index your samples by some unique key rather than integers.


Python XGBoost predict_proba returns very high or low probabilities I trained my data with XGBoost in python with GridSearchCV as follows:

parameters = {'nthread':[6],
              'objective':['binary:logistic'],
              'learning_rate': [0.01, 0.1],
              'max_depth': [5,8,13],
              'n_estimators': [200,500,1000,3000],
              'seed': [1337]}

xgb_model = xgb.XGBClassifier()

clf = GridSearchCV(xgb_model, parameters, n_jobs=-1,
                   cv = StratifiedKFold(shuffle=True,n_splits=5),
                   scoring='accuracy',
                   verbose=2, refit=True)

clf.fit(scaled_X_train.values, y_train)

On the test test I got 0.9 accuracy which is acceptable. However when I predict probabilities with predict_proba I saw that probabilities mostly lie between 0-0.1 and 0.9-1 ranges for 0 and 1 classes respectively.

Since I try to get scores based on the model, those dense probabilities are not so useful.

So what is the main reason of this dense probability distribution? Is this a bad thing? And how can I improve my workflow so that probabilities get wider score range?

Thanks in advance!

If you are after well calibrated scores (that is, the scores outputted by your model can be interpreted as probabilities in some sense, at least according to what you have observed empricially in your training set) then accuracy is not the right function to optimize (i would argue that it is never a good function to optimize, but I won't get into this). Accuracy can be optimized by providing scores that are not necessarily reflective of the empirical probabilities observed in your dataset: ex: suppose the true label = (1, 1, 0, 1) and you have two classifiers (0.51, 0.51, 0.3, 0.51) vs (0.9, 0.9, 0.1, 0.9). Both have the same accuracy assuming 0.5 threshold but clearly very different scores.

Switch your objective to log loss which is optimized only when you feed it well calibrated/true underlying probabilities. Furthermore, look into probability calibration (like platt/isotonic/multiple width binning) if you find that log loss performance is not satisfactory.


What is Difference Between np.zeros() and np.empty() Can Anyone Please Explain me the Difference Between them In Terms Of Any Operation or Computation and Values Stored in them

From the documentation:

  empty, unlike zeros, does not set the array values to zero, and may
  therefore be marginally faster. On the other hand, it requires the
  user to manually set all the values in the array, and should be used
  with caution.

np.zeros

Return a new array setting values to zero.

>>> np.zeros((2, 2))
array([[0., 0.],
       [0., 0.]])

np.empty

Return a new uninitialized array.

>>> np.empty((2, 2))
array([[1.35807735e-312, 1.35807731e-312],
       [1.99637364e-310, 8.69169476e-311]])


Understanding autoencoder loss function I've never understood how to calculate an autoencoder loss function because the prediction has many dimensions, and I always thought that a loss function had to output a single number / scalar estimate for a given record. However, on GitHub I recently came across a repository that has implemented an example autoencoder in TensorFlow, and the squared error implementation is a lot simpler than I thought:

cost = tf.reduce_mean(tf.pow(y_true - y_pred, 2))

The TensorFlow documentation on reduce_mean says, among other things:

  If axis has no entries, all dimensions are reduced, and a tensor with
  a single element is returned.

Can I conclude from all of this that the squared error of an autoencoder prediction is just the average across all of the record's dimensions?

Yes, you are correct in thinking that the squared error of an autoencoder prediction for a single example is the average of the squared error of the prediction for all dimensions. Similarly, the squared error for a whole batch of examples will be the average of the error of each example in the batch.


How to identify clusters after multiple runs? Suppose I run an unsupervised clustering algorithm. After multiple runs, I find clusters and would like to know if the same cluster was found more than once.

For example:

I can figure out A-orange, B-green and C-blue are probably the same because their centroids are close together.

However, take the following example:

How can I programmatically figure out that A-blue and C-orange are the same? That A-orange and C-blue are the same? But B-orange and B-blue are not similar to any in A and C?

Use the famous Hungarian algorithm.

It computers the best match permutation.

You can find more details on Wikipedia:

https://en.wikipedia.org/wiki/Hungarian_algorithm


Output probabilities for each class on tensorflow sigmoid function I have a piece of code that uses tf.nn.softmax to predict whether does a image belongs to either class 0, 1, 2... etc.

However, I want to edit the code to using sigmoid as the activation function and outputing all the probabilities, and setting those with probabilities >0.5 as one of the classes identified in the image.

This is the code which I am trying to run: https://github.com/satyenrajpal/Concrete-Crack-Detection

I believe this is the code snippet where my edit should be made:

            for counter,image in enumerate(test_images):
                #break up images into 128*128
                broken_image,h,w,h_no,w_no = break_image(image,128)

                output_image = np.zeros((h_no*128,w_no*128,3),dtype = np.uint8)

                feed_dict = {x: broken_image}
                batch_predictions = sess.run(predictions, feed_dict = feed_dict)

                print("here is one loop")

                print(batch_predictions)

#                file = open("test.txt","w")
#                bpred_str = batch_predictions.astype('str')
#                file.write(bpred_str)
#                file.write(" ")
#                file.close()

                results=np.concatenate((results, batch_predictions))

                matrix_pred = batch_predictions.reshape((h_no,w_no))
                #Concentrate after this for post processing
                for i in range(0,h_no):
                    for j in range(0,w_no):
                        a = matrix_pred[i,j]
                        output_image[128*i:128*(i+1),128*j:128*(j+1),:] = 1-a

                cropped_image = image[0:h_no*128,0:w_no*128,:]
                pred_image = np.multiply(output_image,cropped_image)

I tried to print batch_predictions, but instead it prints out something like:

[1 1 0 1 1 0]

Another snippet is:

#Predict the class
y_pred = tf.nn.sigmoid(layer_fc2)
print("a: %s",y_pred)
#print("Class Probability: %s"%(sess.run(y_pred)))
self.y_pred_cls = tf.argmax(y_pred, dimension=1,name="predictions")

#Cost Function
cross_entropy = tf.nn.sigmoid_cross_entropy_with_logits(logits=layer_fc2, labels=self.y_true)
print("b: %s",cross_entropy)
cost = tf.reduce_mean(cross_entropy)
optimizer = tf.train.AdamOptimizer(learning_rate=1e-4).minimize(cost)

#Predict
correct_prediction = tf.equal(self.y_pred_cls, self.y_true_cls)
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
probabilities=tf.nn.sigmoid(y_pred)
print("Class Probabilities: %s", probabilities)
return optimizer, accuracy

I also tried to print the output of tf.nn.sigmoid(y_pred), but it gave:

Class Probabilities: %s Tensor("Sigmoid_1:0", shape=(?, 2), dtype=float32)

I need help on how to print out the individual probabilities of the classes when I run the model on a unlabelled data. Thank you in advance!

You need to print output of sigmoid before tf.argmax is applied. It can be done with "tf.print", tf.print is a network node that does modify values.

y_pred = tf.nn.sigmoid(layer_fc2)
y_pred = tf.print(y_pred ,[y_pred ])
self.y_pred_cls = tf.argmax(y_pred, dimension=1,name="predictions")

Docs : https://www.tensorflow.org/api_docs/python/tf/print


Stochastic gradient descent and different approaches While dealing with stochastic gradient descent, I got confused by several sources which explained it quite differently.

One source explained stochastic gradient descent as standard gradient descent approach but with substantial smaller training set.
The other approach first sums the gradients of all training set and then multiplies with learning rate and updates it. $Q_{t+1}<-Q_t - \tau * \frac{n}{|S|}*\sum_{j\in S}\nabla L_j(Q_t)$

What is the difference? and How does it work with matrices since I take just couple of training examples? (see example below)

Here is an example:
I have sparse matrix M with user x movies where the value in the matrix represents how much the user rates the movie. I decomposed it into matrices P and Q and using stochastic gradient descent I want to optimize P and Q by minimizing the loss function.
Each value of M can be reconstructed by multiplying vector of Q and P.

Let's say I take batches of 10 so it means I take just 10 values from matrix M and respective rows and columns of P and Q which I will optimize.
If I take the second approach there is sum of gradients of all point and then I do the update but how can I sum those vectors since they do not correspond to the same place in matrix Q or P?
In this case, should I then just make new P and Q just with vectors that corresponds to taken values of M? but then it does not make sense because I should optimize the whole Q and P matrices right?

I hope I made it clear enough.

I think you're actually asking two separate questions: 1) What is stochastic gradient descent (SGD)? and 2) how do I find the matrices $P$ and $Q$ in the matrix factorization of $M$?

For question 1, I don't think either of your explanations is correct. (Explanation 2 sounds to me like normal (batch) gradient descent, although I found your description a little confusing.)

Regardless, both the Wikipedia article and Chapter 5 of the Deep Learning book provides good explanations of SGD. In many machine learning problems, the loss function, $L$ can be decomposed as the sum of losses for individual training examples:

\begin{equation}
L(x, y; \theta) = \sum\limits_{i=1}^{m} L(x^{(i)}, y^{(i)}; \theta)
\end{equation}

Therefore, in normal gradient descent, the number of terms in the gradient scales with the number of training examples, $m$. Computing this term can be very costly as $m$ grows large. In stochastic gradient descent, we instead approximate the gradient by using a "mini-batch" of samples to compute it. So, instead of

\begin{equation}
\nabla_{\theta} L(x, y; \theta) = \sum\limits_{i=1}^{m} \nabla_{\theta} L(x^{(i)}, y^{(i)}; \theta)
\end{equation}

we use

\begin{equation}
\nabla_{\theta} L(x, y; \theta) \approx \sum\limits_{i=1}^{m_{mini}} \nabla_{\theta} L(x^{(i)}, y^{(i)}; \theta)
\end{equation}

where $m_{mini} \ll m$. The key point is that, when running SGD, we randomly shuffle the training examples and iterate through different minibatches during training. So, SGD is not equivalent to normal (batch) gradient descent for a smaller training set. In SGD, we train on all of the training examples, but it just takes us $m/m_{mini}$ iterations to get there. For instance, if $m_{mini} = 1$, and we took $m$ SGD steps, each step would correspond to using a single training example to approximate the gradient, and all $m$ steps would involve using a different training example.

For question 2, solving for the parameters in matrix factorization is usually done via alternating least squares (ALS) or SGD. Here's an excellent blog post explaining how both techniques work in the context of matrix factorization. The post is too long to summarize, but the author carefully and slowly derives all of the relevant equations from scratch, so I recommend you read it!


Does any cloud service provide free GPU access? Is there any cloud developer platform which provides free access to an NVIDIA GPU instance, maybe only for a limited trial period? I'd like to work on it before committing to a paid option. I have tried AWS, Google Cloud Developer Platform and IBM Bluemix, but none of them have this option.

Nvidia has it's own online labs where they can help you learn GPU based data processing. Might be worth checking out for you if you want to learn and play around within the online lab environment.

https://developer.nvidia.com/dli/onlinelabs

All the best.


Developing KNN with custom distance matrices I am trying to predict the extent of suitability (0.5 to 3.5) for a section of non-fiction books based on a few parameters. This is for a set of subscribers that we want to predict books suitability: based on score I plan to display star ratings: most suitable(2.5 to 3.5), mildly suitable(1.5 to 2.5), not relevant (0.5 to 1.5).
Example dimensions:

price: continuous

subject: categorical (humor, adventure, mystery, sports, non-fiction)

hardbound: binary

recency of book: categorical

We initially start with a default profile for all user and after getting feedback from him (as 1, 2 or 3 stars), build a customized profile.
Given that we will initially start with small volume of user data, looking to do a k-NN to identify nearest neighbors on the dimensions, and calculate suitability score based on weighted suitability of those neighbors.
Given the categorical data, I wanted to create custom distance matrices so that I could provide the relative distance between categories. (e.g. to indicate that humor and adventure are close preferences; sports and nonfiction are far apart)
Two questions:

Is my current approach valid- any feedback or pointers?
What specific k-NN implementation (package) would allow me to provide my custom distance matrices?

I would personally take a look at collaborative filtering as this will take into account the information of the user itself, but also the information of similar users. If let's say, my best friend and I have the same taste in books, knowing his taste can help you guess mine! I'd recommend watching the Andrew Ng lectures on Coursera about recommender systems.

KNN might seem like a natural choice if you don't know about recommender systems (I have been there!) but please, make sure to read about them before you decide to use KNN.


How many Hidden Layers and Neurons should I use in an RNN? I am very new to neural networks and machine learning and I have been making a Bitcoin price predictor to learn it. I was wondering about the number of hidden layers I'd need in a recurrent neural net using LSTM cells.

I have 60 inputs for 30 previous days' close prices in 12-hour intervals and require 1 output for the future 12 hours.

I am doing this with Keras in python 3.6.
Any help would be awesome!

Number of layers is a hyperparameter. It should be optimized based on train-test split. You can also start with the number of layers from a popular network. Look at kaggle.com and see how many layers do they use in competitions.


Pearson vs Spearman vs Kendall What are the  characteristics of the three correlation coefficients and what are the comparisons of each of them/assumptions?
Can somebody kindly take me through the concepts?

Correlation is a bivariate analysis that measures the strength of association between two variables and the direction of the relationship. In terms of the strength of the relationship, the value of the correlation coefficient varies between +1 and -1. A value of ± 1 indicates a perfect degree of association between the two variables. As the correlation coefficient value goes towards 0, the relationship between the two variables will be weaker. The direction of the relationship is indicated by the sign of the coefficient; a + sign indicates a positive relationship and a – sign indicates a negative relationship.
Pearson's correlation coefficient and the others are the non-parametric method, Spearman's rank correlation coefficient and Kendall's tau coefficient.
Pearson's Correlation Coefficient
$$
r = \frac{\sum(X - \overline{X})(Y - \overline{Y})}
{\sqrt{\sum(X-\overline{X})^{2}\cdot\sum(Y-\overline{Y})^{2}}}\\
~ \\
\begin{align}
    Where, ~ \overline{X} &= mean ~ of ~ X~variable\\
    \overline{Y} &= mean ~ of ~ Y ~ variable\\
\end{align}
$$
Assumptions:

Each observation should have a pair of values.

Each variable should be continuous.

It should be the absence of outliers.

It assumes linearity and homoscedasticity.

Spearman's Rank Correlation Coefficient
$$\rho = \frac{\sum_{i=1}^{n}(R(x_i) - \overline{R(x)})(R(y_i) - \overline{R(y)})}
{\sqrt{\sum_{i=1}^{n}(R(x_i) - \overline{R(x)})^{2}\cdot\sum_{i=1}^{n}(R(y_i)-\overline{R(y)})^{2}}}
= 1 - \frac{6\sum_{i=1}^{n}(R(x_i) - R(y_i))^{2}}{n(n^{2} - 1)}\\
~ \\
\begin{align}
    Where, ~ R(x_i) &= rank ~ of ~ x_i\\
    R(y_i) &= rank ~ of ~ y_i\\
    \overline{R(x)} &=mean ~ rank ~ of ~ x\\
    \overline{R(y)} &=mean ~ rank ~ of ~ y\\
    n &= number ~ of ~ pairs
\end{align}
$$
Assumptions:

Pairs of observations are independent.

Two variables should be measured on an ordinal, interval or ratio scale.

It assumes that there is a monotonic relationship between the two variables.

Kendall's Tau Coefficient
$$
\tau = \frac{n_c - n_d}{n_c + n_d} = \frac{n_c - n_d}{n(n-1)/2}\\
~ \\
\begin{align}
    Where, ~ n_c &= number ~ of ~ concordant ~ pairs\\
    n_d &= number ~ of ~ discordant ~ pairs\\
    n &= number ~ of ~ pairs
\end{align}
$$
Assumptions:

It's the same as assumptions of Spearman's rank correlation coefficient

Comparison of Each Correlation Coefficients
Pearson correlation vs Spearman and Kendall correlation

Non-parametric correlations are less powerful because they use less information in their calculations. In the case of Pearson's correlation uses information about the mean and deviation from the mean, while non-parametric correlations use only the ordinal information and scores of pairs.

In the case of non-parametric correlation, it's possible that the X and Y values can be continuous or ordinal, and approximate normal distributions for X and Y are not required. But in the case of Pearson's correlation, it assumes the distributions of X and Y should be normal distribution and also be continuous.

Correlation coefficients only measure linear (Pearson) or monotonic (Spearman and Kendall) relationships.

Spearman correlation vs Kendall correlation

In the normal case, Kendall correlation is more robust and efficient than Spearman correlation. It means that Kendall correlation is preferred when there are small samples or some outliers.

Kendall correlation has a O(n^2) computation complexity comparing with O(n logn) of Spearman correlation, where n is the sample size.

Spearman’s rho usually is larger than Kendall’s tau.

The interpretation of Kendall’s tau in terms of the probabilities of observing the agreeable (concordant) and non-agreeable (discordant) pairs is very direct.

Example Python Implementation


What is the term for when a model acts on the thing being modeled and thus changes the concept? I'm trying to see if there is a conventional term for this concept to help me in my literature research and writing.  When a machine learning model causes an action to be taken in the real world that affects future instances, what is that called?

I'm thinking about something like a recommender system that recommends one given product and doesn't recommend another given product.  Then, you've increased the likelihood that someone is going to buy the first product and decreased the likelihood that someone is going to buy the second product.  So then those sales numbers will eventually become training instances, creating a sort of feedback loop.

Is there a term for this?

There are three terms from social science that apply to your situation:

Reflexivity - refers to circular relationships between cause and effect.  In particular, you could use the definition of the term adopted by George Soros to refer to reverse causal loop between share prices (i.e. present value of fundamentals) and business fundamentals.  In a way, the share price is a "model" of the fundamental business processes.  Usually, people assume that causality is one-way, from fundamentals to share price.
Performativity - As used by Donald MacKenzie (e.g. here), many economic models are not "cameras" -- taking pictures of economic reality -- but in fact are "engines" -- an integral part of the construction of economic reality. He has a book of that title: An Engine, Not a Camera.
Self-fulfilling Prophecy - a prediction that directly or indirectly causes itself to become true, by the very terms of the prophecy itself, due to positive feedback between belief and behavior. This is the broadest term, and least specific to the situation you describe.

Of the three terms, I suggest that MacKenzie's "performativity" is the best fit to your situation.  He claims, among other things, that the validity of the economic models (e.g. Black-Scholes option pricing) has been improved by its very use by market participants, and therefore how it reflects in options pricing and trading patterns.


KMeans clustering for Image Data I am trying to cluster the sample of Imagenet Dataset using K-Means clustering.

In this approach, I have used the below 2 approaches to get the optimal number of clusters.

Elbow method   From the Graph it seems like the best number of clusters is from 6 to 10.
Silhouette score

Cluster : 2 | Silhouette score : 0.036273542791604996
Cluster : 3 | Silhouette score : -0.00300691369920969
Cluster : 4 | Silhouette score : 0.0025101888459175825
Cluster : 5 | Silhouette score : -0.005924953147768974
Cluster : 6 | Silhouette score : -0.00808520708233118
Cluster : 7 | Silhouette score : -0.006091121584177017
Cluster : 8 | Silhouette score : -0.00549863139167428
Cluster : 9 | Silhouette score : -0.014739749021828175
Cluster : 10 | Silhouette score : -0.021131910383701324
Cluster : 11 | Silhouette score : -0.04057755321264267
Cluster : 12 | Silhouette score : -0.012825582176446915
Cluster : 13 | Silhouette score : -0.012340431101620197
Cluster : 14 | Silhouette score : -0.032936643809080124
Cluster : 15 | Silhouette score : -0.04154697805643082
Cluster : 16 | Silhouette score : -0.04323640838265419

Where as in Silhouette analysis it looks like only for cluster 4 it is showing better value. Rest of the clusters are seems like samples are wrongly assigned to wrong clusters.

In such cases, which metric needs to be considered?

Updates:
I have reduced the dimension of the features using PCA. Below are the updated graphs for Elbow and Silhouette analysis. However I do not see any improvement in the clustering. As per Silhouette the samples are not being assigned to the closest cluster.

Elbow

Silhouette

Updating the question with latest graphs, VGGNet has been replaced by Resnet50

Code snippet:

        for i in range(lower_range, upper_range):
            # For Elbow curve
            kmeans_cluster = KMeans(n_clusters = i)
            kmeans_cluster_fit = kmeans_cluster.fit(features_list_np)
            loss = kmeans_cluster_fit.inertia_
            error.append(kmeans_cluster_fit.inertia_)

            # For Silhoutte analysis
            preds = kmeans_cluster.fit_predict(features_list_np)
            score = silhouette_score(features_list_np, preds)
            score_list.append(score)

            sample_score = silhouette_samples(features_list_np, preds)
            sample_score_list.append(sample_score)

            print("Cluster : {} | Loss : {} | Silhoutte Score : {}".format(i, loss, score))

            zero_samples = 0
            positive_samples = 0
            negative_samples = 0

            for each_sample in sample_score:
                if each_sample == 0:
                    zero_samples += 1
                if each_sample > 0:
                    positive_samples += 1
                if each_sample < 0:
                    negative_samples += 1

            print("Cluster : {} | Silhouette sample distribution - Zero : {} | Positive : {} | Negative : {}".format(i, zero_samples, positive_samples, negative_samples))

            zero_list.append(zero_samples)
            pos_list.append(positive_samples)
            neg_list.append(negative_samples)

Also should the importance should be given to Silhouette Score or the Silhouette Samples. Because in some of the cases, Silhouette score is less but the number of samples having positive values are more.

Thank you,
KK

Neither, there is not enough discriminatory information in data (yet)

Dont squeeze the data until it tells you the truth. You can change the metric (malahobian distance for example) and the algo but you cant expect it to show miracles.

Using elbow method, as you increase the number of clusters it will always become more homogenous. You dont have a "kink" indicating optimal clustering number.

And with Silhouete (be careful those are averaged Silhoute score per cluster, in other words for k=4 you have 4 scores indicating wether points are lying inside/border/should- be-other-cluster that are averaged) you get that all of the points (in average) lie on the border of the clusters, without clear distinction of the clusters (thats what 0 means).

Advice Find better quantitative representation of data, new feature, reduce noise etc...


How to choose feedforward architecture for few number of features but very large instance? Assume I have 1 million of data instance and each instance contains 100 feature. For each instance, I also have a lable.
The size of X will  be 1,000,000 X 100 and the y will be 1,000,000 X 1. The value of y is binary.
Under this case, how to pick feedforward architecture?
Such as 100-50-1 or 5000-2500-50-1, any suggestion.
Thanks.

Is your set of possible labels discrete or continuous?
If it's discrete and you have L possible different labels, then you should have L units in the output layer (then do a softmax if you want probabilities).
If it's continuous, then you should have 1 output unit.
Lastly, the number of units in the input layer is equal to the flattened dimension of one instance of the input. So if your data is 100 dimensions, then you must have 100 units in the input layer.


Learning capacity: Deep Learning vs Traditional (Shallow) Learning I am currently doing a course in coursera in which Andrew Ng draws the following image:

Does anybody know any references/reasoning that justify the drawn graph? Were any experiments conducted to obtain it? If so, could you point me to them?

I have already asked in the coursera forum and sent an e-mail to the support team with no avail.

Thank you

I believe this statement can be supported with the concept of VC dimension. This blogpost provides a simplified explanation of the term. VC dimension can be understood as an ability of a classifier to learn complex dependencies in the data. From other hand, models with a huge VC bound tend to overfit on small amounts of data.

However, the plot you provided here depicts what is happening when our training set grows indefinitely - the win situation for models with a large VC dimension.

The VC bound of neural networks is something like polynomial from number of weights and connections between them. While VC dimension of a SVM classifier is linear to the dimensionality of the space it operates in.

Thus, neural networks can benefit more from larger amounts of training data. And the more weights you have, the better will be the result. Of course, given that the number of training samples is at least 10 times more than number of weights (a rule of thumb), so you do not face overfitting. That is why data augmentation and regularization are so important in deep learning.


Detect constant (zero-slope) sections in a noisy step function I have data that has the shape of a step function, which is obstructed by noise. I would like to identify the sections with constant slope in it.

The noise does not necessarily have 0 mean, and the steps are not always abrupt. Also, the points on the x-axis are not equidistantly spaced.

I was thinking about trying one-sided differentiators (like here) but not sure if they would work with non-equidistantly spaced data.

I feel like some sort of a global approach based on optimization would be an elegant way to solve that but so far I couldn't come up with one.

For what you're trying to do, first-order differences without interpolation should work just fine. Once you've done that, the problem reduces to a simple anomaly detection task. Here's a demonstration using topoligical anomaly detection. Briefly:

  The technique is essentially a density based outlier detection algorithm that, instead of calculating local densities, constructs a graph of the data using nearest-neighbors. The algorithm is different from other kNN outlier detection algorithms in that instead of setting ‘k’ as a parameter, you instead set a maximal inter-observation distance (called the graph “resolution” by Gartley and Basener). If the distance between two points is less than the graph resolution, add an edge between those two observations to the graph. Once the full graph is constructed, determine which connected components comprise the “background” of the data by setting some threshold percentage of observations ‘p’: any components with fewer than ‘p’ observations is considered an anomalous component, and all the observations (nodes) in this component are outliers.

First, let's generate some data:

n_obs = 100
spike_perc = .1

###################

set.seed(123)

n_spikes = floor(spike_perc * n_obs)
inter_arrival = rexp(n_obs)
xv = cumsum(inter_arrival)

mu_spike = 5
sd_spike = sqrt(mu_spike)
sd_noise = mu_spike/75
spike_events = sample(n_obs, n_spikes)

flat_noise = rnorm(n_obs, 0, sd_noise)
spike_ampl = abs(rnorm(n_spikes, mu_spike, sd_spike))

yv = rep(0, n_obs)
yv[spike_events] = yv[spike_events]  + spike_ampl
yv = cumsum(yv)
yv = yv + flat_noise

Here's what our data looks like (spike events in red):

Calculate first-order differences (appending 0 to preserve index mapping):

dat = c(0,diff(yv))

Find us some outliers:

library(igraph)

# Helper function. Equivalently, use mefa::stack
row_col_from_condensed_index = function(n, ix){
  nr=ceiling(n-(1+sqrt(1+4*(n^2-n-2*ix)))/2)
  nc=n-(2*n-nr+1)*nr/2+ix+nr
  cbind(nr,nc)
}

# Algorithm parameters, may require tuning
rq = .2 # distance quantile for thresholding (higher->fewer outliers)
p = .1  # percent of population necessary for a component to be flagged as an inlier
method='euclidean'

# calculate inter-observation distances and threshold to define adjacency matrix
n=NROW(dat)
d = dist(dat, method=method)
r = quantile(d, rq)
edges = t(sapply(which(d<=r), function(x) row_col_from_condensed_index(n,x)))

# build adjacency graph
g = graph.empty(n, directed=FALSE)
g[from=edges[,1], to=edges[,2]]=1
V(g)$name = 1:n

# Flag outliers based on component size
components = clusters(g, mode="strong")
csize = components$csize[components$membership]
outliers = V(g)[csize<=p*n]$name

And here's how we did (flagged outliers in red):

If you go this route, you may need to tweak the rq and p parameters a bit. If you'd rather not deal with the quantile thresholding parameter (rq), an alternative approach is to use a nearest neighbor graph, in which case you'll have to tinker with the 'k' parameter (of KNN). That approach would replace the distance calculations and graph instantiation above with:

library(cccd)

g = nng(data.frame(dat), k=5, use.fnn=TRUE)


How to use flow_from_directory in keras for csv flow_from_directory in Keras requires images to be in different subdirectories. However, I have the images in a single directory with a csv file specifying the image name and target classes.

How do I use flow from directory directly from csv files named train.csv and test.csv?

Something Like This Should Do The Job
When you are doing something new, Mistakes are likely..
Use At Your own Risk Or Try It Out on A Sample And Try it on a Seperate Directory Completely

import pandas as pd
import os
import numpy as np
import shutil

# source is the current directory
# Open dataset file
dataset = pd.read_csv('dataset.csv')
file_names = list(dataset['filenames'].values)
img_labels = list(dataset['labels'].values)

folders_to_be_created = np.unique(list(dataset['labels'])).values

source = os.getcwd()

for new_path in folders_to_be_created:
    if not os.path.exists(".//" + new_path):
        os.makedirs(new_path)

## Be sure that there is nothing else in your directory except the data, csv and the code file, IT's Better to only have your data in that directory and reference the CSV file from a different Directory...

folders = folders_to_be_created.copy()

for f in range(len(file_names)):

  current_img = file_names[f]
  current_label = img_labels[f]

   ## **Check this Line Accordingly**

  shutil.move("path//to//current//file", "path//to//new//destination//folder//current_label//")

Link for Shutil

Another Link


Suitable modelling approach to incorporate different signals I am working with data in the form of content that is being reviewed to decide if it violates the policy standards or not. In the review process there are several methods to increase the chances of making the correct decision. Content might be sent for multi-review in which case it is reviewed additional times to help get to a correct decision. We can also check the historical accuracy scores of the humans reviewing the content, higher accuracy scores indicate a higher likelihood of a correct decision. Similarly, we can check some other factors such as length of review time, the type of the original content etc.
There is a separate audit process which evaluates a subset of decisions and decides on the policy correct decision in that instance. We can compare these policy correct decisions to the original decisions to evaluate the marginal lift in accuracy from the multi-review system, reviewer accuracy etc. The goal is to develop a system which incorporates these signals and outputs a score that represents our confidence that the review decision is correct.
I have done analysis comparing the marginal improvement in accuracy based on the individual signals, I have also calculated the margin of error for these scores depending on the number of samples we have. For example, when content gets multi-reviewed it is x% more like to be the correct decision with a margin of error of 0.5%. Some signals have a higher margin of error since we had less samples.
I'm trying to figure out the best way to incorporate all of the signals into a single score. I am not sure if regular classifiers will apply since only a portion of posts get multi-reviewed and we only have historical reviewer accuracy scores for a portion.
The way I have been thinking about it in my head is in the form of a decision tree, if the content has been multi-reviewed yes/no, if we have an accuracy score yes/no. Perhaps a decision tree / random forest model would be suitable. It would be great if the model was interpretable and the output could be explained. I would also be open to a more straight forward solution without the need for complex modelling.
Thanks for any advice you can provide on this :)

Combing multiple models together is called typically ensembling.
You are describing hierarchical ensembling. There are simpler methods for ensembling, such as voting and weighted voting.


How to standardize my data (Univariate Time Series Forecasting using Keras LSTM)? Let be $X = (X_1,...., X_n)$ an univariate time serie. I would like to know how to standardize my data when I split it into train and test data. Let me explain you how I tranform $X$ so that I can fit an LSTM neural net. From $X$ I make a new input data and its corresponding output data. So, we have:
$X = ( (X_1, ..., X_m), ... , (X_{n-m}, ..., X_{n-1}) )$
$Y = (X_{m+1}, ..., X_n)$
$\text{Card}X = \text{Card}Y$
Let's set $p$ the size of my test set. If I use Python's notation, we have:
$X_{train} = X[:-p]$
$X_{test} = X[-p:]$
Idem for $Y$ ... Now, I am wondering how to standardize my data. I think that standardizing $X$ before splitting the data into train and test sets could lead to over-fitting since we a transformation that involves all $X_i$. Basically, I am not sure that the sum (mean, standard deviation) will drown the information. In this case I think it could be better to just compute the mean and the standard deviation in the training set and use them to standardize both of the train and test sets. For me it makes no sense to standardize them separately since $\text{Card}X_{test} << \text{Card}X_{train}$. But may be I am wrong. I would also like to know whether I have to standardize $Y$ and $X$ or just $X$. When I am working with MLP neural net I used to just normalize the input data..

So, thank you first for reading and if you have any ideas or remarks, any questions to ask, please let me know. I can explain more it is up to you :)

P.S. I don't find a 'standardization' tag and I thus use the tag named normalization.

Welcome kakarotto! The first thing, standarize with respect to the training set only, then use the statistics to standarize the other sets. As a rule of thumb, we should keep in mind that the test set does not exits while training... (although we have access to it) When training, if the network does not see the test set we (the humans) do either…

If the test set was well selected, and so the whole dataset, the training set and the test set should come from the same probability density function and, if enough data is provided, the statistics you get from the training set would be close to the statistics of the pdf (assuming unbiased estimators for these statistics, that should be the case for the mean and standard deviation)

Based on my experience, only standarize the input data. With the output data, I usually scale it (min-max scaler) if I know the boundaries (it they even exist…). Another usefull trick that usually worked for me its to not feed the net with the input data rather with its relative differences, for example:

$X = (0, X_1 - X_0, X_2 - X_0,\dots,X_m-X_0)$


Split into test and train set before or after generating document-term matrix? I'm working on simple machine learning problems and I trying to build a classifier that can differentiate between spam and non-spam SMS. I'm confused as to whether I need to generate the document-term matrix before splitting into test and train sets or should I generate the document-term matrix after splitting into test and train?

I tried it both ways and found that the accuracy is slightly higher when the I split the data before generating the document-term matrix. But to me, this makes no sense. Shouldn't the accuracy be the same? Does the order of these operations make any difference?

I gave a lot of thought about the question. I agree with you. But the slight difference might come if there are any random variable operation happens during the training. What model are you using for training?


In Machine Learning, what is the point of using stratified sampling in selecting test set data? I am currently learning machine learning via this book "Hands-On Machine Learning with Sci-kit learn and Tensorflow" by Aurelien Geron.

In page 76 and 77, the author talks about using stratified sampling so that your test set would be more representative of the whole data. I didn't really understand the point of this since it would not affect the accuracy of your training model ? or how would selecting better test sets affect the accuracy of your training model ?

When the distribution of your data is balanced or you have enough samples of each class, a normal shuffle split will work well. But if your data distribution is unbalanced and one of the classes is in minority, you do stratified sampling so as to ensure that your train and test splits represent the true nature of your data. For more details take a look here.


What are the current killing machine learning methods? I was wondering whether we could list machine learning winning methods to apply in many fields of interest: NLP, image, vision, medical, deep package inspection, etc. I mean, if someone will get started a new ML project, what are the ML methods that cannot be forgotten?

The question is very general. However, there are some studies being conducted to test which algorithms perform relatively well in a broad range of problems (I'll add link to papers later), concerning regression and classification.

Lately Random Decision Forests, Support Vector Machines and certain variations of Neural Networks are being said to achieve the best results for very broad variety of problems.

This does not mean that these are "the best algorithms" for any problem, that does not exist, and actually is not very realistic to pursue. Also it must be observed that both RDF and SVM are rather-easy methods to initially grasp and obtain good results, so they are becoming really popular. NN have been used intensively since couple of decades (after they revived), so they appear often in implementations.

If you are interested in learning further you should look for an specific area and deal with a problem that can be solved nicely by machine learning to understand the main idea (and why is impossible to find  the method).

You will find common the task to try to predict the expected behavior of something given some known or observable characteristics (to learn the function that models the problem given input data), the issues related to dealing with data in high-dimensional spaces, the need for good quality data, the notable improvements that can give data pre-processing, and many others.


Looking for a machine's breakdown dataset I am searching for a dataset that contains a machine's (any mechanical machine) operational variables (e.g. vibration, temprature, etc. ) versus it's operational status (e.g. functioning, malfunction, breakdown).
I have failed to find such dataset with googling.
Is there such a dataset?

I would recommend the NSSDC NASA data of 550 space spacecraft

https://nssdc.gsfc.nasa.gov/nssdc/obtaining_data.html
https://pds.jpl.nasa.gov/


What are the steps and correct order of the operations in Machine Learning? [from Getting data to optimising models] I've followed lots of tutorials on Machine Learning but in each of these, they go for a different strategy so it's quite confusing for me. I want to Know that what are the operations involved and what are the correct ordering of these.

AS of now, I think the process and the ordering are ->

Get Data
Delete Duplicates
Find Missing Values and Outliers
Create New Features
Deal with missing values and Outliers
Build a base model
Find the best features to select
Try and find different Models
Select the BEST model
Hypertuning of the Model

Please Do Provide if something is missing and correct the sequence.

Yes learning online from different tutors can lead to different strategies. But this stats that the flow depends upon the type of data. Not everytime you have to create features or deal with outliers.

In case of images you reduce noise by applying different methods & filters, in case of text you remove stopwords, punctuation etc. So it depends upon data & its amount. General way to proceede for me is as follow

Understand definition & get domain knowledge
Gather data
Pre-process it (Involves removing noise, applying transformation, pruning data etc)
Do feature selection if necessary
Do data visualisation
Do different model experiments with sample data.
Fix 2 3 models for case
Train with real data
Use different matrices to evaluate (depends upon the data & definition)
Make model accessible using API/Direct to client

This is how in industry i have seen everyone around me works & i follow the same. Has helped me but this is way general, here you have much to change based upon definitions & tasks.


Supervised or unsupervised learning for predicting energy consumption for new buildings I’m working on an model for auto dimensioning district heating pipes for new district heating areas (new customers). I have energy consumption data on hourly basis and describe data about these consumers (e.g. building year, renovation year and floor area) for a lot of district heating consumers. I want to predict dimensional load (energy consumption) for new building connected to the grid based on the describe data (e.g. building year, renovation year, floor area). I have through about using supervised or unsupervised learning, but do not know which of them that fit my needs? Is there an mode (e.g. clustering, SMV, …) that are better than other for this purpose?

Building on The Lyrist answer, I would try to organize all that data and create as many instances as possible of Input-Output. If I understand your case correctly, the input would be the multiple data point that you feel have an impact on the load (if you're not sure about it's impact, chuck it in for now) that you would set up as a vector of values. Then the output would be a single value of load, such as Kwh/day (that doesn't sound right), but just get one 'value' you're interested in predicting. Don't forget to keep training, validation and test sets.

Now you can frame your problem both as a classification problem by creating categories, such as from 0 to 10, 11 to 20, etc, or go for a regression problem, predicting a continuous value. Do lookup past application of similar problem, as they are, to some extent, application specific. Note that the some algorithm might work better with regression, other with classification. Personally, I think that creating buckets of classification might be easier, unless predicting an exact value is important to you.

Now regarding the specific model to go for, it usually is recommended to start with simple model (SVM's is a popular one) and build to more complicated model (kerneled SVM), if you enough data! Resist the temptation of going straight for a 12 layers MLP, as those old SVM's.


Different results obtained for OneVsOneClassifier (or OneVsRestClassifier) when using ordinary KFold and StratifiedKFold cross validation When I fitted a OneVsOneClassifier (or OneVsRestClassifier), I noticed I obtained different results when I used ordinary KFold and StratifiedKFold cross validation. The testing set performance is much lower when ordinary KFold is used compared to when StratifiedKFold is used.

Questions:

1) Am I right to say for OneVsOneClassifier (or OneVsRestClassifier) strategy to work, the proportion of each class needs to be preserved in training and testing sets?

2) Is it better to use ordinary KFold or StratifiedKFold when I want to use OneVsOneClassifier (or OneVsRestClassifier)?

Many thanks.

TL;DR

Use StratifiedKFold and the best performing model (either OneVsOneClassifier or OneVsRestClassifier).

Splitting your training and test set.

Below is are the two approaches you could use for splitting your train and test set. The KFold and the StratifiedKFold:

The main difference between the two is that StratifiedKFold will ensure that the class distributions are always the same in train and test. If you would use simple KFold, it might be possible that a class will appear in the test set, but never in the training set. If that happens, all the samples from that class will get wrongly predicted. So, StratifiedKFold at least gives your model a chance to see one sample from each class.

Regardless of the model you pick, use StratifiedKFold to evaluate your model.

OneVsOne or OneVsRest

It is difficult to say which model would perform best because it will depend on your data's distribution. So, I would say for you to evaluate both and pick the best performing one.

What is interesting, is that you get to choose what best performing means. It could be
the mean precision/recall/F1 score in a cross-validation setting. But it could also be the one that uses less memory, or the fastest. In general it is a mix of all these things with tradeoffs between them. In general, the OneVsOneClassifier performs better in the F1 score, but it is slower than the OneVsRestClassifier, but please not the words "in general". Perform your own tests and see which one works for you.

I usually prefer the OneVsOneClassifier, because it tries to model every class against each class. This way, classes are self contained and finding boundaries around that space is more convenient. When you have a OneVsRestClassifier, you need to model the "rest" class, which can be really complex since it is made up of various classes, which can become problematic.


Understanding pseudocode for co-democratic learning I am reading this blog from Sebastian Ruder blog-link, researcher scientist at Deepmind, and
having problems understanding this pseudocode for Democratic Co-learning. Can somebody help me understand what is going on; specifically, I'm just trying to understand line 1-6.

Description of Democratic Co-learning from blog:

Democratic Co-learning Rather than treating different feature sets as views, democratic co-learning (Zhou and Goldman, 2004) [18] employs models with different inductive biases. These can be different network architectures in the case of neural networks or completely different learning algorithms. Democratic co-learning first trains each model separately on the complete labelled data L. The models then make predictions on the unlabelled data U. If a majority of models confidently agree on the label of an example, the example is added to the labelled dataset. Confidence is measured in the original formulation by measuring if the sum of the mean confidence intervals w of the models, which agreed on the label is larger than the sum of the models that disagreed. This process is repeated until no more examples are added. The final prediction is made with a majority vote weighted with the confidence intervals of the models. The full algorithm can be seen below. M is the set of all models that predict the same label j for an example x.

I guess line 1-3 is the training of several models on labeled data. Is i each sample in the labeled dataset? Is mi a trained model using all the samples from the labeled dataset?
I guess that line 4-6 is going through each sample in the unlabeled dataset but what's j? Is it iterating through each label?

This line I'm especially confused about is M <- {i | pi(x) = j}. Is i being assigned the output of the model from the unlabeled sample and then being assigned to M and what is M?

Let's try to make some sense of what is going on here:

In the first 3 lines as you say, we build $n$ models, being $m$ a single model.

The second loop starts iterating over the unlabelled sample set $U$. For each unlabelled sample $x$, we iterate over the possible labels (e.g $\{0,1\}$ for binary classification, or $dog\space vs.\space cat$, etc...) from the set $C$ and we call it $j$.

$M$ thus is the set of models $m$, that predict that sample $x$ is of the class $j$

After that, if the number of models $|M|$ (where $0\leq |M|\leq n$) that agree that label is $j$ are more than half of the models available ($|M|>n/2$) and the weighted mean ($w$ is the confidence intervals
of the models) of the models that agree on this prediction is higher than the models that disagree (line 7) → then the sample $x$ is included in $L$ with label $j$ (line 8).


Fast Python implementation of the gradient descent I'm looking for fast Python implémentations of gradient descent optimization algorithm. I have a convex problem , with no constraint, so for now I'm using the BFGS algorithm implemented in scikit-learn ( minimize ).

Is there anything faster / scalable on multi-core systems ?

Parallel gradient descent has been implemented in this repository in Python. It should have a familiar interface, since it's being developed for implementation as a scikit-learn feature.


Is the F1 Score sensitive to the threshold? Is the F1 score sensitive or indifferent to the threshold (for defining positive or negative)?

Yes, it is. The formula:

$$F_1 = \frac{2TP}{2TP + FP + FN}$$

There is no symmetry. You have only $TP$ in the nominator which means that $F_1=0$ if $threshold=1$.


What is the efficient way to use apply method in column of pandas Dataframe for large dataset? I have a dataset of approximate 1 hundred thousand records. I want to use apply method in each of the records for further data processing but it takes very long time to process (As apply method works linearly).
I have tried this in Google Colab by selecting GPU settings but still it is very slow. I also try "swifter.apply" but still it is not as efficient.
Is there any way to do this?

You can try pandarallel it works very efficiently for parallel processing. You can find more information about it here.

You should not use this if your apply function is a lambda function. Now assuming you're trying to apply it on a DataFrame called df:

from pandarallel import pandarallel

pandarallel.initialize(nb_workers=n) #n is the number of worker used for parallelization, you can leave it blank and it will use all the cores

def foo(x):
   return #what ever you're trying to compute

df.parallel_apply(foo, axis=1) #if you're applying to multiple columns
df[column].parallel_apply(foo) # if its just one column

Another option you can try is using the python multiprocessing library, here you will break your dataframe into smaller chunks and run them together.

import numpy as np
from multiprocessing import cpu_count, Parallel

cores = cpu_count() #Gets number of CPU cores on your machine
partitions = cores #Define number of partitions

def parallelize(df, func):
    df_split = np.array_split(df, partitions)
    pool = Pool(cores)
    df = pd.concat(pool.map(func, df_split))
    pool.close()
    pool.join()
    return df

Now you can run this parallelize function on your df:

df = parallelize(df, foo)

The more number of cores you have the faster this will be!


Hierarchical dirichlet process results I am thinking about using hierarchical dirichlet process to model a patent dataset. I've seen that HDP uses a base distribution and assumes that every topic comes from that base distribution.
The problem is: first I'm wondering what are the main results from the HDP procedure (in the case of LDA we obtain two matrices that we can use to construct word clouds and graphs but in this case I'm not sure about the results) and what is the exact procedure?

TL;DR Ist just a bayesian varianat that generalises LDA.
See here
Basically in the bayesian Setting you dont Need the params, but you can infer them from the data. Using the same dirichlet generative process logic you can cluster grouped data.


Filter Data for clustering I am trying to synthetize clients Data in order to do clustering. My problem is for 1 customer I have severals rows. I would like to synthetize informations to get 1 row per customer. This clustering is about how customers use fidelity program. Here is a picture of my table : By Column (left to right) :

1) CustomerID 2)Date at which their use their points 3) Category number (Ex: 1 is gift card, 2 is a flight etc) 4) How many points they used 5) How many items they purchased with points

My question is how could I have 1 customer per row without loosing informations.
Maybe Pivot Table? But I don"t know how it work exactly. I am new to statistic btw.

Thank you

Cédric

If you can afford to do the full join once, do it and learn which columns are useful through feature selection. Then you can only SELECT these columns for subsequent iterations, when the database is updated.

Here's a survey: Feature Selection for Clustering: A Review


How to label overlapping objects for deep learning model training I am training yolov3 to detect a custom object (chickens). In a lot of my training images I have overlapping chickens (can only see a partial chicken etc). Is there a common practice for how to label the data (bounding box) in these cases? Should you only label the portion of the image which you can see?

There is no common practice in labeling the bounding boxes. It is always problem dependent. For example, if you want to count the chickens then you should also label the whole chicken as one instance of a chicken.
If you simply what to detect if there is a chicken in the picture you should label the unoccluded part.

You have to think about your problem. What is the goal of the algorithm? Could a human do the task without imagining where the rest of the object is? You should also consider the pixel imbalance for your problem. In general, the first method is a harder task than the second method because even humans have problems in labeling the bounding box for occluded objects. Hence, you will have a lot of variance due to this factor. If you label only what you see the bounding box labeling will be more reliable. As far as I know, the PASCAL Visual Object Classes data set which was used in the YOLO publication did only label what you can see and not what is occluded.

BTW I hope your task aims to improve the live quality of the chickens. It would be a shame if machine learning would be used to harm them.


What loss function does the 'multinomial' distribution with the gbm package in R use? All distributions in the gbm package in R are associated with a loss function. For example, when we set distribution = 'binomial', the loss function chosen internally is the logistic loss function. Can anyone explain how multi-class classification works with gbm and the loss function that is being used for it i.e. when we set distribution='multinomial'? Is it using one-vs-all or all-vs-all under the hood for doing its multi-class classification?

Here is the relevant source code. I'm pretty sure it's using cross-entropy for multiclass:

    //  GBM by Greg Ridgeway  Copyright (C) 2003

#include "multinomial.h"

CMultinomial::CMultinomial(int cNumClasses, int cRows)
{
   mcNumClasses = cNumClasses;
   mcRows = cRows;

   madProb = new double[cNumClasses * cRows];
}

CMultinomial::~CMultinomial()
{
   if(madProb != NULL)
   {
      delete [] madProb;
   }
}

GBMRESULT CMultinomial::UpdateParams
(
   double *adF,
   double *adOffset,
   double *adWeight,
   unsigned long cLength
)
{
   // Local variables
   unsigned long ii=0;
   unsigned long kk=0;

   // Set the probabilities for each observation in each class
   for (ii = 0; ii < mcRows; ii++)
   {
      double dClassSum = 0.0;
      for (kk = 0; kk < mcNumClasses; kk++)
      {
         int iIdx = ii + kk * mcRows;
         double dF = (adOffset == NULL) ? adF[iIdx] : adF[iIdx] + adOffset[iIdx];
         madProb[iIdx] = adWeight[iIdx] * exp(dF);
         dClassSum += adWeight[iIdx] * exp(dF);
      }

      dClassSum = (dClassSum > 0) ? dClassSum : 1e-8;

      for (kk = 0; kk < mcNumClasses; kk++)
      {
         madProb[ii + kk * mcRows] /= dClassSum;
      }
   }

   return GBM_OK;
}

GBMRESULT CMultinomial::ComputeWorkingResponse
(
    double *adY,
    double *adMisc,
    double *adOffset,
    double *adF,
    double *adZ,
    double *adWeight,
    bool *afInBag,
    unsigned long nTrain,
    int cIdxOff
)
{
    unsigned long i = 0;

    for(i=cIdxOff; i<nTrain+cIdxOff; i++)
    {
       adZ[i] = adY[i] - madProb[i];
    }

    return GBM_OK;
}

GBMRESULT CMultinomial::InitF
(
    double *adY,
    double *adMisc,
    double *adOffset,
    double *adWeight,
    double &dInitF,
    unsigned long cLength
)
{
    dInitF = 0.0;
    return GBM_OK;
}

double CMultinomial::Deviance
(
    double *adY,
    double *adMisc,
    double *adOffset,
    double *adWeight,
    double *adF,
    unsigned long cLength,
    int cIdxOff
)
{
    unsigned long ii=0;
    double dL = 0.0;
    double dW = 0.0;

    for(ii=cIdxOff; ii<cLength+cIdxOff; ii++)
    {
        dL += -adWeight[ii] * adY[ii] * log(madProb[ii]);
        dW += adWeight[ii];
    }

    return dL/dW;
}

GBMRESULT CMultinomial::FitBestConstant
(
    double *adY,
    double *adMisc,
    double *adOffset,
    double *adW,
    double *adF,
    double *adZ,
    unsigned long *aiNodeAssign,
    unsigned long nTrain,
    VEC_P_NODETERMINAL vecpTermNodes,
    unsigned long cTermNodes,
    unsigned long cMinObsInNode,
    bool *afInBag,
    double *adFadj,
   int cIdxOff
)
{
      // Local variables
    GBMRESULT hr = GBM_OK;
    unsigned long iNode = 0;
    unsigned long iObs = 0;

   // Call LocM for the array of values on each node
    for(iNode=0; iNode<cTermNodes; iNode++)
    {
        if(vecpTermNodes[iNode]->cN >= cMinObsInNode)
        {
         // Get the number of nodes here
         double dNum = 0.0;
         double dDenom = 0.0;
         for (iObs = 0; iObs < nTrain; iObs++)
         {
            if(afInBag[iObs] && (aiNodeAssign[iObs] == iNode))
                {
               int iIdx = iObs + cIdxOff;
                    dNum += adW[iIdx] * adZ[iIdx];
               dDenom += adW[iIdx] * fabs(adZ[iIdx]) * (1 - fabs(adZ[iIdx]));
                }
         }

         dDenom = (dDenom > 0) ? dDenom : 1e-8;

         vecpTermNodes[iNode]->dPrediction = dNum / dDenom;
        }
    }

    return hr;
}

double CMultinomial::BagImprovement
(
    double *adY,
    double *adMisc,
    double *adOffset,
    double *adWeight,
    double *adF,
    double *adFadj,
    bool *afInBag,
    double dStepSize,
    unsigned long nTrain
)
{
    double dReturnValue = 0.0;
    double dW = 0.0;

   unsigned long ii;
   unsigned long kk;

   // Calculate the probabilities after the step
   double *adStepProb = new double[mcNumClasses * mcRows];

   // Assume that this is last class - calculate new prob as in updateParams but
   // using (F_ik + ss*Fadj_ik) instead of F_ik. Then calculate OOB improve
   for (ii = 0; ii < mcRows; ii++)
   {
      double dClassSum = 0.0;
      for (kk = 0; kk < mcNumClasses; kk++)
      {
         int iIdx = ii + kk * mcRows;
         double dF = (adOffset == NULL) ? adF[iIdx] : adF[iIdx] + adOffset[iIdx];
         dF += dStepSize * adFadj[iIdx];
         adStepProb[iIdx] = adWeight[iIdx] * exp(dF);
         dClassSum += adWeight[iIdx] * exp(dF);
      }

      dClassSum = (dClassSum > 0) ? dClassSum : 1e-8;

      for (kk = 0; kk < mcNumClasses; kk++)
      {
         adStepProb[ii + kk * mcRows] /= dClassSum;
      }
   }

   // Calculate the improvement
    for(ii=0; ii<nTrain; ii++)
    {
        if(!afInBag[ii])
      {
         for (kk = 0; kk < mcNumClasses; kk++)
         {
            int iIdx = ii + kk * mcRows;
                dReturnValue += adWeight[iIdx] * adY[iIdx] *
                               (log(adStepProb[iIdx]) - log(madProb[iIdx]));

            dW += adWeight[iIdx] * adY[iIdx];
         }
      }
    }

    return dReturnValue/dW;
}


Backpropgating error to emedding matrix I understand the backpropagation algorithm of neural networks, and how the error propagates backwards in layers.  That is, I understand that given a 3-layer feed forward network, the amount to change W1 is dependent on the weights from layers 2 and 3, as well as the derivatives of their activation functions.

Question: When your first layer is an embedding layer (i.e., consider initializing the embedding matrix with glove), how does the network update that matrix using backpropagation? How do you represent that layer as an equation consisting of the input and some weight matrix?

An embedding layer is in fact a linear layer. It maps the input, using a matrix multiplication, to the output, without any activation function after the multiplication. Therefore, the backpropagation is exactly as you would do with linear layer.

Why don't we just call it linear layer, then?

At theory level, an embedding layer performs a matrix multiplication to the input. However, in practice, the coded implementation is slightly different. This is due to the fact that the input, as a category, is incoded in a one-hot way, and the matrix multiplication by a one-hot encoded vector is as easy as a look-up, so there is no need to multiply the whole matrix.


How to change a Pandas dataframe into feature vector? I have a Pandas dataframe with 10 columns, 9 of which are features to be used to predict the 10th column.

How is it ossible to convert this Pandas dataframe into X and y vectors to use in a linear regression problem?

If you have your dataframe loaded as the variable df, you can simply use this

X = df[['A','B','C']]
y = df['Z']

where A, B and C are your independent variables and Z is your dependent variable.


Does YOLO give preference to color over shape or vice-versa while detecting an object? If you train your YOLO model only on grayscale images to detect car, then would it able to recognise a car in a colored image also. If so, then can I assume that YOLO consider only object shape not color? Kindly clarify.

YOLO has 3 input channels (RGB typically) and is a CNN, a neural net that uses stacked (trainable) convolution filters. Single convolution filters act as feature detectors and can be (trained to be) sensitive for straight edges from dark to light or a gradual passage from green to red. By stacking those you can build filters for more complicated features (such as faces).
TL;DR The network can learn to differentiate green and red circles if you want to.


Why can't i normalize my data using preprocessing.normalize? code:

df['Miles'] = preprocessing.normalize(df['Miles'])

Error msg:

ValueError: Expected 2D array, got 1D array instead:
array=[33.3304678  33.3304678  33.3304678  ...  3.00959404  3.00959404
  3.00959404].
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.

Sklearn.preprocessing.normalize is vector norm normalization. By default it use L2 Norm which is $x_{norm}=\sqrt{\Sigma{x_i^2}}$. For example, if there's one vector $(1,2,3)$, then the $x_{norm}=\sqrt{1^2+2^2+3^2}=3.7416$, then if you normalize the vector, it would be $(\frac{1}{3.7416},\frac{2}{3.7416},\frac{3}{3.7416})$.

If that's what you want, try

# convert to numpy array with shape(rows,1)
df_array = df['Miles'].values.reshape(-1,1)
df['Miles'] = preprocessing.normalize(df_array,axis=0)

Note that setting axis=0, otherwise it would be done column-wise and you will get all 1 vector.

I think you may want to scale the vector rather than normalization, try mean-std scale by sklearn.preprocessing.scale, max-min scale by sklearn.preprocessing.minmax_scale. Be sure that you reshape the array first.


What are tokens and tokenizations? I'm a high school senior who is new to data science, and would like to get into natural language processing. I currently know nothing about NLP, and the information online can be overwhelming. What are tokens? What are they used for / why do we need to tokenize text?

Well, I think it is better to watch a course about it.
But, for instance, when people comment on different products on Amazon, there is a way to understand are their comments positive or negative, therefore their comments are used as data, analyzed and then labeled as positive or negative. Further, using this approach, one can find whether an email is spam or not, etc.

In order to analyze a text, we need to split it into its elements, meaning, its words (usually) each of these elements is called a token.

after splitting the text to words, useless words that do not represent anything, some signs, etc (the, ".", "my",...) are omitted, then we create bags of words, that are the most important words in the whole dataset (all comments), then each word is a column in the table and each row of the table is a comment and has 1 in a column if it contains that specific word. there is also a label column that shows if the comment is positive or not. then having this dataset, we can fit a classification model to the dataset.


How to get tf tensor value computed in loss function in keras train_on_batch without computing it twice or writing custom loop? I have a model and I've implemented a custom loss function something along the lines:

def custom_loss(labels, predictions):
    global diff
    #actual code uses decorator so no globals
    diff = labels - predictions
    return tf.square(diff)

model.compile(loss=custom_loss, optimizer=opt.RMSprop())

...

model.train_on_batch(input, labels)

#

How to get diff after I've run train_on_batch without causing
it to rerun predict a second time behind the scenes(unnecessary slowdown) and mess up with trainable/batchnorm etc(possible problems)?

I want to avoid making a manual raw tensorflow train_op loop etc, keeping track of learning phase and whatnot. Is this my only choice?

I'm using tensorflow 1.14's keras module

I've solved it(discovered control dependencies and remembered variables)
Basically, I create an assign diff to a variable operation, and with the help of control_dependencies force tf to do this operation every time op is calculated, this way when I get this variable, it doesn't cause graph recalculation

diff_var = tf.Variable()

def custom_loss(labels, predictions):
    diff = labels - predictions
    diff_var_op = diff_var.assign(diff)
    with tf.control_dependencies([diff_var_op]):
        return tf.square(diff)

test code

import tensorflow as tf
sess = tf.Session()
var1 = tf.Variable(1, dtype=tf.float32)
var2 = tf.Variable(2, dtype=tf.float32)
counter = tf.Variable(1, dtype=tf.float32)
var2_op = tf.square(var2)*counter

diff_var = tf.Variable(10, dtype=tf.float32, trainable=False)

diff = var2_op - var1
diff_var_op = diff_var.assign(diff)

with tf.control_dependencies([diff_var_op]):
    op = tf.square(diff)

sess.run(tf.global_variables_initializer())

print('diff var:', sess.run(diff_var))     #10
print('counter:', sess.run(counter))       #1
print('op:', sess.run(op))                 #9
print('diff var:', sess.run(diff_var))     #3
print('-')
counter_op = tf.assign_add(counter, 1)
print('counter:', sess.run(counter))       #1
print('diff var:', sess.run(diff_var))     #3  #still the same
print('var2:', sess.run(var2_op))          #4
print('-')
sess.run(counter_op)
print('after counter_op')
print('counter:', sess.run(counter))       #2
print('var2:', sess.run(var2_op))          #8
#still the same even though var2 has changed because of counter_op
print('diff var:', sess.run(diff_var))     #3
print('op:', sess.run(op))                 #49 #running full op
print('-')
print('after op')
print('diff var:', sess.run(diff_var))     #7
#variable changed, no operations involved


Find threshold in rate to determine reason for lost customer I'm not sure if anybody will be able to help with this, or even if I'll be able to explain it well but I am stuck so here goes....

I have a set of customers, some are lost, some are still active. We have a model which looks through the data and assigns reasons for why they became lost. One of the reasons to determine is if them not using our software caused them to become lost - we have software to automate their orders but they can ignore this and process orders manually.

I have created 2 variables; software usage rate and software usage vs their country average (as a ratio).

My aim is to determine the point, in either or both of these, which is basically the threshold/marker to say the software was an issue. I am stuck as there are in essence no results - we know if a customer is lost, but we can only speculate/model on the reasons.

All I've been doing so far is looking at the distributions of lost and active manually.

Any help would be great.

Thanks

You could use a two-sample $t$-test to determine whether software usage is significantly lower for lost users.

Plotting 'software usage rate'/'software usage rate relative to country average' against 'percentage of lost users' might give you an idea about a suitable threshold.


Is there any working alternatives to Netvizz? I'd like to download my friends data as a graph/network from Facebook. In Feb 2015, Facebook banned the awesome application Netvizz. Do you know any alternative application for my purpose?


Is Recursive Feature Elimination finding best features subset? On a set of 9 features I have applied Recursive Feature Elimination (RFE) algorithm using SVM estimator, following approach from (1). When requesting a subset of size 1 to be found, then RFE returned feature X.
However, when I trained SVM over each feature individually, I found another feature Y to have higher accuracy than SVM trained over X.
I thought that RFE finds features with the highest accuracy.
Is my understanding of RFE wrong?
(1): Gene selection for cancer classification using support vector machines

No, RFE cannot guarantee that it finds the feature subset with optimal score.
As with most greedy processes, the point of RFE is to reduce the computational cost (fitting a model for each of the $2^m$ feature subsets), at the cost of perhaps not finding the actual optimum (but hopefully "close enough").
See also https://stats.stackexchange.com/questions/232461/question-about-recursive-feature-elimination


Name of this algorithm for supervised cluster assignment Is there the name for an algorithm of cluster assignment that is based uniquely on the distance between the data point to classify and the center of the cluster?

Let me be more clear:

Let's say that I have two clusters $A,B$ made by $N$ points $x(i)_A$ and $x(j)_B$. I have to assign a new point $x_{new}$ to it's cluster.

The point is assigned to the cluster that minimize the distance between the point and the center of the cluster:

$$ min_{A,B} ( d_A(x_{new}, \frac{1}{N}\sum_{i \in A}x_i), d_B(x_{new}, \frac{1}{N}\sum_{j \in B}x_j) ) $$

Does this procedure has a name? Is there an algorithm similar to this? The closest algorithm that I can think of is k-Nearest Neighbour, but it's easy to think of cases where kNN performs properly and this algorithm performs very poorly. For instance, this algorithm as such will perform badly on data shaped non-uniformly / non-gaussian.

This question arise from a paper that offer a procedure for doing supervised quantum machine learning, and claims that:

  Consider the task of assigning a post-processed vector $u \in R^n$ to
  one of two sets $V, W$, given $M$ representative samples $v_j ∈ V$ and
  $M$ samples $w_k ∈ W$. A common method for such an assignment is to
  evaluate the distance $|u−\frac{1}{M}\sum_{j} v_j |$ between $u$ and
  the mean of the vectors in $V$ , and to assign $u$ to $V$ if this
  distance is smaller than the distance between $u$ and the mean of $W$.

K means clustering kind of works like this.
It uses the Lloyd algorithm, which basically has two steps.
Per each new sample:
 1) Choose the best cluster, by calculating the minimum distance to available clusters mean.
 2) Recalculate chosen cluster mean.