Detecting hand-written objects from images Problem Statement: Identify the sub-region in an image with hand-written text/scribbling
(Attaching an example for reference. Masked some text for privacy)

Once I get the pixel values, I want to replace them with white pixels, so I remove the scribblings completely from the image.

(PS: I don't have a lot of training images to train a CNN. So, asking here in case the problem can be solved via a classical CV approach)

You could try to run some OCR like tesseract. It'll give you locations of detected text. It'll probably fail for handwritten and skewed text. You'll get your mask, though this is a bit hacky as some false negatives may appear.


Keras - How to classify 1D time series Say I have a training dataset composed by 128 1-Dimensional time series in form of numpy arrays.

They all correspond to a certain action that I label action_1 and I want to recognize.

What would be the most efficient way to accomplish the following taks:

Train a model with the training dataset (the 128 1-D numpy arrays)
Ask the model to predict the action in a new test entry (a 1-D numpy array)

This is my very first attempt at Machine Learning, thank you in advance for any indications.

I'm not sure about what's your question exactly. If you are interested in the model architecture time series are the typical example for using RNNs (recurrent neural networks) or some more advanced form of them i.e. LSTM or GRU. They are all implemented in keras (https://keras.io/layers/recurrent/).

You may also can experiment with CNNs (convolutional neural networks). Usually they are used for images but in certain cases they can even outperform RNNs. They are also implemented in keras. (https://keras.io/layers/convolutional/)


Object detection/recognition, pre-processing error My Imports:
# Importing modules
import numpy as np
import pandas as pd
import os
import matplotlib.pyplot as plt
import cv2

from keras.utils import to_categorical
from keras.layers import Dense,Conv2D,Flatten,MaxPool2D,Dropout
from keras.models import Sequential

from sklearn.model_selection import train_test_split

My Code:
np.random.seed(1)

train_images = []
train_labels = []

shape = (108,108)

label_path = 'Beer/ModelSet/'

train_labels.append('miller_lite')
train_labels.append('stella_artois')
train_labels.append('michelob_ultra')
train_labels.append('belgian_blue')

for folder in os.listdir(label_path):
    for files in os.listdir(label_path+folder):
        img = cv2.imread(os.path.join(label_path,folder,files))
        train_images.append(img)

train_labels = np.asarray(pd.get_dummies(train_labels).values)

train_images = np.asarray(train_images)

x_train, x_val, y_train, y_val = train_test_split(train_images, train_labels, random_state=1)

The part that fails
    x_train, x_val, y_train, y_val = train_test_split(train_images, train_labels,
random_state=1)

The reason it fails
ValueError: Found input variables with inconsistent numbers of samples: [20000, 4]

My Error:
2020-08-03 23:47:11.117431: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
Traceback (most recent call last):
  File "/Attempt-2.py", line 40, in <module>
    x_train, x_val, y_train, y_val = train_test_split(train_images, train_labels, random_state=1)
  File "python3.8/site-packages/sklearn/model_selection/_split.py", line 2127, in train_test_split
    arrays = indexable(*arrays)
  File "python3.8/site-packages/sklearn/utils/validation.py", line 293, in indexable
    check_consistent_length(*result)
  File "python3.8/site-packages/sklearn/utils/validation.py", line 256, in check_consistent_length
    raise ValueError("Found input variables with inconsistent numbers of"
ValueError: Found input variables with inconsistent numbers of samples: [20000, 4]

The Goal:
Create a classification model that distinguishes between 4 different brands of beer bottles. MillerLite,StellaArtois,MichelobUltra, and BelgianBlue.
Background Note:
I have never had any practical/work experience with either software engineering, data science, software development, machine learning etc. I am just a student messing around for my own amusement/fun.
The Question:
How exactly do I fix this? I understand the problem is that matrix x is not the same size as matrix y. X is 20000 images of size 108,108 and 3 channels RGB. Y is the label matrix:
[[1 0 0 0][0 1 0 0][0 0 1 0][0 0 0 1]]
in order to split into train/test images the error says I need to have the same len/size array/matrix for both x and y.

You have to create labels for each of the images and then split it into train and test. I believe you have 20,000 images - so you have to also have 1 label for each image not jus the 4 categories alone in an array. One of the most important steps in training a DNN model to do image classification (as is your case) or any image related task is creating labels (in your case) and otherwise annotations for other problem statements. I think you have just created a array of what your labels are and havent associated them with the actual image itself.


group_by: How to Transform my Data I'm coming from Stata and struggling to get used to group_by in dplyr. Perhaps using group_by is the wrong approach, but if you know Stata, you'll understand why I'm trying to use it (I think).

I'd like to transform the data as shown below:

Essentially, I'm trying to collapse each NAICS code into a single row, and to create columns for the 5 statistics under CONC and HHI for each NAICS code (the 4 values for CONC and one for HHI, ignoring the Xs)

The data is here. The code I'm trying is as follows:

ManufMktConc <- read.csv("Manufacturing.csv",
                            stringsAsFactors = FALSE)

ManufMktConc %<>%
  tbl_df %>%
  slice(-1) %>%
  filter(CONCENFI.display.label!="All companies") %>%
  select(-GEO.id,-GEO.id2,-GEO.display.label,-COMPANY,-RCPTOT,-YEAR.id)

ManufMktConc_byInd <- group_by(ManufMktConc,NAICS.id)

ManufMktConc_byInd %<>%
      arrange(CONCENFI.id) %>%
      mutate(FourFirm = ManufMktConc_byInd$CCORCPPCT[ManufMktConc_byInd$CONCENFI.id=="856"])

That gives me the following error:

> ManufMktConc_byInd %<>% arrange(CONCENFI.id) %>% mutate(FourFirm=ManufMktConc_byInd$CCORCPPCT[ManufMktConc_byInd$CONCENFI.id=="856"])
Error: incompatible size (651), expecting 4 (the group size) or 1

So I can tell that I am asking R to stick the 651 instances of ManufMktConc_byInd$CONCENFI.id == "856" into the group of 4. I guess this is where my understanding of group_by falls apart. Why isn't my logical statement only applied within each group?

Thank you.

ManufMktConc <- read.csv("Downloads/Manufacturing.csv",
                         stringsAsFactors = FALSE)

ManufMktConc %<>%
  tbl_df %>%
  slice(-1) %>%
  filter(CONCENFI.display.label!="All companies") %>%
  select(-GEO.id,-GEO.id2,-GEO.display.label,-COMPANY,-RCPTOT,-YEAR.id)

##  use melt and cast from reshape2
require(reshape2)

# melt makes the data set tall and thin using id variables and measure variables
#
ManufMktConc_molten <- as.data.frame(ManufMktConc) %>%
  melt(id.vars=1:4, measure.vars=5:6) %>%
  filter(value!="X")
ManufMktConc_molten[1:5,]

    NAICS.id       NAICS.display.label CONCENFI.id CONCENFI.display.label  variable value
1      311        Food manufacturing         856    4 largest companies CCORCPPCT  16.3
2      311        Food manufacturing         857    8 largest companies CCORCPPCT  24.2
3      311        Food manufacturing         858   20 largest companies CCORCPPCT  38.4
4      311        Food manufacturing         859   50 largest companies CCORCPPCT  50.9
5     3111 Animal food manufacturing         856    4 largest companies CCORCPPCT  30.2

# make a new column with the eventual column header. (Note this is different that your example)
ManufMktConc_molten$label <- paste0(ManufMktConc_molten$variable,
                                    trimws(substr(ManufMktConc_molten$CONCENFI.display.label,1,2)))

# cast it into multiple columns (something like a pivot in Excel).
ManufMktConc_result  <- ManufMktConc_molten %>%
  cast(NAICS.id + NAICS.display.label ~ label) %>%
  select(1,2,4,6,3,5,7)     ## reorder columns

ManufMktConc_result[1:5,]

      NAICS.id             NAICS.display.label CCORCPPCT4 CCORCPPCT8 CCORCPPCT20 CCORCPPCT50 VSHERFI50
    1      311              Food manufacturing       16.3       24.2        38.4        50.9     110.7
    2     3111       Animal food manufacturing       30.2       40.7        57.8        71.5     368.6
    3    31111       Animal food manufacturing       30.2       40.7        57.8        71.5     368.6
    4   311111  Dog and cat food manufacturing       67.8       80.6        89.6        96.5    2019.4
    5   311119 Other animal food manufacturing       24.3       36.2        51.5        68.2     228.3


XGBoost in final model what does "yes=3,no=4" mean? XGBoost v0.6 Binary Classification

$ cat predin.txt
0 1:10 2:10 3:10
1 1:10 2:10 3:100
1 1:10 2:100 3:10
1 1:100 2:10 3:10

In other words, the negative instance label 0 requires low values for all three features.

./xgboost/xgboost m.conf
./xgboost/xgboost m.conf task=dump model_in=0002.model name_dump=dump.raw.txt
$ cat dump.raw.txt
booster[0]:
0:[f1<55] yes=1,no=2,missing=1
    1:[f2<55] yes=3,no=4,missing=3
        3:[f3<55] yes=5,no=6,missing=5
            5:leaf=-1.5
            6:leaf=1.5
        4:leaf=1.5
    2:leaf=1.5
booster[1]:
0:[f1<55] yes=1,no=2,missing=1
    1:[f2<55] yes=3,no=4,missing=3
        3:[f3<55] yes=5,no=6,missing=5
            5:leaf=-0.784694
            6:leaf=0.784694
        4:leaf=0.784694
    2:leaf=0.784694

What does "yes=3,no=4" mean?

I assume it should read as follows:

if condition is true, go to branch 3
otherwise go to branch 4

I am only guessing, but seems plausible


Can the Generative Adversarial Network useful for Outlier detection and Outlier explanation in a high dimentional numerical data? I have been building a model to find explanation of Outliers in a high dimensional numerical data, generated from many sensors. The data contains more than 350 different fields and each field has numerical values (Float or Integer). It looks like: 350 columns and many rows. I want to find the outliers/ anomalies in the data and also the Explanation why those values are outliers.

I was reading about Generative models and found that "They have the potential to understand and explain the underlying structure of the Input data even when there are no labels." I would like to know if it will be good to use the GANs for outlier detection and explanation on numerical data?

See this answer by Ian Goodfellow (the creator of GANs) to the same question at Quora:

There are definitely some papers about it, such as [1703.05921] Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery. I don’t work on anomaly detection so I haven’t read these papers and don’t know a lot about how they work.
It’s worth mentioning that there are also papers on anomaly detection using inputs that resemble adversarial examples: https://arxiv.org/pdf/1706.02690.pdf
For GANs, one thing to keep in mind is that the discriminator is not a generalized detector of weird things. It is trying to tell whether a sample came from the real data or one specific non-data distribution: the generator. Because of that, it seems like the discriminator would only be useful for anomaly detection if you think you can make your generator resemble the anomalies you expect to need to detect.

Nevertheless, neural networks in general are black boxes (i.e. not very interpretable), so they might detect outliers but you would not be able to tell what characterized them as so.


What algorithm to use I am stuck on what algorithm to use. I want to train my program on a dataset where i have an input image, and an output image which is a modified version of the input. The whole context of the image is important, and each pixel counts.

After training i want to input an image, and get an output similar to the training data.

You can use Deep convolution GAN network of Deep learning. It would be a generative model.

you can get detailed help from :

https://medium.com/@keisukeumezawa/dcgan-generate-the-images-with-deep-convolutinal-gan-55edf947c34b


Vocabulary List From word2vec and GloVe Is there a way I can access just the vocabulary list of pre-trained vectors for word2vec and GloVe? I do not need the entire n-dimensional embeddings.

In short: Yes, you can.

You need to first load the vectors using the Gensim module in Python.

# Load Google news vectors
word2vec_path = "path_to_the_vectors/GoogleNews-vectors-negative300.bin"
word2vec = gensim.models.KeyedVectors.load_word2vec_format(word2vec_path, binary=True)

# contains the list of all unique words in pre-trained word2vec vectors
w2v_vocabulary = word2vec.vocab


How to adjust cofounders in Logistic regression? I have a binary classification problem where I apply logistic regression.

I have a set of features that are found significant.

But I understand that Logistic regression doesn't consider feature interactions.

While I read online that a lack of feature interaction can be accounted by adjusting logistic regression for confounders.

Currently I did this and got the significant features.

model = sm.Logit(y_train, X_train)
result=model.fit()
result.summary()

But how do I adjust for confounders? Any pointers to tutorial for non-stat person like me would really be helpful.

Can someone let me know on how can I do this?

While I'm not sure what you mean when you say "adjust for confounders", I suppose your question is about model choice (or variable/feature selection).

Here are some thoughts on this problem:

Clearly define what you want to achieve: If you want to achieve a good prediction (so you are not up to causal modeling), choose a suitable metric to measure model fit. For a classification problem, you could look at the confusion matrix or AUC (but there are many more options).
Choose a baseline model which you believe is a good starting point for your problem. Look at relevant metrics of this model.
Try to come up with improvements: You could - for instance - work on your features and see if your "new" model performs better (based on the chosen metrics) compared to the baseline model. You can also add interaction terms (between variables) by simply multiplying variables. So when you have a model $y=\beta_0+\beta_1 x_1+\beta_2 x_2 + u$, you can also go for $y=\beta_0+\beta_1 x_1+\beta_2 x_2 + \beta_3 x_1 x_2  + u$ (Note: $\beta$ are the regression coefficients, $u$ is the error term).

Make sure you train your model on one part of your data and you test your model on another part of the data (train/test split).

Some notes on feature selection: Generally it is not a good idea to select features only by looking on significance. Why? Significance tells you if the estimated confidence interval of the coefficient of some variable/feature "crosses zero" (the interval has positive and negative values). This says little about the contribution of a variable to overall model performance. Variables can also be "jointly significant", so even non-significant (single) features can be important (in interplay with other features). Alternatively, a linear transformation of some feature may make it significant, e.g. in the case of adding polynomials (squared terms for instance) or making a logistic transformation. So in essence: don't kick out features only based on significance. By doing so, you have a fair chance of introducing the "omitted variable bias".

In case you have a lot of possibly relevant features and you are sick of doing model selection "by hand", you may also look into Lasso (or Ridge) regression. Under this approaches features are "shrunken" (automatically) when they are not so useful for a good prediction. Here is a very good intro to Lasso/Ridge/Elastic Net by Trevor Hastie and Junyang Qian. The code is in R, but the tutorial is very good.

You would surely gain from looking at the book "Introduction to Statistical Learning". Chapter 4 covers Logit in a very instructive way. Plus there is Python code for the Labs in the book, which could give you a good starting point. Reading your question, the book and the Lab code from the book, would be a natural statring point for you.


List of Correlations to Correlation Matrix I am using python to do some data analysis and I need to represent the following table as a correlation matrix. The correlation value is a value between -1 and 1.

+-------------------+----------------------------------+--------------------------+
| Correlation Value |            Variable 1            |        Variable 2        |
+-------------------+----------------------------------+--------------------------+
| -0.07035          | ELECTRICAL_CONDUCTIVITY_AT_10_CM | COVER_OPEN_WATER_PERCENT |
| nan               | ELECTRICAL_CONDUCTIVITY_AT_10_CM | Heat_1988-2013_trnd      |
| nan               | ELECTRICAL_CONDUCTIVITY_AT_10_CM | 1972                     |
| -0.059789         | ELECTRICAL_CONDUCTIVITY_AT_10_CM | BEO_HC14                 |
| nan               | ELECTRICAL_CONDUCTIVITY_AT_10_CM | st_6_avg                 |
| 0.214723          | ELECTRICAL_CONDUCTIVITY_AT_10_CM | slope_angle              |
| 0.247674          | ELECTRICAL_CONDUCTIVITY_AT_10_CM | total_mn                 |
| 0.01978           | ELECTRICAL_CONDUCTIVITY_AT_10_CM | FD13_low_flux            |
| 0.136542          | ELECTRICAL_CONDUCTIVITY_AT_10_CM | 19.1.1                   |
+-------------------+----------------------------------+--------------------------+

What tool would be best to accomplish this task? I attempted to use matplotlib to create a 2d histogram but as far as I could tell it requires 2 numerical numpy arrays. Are there any other tools or functions that could create a 2d histogram depending on the level of the correlation value?

Well, if this is the whole table, I don't see how a matrix is useful, since one of the variables you're looking at doesn't change...you can use something like this to show the correlation to each category of variable 2 https://seaborn.pydata.org/generated/seaborn.countplot.html


Reason behind the sum of rate factors for calculating cost function derivative Suppose we have a network of neurons like below:

We make a little change in weight w[l][j][k] on our network, and it can make change on our cost function from many paths

now we just pick one of that paths from our first choosed weight to -> it's activation's neuron to -> one of it's weights on the next layer to -> it's activation and so on...

we can imagine an equation (using chain rule) like below for this only one path and we call it a rate factor:

but it is only for one path of changing the cost as we changing that weight, so we need calculate the rate of cost changing by all possible related paths (all the rate factors) in the famous book of michael nielsen wrote that we calculating the rate of cost changing by doing sum all over the rate factors like below:

but really i don't understand exactly how can we calculate that slope of the cost with respect to that weight by summing all that rate factors how can relate that sum to our equation

a little later on that book we had a proof for an equation for the error in layer 'l' in terms of the error in the next layer, for that we have:

and here we do a sum over all related neurons errors too.
can someone give me a proof or an explain for that sum
how that rate of cost changing can relate to sum over all that related rate factors, why we use sum?

of course the neurons's activations are related to each other by calculating sum of all neurons in last layer using weights and biases

but using this for calculating that derivative is not so clear and tangible for me but any explanation about that sum over the derivatives can be very helpful for me.

thank you.

First, disclaimer: you're presenting fairly hairy computations and it's a little hard to read your question. I'll try answering nonetheless.

i don't understand exactly how can we calculate that slope of the cost with respect to that weight by summing [...]

If I interpreted the sums correctly then the sums in your question seem to deal with the Jacobian matrix that is used when applying the chain rule in multiple dimensions.
Intuitively, the weight $w_{j,k}^{l}$ will be referenced by all the paths originating from the neuron utilizing $w_{j,k}^l$, and will then be expressed by nodes in the "final" expression of $C$ via multiple linear combinations (sums). I'm fairly certain that if you actually followed the gradient $\partial C\over\partial w_{j,k}^l$ starting at the sum $C=a^L_{1}w_{2,k}^L+a^L_{2}w_{2,k}^L$ it would work out to be the sum you show. This also, I believe, would apply to the second sum you present.


Clustering strings inside strings? I am not sure whether I formulated the question correctly. Basically, what I want to do is:

Let's suppose I have a list of 1000 strings which look like this:

cvzxcvzxstringcvzcxvz

otortorotrstringgrptprt

vmvmvmeopstring2vmrprp

vccermpqpstring2rowerm

proorororstring3potrprt

mprto2435string3famerpaer

etc.

I'd like to extract these reoccuring strings that occur on the list. What solution should I use? Does anyone know about algorithm that could do this?

Interesting question! I have not encountered it before so here is a solution I just made up, inspired by the approach taken by the word2vec paper:

Define the pair-wise similarity based on the longest common substring (LCS), or the LCS normalized by the products of the string lengths. Cache this in a matrix for any pair of strings considered since it is expensive to calculate. Also consider approximations.
Find a Euclidean (hyperspherical, perhaps?) embedding that minimizes the error (Euclidean distance if using the ball, and the dot product if using the sphere). Assume random initialization, and use a gradient-based optimization method by taking the Jacobian of the error.
Now you have a Hilbert space embedding, so cluster using your algorithm of choice!

Response to deleted comment asking how to cluster multiple substrings: The bulk of the complexity lies in the first stage; the calculation of the LCS, so it depends on efficiently you do that. I've had luck with genetic algorithms. Anyway, what you'd do in this case is define a similarity vector rather than a scalar, whose elements are the k-longest pair-wise LCS; see this discussion for algorithms. Then I would define the error by the sum of the errors corresponding to each substring.

Something I did not address is how to choose the dimensionality of the embedding. The word2vec paper might provide some heuristics; see this discussion. I recall they used pretty big spaces, on the order of a 1000 dimensions, but they were optimizing something more complicated, so I suggest you start at R^2 and work your way up. Of course, you will want to use a higher dimensionality for the multiple LCS case.


How to calculate events per day in R including dates when no events occurred? I would like to create a data frame in which in the first column I will have all the dates from a certain period of time and in the second the number of events that occurred on each date including dates when no events occurred. I would also like to count the events to which specific factors have been assigned

The first data frame in which I have the events with dates for a given date:

Row Sex Age      Date
1    2   36   2004-01-05
2    1   47   2004-01-06
3    1   26   2004-01-10
4    2   23   2004-01-20
5    1   50   2004-01-27
6    2   35   2004-01-28
7    1   35   2004-01-30
8    1   38   2004-02-06
9    2   29   2004-02-11

Where in the column "Sex" 1 means female and 2 male.

Second data frame in which I have dates from the examined period:

Row    Date
 1  2004-01-05
 2  2004-01-06
 3  2004-01-07
 4  2004-01-08
 5  2004-01-09
 6  2004-01-10
 7  2004-01-11
 8  2004-01-12
 9  2004-01-13
10  2004-01-14

I want to get a data frame that looks like this:

Row    Date       Events (All)   Events (Female)   Events (Male)
 1  2004-01-05         1                0                1
 2  2004-01-06         1                1                0
 3  2004-01-07         0                0                0
 4  2004-01-08         0                0                0
 5  2004-01-09         0                0                0
 6  2004-01-10         0                1                0
 7  2004-01-11         0                0                0
 8  2004-01-12         0                0                0
 9  2004-01-13         0                0                0
10  2004-01-14         0                0                0

Could anyone help?

I made the assumption that:

Events (All) =   Events (Female)  + Events (Male)

I also manipulated the data so that there are more than one event per day.

As a result, df looks like:

  Sex Age       Date
1   2  36 2004-01-05
2   1  47 2004-01-05
3   1  26 2004-01-10
4   2  23 2004-01-10
5   1  50 2004-01-27
6   2  35 2004-01-27
7   1  35 2004-01-30
8   1  38 2004-02-30
9   2  29 2004-02-30

The following code should achieve the desired results.

library(tidyr)
df = read.csv([Path to dataset], stringsAsFactors = FALSE)

df %>% group_by(Date, Sex) %>%
  summarise(sex_count = n()) %>%
  spread(Sex,sex_count, fill=0) %>%
  rename( event_female = '1', event_male = '2') %>%
  mutate(event_all = event_female + event_male ) %>%
  select(event_all, event_female, event_male)

output:

 Date       event_all event_female event_male
  <chr>          <dbl>        <dbl>      <dbl>
1 2004-01-05         2            1          1
2 2004-01-10         2            1          1
3 2004-01-27         2            1          1
4 2004-01-30         1            1          0
5 2004-02-30         2            1          1
```


Type of regression with nominal, ordinal, interval and ratio data Statement of problem: An ambulance is at the hospital dropping off a patient. The goal of the paramedic is to get released from the hospital as soon as possible. I am curious, what are the factors in how long an ambulance off loads a patient at the hospital? Can I predict how long an offload will take given certain variables. And how confident can I be in this model? The Dependent Variable is HospitalTime, it is a ratio type of data and is measured in seconds. The Independent Variables are:

Hospital, a nominal type of data recoded into integers, 1 would stand
for Lee Memorial.
Ambulance, a nominal type of data recoded into integers, 9 would
stand for ambulance #9
PatientPriority is an ordinal type of data recoded into integers. A 1
is a high priority, 2 is a medium priority and 3 is low acuity.
MonthOfCall is an interval type of data recoded into integers. A 6
would be June and 12 is December. A 12 (December) is not twice as
much as a 6 (June) in this case.
HourOfCall is an interval type of data recoded into integers. Once
again, an offload happening at 10:00 pm is not more than something
happening at 10:00 am.
Officer1 and Officer2 are nominal data and are integers representing
an EMT and a paramedic.

My question is this: Given this type of data and my goal to predict the off loading time at the hospital, what kind of regression model should I look into?

I have looked at my statistics books from university days and they are all using ratio data. My data is mixed with nominal, ordinal, interval and ratio.

I have as much data as you could ask for. I have at least 100,000 observations.

Can you please push me in the right direction? What kind of model should I use with this type of data?

Shown below are observations to give you a tiny peek at my data:

IncidentID,HospitalTime,Hospital,Ambulance,PatientPriority,MonthOfCall,HourOfCall,Officer1,Officer2
757620,1849,7,11,2,10,10,234,771,chr(10) 802611,2625,7,11,3,1,18,234,777,chr(10)
765597,1149,7,12,3,11,2,234,777,chr(10) 770926,1785,7,12,3,11,15,234,777,chr(10)
771689,3557,7,12,2,11,14,234,777,chr(10) 822758,1073,7,20,3,3,13,777,307,chr(10)
767249,2570,7,22,2,11,11,560,778,chr(10) 767326,1998,7,22,1,11,18,560,777,chr(10)
785903,1660,7,22,3,12,12,234,777,chr(10) 787644,2852,7,22,3,12,17,234,777,chr(10)
760294,1327,7,23,2,10,14,498,735,chr(10) 994677,3653,7,32,2,2,15,181,159,chr(10)
994677,3653,7,32,2,2,15,181,159,chr(10) 788471,2053,5,9,2,1,3,498,777,chr(10)
788471,2053,5,9,2,1,3,498,777,chr(10) 759983,1342,5,11,2,10,8,474,777,chr(10)
791243,1635,5,11,2,1,18,234,777,chr(10) 800796,1381,5,11,3,1,11,234,777,chr(10)

P.S. This question is cross-posted in Stack-Overflow under the same title and author.

I think I'd try to model the rate at which the ambulances are released instead of the time it takes. That would let me use Poisson regression, which is the canonical type of GLM for rates (in R's GLM, set family = "poisson".) However, in order to use PR, the data needs to have its variance equal to its mean, or at least close to it.


Which decision tree algorithm does H2O use? Does H2O's plain random forest use CART, C4.5, 5.0, or something else? I cannot find this information. sklearn's docs say they use a modified version of CART, and I assume H2O also uses something like CART.

I have heard H20 Driverless AI uses an ensemble of xgboost and lightgbm models. In some situations it uses tensorflow neural nets as well.


What is the best question generation state of art with nlp? I was trying out various projects available for question generation on GitHub namely NQG,question-generation and a lot of others but I don't see good results form them either they have very bad question formation or the questions generated are off-topic most of the times, Where I found one project that actually generates good questions

bloomsburyai/question-generation

It basically accepts a context(paragraph) and an answer to generate the question and I am trying to validate the questions generated by passing the generated question along with the paragraph to allenNLP

Answer generation for a question

And then I am trying to make sure the generated answers are correct for the questions generated with calculating the sentence embedding for both the answers(AllenNLP and PotentialAnswer) using Universal Sentence Encoder and a cosine distance to get how similar the answers match and the filtering question that has least cosine distance.

Wanted to know if this is the best approach or Is there a state of the art implementation for question generation? Please suggest

For your first part of the question as to which question generation approaches are good - Neural question generation is being pretty popular (as of 2018/2019) among NLP enthusiasts but not all systems are great enough to be used directly in production. However, here are a few recent ones which reported the state-of-art performances in 2019 and have shared their codes too:

https://github.com/ZhangShiyue/QGforQA
https://github.com/PrekshaNema25/RefNet-QG

This one is a 2020 one (now that NLP performances have been improved with Transformers)
3. https://github.com/patil-suraj/question_generation
Besides, if you want more control as to understand and fix for wrongly generated questions, I would suggest the more traditional rule-based approach like the below which is more reliable than the above neural ones ones and generates a larger amount of question-answer pairs than the above 2:

http://www.cs.cmu.edu/~ark/mheilman/questions/
https://bitbucket.org/kaustubhdhole/syn-qg/src/master/

To answer your second question, if your QG model is generating an answer, then it makes sense to use cosine similarity. Assuming your question generation is at the sentence level, you will mostly have short answer spans and hence averaging Glove or Paragram word vectors might serve you better results than the Universal Sentence Encoder.


Is there Orange support for MySQL and SQLite? I would like to be able to directly access both MySQL and SQLite databases as part of an Orange workflow. I see there is support for PostgreSQL and (possibly)

In order to be more precise, I have just finished the example of a Python (ver. 3.7) Script to include into the workflow to extract data from a MySQL server via an ODBC connector as following :

from Orange.data import Table, Domain, ContinuousVariable
from pyodbc import connect

connecteur = connect(DSN='basepython') # "table1" from a MySQL server via ODBC connector.
curseur = connecteur.cursor() # Creation of the cursor for data swept.

# Execution of the SQL Query.
curseur.execute("SELECT id, champ1, champ2, champ3 FROM table1")

# All data of "table1" are saved in "donnees".
donnees = curseur.fetchall()

liste_globale = [] # Final list : It will contain the final data.
liste_intermediaire = [] # Temporary list : Only used for conversion.

# Whereas having tuples as list elements, you must convert them into lists included into the main list.
for partie in donnees:
    for i in range(len(partie)):
        liste_intermediaire.append(partie[i])
    liste_globale.append(liste_intermediaire)
    liste_intermediaire = []

# Creation of the data domain : Names of the variables seen in Orange WorkFlow.
identite = ContinuousVariable("Identification")
colonne1 = ContinuousVariable("Colonne1")
colonne2 = ContinuousVariable("Colonne2")
colonne3 = ContinuousVariable("Colonne3")

domaine = Domain([identite, colonne1, colonne2, colonne3])

# Final data
out_data = Table(domaine, liste_globale) # Data seen at the output of the Python_Script widget.
print(out_data) # This line is only used for debugging, you can comment it.

# Don't forget to close the connection at the end.
connecteur.close()

This example is used with the following workflow :

Of course, if you want to connect both MySQL server and SQLite in the same time, you can adapt this script with two connectors, for example :

connecteur1 = connect(DSN="MySQL_ODBC_DSN")
connecteur2 = connect(DSN="SQLite_ODBC_DSN")

And you can work with these two independent connectors as following :

curseur1 = connecteur1.cursor()
curseur2 = connecteur2.cursor()

curseur1.execute("SELECT * FROM mysql_table")
curseur2.execute("SELECT * FROM sqlite3_table WHERE id='2'")

donnees1 = curseur1.fetchall() # If there are several rows to save.
donnees2 = curseur2.fetchone() # If there is only one row to save.

And at the end, don't forget to close each connection :

# Closing of the two opened database connectors.
connecteur1.close()
connecteur2.close()


Machine learning methods for panel (longitudinal) data I have a panel data set, for example:

obj  time        Y       x1      x2
1    1         0.1     1.28    0.02
2    1        0.11     1.27    0.01
1    2        -0.4     1.05    -0.06
2    2        -0.3     1.11    -0.02
1    3        -0.5     1.22    -0.06
2    3        1.2      1.06    0.11

I`m new at ML and until recently I did not know that this is a special (panel) data type. I predicted the value of a variable $Y(t+1)$ by values $x_1(t)$ and $x_2(t)$ (time lag) using linear regression model and MLP. But now I read some information about panel data analysis and realized that the methods I used were not suitable. At the moment I found that fixed/random effects models are suitable for panel data analysis. So, I have several questions:

What other methods are correctly used to analyze panel data (I`m interested in neural network models)? I read that these methods must take into account the dependency between particular object values and previous occurring values of this object (
what is in models with fixed and random effects).
I also tried to use MLP by feeding 2D data to it. I divided the panel data into $k=time$ $quants$ $count$  2D blocks and passed this data to the MLP input. For example above, $k=3$, ($input$ $layer$ $size= 4 =
number$ $of$ $predictors* block$ $objects$ $count$). In this case $batch$ $size=1$.
If I make the $batch$ $size=2$ and feed the neural network with 1D data ($input$ $layer$ $size=2$ for example above, will there be any difference? In both cases, the weights of the neural network will be rebuilt after the observations on all objects are transmitted in one quantum of time.

Panel data = multi-object time series.
in other words you have time series problem (time) for different objects (obj) that you are trying to predcit (Y).
If I were you I would just dissect this problem and start thinking in terms of time-series + another discriminative column called obj. What approaches do you know there? Here is really cool and MODERN time-series tutorial, check it out.
Regarding NN, why are you trying to squeeze it in so hard? Let the data tell you what algo can model it. Personally given these 3 features NN is just too much and you can achieve similiar results with less complexity using some simpler/less expensive approaches.


How do I find the common values in two different dataframe by comparing different column names? If the data frames are as follows :
df1
column names = UniqueID  Age Gender
values = 213,21,F
         145,34,M
         890,23,F
df2
column names = ID Occupation Bill
               145 Teacher 45RS
               890 Student 70RS
               456 Teacher 90Rs

I want to use a function which finds the common values in "UniqueID" from df1 and "ID" from df2 and gets stored in df3. Also, the only columns which exist in df3 should be of "common ids" and the columns of df2.

You can use pandas.merge() to get the common rows based on the columns. Try this:

df3 = pd.merge(df1, df2, how='inner', left_on='UniqueID', right_on='ID')

However, this will give you all the columns. After that you can use

df3.drop([column_names], inplace=True)

to drop the columns that are not required.


Choice of f1 score for highly imbalanced dataset? I am confused whether to use f1 score with 'micro' average or 'macro' average for better evaluation.
Given my dataset is highly imbalanced(600:100000)

To summarise this answer,

Macro calculates and F1 score for each class then averages them.

Micro calculates the recall/precision for each class, averages them then calculates the F1 score.

Micro average is preferable in a multi class situation where one class heavily outweighs the others.


identifying input sequence using neural network I fount simple neural scenario sayinh : network having 3 layers, 3 inputs and 2 outputs. It should be trained to recognize a simple pattern - if inputs are correspondingly 6.0,7.0 and 8.0 then outputs should be 3.0 and 4.0, otherwise outputs should be 23.0

What I understand is, hidden layer should Check input pattern in sequence with simple if else condition.

if in1 == 6 and in2 == 7 and in3 ==8:
    out1, out2 = 3, 4
else
    out = 5

Do I understand the problem correctly or something I am missing?

A layer in a neural network does not process logic of the kind you present. It cannot "check the input pattern with a simple if else pattern". You could make a model that processed your example inputs to example outputs as you show, but it would not be a neural network.

In machine-learning, neural networks have a formal, numeric structure. There are a few variations, but essentially in your diagram:

Each circle is a "neuron" which may have a numeric output value, also called its activation.
Each arrow is a connection between neurons that shows where values are passed between neurons. Each connection has a strength or weight associated with it.
When you have many arrows going into a neuron, you can calculate its new activation starting with the sum of weights times the activation values of neurons that they are connected from. Call this sum $z$. Then the activation of the neuron you are looking at is $a = f(z)$ where f is an "activation function" (also called "transfer function").

The above very roughly covers how many neural networks are constructed. As you can see though, there is no room in that for having your if/else statement.

Instead, neural networks like the one you show (technically a feed forward network also called a multi-layer perceptron or MLP) are most often trained by example. This involves repeatedly giving the network inputs where you know the output you want. To start with the network will give completely incorrect outputs, and you then adjust the weights to try and make the result closer. This is repeated very many times until the network becomes the best model it can for your purpose. A very common algorithm used to train a neural network in this way is backpropagation and gradient descent.

For your problem as described, a neural network is not a good model, and would not be used in the real world. That is because you already know the full logic of a simpler model that covers all possible inputs. At best with a neural network you will get an approximation to that already-perfect model.

However, if this is for the purposes of understanding neural networks, your approach would be roughly this:

Generate some training data with example inputs and outputs. Most example outputs will be the 23,23 default values, but when inputs are the special values 6,7,8 then the training outputs should be 3 and 4.
Build an initial network. Normally you would use an appropriate library - it is important you learn how the library works and understand any limitations. E.g. many activation functions have a maximum output of 1, so cannot learn the values 3,4, or 23.
Train the network to fit the example data. There are very many options there, depending on the library you are using. Usually it is just a matter of selecting parameters for the learning that help it progress smoothly. You may have to run training for a large number of repeats over the whole dataset (called epochs)
Test that your network works as you expect. Even if you have trained it well, the response of the network is never as precise as your problem statement. If you gave it inputs of 5.8,7.2,8.1 (i.e. close to 6,7,8) then it would likely output something like 7.2,8.5 and not 23,23 .  . . it will interpolate between values.


LSTM for financial data I'm using LSTM to predict financial data. As input data I use log returns and I want to predict the next day market movement. Do I need to retrain the ANN every day in order to keep time consistency or I can simply train ANN once for example with the data from 2010 to 2018 and predict market movement in 2019?

I'm using daily data

An interesting idea would be to train the model with data between 2010 and 2018 and then keep training it every day to keep it updated.

Interesting related works can be found here and here.

Anyway, you need to decide what you want to predict it: do you want a daily output, monthly or yearly?


Using MultiLabelBinarizer for SMOTE This is my first NLP project. I'm trying to use SMOTE for a classifier with 14 classes. I need to convert the classes into an array before using SMOTE. I tried using MultiLinearBinarizer but it does not seem to be working. From the stack trace, it seems like everything is getting converted.  Do I need to convert something else to an array? How would I do that?

from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.preprocessing import MultiLabelBinarizer

nb = Pipeline([('vect', CountVectorizer()),
               ('tfidf', TfidfTransformer()),
               ('clf', MultinomialNB()),
              ])
nb.fit(X_train, y_train)

mlb = MultiLabelBinarizer()
print(mlb.fit_transform(df["Technique"].str.split(",")))
print(mlb.classes_)

import imblearn
from imblearn.over_sampling import SMOTE

smote = SMOTE('minority')

x_sm, y_sm = smote.fit_sample(X_train, y_train)
#print(x_sm.shape, y_sm.shape)
pd.DataFrame(x_sm.todense(), columns=tv.get_feature_names())

I'm getting the error ValueError: could not convert string to float: 'left left center'

Here is the stack trace

[[1 0 0 ... 0 0 0]
 [1 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]
 ...
 [0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]]
['Appeal_to_Authority' 'Appeal_to_fear-prejudice' 'Bandwagon'
 'Black-and-White_Fallacy' 'Causal_Oversimplification' 'Doubt'
 'Exaggeration' 'Flag-Waving' 'Labeling' 'Loaded_Language' 'Minimisation'
 'Name_Calling' 'Red_Herring' 'Reductio_ad_hitlerum' 'Repetition'
 'Slogans' 'Straw_Men' 'Thought-terminating_Cliches' 'Whataboutism']
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-45-68681190410f> in <module>()
     10 smote = SMOTE('minority')
     11
---> 12 x_sm, y_sm = smote.fit_sample(X_train, y_train)
     13 #print(x_sm.shape, y_sm.shape)
     14 pd.DataFrame(x_sm.todense(), columns=tv.get_feature_names())

8 frames
/usr/local/lib/python3.6/dist-packages/numpy/core/_asarray.py in asarray(a, dtype, order)
     83
     84     """
---> 85     return array(a, dtype, copy=False, order=order)
     86
     87

ValueError: could not convert string to float: 'left left center'
```

You can't fit X_train into y_train without encoding.
Try something like this for features:

from sklearn.preprocessing import OneHotEncoder
enc = OneHotEncoder(handle_unknown='ignore')
enc.fit(X)

and label encoding for labels.


NoSQL engine/service recommendation for geolocation data First of all, I am new in this field we call big data, so my questions may be naive.

In order to build an application, which deals with geolocation data, which could be : latitude and longitude coordinates and Geography SQL Server column types.

I need to have the following elements made easy:

Scalability : be prepared to receive huge amount of data, adding servers to the system have to be easy
Proximity requests : in example, how much points are in a circle (at meter scale).
Data must be accessible rapidly after being written.

I've been looking around for existing solutions, which are "Hadoop friendly" (Hortonworks, Cloudera) and available DBMS, like Cassandra.
I have found some interesting information, but I still think it's hard to decide, which one to choose.
It also need drivers for NodeJS & .NET (Hadoop with Cassandra seem to be OK with that).
I've also looked around the MongoDB ecosystem, but, again, I feel that it is hard to know where to look at. By (little) experience with Mongoose, MongoDB can be disqualified by the third point because data writes are slow. But my model could certainly be improved.

Do any of you have some recent experiences, manipulating massive amount of geolocation data? I would appreciate sharing them here as well as any quality and recent literature on the subject.

One approach can be to use a no-SQL database running on top of a distributed store (like Cassandra or Hbase). Add an external index which support spatial indexing (say elastic) for fast search. This makes your solution scalable (due to distributed store) and fast enough for spatial search queries.


Sampling in Text Classification: can the results be considered 'reliable'? I am testing different models (SVM, Logistic Regression, Naive Bayes, Random Forest) for predicting the class of a spam email. My target is a binary variable.
I am analysing only text, no other fields.
My dataset includes
Label
0.0    3333
1.0     768

As you can see there is a big problem with classes imbalanced.
I read about the use of downsampling and upsampling, so I applied them before training and testing the dataset.
I got good results in terms of F1, recall and accuracy for upsampling (above 88%; max 97%), bad for downsampling (<=76%).
For instance:
Down
              precision    recall  f1-score   support

         0.0       0.79      0.43      0.56       102
         1.0       0.61      0.87      0.76       114

Confusion Matrix:
 [[ 49  60]
 [ 12 100]]

Up
              precision    recall  f1-score   support

         0.0       1.00      0.85      0.91       873
         1.0       0.87      1.00      0.94       884

Confusion Matrix:
 [[772 141]
 [  20 822]]

I would like to ask you if these values can be considered good results or they can't. I am considering  a publication (not only to include similar analysis), so I would like to check if such results can be considered reliable, despite of the imbalance.
Any suggestions and advice will be greatly welcome.

There seems to be a mistake in your method:

I read about the use of downsampling and upsampling, so I applied them before training and testing the dataset.

It's incorrect to change the distribution of the test set. When resampling, the resampling should be applied only on the training set. The goal is to force the model to take into account the two classes, because in case of imbalance the model tends to focus on the majority class. However the true proportion of the class in the "real dataset" is still the same, and the test set should follow this true proportion. Otherwise the performance looks artificially good on the test set, even though the classifier will make more mistakes with real data since it doesn't have the same distribution.
So the performance values that you obtain on a resampled dataset are meaningless, I'm afraid.

I am considering a publication (not only to include similar analysis), so I would like to check if such results can be considered reliable, despite of the imbalance.

If you are considering a peer-reviewed publication, you must also make sure that your contribution is original (i.e. new) and brings some advantage over the existing methods. This means that you need to know the state of the art in spam classification (there are a lot of papers already published about this task) and show what your method improves something compared to the existing methods. Ideally this is done by proving that your new method obtains better performance than the state of the arts methods using a benchmark dataset. But it's usually hard to beat state of the art performance on a well-known problem.


Does Box plot with many outliers effects the result? I am using Haberman's cancer survival dataset https://www.kaggle.com/gilsousa/habermans-survival-data-set
to draw a box plot. Here Surv_status is the target variable which has two classes and axil_nodes_det is the feature. I am getting the plot as follows with too many outliers from the first class

I want to know whether more outliers will affect the prediction of output

As the comments suggest, it's not always helpful to think of points outside the whiskers as "outliers". What you are seeing in these boxplots is a strong positive skew. And yes, a strongly skewed target is typically harder to predict than a less-skewed target.

In traditional survival analysis, it's common to model this skewed data using a parametric probability distribution that naturally produces positive and positive-skewed data, e.g. the Weibull distribution.

Otherwise, you can also try a Box-Cox or inverse hyperbolic sine (IHS) transformation on the survival time to reduce the skew.

For visualizing strongly-skewed data, you can either apply one of the above skew-reducing transformations, or use an "adjusted" boxplot, which have an R implementation.


Building Image Dataset In a Studio I'm currently working in a problem of Object Detection, more specifically we want to count and differentiate similar species of moths.

We are already testing some detection algorithms:

YOLOv3
RetinaNet
Mask R-CNN

For the datasets, first we'll scrape the web for those images to have a starting model.

Then we'll get plenty of pictures from our devices that will look like this:

Unfortunately it will take 6 months to get those pictures.

Currently we have some samples of moths, and we have the resources to take pictures of them in a studio. We want to simulate the field conditions as well as taking close ups of the moths in different angles and lighting conditions. This experiment would also allow for us to get the accuracy of classification from an expert entomologist.

Is the effort of taking studio photos worth it?

Would these pictures significantly increase the models accuracy?

Is there any opportunity for publication?

Is there any paper that uses studio photos as well as field photos for object detection?

Also, there's plenty of opportunities for publication here in the company, so they would go the extra mile if we publish something, or if these pictures would improve the final accuracy of the models.

Disclaimer:

This is a question that is probably going to be flagged since it is too broad and answers will be mainly based on opinion.

Is this effort worth it?

That is subjective, what commercial/social use does this have? Is this kind of detection really relevant? What can you do with those detections and classifications? That is something only an entomologist could answer properly, you will get better answers for this one with them.

If you can find this kind of motivations for your work, you probably get a good dataset with good use. But if you don't even know what kind of applications you're aiming to you should start doing more research on motivation.

Would these pictures significantly increase the model's accuracy?

It is highly probable that it will, but you can't be sure until you try. Also, train your models with recently acquired images during the acquisition process to make sure that those images are having an impact on your models.

Is there any opportunity for publication?

Surely there is, if it will be cited multiple times or if the paper will be well received by Machine Learning community? Probably. If not, that probably would get you a top publication on entomology.

Is there any paper that uses studio photos as well as field photos for object detection?

Well, there are many. These controlled datasets are pretty easy to come up with and have been used extensively see BioID for example, until today many publications use it for benchmarking on Eye Pupil Detection and Facial Landmark Detection. In the wild datasets (usually collected from the web) are usually more challenging and also more fitted to real-world application, but it is not always possible to acquire them easily.

Last year CEFET-RJ and UFRJ published papers in IEEE with created datasets for Aedes Aegypti breeding sites detection. This dataset was created using drones and manually annotated by CEFET-RJ and UFRJ undergrad students.


LinearRegression with multiple binary features sometimes performs poorly I have a dataset comprising a number of binary features which are the dummies (as in, pd.get_dummies()) of categorical features. SalePrice is my target variable.

I'm literally just fitting a sklearn LinearRegression model with that data a thousand times to get an average of the score, and I'm getting a weird result. The relevant bit of my code looks like this:

import numpy as np
scores = np.array([])

for i in range(1000):

    x3_train, x3_test, y3_train, y3_test = train_test_split(
            df3.drop('SalePrice', axis=1),
            df3.SalePrice,
            test_size=0.33
    )

    lr3 = LinearRegression()
    lr3.fit(x3_train, y3_train)

    scores = np.insert(scores, 0, lr3.score(x3_test, y3_test))

print(scores.mean())

Now the weird result is that the average result is super poor, because every so often the model just tanks completely but most of the time performs "reasonably" (still terrible but that's not a surprise as it's incredibly basic and not tuned at all, I'm just comparing the effect of treating a set of features in different ways). For example the first 30 runs generated these scores:

0     5.907010e-01
1     6.044523e-01
2     5.178049e-01
3     5.622240e-01
4     5.810432e-01
5     5.131722e-01
6     5.772946e-01
7     4.674152e-01
8     4.962015e-01
9     4.887872e-01
10    5.144772e-01
11    5.676829e-01
12    5.122566e-01
13    5.453985e-01
14    5.355022e-01
15    5.888459e-01
16    5.552912e-01
17    5.615658e-01
18    5.472429e-01
19    5.810185e-01
20    5.334900e-01
21    5.493619e-01
22    5.567195e-01
23    5.514374e-01
24    4.916478e-01
25    4.580718e-01
26    5.286095e-01
27    5.761865e-01
28    5.638573e-01
29   -1.809208e+24
Name: lr3, dtype: float64

I guess my question is what is likely to be happening on that 30th run through such that the model performs so poorly? I'm comparing this model to others that treat the data differently (e.g. simply encode using .astype('category').cat.codes) and whilst there's relatively minor variations in the "usual" range of scores (they're all sort of 0.44 - 0.63) those other models don't have this occasional complete tanking.

You should always consider normalizing your output to some predefined range, otherwise there is a possibility of the gradients exploding as the loss will be of high magnitudes. It also becomes hard to output such a wide range. Try transforming your output using some StandardScaler, or a RobustScaler if there are significant outliers, and try again.


Need to kickstart learning rates I was just looking at the docs on Pytorch for different available schedulers and I found one that I am having some trouble understanding here.

The others seem to make sense: As training progresses, the learning rate gradually decreases. But in my study so far, I am yet to come across a model that needs such a "kickstart" mechanism.
Could someone please help me figure out why we need this?

This technique of gradually increasing the learning rate linearly during early phase of training is called learning rate warmup.
This is typically needed in order to prevent overfitting during the initial phase of training where a certain subset of observations may significantly skew your model towards some features and push you towards a bad local optimum.
The rectified adam paper suggests that this warmup heuristic serves as a variance reduction technique.

Due to the lack of samples in the early stage, the adaptive learning rate has an undesirably large variance, which leads to suspicious/bad local optima

They perform a few experiments that validates their hypothesis.


about high-dimensional regression data I am doing experiments on the high-dimensional regression. However, it is hard to obtain the practical or synthetic high-dimensional data. I have checked on UCI website as well as some papers with related to this topic. Currently, I have not obtained satisfactory datasets.

Could someone give me some hints on how to search and obtain the high-dimensional data? Thanks in advance.

Please check this website: http://dataportals.org/, its a collection of data portals from around the world. Might find here what you are looking for.


Tool to Generate 2D Data via Mouse Clicking Often when I am learning new machine learning methods or experimenting with a data analysis algorithm I need to generate a series of 2D points. Teachers also do this often when making a lesson or tutorial.

In some cases I just create a function, add some noise, and plot it, but there are many times when I wish I could just click my mouse on a graph to generate points. For instance, when I want to generate a fairly complex relationship between x and y, it's a hassle to think of the right formulation to generate the points programmatically.

Does there exist a tool that will allow me to generate data points using my mouse, with an option to export to CSV (or other simple format)?

For example, I am currently learning how to use mutual information and entropy as a metric of dependence between variables. I'd like to see what happens when I have data that is clearly dependent but does not have a linear relationship, so I drew this image:

Now I just need a way to export the coordinates of the points to CSV. I realize this is a simple program and I could write my own, but surely someone has already done this and created a tool to do so? It could be a website, an .exe, Python source, or any other application.

I recently discovered this site: http://www.librec.net/datagen.html

Outputs list of points, and color ID (class) for each point.

Screenshot:


Using LSTM's on Multivariate Input AND Multivariate Output I have a very small dataset, only about 40 rows, that has historical usage data for a few categories (roughly 20). I strongly suspect that these categories are dependent in a partial-zero-sum-game fashion: if the usage of one category goes up, I'm expecting that of another to go down. My goal is to predict the next row of all categories.

I've looked at this post, but it's not predicting multiple variables. I also looked at this post, but it's still univariate output (albeit multiple time steps) and multivariate input. So far, I've been basing my approach on the typical LSTM post here at machinelearningmastery, but it's also a single-output-variable example, and a number of the functions used, such as scaler.inverse_transform don't appear to broadcast very well. I'm even having difficulties trying to scale back my full example to match his!

Any tips for scaling LSTM's up to multivariate output? Can the keras LSTM do this natively? If so, how would the code change?

Thanks for your time!

You can simply change the number of units at the last layer if you want to predict multiple outputs (or just said differently, one output with multiple features). For example from the link you shared, you can change the units of the Dense layer:

model = Sequential()
model.add(LSTM(50, input_shape=(train_X.shape[1], train_X.shape[2])))
model.add(Dense(2))
model.compile(loss='mae', optimizer='adam')

The above model would now predict the next step of an output with 2 "features".
Note that you output should be of shape num_samplesx2 now.


Should we use discrete or continuous input for decision trees I have 2 datasets, a continuous dataset(75 datapoints and 14 variables) and a discretized dataset which was made by placing the continuous datasets into buckets. I have built a decision tree classifier (using the python sklearn package) and the classifier works much better for the discrete dataset rather than the continuous dataset.

I have also read in a few papers that sometimes it is preferable to use discrete datasets. But I don't know why. I Would appreciate any input or an explanation.

Discrete is the way to go. The reason is simple if you visualize a decision tree it involves drawing a decision boundary based on a set of constraints which are in the form of features. It would be much easier to draw these decision boundaries based on discrete features compared to its continuous counterpart. If the values are continuous it becomes difficult for the classifier to effectively draw this boundary and might have some skew in its results.

There is a useful video series in Udacity, Intro into Machine learning, please refer the decision tree section they show a really good visualization on how decision trees work.

Link: https://in.udacity.com/course/intro-to-machine-learning--ud120

Please check that out it may help you understand better.


generate a report in R where the value in each Column is to be derived from different tables in sybase databasea I work in a stress testing team. my boss has asked me to generate an interactive report every day using different tables in a Sybase database. I know the table structure and sql queries, but can't get my head around implementing in Rstudio using shiny or markdown.

Please advise.
thanks,

Here is a good overview of recommended ways:

http://r4ds.had.co.nz/r-markdown-formats.html#r-markdown-formats

I suggest using the package flexdashboard (heading "Dashboards"). Interactivity can be provided by htmlwidgets you can find an overview here:

http://www.htmlwidgets.org/

As an example, install:

install.packages('flexdashboards') # the dashboard
install.packages('plotly') # for interactive graphs
install.packages('DT') # for interactive tables
install.packages('tidyverse') # for data transformation etc

now create a dashboard as explained on

https://rmarkdown.rstudio.com/flexdashboard/

and fill it with your graph or tables. Here I use the starwars dataset provided in the package dplyr

---
title: "Single Column (Fill)"
output:
  flexdashboard::flex_dashboard:
    vertical_layout: fill
---

### Chart 1

```{r}
library(tidyverse)
library(DT)
datatable(starwars %>% select(-films, -vehicles, -starships))
```

### Chart 2

```{r}
g <- starwars %>%
  ggplot(aes(height, mass, color = gender)) +
  geom_point() +
  ggtitle("Mass vs. Height by gender")

library(plotly)
ggplotly(g)
```

output will be a html(interactive) file with table and graph:


Should I resize my test images when feeding to a CNN model? I am new to deep learning, I have made my first CNN network which accepts an input of (32,32,3) while training.
Now I want to test some random images on my trained model.
Do I have to resize my test images to (32,32,3). OR Can I leave the images as it is. OR Can I resize all of them to a higher resolution?
Plz give reasons.

Firstly let's start with the last question. It is normally not possible to increase the size of an image, that is, increasing its resolution without using AI solutions. But one famous solution to overcome this problem is padding the image to increase its size. But depending on the filter size, it may poorly perform if the original image is too small compared to the original input size, and filter dimensions are big. Also, in any case, if a model designed to accept the high resolution, would not perform similarly with small images because it will not get the same size of input or information (padding will just contain dummy data).
However, you can build a model that can accept different sizes, although the one that you have built will not. Your model is configured to receive 32x32x3, that is, 3072 input nodes. Thus, you have to resize all of your high-resolution images to 32x32x3 and pad the smaller images.


How can I know if my NN TensorFlow model is overfitted or not? I am new with TensorFlow (Python) and I can not juge my obtained results in terms of training and testing accuracy

I am using the GradientDescentOptimizer with a learning coeff equal to 10^(-4) and I have executed the following code :

for gg  in range (1501):
    session.run(optimizer, feed_dict=train_data)
    train_accuracy = session.run(accuracy, feed_dict=train_data)
    if gg % 100 ==0 :
        #print(train_accuracy)
        session.run(optimizer, feed_dict=test_data)
        test_accuracy = session.run(accuracy, feed_dict=test_data)
        print(test_accuracy)

One time I have commented the testing accuracy to print only the training accuracy and one time I did the contrary.
I had stored them in the following table:

My problem is that I am not able to detect overfitting if it exists or not
Can you help me please!

Short answer, no it does not look like you are overfitting. Your validation (you call this test) accuracy seems to follow your training accuracy quite closely.

Easiest way to check for yourself is to plot the training and validation accuracy over each iteration, then look at how the curves behave. What you want to see is a test curve that closely follows your validation curve.

Overfitting can come in too ways, your validation curve being quite a bit lower than your training curve and/or a validation curve that is decreasing when the training curve is increasing.

Both these behaviors can be seen in this picture:

Image taken from Stanford's CS231n

Edit: As gunes points out you seem to have a programming error that should be addressed before any checks for overfitting.


ML: How to think feature selection? What is the basic philosophy behind feature selection and modelling? How do you actually start? Could you please share your real (practical) inputs?

Bit of background:
I am actually trying to analyse incoming service desk requests (emails) and predict/assign it to the concerned department (team).

I have identified 2 features, but they are giving ~70% prediction accuracy, because of few ambiguous entries. So, need to add more features to strengthen my model.

I am sure your inputs will help newbies (like me) to get the initial idea.

In a lot of cases, you deal with too many features and you either try to reduce the dimensionality of your feature space with techniques like PCA or you try to select only the relevant ones. The latter is known as feature selection. However, what you have is a situation where you don't have a lot of features and the information you get from those does not seem to be enough for a solid classification.

Generally speaking, the best options you have in this case are cleaning your data as good as possible and gathering more features. This can either be done by using new sources of information or by deriving new information from existing sources.
Usually, you derive the best feature candidates for your model from the literature of the respective subject area. Domain knowledge helps a lot!

In your case, I would guess that variables like time of day, weekday and day of the month could provide some valuable information, apart from the mail content of course. But these are simple guesses. In the end, you will have to dig into the literature and find the right features by trial and error.

Since you are trying to classify incoming emails, I assume you have access to the mail subject and the mail body. In this case, I would recommend starting with a simple Naive Bayes algorithm, since it is easy to use, effective and there are many implementations and guides how to use them.  Also, Naive Bayes can easily be applied to multiclass problems like yours.


Is there any augmentation tool for images and bounding boxes? I don't have a lot of training data and I'm looking for some tools in python or executable program like labelimg that do some heavy augmentation on images, even better if they also change bounding boxes coordinate accordingly. Any help will be appreciated!

I think you should look into imgaug. It supports most image augmentation and does have support for bounding boxes.

Docs: https://imgaug.readthedocs.io/en/latest/


How can I find correlation between features? The problem

I want to figure out how routers correlate between each other. Like, if a specific error occurred in router A, and almost at the same time the error occurs in router B, they probably have some connection with each other (are at one line).

The Data

Suppose I have a dataframe that looks like this:

|Router|Error|Duration|Timestamp          |
|DB-XX |GSM  |26.5374 |2019-05-01 00:20:14|
|DT-XY |AUC  |15.5400 |2019-05-01 01:15:01|
|DR-YY |AUC  |02.0333 |2019-05-01 01:17:13|
|DP-YX |LOC  |45.2609 |2019-05-01 00:01:10|

The question

What is the best way to deal with it? Regression (one vs the rest) for each router? The problem is, that there are hundreds of models and I also want to reduce computational costs...

Correlation is a metric that can be used when two features have the same number of total observations. Given your scenario, each router can fail for different number of times. So, we can't estimate correlation between two routers since they will be having different number of observations.

One thing you can do would be to convert every router into a time series signal, using the provided duration and start time. You can they compute the correlation then. But, it might not be really informative of the actual situation, as the signal will be predominantly sparse


Strategies for handling unlabeled data which is slightly different from the labeled data Suppose you have a dataset with the following properties:

The number of samples is fairly large (~100K samples)
There are ~150 contextual features and 1 feature consisting of a text-string (which can, of course, be split into any number of features depending on the pre-processing of the text). It is expected that the text-string will have really great predictive power
Samples are divided into 3 categories (prior to you receiving the data) based on a few of the contextual features with category A containing ~5% of the samples, category B containing ~20%, and category C containing the remaining 75%
Category A is entirely labeled, category B is partly labeled (with only a small proportion being unlabeled), and category C is entirely unlabeled
The features used to categorize the samples are likely to influence the probability of a sample belonging to class 0 or class 1.
The samples are not completely different between categories (that is to say, we're not talking cats versus dogs). E.g.: Two very similar samples might end up in different categories based on very small differences on a numerical feature with a large range

The purpose is to build a classifier which will correctly classify the samples. That might look like a semi-supervised learning-problem, but I am worried about the structural differences between the categories. Hence my question: Which strategies could be employed to build a classifier performing well on all of the samples?

Of course I could just be conservative and only deal with the labeled data, but there is great value in also being able to predict the unlabeled data (e.g. the 75% of the data in category C). That's why I'll try to pick your brains for creative solutions!

There is no straightforward way to deal with this scenario, but here are some ideas:

Augment the A/B/C label set with an extra label "U" for "unknown". Now all samples are labeled.
Predict A/B/C labels from the unlabeled samples from the other pieces of data in the sample. Now all samples are labeled.
Train a system for data with labels and another system for data without them. Use them according to the availability of the label. It would be great if your ML approach allows for sharing parts of the model for labeled data and the one for unlabeled data (e.g. neural networks).


How pairplot is constructed? Based on what rule? Why people use it? I am curious how pairplot in seaborn library is constructed. I have seen similar plot, such as Pearson Correlation plot. What is the difference between these two plots? What is exact the use case of them?

Differences:

Pairplot

If you have m attributes in your dataset, it creates a figure with (m)x(m) subplots.

The main-diagonal subplots are the univariate histograms (distributions) for each attribute.

For a non diagonal subplot, assume a position (i,j). This plots all the samples of the dataset on a coordinate system with axes the attributes i and j. In other words, it projects the dataset on these two attributes only. This is particularly interesting to visually inspect how the samples are spread with respect to these two attributes ONLY. The "shape" of the spread can give you valuable insight on the relation between the two attributes. For example, if the samples are spread on a linear pattern, it means that the two attributes are linearly related in your dataset. But you can also visually recognize nonlinear relations, such as quadratic, etc.

Correlation (Pearson)

It is a mathematical measure of strictly LINEAR correlation between two attributes. The closer it is to 1 or -1, the stronger the linear relation. The closer it is to 0, the weaker the relation. If there is a value close to 1 or -1, then you can safely assume that these two variables are linearly connected. If it is close to 0, you can assume the absence of linear relation BUT you cannot infer anything about nonlinearities (they might exist even if correlation is very low).

The main-diagonal elements of the correlation matrix show the autocorrelation of each attribute, thus a measure of its normalized spread around its mean value.

The non diagonal elements (i,j) show the linear correlation between the attributes i and j. Usually, correlation matrix is plotted on a heatmap (seaborn) that helps with the visualization.

Note

If you use Pearson correlation and you find that for the selected attributes (i,j) the value is close to 1 or -1, you should expect the samples forming a straight line in the (i,j) and (j,i) subplots of your Pairplot.


find bigrams in pandas I have a DataFrame with 4 columns: 'Headline', 'Body_ID', 'Stance', 'articleBody', with 'Headline' and 'articleBody containing cleaned and tokenized words.  I want to find bi-grams using nltk and have this so far:

bigram_measures = nltk.collocations.BigramAssocMeasures()
articleBody_biGram_finder = df_2['articleBody'].apply(lambda x: BigramCollocationFinder.from_words(x))

I'm having trouble with the last step of applying the articleBody_biGram_finder with bigram_measures. I've tried multiple iterations of lambda with list comprehension but am getting nowhere.

my most recent attempts:

df_2['articleBody_scored'] = score_ngrams(bigram_measures.raw_freq) for item in articleBody_biGram_finder

df_2['articleBody_scored'] = articleBody_biGram_finder.apply(lambda x: BigramCollocationFinder.score_ngrams(bigram_measures.raw_freq))

df_2['bigram_scored'] = df_2['bigram_finder'].apply(lambda x: x.score_ngrams(bigram_measures.raw_freq))


Is there a model that can adapt to additional new training data with different columns? My training data comes in batches. Sometimes, new batches (completely new samples) come with new columns that are not in old batches, or they may be missing some of the old columns.

For example, suppose there are two ingestions. In the 1st ingestion, we have ETL on a set of fields. In the 2nd ingestion, we have added a new field and we are not allowed to ingest and update the old records again (they may have been deleted for good).

Ideally, I want to train a classifier using all batches of data. What kind of algorithms would perform well under this scenario.

A tree-based algorithm can do that.

The point is that you need to train the model with the union of the possible columns that can exist the different batches.

Moreover you need to account for missing values so that the model can learn to recognize a missing and handle them: you need to recode the missings with a proper value, for example you can create a new level for categorical variables and recode the numerical in the standard way (zero, mean, extreme value, etc.)


Plotting multiple precision-recall curves in one plot I have an imbalanced dataset and I was reading this article which looks into SMOTE and RUS to address the imbalance. So I have defined the following 3 models:
    # AdaBoost
    ada = AdaBoostClassifier(n_estimators=100, random_state=42)
    ada.fit(X_train,y_train)
    y_pred_baseline = ada.predict(X_test)

    # SMOTE
    sm = SMOTE(random_state=42)
    X_train_sm, y_train_sm = sm.fit_sample(X_train, y_train)
    ada_sm = AdaBoostClassifier(n_estimators=100, random_state=42)
    ada_sm.fit(X_train_sm,y_train_sm)
    y_pred_sm = ada_sm.predict(X_test)

    #RUS
    rus = RandomUnderSampler(random_state=42)
    X_train_rus, y_train_rus = rus.fit_resample(X, y)
    ada_rus = AdaBoostClassifier(n_estimators=100, random_state=42)
    ada_rus.fit(X_train_rus,y_train_rus)
    y_pred_rus = ada_rus.predict(X_test)

I then plotted the precision-recall curve for these 3 models. I chose this curve as I want to visualise how the models are performing, and I am not very interested in true negatives (the negative class is the majority class).
To plot the curve, I used ScikitLearn's plot_precision_recall_curve method, like so:

    from sklearn.metrics import precision_recall_curve
    from sklearn.metrics import plot_precision_recall_curve
    import matplotlib.pyplot as plt

    disp = plot_precision_recall_curve(ada, X_test, y_test)
    disp.ax_.set_title('Precision-Recall curve')

    disp = plot_precision_recall_curve(ada_sm, X_test, y_test)
    disp.ax_.set_title('Precision-Recall curve')

    disp = plot_precision_recall_curve(ada_rus, X_test, y_test)
    disp.ax_.set_title('Precision-Recall curve')

This resulted in 3 separate plots.

However, I want to have these 3 curves in one plot such that they can be easily compared. So I want a plot like the one in the article:

But I am not sure how to do this as the plot_precision_recall_curve method only takes one classifier as input.
Some help would be appreciated.

Try using Matplotlib gca() method in this way you can indicate what axis you want to plot in
from sklearn.metrics import precision_recall_curve
from sklearn.metrics import plot_precision_recall_curve
import matplotlib.pyplot as plt

plot_precision_recall_curve(ada, X_test, y_test, ax = plt.gca(),name = "AdaBoost")

plot_precision_recall_curve(ada_sm, X_test, y_test, ax = plt.gca(),name = "SMOTE")

plot_precision_recall_curve(ada_rus, X_test, y_test, ax = plt.gca(),name = "RUS")

plt.title('Precision-Recall curve')


How can I plot line plots based on an input python dataframe? I need help to create a plot using 3 different columns from a dataframe. my dataframe looks like this:

index      CMPGN_NM  COST_SUM    SUMRY_DT

2   GSA_SMB_SMB_Generic_BMM      8985  2018-05-17

3   GSA_SMB_SMB_Generic_BMM      7456  2018-05-18

4   GSA_SMB_SMB_Generic_BMM      5761  2018-05-19

10  GSA_SMB_SMB_Generic_BMM      4251  2018-05-20

5   GSA_SMB_SMB_Generic_BMM     10521  2018-05-21

6   GSA_SMB_SMB_Generic_BMM     10216  2018-05-22

7   GSA_SMB_SMB_Generic_BMM     11023  2018-05-23

9   GSA_SMB_SMB_Generic_BMM     11242  2018-05-24

8   GSA_SMB_SMB_Generic_BMM      8817  2018-05-25

1   GSA_SMB_SMB_Generic_BMM      6937  2018-05-26

0   GSA_SMB_SMB_Generic_BMM      4581  2018-05-27

I would like the output to look like  the graph as below

Here's a solution:

I've created a sample dataframe with some arbitrary values. Here it is:

import pandas as pd
import numpy as np
from datetime import datetime

test = pd.read_csv('/home/sagar/Desktop/test.csv')
# Convert your date from 'str' to 'datetime' format
test['SUMRY_DT'] = test['SUMRY_DT'].map(lambda x: datetime.strptime(x, '%Y-%m-%d'))
# Set it as your dataframe index
test.set_index('SUMRY_DT', inplace=True)
test
            CMPGN_NM                    COST_SUM
SUMRY_DT
2018-05-17  GSA_SMB_SMB_Generic_BMM     8985
2018-05-18  GSA_SMB_SMB_Generic_BMM     7456
2018-05-19  GSA_SMB_SMB_Generic_BMM     5761
2018-05-20  GSA_SMB_SMB_Generic_BMM     4251
2018-05-21  GSA_SMB_SMB_Generic_BMM     10521
2018-05-22  GSA_SMB_SMB_Generic_BMM     10216
2018-05-23  GSA_SMB_SMB_Generic_BMM     11023
2018-05-24  GSA_SMB_SMB_Spark           11242
2018-05-25  GSA_SMB_SMB_Generic_BMM     8817
2018-05-26  GSA_SMB_SMB_Generic_BMM     6937
2018-05-27  GSA_SMB_SMB_Generic_BMM     4581
2018-05-10  GSA_SMB_SMB_Spark           7089
2018-05-13  GSA_SMB_SMB_Spark           2121
2018-05-11  GSA_SMB_SMB_Spark           234
2018-05-12  GSA_SMB_SMB_Spark           11077

# Plot your data
test.groupby('CMPGN_NM')['COST_SUM'].plot(legend=True)

With the actual data, your chart would resemble the picture you have provided.

Hope this helps.


Downsampling audio files for use in Machine Learning I'm trying to use the work (Neural Networks) done in this repo:
https://github.com/jtkim-kaist/VAD

It says this:

  Note: To apply this toolkit to other speech data, the speech data
  should be sampled with 16kHz sampling frequency.

I've got speech data at 48khz. I've read in places that reducing sampling rate is a complicated process, you can't just remove every nth datapoint, you have to filter things...

Is this necessary if I only intend to use the data in the Neural Network toolkit provided by the repo I linked?
If so, is there an industry standard method for changing sample rate?

I realise that it probably depends on what features are being used. However the feature that is used is this:

  MRCG (multi resolution cochleagra) concatenates the cochleagram features at multiple spectrotemporal resolutions

This is a ruddy complicated feature! Lets pretend we're just using a Melspectogram (unless you're willing to answer the question from the perspective of MRCG's).

Neural networks are likely to use features of a Melspectogram that we wouldn't think of. This makes me think it is unwise to train the Neural Net using downsampled speech data unless we intend to predict using 48khz data downsampled to 16khz forever after...

What do you think? Can I use my 48khz data - downsampled with no filtering - with the expectation that the model will work for prediction on real 16khz data?

And then for future readers sake, how about the other way? Say I had an 8khz file, could I increase the sample rate to 16khz without filtering?

Resampling of audio is a standard process and there are many implementations available. In Python you can use librosa, or you can write a script that uses ffmpeg or similar.

If you want to reuse an already trained model this is critical, as the neural network will have learned features based on 16kHz input. It might work OK if you re-train the model with your data using 48kHz. It will be slower to train and predict, but performance can be similar.
However your would need to update the audio_sr parameter of the audio feature extraction. Otherwise the length of window in milliseconds would be wrong, probably no longer matching what is typical for speech, resulting in degraded performance.


Using the validation data I'm unclear on the exact process of using the validation data.

Let's say that I fit my neural network model and adjust hyperparameters using the training set and validation set. Do I then evaluate the test set on this model? Or do I recombine the validation and training sets and fit a fresh model with the hyperparameters that I found during the validation phase, and then evaluate on the test data? I have seen a number of different notebooks and examples that do both ways.

Surely, once I've found my hyperparameters, it makes sense to fit a fresh model using the full training set (recombined with validation set), since the validation loss has no effect on the weights.

The common procedure is this:

Fit the model on train split and adjust hyperparameters using the training split and validation split, as you already said.

Then, to get a "final measure" of performance, train model on both training and validation splits using the hyperparams found during validation phase, and evaluate on test split.


How to deal with large data sets So i'm very new to this, forgive my silly questions.

I've got some data I need to analyse, I would usually just use excel it doesnt seem to be able to do the job.

I have .csv files containing data for every seconds over a year period.
The data consists simlpy of  timestamp and valueX.

I need to analyse these values to see if values X goes below a certain value, and if it does I have various calculations to carry out. Value A is proportional to X and just instantaneous, then value B is a counter proportional to A.

Could you suggest a method/ language / software that would be the best and most accessible to do this? I don't have much experience in data analytics/big data apart from using excel and a bit of matlab and python.

thanks in advance

EDIT:
Thanks for the replies, my data is like this...

I have a text file for January values.

01/02/2016 00:00:00,49.972332
01/02/2016 00:00:01,49.9690056
01/02/2016 00:00:02,49.9600029
01/02/2016 00:00:03,49.9490013
01/02/2016 00:00:04,49.9430046

Different text files for each month, some files have text or notes at the top which I'd need to clean up.

For some context the second value is grid frequency, If it goes below a certain threshold, it constitutes an grid fault. I need to know how long these faults occur for, how often, how far the f value falls etc.

EDIT 2:
Thanks for the tips folks, I've been able to read in my data, clean it up and combine it into one dataframe.
I'm now try to add some extra column to the dataframe that are dependent of the col 1 value but having some trouble.

    "Read in month csv files and create dataframes"
df_jan_short = pd.read_csv('2016_01_short.csv', header = None ,parse_dates = [0], index_col = 0, names = ['timestamp','Freq'],squeeze= True)
df_feb_short = pd.read_csv('2016_02_short.csv', header = None ,parse_dates = [0], index_col = 0, names = ['timestamp','Freq'],squeeze= True)

    "Combine dataframes into one dataframe for year" months_short = [df_jan_short, df_feb_short] year_short = pd.concat(months_short)

    "Repalce NaN values in df with 50.00 Hz"
year_short.fillna(50.00,inplace = True)

    "Check what range freq is in"

year_short['Case'] = 'No Case'
year_short['Case'](year_short['Freq'] > Fc & year_short['Freq'] < Ff ) = 'A' year_short['Case'](year_short['Freq'] > Fa & year_short['Freq'] < Fc ) = 'B'
year_short['Case'](year_short['Freq'] < Fa ) = 'C'
year_short['Case'](year_short['Freq'] > Ff & year_short['Freq'] < Fh ) = 'G'
year_short['Case'](year_short['Freq'] > Fh ) = 'H'

But I get the following error...

runfile('C:/Users/ShaneOKeeffe/Documents/Grid Freq/Py/4.py', wdir='C:/Users/ShaneOKeeffe/Documents/Grid Freq/Py')
Traceback (most recent call last):

  File "C:\ProgramData\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 2862, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-74-3a0fbc1f23f6>", line 1, in <module>
    runfile('C:/Users/ShaneOKeeffe/Documents/Grid Freq/Py/4.py', wdir='C:/Users/ShaneOKeeffe/Documents/Grid Freq/Py')

  File "C:\ProgramData\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 710, in runfile
    execfile(filename, namespace)

  File "C:\ProgramData\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 101, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "C:/Users/ShaneOKeeffe/Documents/Grid Freq/Py/4.py", line 48
    year_short['Case'](year_short['Freq'] > Fc & year_short['Freq'] < Ff ) = 'A'
                                                                                ^
SyntaxError: can't assign to function call

Welcome! I think you asked in the right place.

If you have familiarity with Python, you might look at the numpy and Pandas libraries.  Numpy implements fast array and matrix manipulations, and Pandas arranges numpy objects into tables.  Along with scipy, they make the basis of the Python numerical computing stack.

Without more detail it's difficult to share a code example that'll do what you want, but if you run into any issues feel free to ask.

Edit: if your dataset is REALLY big and won't fit in memory, there are ways around that too, just specify.


Fit Decision Tree to Gradient Boosted Trees for Interpretability I was wondering if there is literature on or someone could explain how to fit a decision tree to a gradient boosted trees classifier in order to derive more interpretable results.
This is apparently the approach that Turi uses in their explain function which outputs something like this:
Turi's explain function:

from their page here.
I know that for random forests you can average the contribution of each feature in every tree as seen in the TreeInterpreter python package, but this seems to be a different method since it is focused on exact splits and one decision tree.
Does anyone know anything more about this method for interpreting gradient boosted trees?

Gradient boosting learns multiple decision or regression trees after each other. The difference with random forests is that the trees correct each other. Each new tree is fitted on the residual produced by the predictions from the earlier trees.

The explain method shows for each prediction (i.e. record) why a particular decision was made. This results in a set of decisions. Each tree makes several decisions. The decisions of all trees are stacked to get the set of rules that justifies a single decision. This information is specific for the gradient boosting model and the specific record.

If you want to know more about the gradient boosting model, you may want to consider feature importance. Feature importance expresses how important the features are. You can use print model.get_feature_importance() for this.


Online Variational Autoencoder when training a VAE, typically one samples from the latent distribution using the reparametrization trick using a fairly large minibatch size (>100) in the decoder/generator half of the VAE. I'm assuming this minibatch size allows the network to 'smooth' out the error and allows us to avoid having to repeatedly sample from the latent space.

However, I'm interested in online scenarios where you are training the VAE on streaming data as it arrives, so the batch size would be 1. In this case, it can take the VAE a long time to converge because the error is highly volatile.

Is there any way to avoid this issue in practice? I am unsure what will happen if I have to repeatedly sample from the latent distribution and then take the mean of those samples (or something) - aside from obvious performance concerns. The other alternative is to wait for enough samples to arrive that I can train them in a larger batch, but even in this case I wouldn't be able to wait for 100+ samples to arrive.

The fact that pieces of data arrive one by one does not define the minibatch size. You can have a buffer of size N (with either FIFO or with any other eviction policy that suits the statistical properties needed) and sample minibatches of size M out it every time you want to update the autoencoder.

Depending on the buffer eviction policy and the sampling strategy, this may also help to avoid local autocorrelation.

Note: this answer was originally a comment to the OP's question.


What does the KFold error mean and how to get confusion matrix from Kfold random forest implementation? from sklearn.model_selection import KFold

num_folds = 10

seed = 77

kf = KFold(n_splits=num_folds,random_state=77,shuffle=False)

rfc=RandomForestClassifier(n_jobs=2,n_estimators=500,criterion="entropy",random_state=9999)

kf.get_n_splits(X)

for train_index, test_index in kf.split(X):

    X_train, X_test = X[train_index], X[test_index]

    y_train, y_test = y[train_index], y[test_index]

    rfc.fit(X_train, y_train)

    print(confusion_matrix(y_test, rfc.predict(X_test)))

print(10* '#')

I get error which I dont understand.
KeyError: "None of [Int64Index([100, 101, 102, 103, 104, 105, 106, 107, 108, 109,\n            ...\n            990, 991, 992, 993, 994, 995, 996, 997, 998, 999],\n           dtype='int64', length=900)] are in the [columns]"

modified my code like below, works now

kf = KFold(n_splits=num_folds,random_state=seed,shuffle=False)
kf.get_n_splits(X)
i=1
print("confusion matrix:")
for train_index, test_index in kf.split(X):

    X_train, X_test = X.iloc[train_index,:], X.iloc[test_index,:]
    y_train, y_test = y.iloc[train_index], y.iloc[test_index]

    rfc.fit(X_train, y_train)
    print (i,"\n",confusion_matrix(y_test, rfc.predict(X_test)))
    i=i+1
    print(10* '#')


Explanation about i//2 in positional encoding in tensorflow tutorial about transformers I was implementing the transformer architecture in tensorflow.
I was following the tutorial : https://www.tensorflow.org/tutorials/text/transformer#setup_input_pipeline
They implement the positional encoding in this way:
angle_rates = 1 / np.power(10000, (2 * (i//2)) / np.float32(d_model))

However in the paper i is not divided by 2 (i//2), is this a bug? , or why is the reason to make this operation?

thanks

It's not a bug, although they added some confusion with this trick. They should better call their argument $j$ instead of $i$, cos what they actually do is they take all values $0 \leq j \leq d_{model} - 1$ and compute $PE(pos, j)$. $j$ сan be either even or odd, but in the right side of the equation it's even, that's why they compute i//2 and multiply back by 2.


Why do CNNs with ReLU learn that well? Convolutional Neural Networks (CNNs) use almost always the rectified linear activation function (ReLU):

$$f(x) = max(0, x)$$

However, the derivative of this function is

$$f'(x) = \begin{cases} 0 &\text{if } x \leq 0\\ 1&\text{otherwise}\end{cases}$$

(ignoring that is not differentiable at $0$, as I think it is done in practice). For inputs > 0 this is fine, but why doesn't it matter that the gradient is 0 at every point < 0? Or does it matter? (Are there publications about this problem?)

If a neuron outputs 0 for every sample of the training data, it is basically lost, correct? Its weights will never be adjusted again?

ignoring that is not differentiable at 00, as I think it is done in practice

yes see  ReLUs are not differentiable at zero

  If a neuron outputs 0 for every sample of the training data, it is basically lost, correct? Its weights will never be adjusted again?

yes see What is the "dying ReLU" problem in neural networks?


Can ridge regression be used for feature selection? I'm trying to figure out whether using Ridge Regression for regularization can be used to cause a more sparse hypothesis however to me it seems like ridge will never actually bring any coefficients to zero, only really close to it.

So can ridge regression cause any coefficients to become zero? Can the number of zeros in a weights vector change from zero to something else?
Or maybe to put it simply, are the number of zero coefficients using ridge regression monotonically increasing or possibly decreasing?

Thanks.

Regularization of Ridge causes its weight to become very close to zero, but not zero. In contrast lasso can make weights equal to zero because of the types of regularization they use.

I would recommend to use lasso if you have lots of features, and you think just few of them are important. Otherwise, even though your model becomes simpler you will have a poor accuracy.

There is ElasticNet which combines both, but it is more expensive since it uses two regularization.

I didn't really understand what you ask by

  Can the number of zeros in a weights vector change from zero to something else?

If you determined number of parameters beforehand their weights are unknown yet, not zero. That's what you are trying to find.

One last thing, for feature selection there are other methods. These(ridge, lasso) are just linear models for regression. If you want to identify which features work best etc. I suggest you to do a research on methods for this.