What algorithm to use for extracting information from bank statements I am trying to find the best way to extract information from bank statements. A bank transaction is not a natural text but still human readable.

I would like to extract a bunch of data if present like payment method, date, amount, vendor/customer name and even information like an order/invoice ID or the reason. I have sets of data that I can use for the training, for instance (100k+ vendors, payment methods words, etc). Also the solution must work with multi languages. The bank transactions could be in different languages (not only english).

Is named entity recognition the best way to go?

here is some sample data I can have in input:

Dividend
POS PURCHASE Non-PIN: RCI*RINGCENTRAL, INC.
IMP.DEB.LEY 83652 - ALICUOTA GENERA
DIRECTDEBIT EQUITY COMMERCE MERCH FEES 158713 62000329526385 CCD
ADVANCE AUTO PARTS JACKSONVILLE

Thanks for your help

Am currently working on something in this domain.

The rough process I am currently following is -

Extract data from PDFs (ubiquitous version of Bank Statements nowadays) into more usable formats. Currently,converting them to TXT files first as an intermediate step.
Generically, bank statements (from a specific bank) tend to be structured in the same format. Hence, while converting the TXT to CSV, you can structure the algo such that it knows what to pick up - based on a rough analysis of the TXT file.

Use 2 different data frames to store the statements content - one df keeping track of the transactional data (storing 3 values - Date, Description and Amount) and the other to keep track of the metadata (storing 2 values - Key and Value)
You can determine the rough structure of the TXT file by going through a few TXTs. e.g. the transactions generally begin after a specific element's occurrence in every bank statement. Ignore the page footers / headers that recur in case of multi-page transactions. Also, another observation would be that all data in a specific column of the transactions (the transactions are read in as a table by my program [link given below]) appear together.
Doing the above enables the transaction and meta data to a relatively structured and processable format

Perform analysis on the Description column or the entire transactional df you have thus generated.

You can use the following code to convert a given PDF to a txt file -

from __future__ import unicode_literals
import os
import sys
reload(sys)
sys.setdefaultencoding('utf8')
from pdfminer.pdfinterp import PDFResourceManager, PDFPageInterpreter
from pdfminer.pdfpage import PDFPage
from pdfminer.converter import XMLConverter, HTMLConverter, TextConverter
from pdfminer.layout import LAParams
from cStringIO import StringIO

def pdfparser(data):
    dest = data[:-3]+"txt"
    print "\n\n\n\n\n",dest
    fp = file(data, 'rb')
    rsrcmgr = PDFResourceManager()
    retstr = StringIO()
    codec = 'utf-8'
    laparams = LAParams()
    device = TextConverter(rsrcmgr, retstr, codec=codec, laparams=laparams)
    # Create a PDF interpreter object.
    interpreter = PDFPageInterpreter(rsrcmgr, device)
    # Process each page contained in the document.
    for page in PDFPage.get_pages(fp):
        interpreter.process_page(page)
        data =  retstr.getvalue()
        data = unicode(data,'utf-8', errors='ignore')
    #print data
    # write data to a file
    print data
    with open(dest, "w") as f:
        f.write(data)

# set the working directory
path = "F:/banking/payments/"
os.chdir(path)
fls=os.listdir(path)

for x in fls:
    if x[-3:]=="pdf":
        pdfparser(path+x)


Multiclass classification problem with multiple targets to be predicted We have a supervised multi-class classification problem where we need to predict two targets for each sample: 'brand' and 'category'. Our features are 'shop_name' which can be any proper noun and several categorical features ('shop_type', 'building_type').
In the training data, 'brand' is not always present, but 'category' is always present. When brand is present, the category is directly dependent on it. For e.g. If the brand is 'Snickers', the category is already known to be 'Foodstuff'. In other cases, the category has to be determined from the input features.
Here is a representative example of our training data:

shop_name
shop_type
building_type
brand
category

Snickers Ltd

huge
Snickers
Foodstuff

Acme Intl
office stationary
medium

Office Equipment

Davidoff cigars
cigarette
big
Davidoff
Smoking

Sample Company
car repair
small

Automobile

Features: shop_name (free text), shop_type (categorical - 100 types), building_type (categorical - 20 types)
Target: brand (categorical - 2500 brands), category (categorical - 250 categories)
How should we structure our classification algorithm to determine brand and category? We have a well known list of about 2500 brands that we are interested in.

You might want to visit the definition of multi-class and multi-label problem. In this case looks like are aiming to solve a multi-label problem. Refer to the following problem, this might help you better understand your situation.  https://stats.stackexchange.com/a/11866/286349


Saving the hidden layers of a trained network I have used a pre-trained VGG-19 model to build an image classifier as part of a MOOC. I have implemented a classifier like this:

classifier = nn.Sequential(nn.Linear(25088, 512),
                           nn.ReLU(),
                           nn.Dropout(p=0.2),
                           nn.Linear(512,350),
                           nn.ReLU(),
                           nn.Dropout(p=0.2),
                           nn.Linear(350,250),
                           nn.ReLU(),
                           nn.Dropout(p=0.2),
                           nn.Linear(250,102),
                           nn.LogSoftmax(dim=1))

model.classifier = classifier

In the next step, I was asked to save the trained model so it could be reused later. My understanding of saving such a model was that I had to create a dictionary with the input size (given by the pre-trained model), the output size (given by my specific problem), the model's state dict and the hidden layers (given by my own code). I tried to do that like this:

checkpoint = {'input_size':25088,
              'output_size':102,
              'hidden_layers':[each for each in model.classifier],
              'state_dict':model.state_dict(),
              'optimizer':optimizer.state_dict(),
              'epochs':epochs,
              'class_to_index':cat_to_name}

The I tried to load it like this:

def load_checkpoint(path):
    checkpoint = torch.load(path)
    model = nn.Sequential(checkpoint['input_size'],
                          checkpoint['output_size'],
                          checkpoint['hidden_layers'])
    model.load_state_dict(checkpoint['state_dict'])
    epochs = checkpoint['epochs']
    model.class_to_idx = checkpoint['class_to_index']
    return model, epochs

test_model, _ = load_checkpoint('trained_model.pth')
print(test_model)

However, this just raises the error:

TypeError: int is not a Module subclass

What did I do wrong? As far as I can tell checkpoint['hidden_layers'] does not contain integers by themselves:

print(checkpoint['hidden_layers'])

Returns:

[Linear(in_features=25088, out_features=512, bias=True), ReLU(), Dropout(p=0.2), Linear(in_features=512, out_features=350, bias=True), ReLU(), Dropout(p=0.2), Linear(in_features=350, out_features=250, bias=True), ReLU(), Dropout(p=0.2), Linear(in_features=250, out_features=102, bias=True), LogSoftmax()]

Which is what I would expect.

Thank you for reading! Any thoughts or comments are highly appreciated!

There are a few mistakes. The following is the way u need to change your code. I am not sure what your model class consists  but I am hoping it is class function with .classifier consisting the network.

def load_checkpoint(model, path):
    checkpoint = torch.load(path)
    model.classifier.load_state_dict(checkpoint['state_dict'])
    epochs = checkpoint['epochs']
    model.class_to_idx = checkpoint['class_to_index']
    return model, epochs

classifier = nn.Sequential(nn.Linear(25088, 512),
                           nn.ReLU(),
                           nn.Dropout(p=0.2),
                           nn.Linear(512,350),
                           nn.ReLU(),
                           nn.Dropout(p=0.2),
                           nn.Linear(350,250),
                           nn.ReLU(),
                           nn.Dropout(p=0.2),
                           nn.Linear(250,102),
                           nn.LogSoftmax(dim=1))

model.classifier = classifier

test_model, _ = load_checkpoint(model, 'trained_model.pth')
print(test_model)

Similarly you need to load optimizer, scheduler and other attributes if you want to retrain the network.

if your intention is to load only part of the network instead of the entire network. Something as below

classifier = nn.Sequential(nn.Linear(25088, 512),
                           nn.ReLU(),
                           nn.Dropout(p=0.2),
                           nn.Linear(512,350),
                           nn.ReLU(),
                           nn.Dropout(p=0.2),
                           nn.Linear(350,250))

model.classifier = classifier
weights = torch.load("weight.pth")
state_dict = model.classifier.state_dict()

for name, param in model.classifier.named_parameters():
    if name in state_dict.keys():
        state_dict[name] = weights["state_dict"][name]

model.classifier.load_state_dict(state_dict)

In this way you can only load the part of the network required.


How to give breakdown of text prediction sentences in Shiny I have created a text prediction app that displays 10 sentences based on the word searched in the toolbar. However, all the sentences are being displayed in a single toolbar horizontally. I would like the sentences to appear sequentially i.e. vertically, one after another
    ui<- shinyUI(fluidPage(

  # Application title
  mainPanel(
   img(src='image.jpg', align = "right"),

  #titlePanel(title=div(img(src="spsimage.jpg"))),

  fluidRow(HTML(" Search Bar")),
  #fluidRow(HTML(" Date: 06-29-2020") ),

  fluidRow(
    br(),
    p("Text predictior app ")),
  br(),
  br(),

  fluidRow(HTML("Enter a word. Press \"Next words\" button to predict the following words") ),
  fluidRow( p("\n") ),

  # Sidebar layout
  sidebarLayout(

    sidebarPanel(
      textInput("inputString", "Enter a word here",value = " "),
      submitButton("Next words")
    ),

    mainPanel(
      h4("Predicted Next Word"),
      verbatimTextOutput("prediction"),
      textOutput('text1'),

    )
  )
)))

server <- function(input, output, session) {
  sentences <- reactive({
    make_sentences(input$inputString)
  })
  output$prediction <- renderText({ sentences() })
}

the output is coming as - Angry look for some time for Angry glance, and the duc d’enghien Angry look was a tone or Angry tears. “Lise!” Was that? The Angry tears. “Lise!” Said princess anna Angry look as much as she Angry look for three of the Angry tears. “Lise!” Said to the Angry look fixed on with an Angry glance, and fireworks are not
I would prefer -

1- Angry look for some time for
2- Angry glance, and the duc d’enghien
3- Angry look was a tone or Angry tears. “Lise!” Was that?
4-The Angry tears. “Lise!” Said princess anna
5-Angry look as much as she
'''''

Maybe you could put the sentences in a dataframe and used renderTable, something like this:
server <- function(input, output, session) {
  sentences <- reactive({
    data.frame(sentences=make_sentences(input$inputString))
  })
  output$prediction <- renderTable(sentences())
}

[edit]
In order to only have them line by line something like this might work:
server <- function(input, output, session) {
  sentences <- reactive({
    paste(make_sentences(input$inputString),sep='<br/>')
  })
  output$prediction <- renderTable(sentences())
}

Alternatively you could have a numbered list by using the corresponding HTML tags: insert the whole list inside <ol> ... </ol> and every sentence in a <li> .. </li>.


Visualizing F-score differences in information extraction I have several corpora and NLP systems (including a few merge ensembles of output of these systems combined in unions and intersections) with which I have extracted the annotation span sets {(begin, end)} for each corpus across all documents within the corpus and compared the span sets to each corpus's respective gold standard and thus obtained standard measures of F-score, precision and recall.

I am trying to qualitatively assess why certain systems don't perform as well as a particular ensemble combination on F-score, so I figured the easiest way would be to generate precision-recall or ROC curves.

The task is just a simple binary classification: either a span of text is annotated (labeled as 1) or it is not (labeled as 0).

I have numpy vectors of the same length for each document in the corpus for both the system predictions and the gold standard, so I plan on using these for y_true and y_predict when trying to generate my ROC curve.

Is this a good approach to observe the behavior of my F-scores, assuming I plot them all on the same graph? If not, any recommendations for a better approach would be most appreciated.

Yes, I think that's a sound approach and a good way to compare different systems.

A ROC curve comparison is usually more informative than the raw performance scores, but it's still quite general. In case you want to observe even more detail, you could also try to look at specific groups of instances. One way to do that is to count for every instance how many systems correctly classify it: an instance almost always correctly classified is "easy", and conversely an instance which is almost always misclassified is "hard". It's often interesting to look at what happens specifically for the "hard" instances with the different systems. You could take a subset of "hard" instances and calculate the performance or ROC curve only on those, in order to distinguish more precisely the best systems.

For the record, if it makes sense for your task you might also want to consider more flexible scoring methods for text spans: currently it seems that your evaluation considers an answer correct only of the exact span is predicted. You could consider counting the fact that a span is partially correct, for instance by counting the number of tokens correctly annotated.


Multi task learning in Keras I am trying to implement shared layers in Keras. I do see that Keras has keras.layers.concatenate, but I am unsure from documentation about its use. Can I use it to create multiple shared layers? What would be the best way to implement a simple shared neural network as shown below using Keras?

Note that all the shape of input, output and shared layers for all 3 NNs are the same. There are multiple shared layers (and non-shared layers) in the three NNs. The coloured layers are unique to each NN, and have same shape.

Basically, the figure represents 3 identical NNs with multiple shared hidden layers, followed by multiple non-shared hidden layers.

I am unsure how to share multiple layers as in the Twitter example, there was just one shared layer (example in API doc).

By using the functional API you can easily share weights between different parts of your network. In your case we have an Input $x$ which is our input, then we will have a Dense layer called shared. Then we will have three different Dense layers called sub1, sub2 and sub3 and then three output layers called out1, out2 and out3.

x = Input(shape=(n, ))
shared = Dense(32)(x)
sub1 = Dense(16)(shared)
sub2 = Dense(16)(shared)
sub3 = Dense(16)(shared)
out1 = Dense(1)(sub1)
out2 = Dense(1)(sub2)
out3 = Dense(1)(sub3)

We can define our model now like this:

model = Model(inputs=x, outputs=[out1, out2, out3])

It will expect a tuple/list of three elements now, one for each output.

You can go a lot further with these concepts. Let's say we would like to learn individual weights for the person layers but still want to have the same weights for the linear combination towards the output layer, we could achieve that by doing this:

out = Dense(1)
out1 = out(sub1)
out2 = out(sub2)
out3 = out(sub3)

EDIT: Concatenating is basically the opposite of what you want to do, it's pasting (intermediate) outputs of different parts of your network into a new layer. You actually want to split off to multiple different parts.


When one model is superior in real world use? I have an NLP neural network that I have developed with Keras for multi-label classification.

I have fit the model several times and save the best results (via best validation accuracy score) after each set of epochs completes. All of my saved models are in the 96%+ validation accuracy score (according to Keras).

However, when I run these models against real-world data where I also know the result (e.g. effectively a second round of validation) one model in particular outperforms the rest. I can take the champion model (96.29% validation accuracy) and put it up against another model (with something like 96.18% validation accuracy) and the champion model can achieve 90%+ accuracy in the second round of validation while the other model - or any other model - will do nowhere near that. This one model will achieve a minimum 8% accuracy above all other models.

I have double-checked my methodology and I'm nearly positive that all models are being created with the same code and process.

Should I be concerned that this one particular model outperforms the rest? Does it indicate anything in particular in my overall methodology?

Maybe I did not get the question but all looks fine. This is how you do model selection. You have several models (either same algorithm with different parameters or different algorithms. does not matter) and then you perform cross validation to get the best model according to empirical errors coming from validation set. The best model wins the game and is chosen. Everything seems to be right.


Anyway to know all details of trees grown using RandomForestClassifier in scikit-learn? I am building a standard RandomForest classifier (named model, see the code below) using scikit-learn package. Now, I want to get all parameters of one Randomforest classifier (including its trees (estimators)), so that I can manually draw the flow chart for each tree of the RandomForest classifier. I wonder if anyone knows how it can be done?

Thank you in advance.

Manh

#Import Library
from sklearn.ensemble import RandomForestClassifier #use RandomForestRegressor for regression problem
#Assumed you have, X (predictor) and Y (target) for training data set and x_test(predictor) of test_dataset
# Create Random Forest object
model= RandomForestClassifier(n_estimators=10, max_depth=5) #n_estimators=1000 oob_score = True
#====
#X, y = input_X, input_y

from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size = 0.2, random_state = 4)

# Train the model using the training sets and check score
model.fit(X_train, y_train)

#Predict Output
y_pred_train = model.predict(X_train)
y_pred_test = model.predict(X_test)

#accuracy
from sklearn.metrics import accuracy_score
print(accuracy_score(y_train,y_pred_train))
print(accuracy_score(y_test,y_pred_test))

I think Terence Parr's answer is now partly outdated. You can get the same (and more) with:

print("Tree depths: ", [t.get_depth() for t in model.estimators_])
print("Tree number of leaves: ", [t.get_n_leaves() for t in model.estimators_])


Punctuation removal effect on n gram detection for keyword generation I'm building ngrams after removing punctuation and lemmatization. The algorithm is to detect keywords in large bodies of text.

I have concerns that 2 documents

  The child played with the red ball.

And

  The sign was red. Balls are the toys of children.

would both contain "red ball". Are there best practices here? Ideally I would not want the second document to have the same value of "red ball" as the second.

The n-gram model is often built after segmenting into words and sentences. If the data is segmented by sentence it's easy to avoid any overlap between sentences: one can simply extract n-grams sentence by sentence independently. In case it's more convenient to extract all the n-grams at once, padding can be used to mark the beginning/end of sentences like this:

  The sign was red #SENT# Balls are the toys of children #SENT#

Dealing with other punctuation signs which don't mark the end of a sentence can be a bit more tricky, especially if you want to keep the possibility of a keyword which spans over some punctuation signs (for instance in "red-handed" or "tl;dr").


Extremely high gain with LightGBM I am working on a binary classification problem. The target variable is not linearly separable, so I've decided to use LightGBM with default parameters (I only play with n_estimators on range from 10 - 100).

When I output Gain (feature importance for LightGBM) it has extremely high values on the x-axis. When I increase the number of estimators x-axis gain grows even higher. It seems to me that the model is overfitted to a single feature [1].

On the other hand, split feature importances seem to have a nice distribution without extreme values on the x-axis[2].

What is the reason for this behavior with extreme Gain values? Regularization?

Its definitely not regularization since default values are 0 (check here)

n_estimators is the number of decision trees you take into bagging. These decision trees take random number of rows AND columns (again depending on the parameters) so it could be that you took some unfortunate combination which should level out with higher number of n_estimators.

Suggestion: use different criteria for feat.importance for example Permutation importance, and then compare.

Final thought, hyperparams can be irrelevant. Have a look at the effect of parameters on certain tasks, with different algorithms. Conclusion: it can be an overkill (sometimes its important!)


Cluster algorithm which minimizes a distance while fulfilling a constraint I have data of 90 persons.

Data of a person:
- x coordinate
- y coordinate
- score (1 to 6)

I want to form groups of 9 people each so that:
- the distance between people in one group is as small as possible
- people with the same coordinates are in different groups
- people within a group should have different scores

I am not expecting a full-fledged algorithm but rather hints in the right direction. So far I am using a simple k-means algorithm which forms groups so that the distance between people gets minimized however the two other constraints are not considered, unfortunately.

I found this article on "Clustering with Constraints" to be a helpful resource to what you're talking about:

Credit:
Clustering with Constraints
Incorporating Prior Knowledge
into Clustering
Adapted from a Tutorial of Sugato Basu and Ian Davidson (SDM 2005)


Tensorflow Model fitting ValueError: Layer sequential expects 1 inputs, but it received 520 input tensors I am trying to train a model using Tensorflow. I am reading a huge csv file using tf.data.experimental.make_csv_dataset
Here is my code:
Imports:
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.layers.experimental import preprocessing

LABEL_COLUMN = 'venda_qtde'

Reading a csv into a tf.data.Dataset:
def get_dataset(file_path, **kwargs):
  dataset = tf.data.experimental.make_csv_dataset(
      file_path,
      batch_size=4096,
      na_value="?",
      label_name=LABEL_COLUMN,
      num_epochs=1,
      ignore_errors=False,
      shuffle=False,
      **kwargs)
  return dataset

Buiding a model instance:
def build_model():
  model = None
  model = keras.Sequential([
    layers.Dense(520, activation='relu'),
    layers.Dense(520, activation='relu'),
    layers.Dense(520, activation='relu'),
    layers.Dense(1)
  ])

  model.compile(loss='mean_squared_error',
                optimizer='adam',
                metrics=['mae'])
  return model

Executing the functions:
ds_treino = get_dataset('data/processed/curva_a/curva_a_train.csv')
nn_model = build_model()
nn_model.fit(ds_treino, epochs=10)

But when the fit function is called, I get the error:
ValueError: in user code:

    /home/machine-learning/.virtualenvs/jupyter-n5c7sT9n/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:806 train_function  *
        return step_function(self, iterator)
    /home/machine-learning/.virtualenvs/jupyter-n5c7sT9n/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:796 step_function  **
        outputs = model.distribute_strategy.run(run_step, args=(data,))
    /home/machine-learning/.virtualenvs/jupyter-n5c7sT9n/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_lib.py:1211 run
        return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
    /home/machine-learning/.virtualenvs/jupyter-n5c7sT9n/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_lib.py:2585 call_for_each_replica
        return self._call_for_each_replica(fn, args, kwargs)
    /home/machine-learning/.virtualenvs/jupyter-n5c7sT9n/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_lib.py:2945 _call_for_each_replica
        return fn(*args, **kwargs)
    /home/machine-learning/.virtualenvs/jupyter-n5c7sT9n/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:789 run_step  **
        outputs = model.train_step(data)
    /home/machine-learning/.virtualenvs/jupyter-n5c7sT9n/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:747 train_step
        y_pred = self(x, training=True)
    /home/machine-learning/.virtualenvs/jupyter-n5c7sT9n/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py:975 __call__
        input_spec.assert_input_compatibility(self.input_spec, inputs,
    /home/machine-learning/.virtualenvs/jupyter-n5c7sT9n/lib/python3.8/site-packages/tensorflow/python/keras/engine/input_spec.py:155 assert_input_compatibility
        raise ValueError('Layer ' + layer_name + ' expects ' +

    ValueError: Layer sequential expects 1 inputs, but it received 520 input tensors. Inputs received: ...

My dataset has 519 features and 1 label and about 17M lines.
Can anyone help me what I am doing wrong?

First of all look at the shape of tensors that your tf.data.Dataset returns then try to set the input_shape of the first Dense layer like:
model = keras.Sequential([
    layers.Dense(520, activation='relu', input_shape=(1, 519)),
    layers.Dense(520, activation='relu'),
    layers.Dense(520, activation='relu'),
    layers.Dense(1)
  ])

or explicitly add the Input layer
or set the number of neurons of the first Dense layer corresponding to the number of features (519)
Also read the docs they are really great:
https://www.tensorflow.org/api_docs/python/tf/keras/Sequential


How to rename columns that have the same name? I would like to rename the column names, but the Data Frame contains similar column names. How do I rename them?

df.columns

Output:

Index([ 'Goods',
       'Durable goods','Services','Exports', 'Goods', 'Services', 'Imports', 'Goods', 'Services']

Here, there are three goods columns that have similar names. How can I rename a specific column?

You can use this:

df.columns = ['Goods_1', 'Durable goods','Services','Exports', 'Goods_2', 'Services', 'Imports', 'Goods_3', 'Services']

or if you have too many columns:

cols = []
count = 1
for column in df.columns:
    if column == 'Goods':
        cols.append(f'Goods_{count}')
        count+=1
        continue
    cols.append(column)
df.columns = cols


General approahces for grouping a continuous variable based on text data? I have a general methodological question. I have two columns of data, with one a column a numeric variable for age and another column a short character variable for text responses to a question.

My goal is to group the age variable (that is, create cut points for the age variable), based on the text responses. I'm unfamiliar with any general approaches for doing this sort of analysis. What general approaches would you recommend? Ideally I'd like to categorize the age variable based on linguistic similarity of the text responses.

Since it is general methodological question, let's assume we have only one text-based variable - total number of words in a sentence. First of all, it's worth to visualize your data. I will pretend I have following data:

Here we see slight dependency between age and number of words in responses. We may assume that young people (approx. between 12 and 25) tend to use 1-4 words, while people of age 25-35 try to give longer answers. But how do we split these points? I would do it something like this:

In 2D plot it looks pretty straightforward, and this is how it works most of the time in practise. However, you asked for splitting data by a single variable - age. That is, something like this:

Is it a good split? I don't know. In fact, it depends on your actual needs and interpretation of the "cut points". That's why I asked about concrete task. Anyway, this interpretation is up to you.

In practise, you will have much more text-based variables. E.g. you can use every word as a feature (don't forget to stem or lemmatize it first) with values from zero to a number of occurrences in the response. Visualizing high-dimensional data is not an easy task, so you need a way to discover groups of data without plotting them. Clustering is a general approach for this. Though clustering algorithms may work with data of arbitrary dimensionality, we still have only 2D to plot it, so let's come back to our example.

With algorithm like k-means you can obtain 2 groups like this:

Two dots - red and blue - show cluster centres, calculated by k-means. You can use coordinates of these points to split your data by any subset of axes, even if you have 10k dimensions. But again, the most important question here is: what linguistic features will provide reasonable grouping of ages.


Count the max number of consecutive 1 and 0 in Pandas Dataframe Hey I have the following Dataset
import pandas as pd
df = pd.DataFrame({
'column1': [0,0,1,0,1,0,0,1,1,0,1,1,1]})

I want to be able to count the number of consecutive 1 and 0 and generate 2 columns as such:
consec0: 1,2,_,1,_,1,2,_,_,1,_,_,_
consec1: _,_,1,_,1,_,_,1,2_,1,2,3

I then want to take the max number of consecutive and create two lists:
max_consec0: 2,1,2,1
max_consec1: 1,1,2,3

My dataset in the end will be just max_consec0 and max_consec1

To check if a value has changed, you can use .diff and check if it's non-zero with .ne(0) (the NaN in the top will be considered different than zero), and then count the changes with .cumsum, like this:
df['counter'] = df.diff().ne(0).cumsum()

Afterward, you can create a second dataframe, where the indices are the groups of consecutive values, and the column values are the value (0 or 1, in your case) and length (which is what you ultimately want):
df2 = df.groupby('counter')['column1'].min().to_frame(name='value').join(
df.groupby('counter')['column1'].count().rename('number'))

The resulting max_consec0, max_consec1 are just the values in the [number] column, filtered by the [value] column:
max_consec0 = df2[df2['value']==0]['number'].tolist()
max_consec1 = df2[df2['value']==1]['number'].tolist()

You can verify that the result is [2, 1, 2, 1] and [1, 1, 2, 3], as desired.


How to calculate which word fits the best given a context and possible words? I have this task for research purposes and searched a while for a framework or a paper which already took care of this problem.

Unfortunately I don't find anything which helps me with my problem.

I have a sentence like

if the age of the applicant is **higher** than 18, then ...

and a list of words like

higher, bigger, greater, wider ...

which are all a

Now I want to find find out, which of the given words approximately fits the best at the predefined position in the sentence.

The best fitting word in this example would be 'greater', but for example 'higher' would be also fine.
In my specific case, I want to show an error message if someone would write 'wider', because this doesn't make sense in this semantic context.

I hope that I explained my problem good enough.

There are two options :

CBOW . Modify Word2Vec CBOW code to save the whole trained model (current implementations only persist embedding layer)

  CBOW Model: This method takes the context of each word as the input and tries to predict the word corresponding to the context.

Intro : https://towardsdatascience.com/introduction-to-word-embedding-and-word2vec-652d0c2060fa
Example : https://www.tensorflow.org/tutorials/representation/word2vec

Train an LSTM / GRU to predict next word (given previous N words)

Karpathy's article is probably the best introduction to text generation with RNN (this works at character level, you will have to modify it to work at word level [Word-Vector level])

http://karpathy.github.io/2015/05/21/rnn-effectiveness/

Example :

https://medium.com/phrasee/neural-text-generation-generating-text-using-conditional-language-models-a37b69c7cd4b


Is MLlib compulsory to work with distributed data? I am new to the field of machine learning and recently learned the basics and working out various algorithms in python using libraries such as pandas, numpy, matplotlib, scikitlearn , etc.
                    I started learning about working Bigdata by distributing it and using Apache Spark's library MLlib to load and apply algorithms on it. So is working with MLlib the only way on Spark or is there any other way to use pandas and other libraries on distributed data?

Yes. There are a million ways to engineer a solution.

You can use pandas for data wrangling and distribute this task via parallel python.

I personally use python for distributed - data acquisition, data wrangling, and algorithmic computation. This way i get the capability of Multi-machine, multi CPU, & multi GPU.

If you want to use python all they way, i recommend recording all the different options in terms of libraries, then order them by most active in development releases, or by whatever criterion you value the most.


What is the ideal database that allows fast cosine distance? I'm currently trying to store many feature vectors in a database so that, upon request, I can compare an incoming feature vector against many other (if not all) stored in the db. I would need to compute the Cosine Distance and only return, for example, the first 10 closest matches. Such vector will be of size ~1000 or so.

Every request will have a feature vector and will need to run a comparison against all feature vectors belonging to a subset within the db (which will most likely be in the order of thousands of entries per subset in the worst case scenario).

Which database offers the flexibility to run such a query efficiently ?

I looked into postgres but I was wondering if there were alternatives that better fit this problem. Not sure it matters much, but I'm most likely going to be using Python.

I found this article about doing it in SQL.

EDIT: I am open to alternative solutions for this problem that are not necessarily tied to SQL.

If it's only a few thousand entries each with a 1,000 features, you may just be able to keep it in RAM if you are running this on some kind of server. Then when you get a new feature vector, just run cosine similarity routine. An easy way to do this is just use something standard like pandas and scikit-learn.

Alternatively you can keep everything in SQL, load it into something like pandas and use scikit-learn.

I'm actually not sure you'll get much of a speed up, if any, by writing the computation in SQL itself.


how to make new class from the test data I have a list of accounts as data set and I need to group the accounts that refer to the same user using many features.

I'm thinking to use machine learning( but I'm new in this domain), because I know the group of each account for the training data set.
ex of training data:

account-id   Feature1    Feature2    class(Group)
1            T1          P4          Gr1
2            T2          P4          Gr1
3            T3          P2          Gr2

The problem is in the testing of data and when a new account arrive for a new group not learned before in the training set.

ex of testing data:

account-id   Feature1   Feature2
4             T5         P5
5             T6         P5
6             T3         P2

The groups of the testing data should be as following:

account-id   Feature1   Feature2   class(Group)
4             T5         P5         Gr3
5             T6         P5         Gr3
6             T3         P2         Gr2

The accounts 4 and 5 are in a new group (Gr3) which is not learned before in the training data.

My question is how could I group the new data under a new class that is not defined before in the learning phase ? and which algorithm can I use to solve this issue ?

I think you need to read about Online learning, it refers to learning when new data is being constantly added. In these cases you need an algorithm that can update itself as new data arrives (i.e. it doesn't need to recalculate itself from scratch). In other words, incrementally.

There are incremental versions for support vector machines (SVMs) and for neural networks. Also, bayesian networks can be made to work incrementally.


Is there any difference between feature extraction and feature learning? It appears to me that "feature extraction" and "feature learning" are equivalent  concepts, however there are 2 separate wikipedia articles dedicated to them that are notably different. In particular, only in the Feature Learning article Neural Networks/Deep Learning are mentioned. However, it seems like they would be equally appropriate for either because autoencoders extract features from the raw (typically image) data and those now extracted features feed into the next layers.

So, what's the real difference between these terms?

Yes I think so. Just by looking at Feature Learning and Feature extraction you can see it's a different problem.

Feature extraction is just transforming your raw data into a sequence of feature vectors (e.g. a dataframe) that you can work on.

In feature learning, you don't know what feature you can extract from your data. In fact, you will probably apply machine learning techniques just to  discover what are good features to extract from your dataset. Then you can extract them them apply machine learning to the extracted features. Deep learning techniques are one example of this.

In the word2vec toolkit, for instance, you extract vectors from documents which can't be easily interpreted by a human, you can't look at it and tell what features have been extracted at all. It's just a mass of vectors which, for some reason, give good empirical results.


Interpreting Loss in neural network: Neural network train loss gradually tappers and validation loss never reaches a minima Unable to improve the network validation loss. Is it overfitting/underfitting. How can I get a better validation loss?.The code is below

def create_model(lr=0.05):
    #tf.random.set_seed(1)

    tf.keras.backend.clear_session()
    gc.collect()

    # Dense input
    dense_input = Input(shape=(len(dense_cols), ), name='dense1')

    # Embedding input
    #Turns positive integers (indexes) into dense vectors of fixed size.
    wday_input = Input(shape=(1,), name='wday')
    month_input = Input(shape=(1,), name='month')

    event_type_1_input = Input(shape=(1,), name='event_type_1')
    item_id_input = Input(shape=(1,), name='item_id')
    dept_id_input = Input(shape=(1,), name='dept_id')
    store_id_input = Input(shape=(1,), name='store_id')
    cat_id_input = Input(shape=(1,), name='cat_id')
    state_id_input = Input(shape=(1,), name='state_id')

    wday_emb = Flatten()(Embedding(7, 3)(wday_input))
    month_emb = Flatten()(Embedding(12, 2)(month_input))

    event_type_1_emb = Flatten()(Embedding(5, 1)(event_type_1_input))

    item_id_emb = Flatten()(Embedding(3049, 3)(item_id_input))
    dept_id_emb = Flatten()(Embedding(7, 1)(dept_id_input))
    store_id_emb = Flatten()(Embedding(10, 1)(store_id_input))
    cat_id_emb = Flatten()(Embedding(3, 1)(cat_id_input))
    state_id_emb = Flatten()(Embedding(3, 1)(state_id_input))

    # Combine dense and embedding parts and add dense layers. Exit on linear scale.
    x1 = concatenate([dense_input,
                     event_type_1_emb,
                     wday_emb ,
                     month_emb,
                     item_id_emb, dept_id_emb, store_id_emb,
                     cat_id_emb, state_id_emb])

    x = BatchNormalization()(x1)

    x = Dense(7142, activation=None,kernel_initializer='lecun_normal',kernel_regularizer= regularizers.l1_l2(0.001))(x)
    x = BatchNormalization()(x)
    x = Activation("selu")(x)
    x = AlphaDropout(0.30)(x)

    x = Dense(714, activation=None,kernel_initializer='lecun_normal',kernel_regularizer = regularizers.l2(0.001))(x)
    x = BatchNormalization()(x)
    x = Activation("selu")(x)
    x = AlphaDropout(0.3)(x)
    x = Dense(34, activation = None,kernel_initializer='lecun_normal',kernel_regularizer = regularizers.l2(0.001))(x)
    x = BatchNormalization()(x)
    x = Activation("selu")(x)
    x = Add()([x,x1])
    outputs = Dense(1, activation="softplus", name='output',kernel_regularizer = regularizers.l2(0.001))(x)

    inputs = {"dense1": dense_input, "wday": wday_input, "month": month_input,# "year": year_input,

             "event_type_1": event_type_1_input,

              "item_id": item_id_input, "dept_id": dept_id_input, "store_id": store_id_input,
              "cat_id": cat_id_input, "state_id": state_id_input}

    # Connect input and output
    model = Model(inputs, outputs)

    model.compile(loss=keras.losses.mean_squared_error,
                  metrics=["mse","mape","mae"],
                  #optimizer=keras.optimizers.SGD(learning_rate=lr_schedule))
                  optimizer=keras.optimizers.RMSprop(learning_rate=lr))
    return model
```

this is a case of overfitting: your model performs very good just after few epochs but only in the training set, while on the validation set it improves too slowly.

There are several tricks that you can use:

1) a simpler network: you can lower the dimensions of the embeddig layers, as example. I also see some of the embeddings you are using can be avoided: the case of the month_emb, which I suppose stands for "month", and with have a dimensionality of 12. Is much better to use months as a simple number, or a categorical feature.

2) got more data: this is the hardest and often impossible way to proceed, but often is the best one.

3) reduce the number of inputs, by transforming/joining some of them.


Can someone spot anything wrong with my LSTM forex model? The model below reads in data from a csv file (date, open, high, low, close, volume), arranges the data and then builds a LSTM model trying to predict next day's close based on a number of previous days close values.

However, validation accuracy is about 53.8% no matter if i...
- change hyperparameters
- make it a deep model
- uses many more features than just close

To test if I made a simple mistake I generated another data source that was a signal I created from adding sin, cosine and a little noise so that i KNEW a model should be able to be trained and it was. The model below got about 94% validation without any tuning.

With that in mind. How come when I try to use it on actual data (eurusd 1-min data) it doesn't seem to work..?

Anyone that sees an error or can point me in the right direction?

import pandas as pd
import numpy as np

fpath = 'Data/'
fname = 'EURUSD_M1_1'
df = pd.read_csv(fpath + fname + '_clean.csv')
# file contains date, open, high, low, close, volume

# "y" is whether the next period's Close value is higher or lower than current Close value
outlook = 1
df['y'] = df['Close']<df['Close'].shift(-outlook)

# Drop all NAN's
df.dropna(how="any",inplace=True)

# Get X and y. To keep it simple, just use Close
X_df = df['Close']

# "y" is whether the next period's Close value is higher or lower than current Close value
outlook = 1
y_df = df['Close']<df['Close'].shift(-outlook)

# Train/test split
def train_test_split(X_df,y_df,train_perc):
    idx = int(train_perc/100*X_df.shape[0])
    X_train_df = X_df.iloc[0:idx]
    X_test_df = X_df.iloc[idx:]
    y_train_df = y_df.iloc[0:idx]
    y_test_df = y_df.iloc[idx:]
    return X_train_df.as_matrix(), X_test_df.as_matrix(), y_train_df.as_matrix(), y_test_df.as_matrix()

X_train_df, X_test_df, y_train, y_test = train_test_split(X_df,y_df,90)

# Scaling
def scale(X):
    Xmax = max(X)
    Xmin = min(X)
    return (X-Xmin)/(Xmax - Xmin)

X_train_scaled = scale(X_train_df)
X_test_scaled = scale(X_test_df)

# Build the model
import tensorflow as tf
from keras.models import Sequential
from keras.layers import LSTM, Dense

# ### Constants
num_time_steps = 5   # Num of steps in batch (also used for prediction steps into the future)
num_features = 1   # Number of features
num_neurons = 97
num_outputs = 1 # Just one output (True/False), predicted time series
learning_rate = 0.0001 # learning rate, 0.0001 default, but you can play with this
nb_epochs = 10 # how many iterations to go through (training steps), you can play with this
batch_size = 32

# Reshaping
X_train_scaled = np.reshape(X_train_scaled,[-1,1])
nb_samples_train = X_train_scaled.shape[0]  - num_time_steps

X_train_scaled_reshaped = np.zeros((nb_samples_train, num_time_steps, num_features))
y_train_reshaped = np.zeros((nb_samples_train))

for i in range(nb_samples_train):
    y_position = i + num_time_steps
    X_train_scaled_reshaped[i] = X_train_scaled[i:y_position]
    y_train_reshaped[i] = y_train[y_position]

model = Sequential()

stacked = False
if stacked == True:
    model.add(LSTM(num_neurons, return_sequences=True, input_shape=(num_time_steps,num_features), activation='relu', dropout=0.5))
    model.add(LSTM(num_neurons, activation='relu', dropout=0.5))
else:
    model.add(LSTM(num_neurons, input_shape=(num_time_steps,num_features), activation='relu', dropout=0.5))

model.add(Dense(units=num_outputs, activation='sigmoid'))

model.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics=['accuracy'])

history = model.fit(X_train_scaled_reshaped,
                    y_train_reshaped,
                    batch_size = batch_size,
                    epochs = nb_epochs,
                    validation_split=0.3)

What you are up against is this fundamental property of most tradable, liquid financial price series, and that is, they are Brownian Motion. In discrete time, it's also known as random walk.

The most important property of Brownian Motion is that it's memoryless, whose mathematical expression is

$E[p_{t+1} | p_{-{\infty} : t}] = p_t$

Recurrent neural net, particularly the LSTM flavor, is very powerful in capturing and modeling a long memory process. In fact, it was invented to deal with state that depends on itself many time steps ago (that famous LSTM paper was dated 20 yrs ago[1]).

What you've demonstrated using RNN (assuming your code and training are free of bug) is precisely this property. Put it another way, there is no way to beat a fair coin in predicting tomorrow's FX price.

Another perspective on your attempt. If such naive model could accurately predict FX time series, it would have been exploited by the whales in the hedge fund industry long ago and the opportunity would cease to exist.

[1] Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735-1780.


What is advantage of using the theory of point processes instead of probability distribution functions? I am interested in applications of point processes in machine learning, so I have been studying the theory of point process. However, I cannot see advantage of the theory of point processes compared with the classical probability theory.

For examples, I know the points of a Poisson point process scatters uniformly on a given domain. But, I'm confused what is the difference between Poisson point processes and uniform distribution functions (especially on a bounded domain).

I have similar questions for not only Poisson point processes but other point processes. Take sample points according to a point process. Can we obtain a similar (random) point set via using a probability distribution function?

What is advantage of using the theory of point processes instead of using probability distribution functions in statistics and machine learning?

Thank you so much in advance.

Machine learning is about finding a model for your data and sometime the Point process is a better model than uniform or normal probability distribution.


How to make a numpy ndarray from several numpy arrays I am not to figure out how to form a ndarray from existing numpy arrays.
Suppose I have three arrays - index, distA, and distB.
Now the I want to form a ndarray of these three arrays

What I have tried -

>>> indices = np.array([[5,7,4], [6,4,8]])
>>> distances = np.array([[0.2, 0.3, 0.4], [0.3, 0.4, 0.5]])
>>> np.column_stack((indices, distances))
array([[5. , 7. , 4. , 0.2, 0.3, 0.4],
   [6. , 4. , 8. , 0.3, 0.4, 0.5]])

But I want to form a table-like structure so that I can retrieve values in distA and distB.

5, 0.2, 0
7, 0.3, 0
4, 0.4, 0.4
6, 0, 0.3
8, 0, 0.5

Is it possible to do this with numpy?

If your purpose is to deal with some data analysis from your arrays and have a nice view of it, try this using pandas:

import pandas as pd

df = pd.DataFrame({'distA' : np.array([0.2, 0.3, 0.4]),
                   'distB' : np.array([0.3, 0.4, 0.5])})

df

In the case you want to make computation with vectors, matrixes, tensors, etc .. NumPy is probably a good option. I didn't understand exactly what you've asked, though. Can you explain me what you're asking?


Best programming language I have a masters in pure statistics. I have no knowledge of any programming language. Which is the best programming language that I should learn about and will help enhance my career in data analysis.

I only have a theoretical knowledge of concepts. Kindly suggest a skill that I should develop in order to become a data analyst.

If you want to become a data analyst then R is the best language for you. It is purely made for statistical analysis. It is a free software, so you can install it in your system without any hassle. You can find about it here-

https://www.r-project.org/about.html

It has nearly 10,000 packages which you can use as per your requirement. You can find many free tutorials for it as well.


What are the advantages/disadvantages of neural net packages in R? I've seen that there are four neural net packages in R:

neural
neuralnet
nnet
RSNNS
H2O

What are the advantages/disadvantages of those compared with each other?

As I found out NeuralNetTools only provides additional tools.

take a look at these blogs- R for Deep Learning - 1 and R for Deep Learning - 2.

Hope it helps!


Hyperparameter tunning for Random Forest- choose the best max depth I'm trying to choose the best parameters for random forest model.
For that goal I hae run my model in loop with only one parameter and each time I have changed the number for the parameter max depth. I have created two charts: one for the model  score and one for the MAE.
this is how the charts look like:

My question is what is the right number to take for that parameter?
I fell likeI want to choose really low number when the training set and the test set are close to each other and high (the second point, when the accuracy is above 90) :

But I see that it gets stabalize much after and i'm afrid that it might lead to overfitting.
Is it true? should I choose bigger number from the stable zone or is ok to take low number as long as the training and test data are having the same accuracy and same error?

In general, the max depth parameter should be kept at a low value in order to avoid overfitting: if the tree is deep it means that the model creates more rules at a more detailed level using fewer instances. Very often some of these rules are due to chance, i.e. they don't correspond to a real pattern in the data.
Overfitting is visible in your graphs from the quite high difference between training and test set performance. One can observe that the performance on the test set increases with the first few values of depth (I'm guessing until around 5 or so), and it starts decreasing after that. So the optimal point (performance and no overfitting) is the point that you mention, it's after this point that the model is overfit.
The stable part of the graph is probably due to another parameter (e.g. min. number of instance per leaf) which stops the model from overfitting even more. If the model was free to overfit as much as it wants, it would probably reach max performance on the training set and very low performance on the test set.
Note: I'm confused why you're using both accuracy and MAE, normally the former is for classification and the latter for regression.


Why categorical cross entropy loss is not correlated with NLP scores? I'm training a deep network for image captioning which is consist of one CNN and three GRUs. During training epoch by epoch model loss (categorical cross entropy) decreases but when I'm measuring bleu,METEOR,ROUGE,CIDEr and SPICE scores,I get best ones in the first epoch that has worst loss. I don't get why this is happening? And if categorical cross entropy is not a suitable loss function for autoencoder then what should I use instead?

There may be no mathematical linkage between categorical cross entropy and BLEU. BLEU if i remember correctly is probably a distance based measure of computer output to human judgement - there's no loss variable anywhere. This is unlike the perplexity score which IS derivable from the categorical cross entropy.
Loss function really depends on what you're comparing against your labels. Categorical cross entropy gives a way to compare the distributions of the prediction and truth labels and calculate an error based on the 'distance' between the distributions. Categorical cross entropy really works well to model the distributions of discrete vector labels, common in NLP.


How to turn a series that contains pandas.datetime.timedelta to floats e.g. hours or days? I want to plot some data that contains timedeltas. However, the numpy plot() and errorbar() return:

TypeError: float() argument must be a string or a number, not 'Timedelta'

The question is how can I efficiently convert the timedeltas to a float or integer number. E.g. when I use minutes as base:

0 days 01:30:00 -> 90

or hours

0 days 01:30:00 -> 1.5

Similar to Series.dt.days which would return 0 in this case.

One way to do this would be to use Series.dt.seconds and ´Series.dt.days´ and  multiply with a factor for the desired unit:

(Series.dt.seconds/3600) + (Series.dt.days*24)  # for values with [hours]


Does Amazon RedShift replace Hadoop for ~1XTB data? There is plenty of hype surrounding Hadoop and its eco-system.  However, in practice, where many data sets are in the terabyte range, is it not more reasonable to use Amazon RedShift for querying large data sets, rather than spending time and effort building a Hadoop cluster?

Also, how does Amazon Redshift compare with Hadoop with respect to setup complexity, cost, and performance?

tl;dr: They markedly differ in many aspects and I can't think Redshift will replace Hadoop.

-Function
You can't run anything other than SQL on Redshift. Perhaps most importantly, you can't run any type of custom functions on Redshift. In Hadoop you can, using many languages (Java, Python, Ruby.. you name it). For example, NLP in Hadoop is easy, while it's more or less impossible in Redshift. I.e. there are lots of things you can do in Hadoop but not on Redshift. This is probably the most important difference.

-Performance Profile
Query execution on Redshift is in most cases significantly more efficient than on Hadoop. However, this efficiency comes from the indexing that is done when the data is loaded into Redshift (I'm using the term indexing very loose here). Therefore, it's great if you load your data once and execute multiple queries, but if you want to execute only one query for example, you might actually lose out in performance overall.

-Cost Profile
Which solution wins out in cost depends on the situation (like performance), but you probably need quite a lot of queries in order to make it cheaper than Hadoop (more specifically Amazon's Elastic Map Reduce). For example, if you are doing OLAP, it's very likely that Redshift comes out cheaper. If you do daily batch ETLs, Hadoop is more likely to come out cheaper.

Having said that, we've replaced part of our ETL that was done in Hive to Redshift, and it was a pretty great experience; mostly for the ease of development. Redshift's Query Engine is based on PostgreSQL and is very mature, compared to Hive's. Its ACID characteristics make it easier to reason about it, and the quicker response time allows more testing to be done. It's a great tool to have, but it won't replace Hadoop.

EDIT:  As for setup complexity, I'd even say it's easier with Hadoop if you use AWS's EMR. Their tools are so mature that it's ridiculously easy to have your Hadoop job running. Tools and mechanisms surrounding Redshift's operation aren't that mature yet. For example, Redshift can't handle trickle loading and thus you have to come up with something that turns that into a batched load, which can add some complexity to your ETL.


feature extraction from single word for classification into nouns and names I would like to write a NN that can classify different kinds of words(e.g. nouns,verbs,names) and am struggling to find information on how to do feature extraction on single words.For example, i would like the NN to learn that "street" is a noun, while "How would i go about doing that? I'm really nwe to this and searching for it always seems to yield only examples of text feature extraction, which is not what I'm looking for.

Thank you in advance kind strangers!

I suppose you could try to learn a mapping from characters (or character-level n-grams) to part-of-speech.  This would be analogous to document classification.  Instead of a document, you have a single word.  And instead of a sequence of words, you have a sequence of letters.

With this formulation, most of the tricks which extract features from text could also be applied to extract features from characters (although your mileage may vary).  At the simple end of the scale, you could try "old-school" techniques like bag-of-words and TF-IDF (except you would have a bag-of-characters, and it would be Character-Frequency Inverse-Word-Frequency).  On the complex end of the scale, you could try to learn embeddings of characters or character n-grams.

However, before you get started, there are a few things I think you should keep in mind:

Do you think there is enough information in sub-word features to classify parts of speech?  I think it's pretty unlikely (at least for English).  Your model might learn some of the easy cases (e.g. an "-ly" suffix often indicates that the word is an adverb), but I don't think it would perform well in general.
This is actually a multilabel classification problem, because there are many words which can serve as more than one part-of-speech.  Also, names (proper nouns) are a subclass of nouns.
What happens if you feed your model a non-existent word?  Should your model try to classify it as a part-of-speech, or do you want it to recognize non-existent words?
If you're new to machine learning, consider putting this aside for a little while.  Jumping headfirst into a difficult classification problem with deep learning is a hard, confusing way to learn.

Finally, are you doing this for experimental/fun reasons, or is this intended to be part of a production application?  If it's the former, then go for it!  Worst case: it doesn't work but you learn something.

But if this project is for anything more serious, then you should not train a ML model to solve this problem.  Your model will never outperform a dictionary :)


How does dropout work during testing in neural network? The below paragraph is picked from the textbook Hands-On Machine Learning with sci-kit learn & Tensorflow. I couldn't understand what the author is trying to convey. It would be really grateful if someone can provide an explanation for below paragraph with an example. I have referred these links Dropout in Deep Neural Networks, https://www.coursera.org/lecture/deep-neural-network/dropout-regularization-eM33A, that somehow aligns with the same objective. However, i am seeking good explanations. Please help me.

  Suppose p = 50%, in which case during testing a neuron will be connected to twice as many input neurons as it was (on average) during training. To compensate for this fact, we need to multiply each neuron's input connection weights by 0.5 after training. if we don't, each neuron will get a total input signal roughly twice as large as what the network was trained on, and it unlikely to perform well. More generally, we need to multiply each input connection weight by the keep probability(1-p) after training.
  Alternatively, we can divide each neuron's output by keep probability during training (these alternatives are not perfectly equivalent, but they work equally well)

During training, p neuron activations (usually, p=0.5, so 50%) are dropped. Doing this at the testing stage is not our goal (the goal is to achieve a better generalization). From the other hand, keeping all activations will lead to an input that is unexpected to the network, more precisely, too high (50% higher) input activations for the following layer.

Consider the neurons at the output layer. During training, each neuron usually get activations only from two neurons from the hidden layer (while being connected to four), due to dropout. Now, imagine we finished the training and remove dropout. Now activations of the output neurons will be computed based on four values from the hidden layer. This is likely to put the output neurons in unusual regime, so they will produce too large absolute values, being overexcited.

To avoid this, the trick is to multiply the input connections' weights of the last layer by 1-p (so, by 0.5). Alternatively, one can multiply the outputs of the hidden layer by 1-p, which is basically the same.


Linear Regression: Why use global basis functions instead of local basis functions I'm looking through an online course about machine learning and the first big topic is finding a model that approximates our data with linear regression.
The model itself is linear function and we look for the coefficients of that linear function.
We introduce non-linearities that the data might have into our linear model by adding feature crosses.

For example, lets say we have two-dimensional features ($x_1, x_2$) and label them as $1$ if they are in the 2nd or 4th quadrant, 0 otherwise. A linear function $y(x_1, x_2) = b +m_1x_1+m_2x_2$ cannot approximate this. If we add $x_3 = x_1x_2$, we can find a model that represents that.

However, this is obviously a very artificial problem and not at representative to solve against any kind of data. In the end, we are trying to find a function $y$ that fits some data. This reminds me of solving differential equations numerically. And there we build the function $y$ as a sum of local basis functions $y(x) = \sum_i m_i\phi_i(x)$.

Why would we use global basis functions instead of local basis functions to introduce non-linearity into our linear machine learning model? Why don't we discretize our feature space and optimize on a linear function space?

I see a couple of parts to your question

Part 1

  Why would we use global basis functions instead of local basis
  functions to introduce non-linearity into our linear machine learning
  model?

Local (non-global) basis functions are used sometimes. For example, radial basis functions (RBF) are sometimes used as a basis in linear regression (see these lecture notes).

However, often it matters that a model is interpretable and more complicated models are harder to interpret / draw certain conclusions from.  For example, imagine you're doing a controlled experiment, you may not care about "just" predicting some variable from (the variable you're manipulating + the variables you're controlling for).  But rather, how much of an effect your manipulation has on the output.

Part 2

  Why don't we discretize our feature space and optimize on a linear function space?

RBFs kind-of discretize feature space.  People don't usually literally discretized feature space because discrete optimization is much much more complicated than continuous optimization.


Best ML approach for huge number of classes I have an problem where the dataset consists of:

400k observations
40k classes (mutually exclusive)

The problem is about predicting what is the supplier of an invoice (from which supplier/shop a purchase was made) based on the text of the invoice.
As you can understand there are hundreds of thousands if not millions of suppliers in a country hence the so big number of (mutually exclusive) classes.
In terms of features, I use a TF-IDF algorithm so I end up with 1.5M features. I apply feature selection and they are eventually relatively reduced in number. In any case, the features approach is something that I can and I am open to change so the number of features should not be considered as fixed.
However, this huge number of classes (along with the number of observations, features) results in me having to use huge amount of hardware to make things run and in many cases the VMs just die because of lack of memory.
All these happen when using ML algorithms such as random forest or SGD.
I am thinking that may be Tensorflow and GPUs can be a solution (?) although again this is a kind of hardware solution so I am wondering if I should reframe somehow the problem to make its solution more scalable.
How would you approach this problem IN AN ML WAY to make things scalable given also that the number of classes will be significantly growing in the future (finally there will be around 1M classes)?

Given data will keep increasing in the future along with the classes. You need to look at fast methods of doing things. Look at approximate KNN to do classification in high-volume data. Approximate KNN will solve your problem. Here's the library for it. Your accuracy will suffer a bit but this method would scale.


AUC ROC Curve multi class Classification Here is the part of the code for ROC AUC Curve calculation for multiple classes.
n_classes= 5

y_test = [0,1,1,2,3,4]   #actual value
pred1 = [0,1,1,1,3,4]    #predicted value

fpr = dict()
tpr = dict()
roc_auc = dict()

for i in range(n_classes):
    fpr[i], tpr[i], _ = roc_curve(np.array(pd.get_dummies(y_test))[:, i], np.array(pd.get_dummies(pred1))[:, i])
    roc_auc[i] = auc(fpr[i], tpr[i])

Error :

I understand that there are only 4 classes are predicted 0,1,3,4 in pred1 and 5 classes 0,1,2,3,4 in the y_test. So this error arises.
How to solve this error.

df = pd.get_dummies(pred1)
df.insert(loc=2,column='2',value=0)
#print(df)

add this before the for  loop and instead of using pd.get_dummies(y_test) use only df


Efficient dynamic clustering I have a set of datapoints from the unit interval (i.e. 1-dimensional dataset with numerical values). I receive some additional datapoints online, and moreover the value of some datapoints might change dynamically. I'm looking for an ideal clustering algorithm which can handle these issues efficiently.
I know sequential k-means clustering copes with the addition of new instances, and I suppose with minor modification it can work with dynamic instance values (i.e. first taking the modified instance from the respective cluster, then updating the mean of the cluster and finally giving the modified instance as an input to the algorithm just as the addition of an unseen instance).
My concern with using the k-means algorithm is the requirement of supplying the number of clusters as an input. I know that they beat other clustering algorithms (GAs, MSTs, Hierarchical Methods etc.) in time&space complexity. Honestly I'm not sure, but maybe I can get away with using one of the aforementioned algorithms. Even that my datasets are relatively large, the existence of a single dimension makes me wonder.
More specifically a typical test case of mine would contain about 10K-200K 1-dimensional datapoints. I would like to complete the clustering preferably under a second. The dynamic changes in the value points are assumed to be smooth, i.e. relatively small. Thus being able to use existing solutions (i.e. being able to continue clustering on the existing one when a value is changed or new one is added) is highly preferred.
So all in all:

Can you think of an algorithm which will provide a sweet spot between computational efficiency and the accuracy of clusters wrt. the problem defined above?
Are there some nice heuristics for the k-means algorithm to automatically compute the value of K beforehand?

I think hierarchical clustering would be more time efficient in your case (with a single dimension).
Depending on your task, you may implement something like this:

Having N datapoints di with their 1-dimension value xi:

Sort datapoints based on their xi value.
Calculate distances between adjacent datapoints (N-1 distances). Each distance must be assigned a pair of original datapoints (di, dj).
Sort distances in descending order to generate list of datapoint pairs (di, dj), starting from the closest one.
Iteratively unite datapoints (di, dj) into clusters, starting from beginning of the list (the closest pair). (Depending on current state of di and dj, uniting them means: (a) creating new cluster for two unclustered datapoints, (b) adding a datapoint to existing cluster and (c) uniting two clusters.)
Stop uniting, if the distance is over some threshold.
Create singleton clusters for datapoints which did not get into clusters.

This algorithm implements single linkage clustering. It can be tuned easily to implement average linkage. Complete linkage will be less efficient, but maybe easier ones will give good results depending on your data and task.

I believe for 200K datapoints it must take under second, if you use proper data structures for above operations.


What is the input size of Alex net In the paper ImageNet Classification with Deep Convolutional Neural Networks, the size of input image is 224x224. The following figure shows the input size.

From caffe, deploy.prototxt file from the directory of the bvlc_alexnet says that the input size is 227x227.

Why is the input size different?

I guess it has been a mistake. Take a look at here.

  The other author's were Ilya Sutskever and Geoffrey Hinton. So, AlexNet input starts with 227 by 227 by 3 images. And if you read the paper, the paper refers to 224 by 224 by 3 images. But if you look at the numbers, I think that the numbers make sense only of actually 227 by 227.


How does decision tree for regression makes a prediction? For classification, it is obvious how a decision tree is used to make a prediction.You just have to find the final leaf. However for regression problems, how can you find the prediction considering the continous aspect of the variable to predict?

Depends on the implementation but commonly used is a certain cursive partitioning method called CART.  The algorithm works as following:  It searches for every distinct values for your predictors and chooses to the split based on what minimize the SSE for two groups of dependent variables.  The difference is within the SSE is usually the difference between the actual value and the average of the sample or the difference between the actual value and the output of a linear regression.  For each group, the method will recursively split the predictor values within the groups.

In practice, the method stops when a certain sample size threshold is met.  The node with the lowest SSE becomes the root node.

Reference: You can read more about tree based regressions here: Tree-based regressions


Does partial transfer learning require a lot of computer power? I want to be sure my understanding of the problem is correct.
I want to do image classification and current state of the art in my field is achieved by transfer learning with VGG16.

Since image on this field are totaly different than image used by VGG16 trainig image, we should unfreeze "a lot" of layer.

Original VGG16 model has more than 130M parameter. it's has something like 13 hidden layer and 3 fully connected layer at the end.
If I unfreeze only the fully connected layer i'll still have more than 80M parameter to train!! and i'll probably need to unfreeze some hidden layer to get better performance since my data are really different

So it's look like transfer learning with VGG16 require a lot of computation power

am I right?
Is it possible to use transfer learning with image really different than training data without a lot of computational power?

The dataset on which VGG16 is trained consists of images from different domains, and its purpose is to classify images by extracting images' feature maps and attributes. Unfreezing few botton layers will help in training them for our specific image usage, however, these will be few parameters to learn than the whole connected network as it will have already learned basic features.

When you choose to train only a few bottom layers of VGG16, it will definitely require less computation.


How to get probability of classification I have the binary classification, I tried several models KNN, SVM, decision tree, and random forest. I have 50 000 samples, X_train has 50 000 rows and 2300 columns. Everything works well, but I want to build some semi-supervised model because I have some unlabeled samples. In this case, I need to get the probability of classification that I tried, but it doesn't work.
At first, I tried it for KNN
from sklearn.neighbors import KNeighborsClassifier
classifier = KNeighborsClassifier(n_neighbors = 1, metric = 'minkowski', p = 2)
classifier.fit(X_train, y_train)
y_pred = classifier.predict(X_test)
print(classifier.predict_proba(X_test[0]))

I get [[1. 0.]]. I don't understand why it is 1? (as first I thought it is 100%, but I get it for all test samples)
Then I tried it for the decision tree
classifier = DecisionTreeClassifier(random_state=0)
classifier.fit(X_train, y_train)
y_pred = classifier.predict(X_test)
print(classifier.predict_proba(X_test[0]))

I get [[1. 0.]] too. Why it is an integer?

It is indeed a probability of 1 because you didn't change the default parameters.

The probability for KNN is the average of all the neighbors. If there is only one neighbor n_neighbor=1 it can only be 1 or 0.

The DecisionTreeClassifier expands until all the training data is classified perfectly if you don't control the depth. Again, this likely led to overfitting and to extreme probability predictions as a result. You should try different values for max_depth and see what works best. You can do say by performing cross validation. (If you are unfamiliar with these I recommend reading up on it first.)


R error using package tm (text-mining) I am attempting to use the tm package to convert a vector of text strings to a corpus element.

My code looks something like this

Corpus(d1$Yes)

where d1$Yes is a factor with 124 levels, each containing a text string.

For example, d1$Yes[246] = "So we can get the boat out!"

I'm receiving the following error:

"Error: inherits(x, "Source") is not TRUE"

I'm not sure how to remedy this.

You have to tell Corpus what kind of source you are using.  Try:

Corpus(VectorSource(d1$Yes))


Does convergence equal learning in Deep Q-learning? In my current research project I'm using the Deep Q-learning algorithm. The setup is as follows: I'm training the model (using Deep Q-learning) on a static dataset made up of experiences extracted from N levels of a given game. Then, I want to use the trained model to solve M new levels of the same game, i.e., I want to test the generalization ability of the agent on new levels of the same game.
Currently, I have managed to find a complex (deep) CNN architecture which is able to converge. This means that after training it for a large number of iterations (by the way, I'm using prioritized experience replay), the training error (square difference between Q-values and Q-targets) is very low.
Since I want the agent to be able to generalize to new, unseen levels, I thought about finding the simplest possible CNN architecture which is able to converge on the training levels, since that model would generalize better (be less prone to overfitting on the training levels).
In supervised learning, my reasoning would be correct. However, I don't know if it holds true in Reinforcement Learning. Given a model which is able to converge (minimize error between Q-targets and Q-values), is that model always learning to solve the levels optimally, i.e., does the model find the optimal policy which maximizes reward? In other words, is it possible for a Deep Q-learning agent to find a non-optimal policy which converges to a very small error between Q-targets and Q-values?
From what I have read, I think that as long as Deep Q-learning converges, the policy found is always the optimal one. Please, correct me if I'm wrong.

Policies found by Deep Q-Learning, even after convergence, are not guaranteed to be optimal. The reason is that the neural networks that approximate the Q function in DQN inherently come with a statistical error (bias and variance), a pointer can be found here.
Furthermore, convergence to the optimal policy for tabular Q-learning is only guaranteed when every action is sampled infinitely often in every state. This may make using any 'converged and optimal policy' in experiments very hard, even when disregarding additional complexities from function approximation in DQN.
Please be aware, additionally, that generalization in supervised learning requires that the training and test data are sampled from the same distribution. Similarly for RL, the training and test environments are assumed to be sampled from the same 'distribution'. Generalization to environments not reflected in the training set is referred to as transfer learning in the RL literature.
This is an important and very interesting line of research though, so please be not be discouraged by this comment.


Time series data and ML - separating training/test data I am using XGBoost to try to predict the direction of the stock market based on social media sentiment. Having read through some studies, I was planning to separate the training/test data by time period, e.g. use 2014-2016 data for training and 2016-2018 data for testing.
Does that make intuitive sense given the nature of the data I am using?
I am happy to provide any further details which would be helpful, thank you.

When you are working with time-series data, the most recent data captures the most relevant information possible, so it is more prudent to include them in training data. So a more prudent decision would be to opt for Roll-Forward Partitioning.
Roll-Forward Partitioning: We start with a short training period and we gradually increase it, at each iteration of training, we train it on the current training period and make it forecast the next interval of data. It will require more training time, but it mimics what we would do during deployment where we would want to keep training our model at regular intervals to keep it up to date.
You can find more about it here, here and here.


Learning rate in logistic regression with sklearn In sklearn, for logistic regression, you can define the penalty, the regularization rate and other variables. Is there a way to set the learning rate?

sklearn.linear_model.LogisticRegression doesn't use SGD, so there's no learning rate.

I think sklearn.linear_model.SGDClassifier is what you need, which is a linear classifier with SGD training.

References

http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html

http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html


MCMC vs Network Theory for Big Data and Data Science applications [Apologies if this post sounds naive, I'm fairly new to the world of data science/big data and very unsure where I'm heading career-wise]

I'm currently an undergraduate MMath [integrated master's] Mathematics student in the UK who has finished the third year of the course [out of four years].

As I have been considering the possibility of doing further research in Mathematics/Statistics/Operational Research/Data Science, I have decided to stay on and complete the Master's component of the course [as it is the only Master's course I can get funding for at this stage]. After the Master's I may continue on and do a PhD.

There are currently two projects that appeal to me that seem to have relevant applications. The first one is on improved MCMC [Markov Chain Monte Carlo] methods, in particular MCMC using Hamiltonian Dynamics. There is scope for some big data applications here.

The other project that I could take part in is one on the centrality/communities detection of networks within network science. This could possibly be useful with applications in Operational Research.

Does anyone have an idea as to which project will be more relevant to data science/analytics?

Both MCMC methods and network analysis play an important role in data science. I think you should go for the project you like more. However, in my experience, community detection in networks is a niche (applied math/graph theory) of network analysis, while MCMC methods involve lots of statistical and computational concepts. Personally, being a statistician, I would go for the MCMC methods.


Read pandas data and convert it to IMMA I'm working on a project in which my data set is in xarray and I need to write it to IMMA format. I converted it from xarray to pandas and am totally stumped on how to go from pandas to IMMA. Any advice? Would it be easier to go straight from xarray to IMMA? I am working in python by the way. Thanks!

Following module provides a Python API to IMMA files:

import IMMA

You could use write()

opfile=open('some_output_file.imma','w')
for ob in obs:
    IMMA.write(ob,opfile)

More info could be found here.


Is there a way to print SVM margin obtained for each kernel used in sklearn? It is suggested in SVM to experiment with different classifiers using the various kernels available.Tuning the parameters is advised too.I am experimenting with various gamma and regularization parameters. Is there a way to print 'margin' for each experiment?

You can check out this example on the sklearn site.


How to extract important phrases (which may contain company name) from resume? I have thousands of CV / resumes with me. We want to build a parser which can extract company names from resume.

So far we have tried

Maintained a list of common words present in companies (Eg. Org, Ltd, Limited, Technologies etc.) and use them to identify probable companies. But this list is limited and many times many companies don't get extracted.
Using HTML of CV we have tried to give more score to probable companies which have a certain feature (like Bold, Italics)

Since CV is not only text and we always have some structural information along with it. There should be better ways to extract information. Maybe training some model which could predict companies mentioned in the resume. We are open to any better approaches/suggestions we could incorporate into our system for better accuracy. The precision so far is really bad (less than 45%).

We have already done the segmentation of work experience in CV. So we are able to extract the segment containing work experience with very high precision.

We also have a comprehensive list of companies (millions). Although it contains duplicates and needs significant amount of cleaning. But yes we have a lot of data

Edit

Other approaches we are trying -
We try to predict the important phrases inside text using N-Grams and then mark them as probable companies. Then we do a search on companies corpus with us to find any match. How useful is this technique ? Any better approaches ?

Sounds like you want named entity recognition.  There are a variety of approaches to NER, and plenty of implementations, like the Stanford NER package.

After you find named entities, determining what the named entity refers to is called concept normalization.


Optimal points of $f(x,y)=x^2 + y^2 + \beta xy + x + 2y$ I am self-learning basic optimization theory and algorithms from "An Introduction to Optimization" by Chong and Zak. I would like someone to verify my solution to this problem, on finding the minimizer/maximizer of a function of two variables, or any tips/hint to proceed ahead.

For each value of the scalar $\beta$, find the set of all stationary points of the following two variables $x$ and $y$
$$f(x,y) = x^2 + y^2 + \beta xy + x + 2y$$
Which of those stationary points are local minima? Which are global minima and why? Does this function have a global maximum for some value of $\beta$.

Solution.
We have:
\begin{align*}
f(x,y) &= x^2 + y^2 + \beta xy + x +2y\\
f_x(x,y) &= 2x+\beta y + 1\\
f_y(x,y) &= \beta x + 2y + 2\\
f_{xx}(x,y) &= 2\\
f_{xy}(x,y) &= \beta \\
f_{yy}(x,y) &= 2
\end{align*}
By the first order necessary condition(FONC) for optimality, we know that if $\nabla f(\mathbf{x})=0$, then $\mathbf{x}$ is a critical point.
Thus,
\begin{align*}
f_x(x,y) &= 2x+\beta y + 1 = 0\\
f_y(x,y) &= \beta x + 2y + 2 = 0
\end{align*}
Solving for $x$ and $y$, we find that:
\begin{align*}
x = \frac{\begin{array}{|cc|}
-1 & \beta \\
-2 & 2
\end{array}}{\begin{array}{|cc|}
2 & \beta \\
\beta & 2
\end{array}}=\frac{-2+2\beta}{4-\beta^2}=\frac{2\beta-2}{4 -\beta^2}
\end{align*}
\begin{align*}
y = \frac{\begin{array}{|cc|}
2 & -1 \\
\beta & -2
\end{array}}{\begin{array}{|cc|}
2 & \beta \\
\beta & 2
\end{array}}=\frac{-4+\beta}{4-\beta^2}=\frac{\beta -4}{4 - \beta^2}
\end{align*}
The second order necessary and sufficient conditions for optimality are based on the sign of the quadratic form $Q(\mathbf{h})=\mathbf{h}^T \cdot Hf(\mathbf{a}) \cdot \mathbf{h}$.
The Hessian of $f$ is given by,
$$Hf(\mathbf{x})=\begin{array}{|c c|}
2 & \beta \\
\beta & 2
\end{array}$$
Thus, $d_1 = 2 > 0$ and $d_2 = 4 - \beta^2$. Thus, $f$ has a local minimizer if and only if $4 - \beta^2 > 0$. $g(\beta) = 4 - \beta^2$ is a downward facing parabola. So, the values of this expression positive, if and only if $-2 < \beta < 2$. The function $f$ has no global maximum.
Question. How do I find the actual global minima?

If the function is convex, then all local minimum are global minimum.
If $4-\beta^2 >0$, then the function is convex and hence the local minimum is indeed the global minimum.
If $\beta = \pm 2$, then we have $f(x,y)=(x\pm y)^2+x+2y$, $f(x, \mp x)=x\mp2x$ of which we can make it arbitrary large or small.
If $4-\beta^2 < 0$, then it is indefinite, the stationary point is a saddle point.
It can't be negative definite, it doesn't have global maximum.


Autoencoder gets ~0% accuracy / doesn't train at all So I wanted to get into the topic of 'Autoencoder', and just tested how well it would work on random vectors of size 200.

model = keras.models.Sequential()
model.add(keras.layers.Dense(50, activation='tanh'))
model.add(keras.layers.Dense(200))

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['acc'])
model.fit(vectors, vectors, epochs=50, batch_size=100)

Loss is somehow negative, and continously decreases. Accuracy is pretty much 0%, with values looking like x.xxxxxxxe-07.

The architecture is supposed to be input=200, hidden-layer=50, output=200.

It really doesn't learn anything. What did I overlook, or what is the actual right approach here?

A few things. Firstly I don't think what you are trying to do is very meaningful. An autoencoder tries to find patterns in your data and take advantage of them to be able to compress and reconstruct the data. You are trying to use it with random data, which means there will be no patterns for the autoencoder to learn. I suggest you try using MNIST or another dataset.

Another thing is that your last activation function, your loss function, your metric and possibly your data does not work together. It is hard to tell what is right since I don't know what random numbers you generated. Probably your numbers are not binary, and you would want:

model.add(keras.layers.Dense(200))

model.compile(optimizer='adam', loss='mse')

Or if your random numbers are binary then you would want:

model.add(keras.layers.Dense(200), activation='sigmoid')

model.compile(optimizer='adam', loss='binary_crossentropy')

Lastly, using accuracy as a metric does not make sense for an autoencoder. Accuracy is only useful when you are performing classification.


LSTM Feature selection process We need to implement a time series problem with the LSTM model.

But, while implementing the same, the main challenge I am facing is the feature selection issue. Because our data-set contains 2300 observations and 600 features. And we already know there are so many features which are not at all relevant. But, no one has the domain expertise to confirm which feature is relevant.

I have tried with…

All 600 features, [shape of X become, 2280 X 20 X 600 (if time step is 20)]. The accuracy is very poor 53-55%.
Considered a single column as a predictor (the feature which we want to predict) [shape of X become, 2280 X 20 X 1 (if the time step is 20)]. The accuracy is also poor.
Also, tried with PCA so that all 600 features are reduced to 20 features [approx 98% data was retained], but not seen any improvement.
Finally, I have gone through a very lengthy process.

a. Out of 600 features, each of them is tried out individually with the LSTM algorithm. The feature which was giving the best result is selected.

b. The selected feature is paired with the rest of 599 features individually. Then, I select the best pair that gives the best output.

c. The best pair is further tried with rest 598 features individually and this process continues...

After doing so, as of now, we get 6 features, the accuracy gets improved to 65%.

My question is, is it the only approach that I can do feature selection in time series problems?

In case of any non-time series problem there are so many ways to do the feature selection. What will be the best way to do feature selection in the time series issue?

Let's give another example...

Let's assume we developed the LSTM model with 10 predictors/features. The model is working very fine and we get 80% accuracy.

Now, somehow 100 more features get added which are not relevant for deriving target y.

So, the accuracy will be dropped as the model will focus on 120 predictors/features, where 100 are not relevant.

Then, how will we eliminate the 100 unwanted features?

Have you tried to do a correlation test between the output and the features used to model it?

I usually use Pearson's Correlation to determine the features with low correlation factors. Then I remove them from the training.

In here, I have DOC_DS used to model DOC_US, I have them both in a data frame df with my other features. From the trails I did on my models, the features with less correlation affects the model accuracy later on.

In Python Environment:

print(np.array_str(np.corrcoef(df.DOC_DS,df.DOC_US)))


Merging dataframes in Pandas is taking a surprisingly long time I'm trying to merge a list of time series dataframes (could be over 100) using Pandas. The largest file has a size of $\approx$ 50 MB. The number of rows and columns vary (for instance, one file could have 45,000 rows and 20 columns, another has 100 rows and 900 columns), but they all have common columns of "SubjectID" and "Date", which I'm using to merge the dataframes. I read the dataframes into the list from CSV files, where I know the datatypes for each column.

However, when I try to merge even 10 of the dataframes, it takes around 7 hours, and for all 100, my kernel crashes. Reading all the files in takes about a minute. These aren't files that I would consider "big data" or even large files, like in the posts here, here, or here, so I'm surprised it's taking so long to merge them all. For all 100 dataframes, I'd expect the final dataframe size to be about 1.5 GB, but even that I think Pandas could handle. My code to merge the dataframes is

merged_df = reduce(lambda l,r: l.merge(r, on=['SubjectID', 'Date'], how='outer', suffixes=['_COPYL', '_COPYR']), df_list)

because I want columns to be added on at if they don't already exist for a subjectID and date (I take care of duplicates later).

EDIT

Some sample dataframes are produced by the following code

import numpy as np
import pandas as pd
from functools import reduce

df1 = pd.DataFrame({'SubjectID': ['A', 'A', 'A', 'B', 'B', 'C', 'A'], 'Date': ['2010-03-14', '2010-03-15', '2010-03-16', '2010-03-14','2010-05-15', '2010-03-14', '2010-03-14'], 'Var1': [np.nan, 1, 4, 7, 90, np.nan, 9], 'Var2': [np.nan, 0, 1, 1, 0, np.nan, 1]})
df2 = pd.DataFrame({'SubjectID': ['A', 'A', 'B', 'B', 'C', 'A'], 'Date': ['2010-03-14', '2010-03-15', '2010-03-14', '2010-05-15', '2010-03-14', '2010-03-14'], 'Var2': [ np.nan, 0, 1, 1, np.nan, 1], 'Var3': [0, 0, 1, np.nan, 0, 1]})

df3 = pd.DataFrame({'SubjectID': ['A', 'A', 'A', 'B', 'B', 'C'], 'Date':['2010-03-14', '2010-03-15', '2010-03-16', '2010-03-14', '2010-05-15', '2010-03-14'], 'Var3': [np.nan, 1, 0, np.nan, 0, 1]})

df1['Date'] = pd.to_datetime(df1['Date'])
df2['Date'] = pd.to_datetime(df2['Date'])
df3['Date'] = pd.to_datetime(df3['Date'])

My code to merge the dataframes is

df_list = [df1, df2, df3]
merged_df = reduce(lambda l,r: l.merge(r, on=['SubjectID', 'Date'], how='outer', suffixes=['_COPYL', '_COPYR']), df_list)
merged_df = merged_df.groupby([x.split('_COPY')[0] for x in merged_df.columns], 1).apply(lambda x: x.mode(1)[0])
merged_df['Date'] = pd.to_datetime(merged_df['Date'])

My expected output is

            Date    SubjectID   Var1    Var2    Var3
0     2010-03-14           A    NaN      NaN    0.0
1     2010-03-14           A    NaN      1.0    1.0
2     2010-03-14           A    9.0      1.0    0.0
3     2010-03-14           A    9.0      1.0    1.0
4     2010-03-15           A    1.0      0.0    0.0
5     2010-03-16           A    4.0      1.0    0.0
6     2010-03-14           B    7.0      1.0    1.0
7     2010-05-15           B    90.0     0.0    0.0
8     2010-03-14           C    NaN      NaN    0.0

So far, I've tried reducing the memory of each column datatype by downcasting, different ways to merge in Pandas and CSV files, converting to a sparse dataframe (my data does have a fair amount of NaN), using Dask, and I'm at the point where I'm considering creating a relational database using sqllite in Python. My versions are python-64 3.7.0, pandas 0.23.4, Ipython 6.5.0, using a Mac OSX Darwin with 32 GB memory. I would think my computer should have no trouble doing this, but it does. Is there really no better way or am I doing something wrong that I'm missing?

There are a couple of points I can maybe give to help.

Firstly, Pandas is not great at merging multiple large dataframes in general because every time you merge a new dataframe to an old one, it makes a copy of both to make a third dataframe - this obviously starts taking a lot of time as your master dataframe grows in each step.

Secondly (untested), you could try to overcome any memory bottlenecks in the merging steps by trying something dumb, like merging two or three files, then writing it to disk. Repeat that until you have 30 larger CSV files, then do it again so you have, say 10 files, and so on. I am not sure at which point the python garbage collection would kick in with you single reduce function, so it could even be the case that you start using SWAP memory, which would make things really slow! This solution could at least prevent that from happening. You should perform each repeat in a new Python session.

One other idea, as you are reading the data from CSV files: you could use a terminal tool to simply paste all the CSV files into one file, then read that into pandas once. Then perform required clean up actions. The cleaning might be a pain (given your merge on requirements, but you would likely overcome the long waiting times for pandas to make the copies. If you files are all in one folder, you can run the following in terminal to put them all in one file

cat *.csv > merged.csv

There are other methods here, in case you need to leave out the header line etc.

Lastly I would recommend having a look at the data.table package, which is now functional in Python - it started in the R world. It has a much more robust memory management system (built on ideas from databases) and is able to store huge amounts of data in a single "DataTable", in cases where Pandas would crash. It is a great package to fill the gap between Pandas and Dask (both awesome packages!).

Here is a video, showing how Python data.table works and also shows some benchmarks.


On which data evaluate an object detection model ? (similar or real life data ?) I'm training an object detection model (SSD300) to detect and classify body poses in thermal images.

Even I have more than 2k different poses, the background does not change much (I have only 5 different points of view).

I trained my model on these images (70% for the training and 30% for validation).

Now, I want to evaluate the model on an unbiased dataset.

Should I keep images of my dataset for this purpose or should I use a real life dataset ?

(A good solution would be to have a real life training set, but I don't have)

I tried both, but as expected, I have an mAP=0.9 when evaluated on similar pictures and mAP=0.5 when evaluated on completely different images.

Bonus question: is mAP a relevant metric when I want to show result to a client ? (e.g a client doesn't understand if I tell him "my model has a mAP=0.7")

Precision-Recall ? (but I have to choose a pose classification threshold...)

The model would perform better when given similar data as its trained on.i think you should try mixing the two datasets and check the results.
To the client, you can show the three results and explain how it is a great product.
In place of mAp, show a chart of precision and recall to explain the accuracy.