using LSTM encoder decoder framework to generate sentences i am trying out multiple approaches to generate domain specific data (sentences).For e.g. if i am building a data set to train a chatbot in the travel domain, i would like to take pairs of sentences like "i would like to book a return trip to Philly commencing on the 24th" and "i am traveling to Phialdelphia for a conference for 3 days beginning the 24th" and use these as the enc and decoder inputs. So once the training is complete, i can feed it the first sentence and generate multiple samples, set a threshold (rather than argmaxing it) and have a fairly large corpus of similar sentences that i could train another classifier with. I personally think its a good analogy to machine translation tasks (where the sampled output is another language but in some sense semantically the same sentence)

While i am trying it out, can anyone tell me if they have tried this approach and seen decent results ? Also any recommendations/ pointers on better approaches? thanks a ton!

I don't know about any research on this topic, but I have thought about similar ideas before. Let's assume the decoder is some RNN that outputs probabilites conditioned on the encoder vector and on what it has generated so far, $P(Word|EncodedVector, DecodedSoFar)$. What you can do is a type of beam search (there is research on this), which is basically greedily take the most likely option so far, but keep all the paths in memory while they still have the potential for a likely enough sentence. The probability of a sequence is just the product of all the predictions that we have sampled. Let's say our model has the following properties (we ignore the encoder):

$P(A) = 0.3$,  $P(B) = 0.5$, $P(End) = 0.2$

Now, condtioned on one sample:

$P(A|A) = 0.6$, $P(B|A)=0.1$, $P(End|A)=0.3$

$P(A|B)=0.7$, $P(B|B)=0.2$, $P(End|B)=0.1$

Now let's say when we have two samples the probability of ending is always 1.

This means we get the following probabilities:

$P(End) = 0.2$

$P(A) = 0.3 * 0.3 = 0.09$

$P(B) = 0.5 * 0.1 = 0.05$

$P(AA) = 0.3 * 0.6 * 1. = 0.18$

$P(AB) = 0.3 * 0.1 * 1. = 0.03$

$P(BA) = 0.5 * 0.7 * 1. = 0.35$

$P(BB) = 0.5 * 0.2 * 1. = 0.1$

But we don't want to iterate over all possibilities, this grows way too quickly. What we can do is build a tree which we can bound by our threshold. Let's say we only want samples with at least probability $0.15$. Our root node is empty, with probability 1. Our terminal nodes are the ones that have $End$ as the last part of the sequence. We start growing our tree with the most likely scenario, which is $B$, which means we have $P(empty)=1$, $P(B)=0.5$. For our next option, we can either look at another start than B or by expanding upon B. $P(A)=0.3$ but $P(BA)=0.35$ so we first expand $B$. We now have $P(empty)=1$, $P(B)=0.5$ and $P(BA)=0.35$. The next step is the highest probability, staying at 0.35 by ending $P(AB-end)=0.35$. The next step is expanding our root by going to $P(A)=0.3$. The next step is expanding root with $P(End)=0.2$. Our next step would be $P(AA)=0.18$ but this is below our threshold. We know there are no sequences that are more likely because of the way we have built our tree, which means we have all sequences above the given threshold. You can use something similar to generate the top k most likely samples.

EDIT: Be aware that this does bias towards shorter sentences if your model allows for them, because every probability decreases the total probability. I have not really thought about how much of this is a problem, it also depends a lot on your generating RNN. You could think of some 'penalties' for short ones but you want the metric to be monotonically decreasing because else you cannot bound your tree properly.


Encode time-series of different lengths with keras I have time-series as my data (one time-series per training example). I would like to encode the data within these series in a fixed-length vector of features using a keras model.
The problem is that my different examples' time-series don't have the same lengths. I haven't found a way of doing that. The problem of the encoder-decoder thing is that if the input lengths vary, the output lengths do this also. But I would like to have an output of length 10 let's say that would encode the data of the time-series, regardless of its length.
My question could have a strong link with: https://deepai.org/publication/towards-a-universal-neural-network-encoder-for-time-series but I haven't found any practical implementation.
Any help is very appreciated.

Figure out the max length you want your time series to be and use linear interpolation to fill in missing values in the shorter series so that time series can be same length.
Example of this is:
def interpolate_to_length(time_series, final_length):
    curr_ts_length = time_series.shape[1]
    idx = np.array(range(curr_ts_length))
    idx_new = np.linspace(0, idx.max(), final_length)
    # linear interpolation
    f = interp1d(idx, time_series, kind='cubic')
    new_ts = f(idx_new)
    return new_ts

df = pd.DataFrame(data = {'col1': [1, 5, 10, 15, 20], 'col2': [1, 6, 11, 13, 15]})
print(df)
transposed_values = df.values.transpose()
print(transposed_values)
print(interpolate_to_length(transposed_values, 10))

Output should be:
   col1  col2
0     1     1
1     5     6
2    10    11
3    15    13
4    20    15
[[ 1  5 10 15 20]
 [ 1  6 11 13 15]]
[[ 1.          2.5743027   4.48331047  6.61728395  8.86648377 11.12391404
  13.34567901 15.55098308 17.76177412 20.        ]
 [ 1.          2.93415638  5.36213992  7.88888889 10.11934156 11.67489712
  12.55555556 13.1399177  13.82304527 15.        ]]


How can I parallelize GloVe reverse lookups in PyTorch? I feel like I'm missing something obvious here because I can't find any discussion of this.  I want to do a lot of reverse lookups (nearest neighbor distance searches) on the GloVe embeddings for a word generation network.  I'm currently just iterating through the vocabulary on the cpu. I've sped it up a bit using a process pool, as shown in the snippet below, but it's still very slow for large vocabs.

Is there a way to move this to to the GPU using cuda?  I've also read that there is a way to turn this sort of thing into one big matrix operation...  Any references would be appreciated.  Thanks!

glove = torchtext.vocab.GloVe(name='6B', dim=wordDim)

def closest(vec):
    dists = [(w, torch.dist(vec, glove.vectors[glove.stoi[w]] for w in glove.itos]
    return sorted(dists, key=lambda t: t[1])[0]

output = # word vectors…
# using a process pool to parallelize the lookup
pool=ProcessPoolExecutor(max_workers=8)
predictedWords = [w for w in list(pool.map(closestWord, output)]

It's usually not very efficient to approach these types of problems in pythonic ways, with list comprehensions and such. This whole process can be done with some matrix math, which will be substantially faster (and able to be computed on the GPU using PyTorch). Using torch.dist with default p=2 will compute the Euclidean distance between two tensors, which is defined as

$$
\text{euclidean distance} = d(\mathbf{v}, \mathbf{w}) = \sqrt{\sum_d \left(\mathbf{v}_d - \mathbf{w}_d\right)^2}
$$

You can do this efficiently in PyTorch for every word in your vocab by broadcasting your query word over the whole matrix of word vectors:

def closest(vec):
    dists = torch.sqrt(((glove.vectors - vec) ** 2).sum(dim=1))
    return dists.argmin() # or glove.itos[dists.argmin()] if you want a string output

However, usually people use the cosine similarity to find the closest word vectors rather than the Euclidean distance. This is the cosine of the angle between the vectors, and can be calculated as

$$
\text{Cosine similarity} = \frac{\mathbf{v}\cdot \mathbf{w}}{\|\mathbf{v}\| \|\mathbf{w}\|}
$$

which can similarly be implemented in PyTorch as the following:

glove_lengths = torch.sqrt((glove.vectors ** 2).sum(dim=1))

def closest_cosine(vec):
    numerator = (glove.vectors * vec).sum(dim=1)
    denominator = glove_lengths * torch.sqrt((vec ** 2).sum())
    similarities = numerator / denominator
    return glove.itos[similarities.argmax()]

Note that we moved the $\|\mathbf{v}\|$ part of the equation outside the function here, so we don't have to recompute the lengths of each glove vector every time we want to run it.


Unknown return value in CNTK 103A I came across this statement from CNTK 103A:
gzfname, h = urlretrieve(src, './delete.me')

I understand all the rest but that gzfname, h = ......
What exactly is the purpose of h, is it a filehead thing?
Code:
def load_or_download_mnist_files(filename, num_samples, local_data_dir):
    gzfname, h = urlretrieve(src, './delete.me')
    if (local_data_dir):
        local_path = os.path.join(local_data_dir, filename)
    else:
        local_path = os.path.join(os.getcwd(), filename)

    if os.path.exists(local_path):
        gzfname = local_path
    else:
        local_data_dir = os.path.dirname(local_path)
        if not os.path.exists(local_data_dir):
            os.makedirs(local_data_dir)
        filename = "http://yann.lecun.com/exdb/mnist/" + filename
        print ("Downloading from" + filename, end=" ")
        gzfname, h = urlretrieve(filename, local_path)
        print ("[Done]")

    return gzfname

gzfname, h is doing tuple unpacking from the return of urlretrieve.  From the docs, urlretrieve returns:

  Return a tuple (filename, headers) where filename is the local file name under which the object can be found, and headers is whatever the info() method of the object returned by urlopen() returned (for a remote object). Exceptions are the same as for urlopen().

And the info() method of the urlopen object returns:

  info() — return the meta-information of the page, such as headers, in the form of an email.message_from_string() instance (see Quick Reference to HTTP Headers)

So, h is meta information about the retrieved page.


How do I get a word frequency list? Can't export from Word Cloud I have a list of survey responses and want to analyze the frequency of words used.

Using Orange, I've loaded the file into a Corpus, then Preprocess Text, then Word Cloud.

The Word Cloud does not show the cloud image, and I can't figure out why. Everything appears to be set up just like the Orange documentation shows online. Word Cloud does show a nice list of words and their frequencies, but I can't copy/paste that into Excel or export that list.

How can I export that list of word frequencies into a table where one column is the word, and the other column is the count? Is there an easier way to do this?

Ad 1) Word Cloud has a silly bug that causes the visualization sometimes not displaying. Press Regenerate word cloud and the visualization will appear.

Ad 2) This is currently not possible in Orange, but it's surprising how we haven't thought about it yet. We will make necessary modifications and the export should be available in the next release.


problem with regular expression I have a dataframe in R containing  a variable for programming languages. This is extracted from a multiple choice questionnaire in a survey. As a result, any programming language may fall into the beginning, middle or end of the whole string.
I added a binary variable for any programming language. Here is a sample of my data:

language <- c(
"Ruby, Assembly, R, Go, Rust",
"Java, Javascript, Ruby, SQL",
"Java, Ruby",
"C, C++",
"PHP, Javascript, Ruby, Assembly, Swift, R, Matlab, Go, Haskell",
"R",
"Perl, Javascript, R",
"Javascript, Ruby, Bash",
"Python, PHP, Javascript",
"Java",
"Java, C"
 )

df <-as.data.frame(language,stringsAsFactors = FALSE)
df$python      <- 0
df$java        <- 0
df$c           <- 0
df$r           <- 0
df$d           <- 0

for(trow in (1:nrow(data)))
{if (length (grep(pattern="Python",x=df[trow,"language"]))  > 0)    {df[trow,"python"] <- 1}}

I have problem with one-word languages such as C, R, D and Java . I made the variables for C#, C++ , Javascript, Ruby and others using the for loop but  I have problems with these four. What regular expression can I use so that it covers Java but not Javascript (in the beginning, middle and end of the string) and covers R but not Ruby and so on? Any other solution for making the binary variables are also welcome.

You could do

reshape2::recast(
  data = setNames(strsplit(language, ", ", T), language),
  formula = L1~value,
  fun.aggregate = length
)
#                                                                L1 Assembly Go R Ruby Rust Java Javascript SQL C C++ Haskell Matlab PHP Swift Perl Bash Python
# 1                                                          C, C++        0  0 0    0    0    0          0   0 1   1       0      0   0     0    0    0      0
# 2                                                            Java        0  0 0    0    0    1          0   0 0   0       0      0   0     0    0    0      0
# 3                                                         Java, C        0  0 0    0    0    1          0   0 1   0       0      0   0     0    0    0      0
# 4                                     Java, Javascript, Ruby, SQL        0  0 0    1    0    1          1   1 0   0       0      0   0     0    0    0      0
# 5                                                      Java, Ruby        0  0 0    1    0    1          0   0 0   0       0      0   0     0    0    0      0
# 6                                          Javascript, Ruby, Bash        0  0 0    1    0    0          1   0 0   0       0      0   0     0    0    1      0
# 7                                             Perl, Javascript, R        0  0 1    0    0    0          1   0 0   0       0      0   0     0    1    0      0
# 8  PHP, Javascript, Ruby, Assembly, Swift, R, Matlab, Go, Haskell        1  1 1    1    0    0          1   0 0   0       1      1   1     1    0    0      0
# 9                                         Python, PHP, Javascript        0  0 0    0    0    0          1   0 0   0       0      0   1     0    0    0      1
# 10                                                              R        0  0 1    0    0    0          0   0 0   0       0      0   0     0    0    0      0
# 11                                    Ruby, Assembly, R, Go, Rust        1  1 1    1    1    0          0   0 0   0       0      0   0     0    0    0      0


GAN discriminator converging to one output I am running a GAN on the MNIST dataset. As the GAN continues to train, the quality of the generator seems to get worse, and it even seems as if it is starting to converge to just one value.

Epoch 7:

Epoch 21:

Why is this happening? I am running the code from this repo here:
https://github.com/bstriner/keras-adversarial/blob/master/examples/example_gan.py

What you are experiencing is called mode collapse, which can occur in GAN Training and is one of its "training instability problems". If you are using Vanilla GAN one effective way is to implement mini batch discrimination, which for my understanding does give the discriminator batches of real and fakes and it has to decide batchwise if its fake or real. So if it encounters the same or nearly same image often it is easy for him to say it's fake.
Other GAN Architecture like WGAN, and BEGAN state that they don't suffer from mode collapse which would be another alternative.

You can also try to expand the GAN to an AC-GAN, which produces classwise samples (0-9) but in AC-GAN it's also possible to suffer from classwise mode collapse.

Here is a pull request for keras implementing minibatch discrimination : https://github.com/keras-team/keras/pull/3677


Does MS PowerBI directly connect with Azure's ML workbench? Someone wants me to try to see if I can establish a workflow for them using PowerBI and Azure's ML, then show them how to use it. The idea is to create kind of a seemless experience for them, so they done have to worry about coding or dealing with ETL, etc.

Right now I can access Azure but I have to go through some red tape to access the cloud version of PowerBI.

Does anyone happen to know if these 2 product services can be directly linked, such that the person I'm trying to help wouldn't have to worry about importing or exporting files (assuming I pipe the data into Azure's ML for them)?

Yes, they can connect natively. You can manage data and then put it all in different services like it is showed next. As you can see you can use SQL database, blob storate and also PowerBI.

Here you can find a tutorial on how to do streaming analytics with Azure and PowerBI.


How to create a Document Categorization Classifier for different contexts of Documents I have a doubt solving a test. The idea here is to demonstrate the NLP and Machine Translation abilities.
The Dataset is a multilingual, multi-context set of documents. The dataset is divided on context categories (Wikipedia, conference_papers, Amazon Reviews, etc.,) and on languages.
The objective is to create a document cartegorization classifier (in Python) for the different contexts of the documents. The classifier has to be done at context level, regardless of the language the documents are written in.
An important fact is that The dataset original has been modified and a document Never is repeated in 2 languages.
I have 2 ideas on mind to solve that:

Train on all the data creating a multilingual classifier
Doing language detection first and use monolingual models later.

What could be a reasonable approach to doing text classification for multiple languages?

This depends on whether you have enough training data for all languages. If yes, doing language ID and language-specific models might be a good choice, especially if there is a BERT-like model available for each language.
An alternative would be, do the language ID and than machine-translate the input into English and only train an English classifier. You can use e..g, high-quality pre-trained Marian models recently published by the University of Helsinki.
Otherwise, I would use pre-trained multilingual representations (probably XLM-R that is much better than Multilingual BERT) to get representation and train a single classifier for all languages. The multilingual representations seem to have even some zero-shot abilities, i.e., the classifiers seem to generalize even for languages that are no in the training data.


How can I make a prediction in a regression model if a category has not been observed already? I'm researching a regression model to predict a target value that has four features, all of which are categorical.

The categories are not fixed, e.g. one is a customer identifier. How could my model handle making a prediction for a customer identifer it hasn't seen before, based on the remaining features it has been trained with?

I have considered having a model for each of the features that could predict which category label is most similar based on the other three remaining features (or multiple similar category labels could be used and take an average of the target values for those).

My only concern with with this method is that it's not that scalable, I'm going to want to extend the model with more and more categorical features.

Is there some technique that could create an 'unknown' label for each of the features so the model can handle this case or would the prediction likely be completely inaccurate?

Technically, you can't. That is one of the limitations of regression models; they are really only effective for values/ranges that they have seen before. Your use of categorical values makes it even more complex. But even with continuous variables, it is not recommended to use regression models for these "unknown" values.


NLP : variations of a text without modifying it's meaning I am currently working on the automation of recurring reports (weekly 30-50 pages reports for around 100 districts). Those reports have a mostly fixed form : maps, graphs, data tables and small zone of text.

Apart for some discussion around colors and legends, it isn't difficult to automate the production of maps / graphs / tables. (I work with Rmarkdown if you want to know)

However, for the text, a simple approach like writing 'r value' in markdown to produce a variable value inside of the text feel 'too automated'. The reports end up having ten sentences like 'During the last quarter (QX 201X) total result was XXX (a +X% growth compared to the same quarter the previous year).'

I'd like to get automatic variations of that phrase without modyfiying it's meaning. I've ended up writing half a dozen variations myself. But (1) it still feels repetitive and unnatural, and (2) doing it for every phrase of the report may take a lot of time.

We have seen a lot of extraordinary things in transfering things for visual representation (see : https://en.wikipedia.org/wiki/Neural_Style_Transfer). So I was wondering if we have similar things for NLP, that would allow a text to be rewritten using a different 'style' (a neutral style -or an absence of style- in my case), keeping it's main content. The main paper I found on the subject is titled 'What is wrong with style transfer for texts?' and shows why style transfer doesn't really work for texts. Given (1) the constraint (keeping the same meaning) and (2) it's formalism (I know which number should be shown), I feel like the problem may be simpler than the whole style transfert.

Any idea where to start to automatically write variations of a text while keeping it's meaning constant ?

Text summarization can be divided into two categories 1. Extractive Summarization and  2. Abstractive Summarization

Extractive Summarization: These methods rely on extracting several parts, such as phrases and sentences, from a piece of text and stack them together to create a summary. Therefore, identifying the right sentences for summarization is of utmost importance in an extractive method.
Abstractive Summarization: Abstractive methods select words based on semantic understanding, even those words did not appear in the source documents. It aims at producing important material in a new way. They interpret and examine the text using advanced natural language techniques to generate a new shorter text that conveys the most critical information from the original text.

What you are looking for is abstractive summarisation. Since you are working in R there is a nice library called lexRank taking an example from here would look something like
#load needed packages
library(xml2)
library(rvest)
library(lexRankr)

#url to scrape
monsanto_url = "https://www.theguardian.com/environment/2017/sep/28/monsanto-banned-from-european-parliament"

#read page html
page = xml2::read_html(monsanto_url)
#extract text from page html using selector
page_text = rvest::html_text(rvest::html_nodes(page, ".js-article__body p"))

#perform lexrank for top 3 sentences
top_3 = lexRankr::lexRank(page_text,
                          #only 1 article; repeat same docid for all of input vector
                          docId = rep(1, length(page_text)),
                          #return 3 sentences to mimick /u/autotldr's output
                          n = 3,
                          continuous = TRUE)

#reorder the top 3 sentences to be in order of appearance in article
order_of_appearance = order(as.integer(gsub("_","",top_3$sentenceId)))
#extract sentences in order of appearance
ordered_top_3 = top_3[order_of_appearance, "sentence"]

> ordered_top_3
[1] "Monsanto lobbyists have been banned from entering the European parliament after the multinational refused to attend a parliamentary hearing into allegations of regulatory interference."
[2] "Monsanto officials will now be unable to meet MEPs, attend committee meetings or use digital resources on parliament premises in Brussels or Strasbourg."
[3] "A Monsanto letter to MEPs seen by the Guardian said that the European parliament was not “an appropriate forum” for discussion on the issues involved."

EDIT: How I like to think about abstractive summarisation: Y
Using encoder-decoder architecture (extendended with transformers) for seq2seq problems you can essentially get an embeding of your text, where same sentences can be embedded differently in different context, giving same/similiar output.


Does feature selections matter to Decision Tree algorithms? Background: Currently I'm working with my thesis project, which is to build a Tree-based ensemble methods for classification on a large data set. Before I started with modelling, I've spent a large amount of time on feature selection using correlation-based criteria to select a subset of features, so that those chosen features have high correlation with the response variable and low correlation with each other.

However, one of my supervisors questioned why I spent so much time in feature selection, as he mentioned, the decision tree algorithm can naturally select which features are most important. Later I checked book Introduction to Data Mining by Tan (2014) and it says clearly that "Feature selection occurs naturally as part of the data mining algorithm, ..., such as decision tree classifiers" .
I felt a bit frustrated by spending time on feature selection. When I eventually run the models with and without feature selection, it does NOT show any significant differences between the results.

My question is: Does feature selections matter at all to Decision Tree algorithms?

For ensembles of decision trees, feature selection is generally not that important. During the induction of decision trees, the optimal feature is selected to split the data based on metrics like information gain, so if you have some non-informative features, they simply won't be selected.

Feature selection can still be important for small datasets, where spurious relationships between features and class labels are more commonly seen.

As a side note: it's common practice to remove some features from datasets for all learning algorithms, like ID fields. These features are especially bad for decision trees, as they usually end up being the feature with the highest information gain (and therefore selected to be split on) while actually containing no generalisable information. So in this way, feature selection can be useful when using decision trees, although it's arguable that the ID field should be considered a feature in the first place.


How much app analytics data to collect? Excuse the potentially dumb question, I've only just started learning about data science.

How do we find out how much data we should collect before using it to start making decisions?

Is there a way of knowing if we should wait to collect more data? Or if the quantity of data can lead to meaningful results?

Background: we have a mobile app and some stats about it, shown below. It's early days and the data set is small. We want to start using this data to make better decisions about improving the product.

Short answer is: as much as possible.

The more data you have, the more likely you can make correct inference. In the worse case, you could simply discard anything that you don't like. It's like money, the more the better - you always have an option to spend or use less.

Statistics could be doggy without sufficient sample size, your statistical power would be affected and any model you have could be biased.

Having said that, you don't want everything from your user. Use your common sense, and ask yourself what exactly you want to do.

Based only on the data in your question, I'd say: (I could be wrong because I don't understand where you got the data)

How many total users you have that didn't drop out? If your total user base is like 10K, 28 users dropping out is not a problem. But if you only have 30 users...
Your sample size is small. Is that one-day installation? In any case, 28 users for a mobile app is a tiny data set. You may want to focus more on marketing.

EDIT:

How many respondent you need is hard to tell. Apple probably needs better data quality than you. If you'd like to do it statistically, you may calculate the minimum sample size and it's related to statistical power.


Error when trying .transform for OrdinalEncoder from Scikit Learn I'm having a lot of issues using scikit learn recently and was hoping someone could help me with my problem. I can use other methods to ordinal encode but i want to figure this one out.
for i in range(len(ordinal_orders)):
   ord_en = OrdinalEncoder(categories = {0:ordinal_orders[i]})
   X_train.loc[:,ordinal_features[i]] = ord_en.fit_transform(X_train.loc[:,ordinal_features[i]].values.reshape(-1,1))

This works fine but when i try and apply this transformation to the test set i get an error.
for i in range(len(ordinal_orders)):
    ord_en = OrdinalEncoder(categories = [ordinal_orders[i]])
    X_test.loc[:,ordinal_features[i]] = ord_en.transform(X_test.loc[:,ordinal_features[i]].values.reshape(-1,1))

i get the value error
AttributeError                            Traceback (most recent call last)
<ipython-input-45-24cb27da6829> in <module>
      1 for i in range(len(ordinal_orders)):
      2     ord_en = OrdinalEncoder(categories = [ordinal_orders[i]])
----> 3     X_test.loc[:,ordinal_features[i]] = ord_en.transform(X_test.loc[:,ordinal_features[i]].values.reshape(-1,1))
      4

/opt/anaconda3/lib/python3.8/site-packages/sklearn/preprocessing/_encoders.py in transform(self, X)
    698             Transformed input.
    699         """
--> 700         X_int, _ = self._transform(X)
    701         return X_int.astype(self.dtype, copy=False)
    702

/opt/anaconda3/lib/python3.8/site-packages/sklearn/preprocessing/_encoders.py in _transform(self, X, handle_unknown)
    105         X_mask = np.ones((n_samples, n_features), dtype=np.bool)
    106
--> 107         if n_features != len(self.categories_):
    108             raise ValueError(
    109                 "The number of features in X is different to the number of "

AttributeError: 'OrdinalEncoder' object has no attribute 'categories_'

The ordinal features and orders are
ordinal_features=['LotShape','ExterQual','ExterCond','BsmtQual','BsmtCond',
'BsmtExposure','BsmtFinType1','BsmtFinType2','HeatingQC','KitchenQual',
'FireplaceQu','GarageQual','GarageCond','GarageFinish','Fence','PoolQC']

ordinal_orders=[
    #LotShape
['Reg','IR1' ,'IR2','IR3'],
    #ExterQual
['Fa','TA','Gd','Ex'],
    #ExterCond
['Po','Fa','TA','Gd','Ex'],
    #BsmtQual
['None','Fa','TA','Gd','Ex'],
    #BsmtCond
['None','Po','Fa','TA','Gd','Ex'],
    #BsmtExposure
['None','No','Mn','Av','Gd'],
    #BsmtFinType1
['None','Unf','LwQ', 'Rec','BLQ','ALQ' , 'GLQ' ],
    #BsmtFinType2
['None','Unf','LwQ', 'Rec','BLQ','ALQ' , 'GLQ' ],
    #HeatingQC
['Po','Fa','TA','Gd','Ex'],
    #KitchenQual
['Fa','TA','Gd','Ex'],
    #FireplaceQu
['None','Po','Fa','TA','Gd','Ex'],
    #GarageQual
['None','Po','Fa','TA','Gd','Ex'],
    #GarageCond
['None','Po','Fa','TA','Gd','Ex'],
    #GarageFinish
['None','Unf','RFn','Fin'],
    #Fence
['None','MnWw','GdWo','MnPrv','GdPrv'],
    #PoolQC
['None','Fa','Gd','Ex']
]

train and test set is just a generic dataset from kaggle, theyre not an issue here.
Any sort of clarification would be nice!

It is really hard to figure out the logic behind what you are doing, it look odd
But assuming you are trying to apply a preprocessing step to a data frame I would go as follows:
from sklearn.compose import make_column_transformer
from sklearn.preprocessing import OrdinalEncoder

ordinal_features=['LotShape','ExterQual','ExterCond','BsmtQual','BsmtCond',
'BsmtExposure','BsmtFinType1','BsmtFinType2','HeatingQC','KitchenQual',
'FireplaceQu','GarageQual','GarageCond','GarageFinish','Fence','PoolQC']

ordinal_orders=[
    #LotShape
['Reg','IR1' ,'IR2','IR3'],
    #ExterQual
['Fa','TA','Gd','Ex'],
    #ExterCond
['Po','Fa','TA','Gd','Ex'],
    #BsmtQual
['None','Fa','TA','Gd','Ex'],
    #BsmtCond
['None','Po','Fa','TA','Gd','Ex'],
    #BsmtExposure
['None','No','Mn','Av','Gd'],
    #BsmtFinType1
['None','Unf','LwQ', 'Rec','BLQ','ALQ' , 'GLQ' ],
    #BsmtFinType2
['None','Unf','LwQ', 'Rec','BLQ','ALQ' , 'GLQ' ],
    #HeatingQC
['Po','Fa','TA','Gd','Ex'],
    #KitchenQual
['Fa','TA','Gd','Ex'],
    #FireplaceQu
['None','Po','Fa','TA','Gd','Ex'],
    #GarageQual
['None','Po','Fa','TA','Gd','Ex'],
    #GarageCond
['None','Po','Fa','TA','Gd','Ex'],
    #GarageFinish
['None','Unf','RFn','Fin'],
    #Fence
['None','MnWw','GdWo','MnPrv','GdPrv'],
    #PoolQC
['None','Fa','Gd','Ex']
]

transformer = make_column_transformer((OrdinalEncoder(categories= ordinal_orders),ordinal_features), remainder = "passthrough").fit(X_train)

Then you can apply the transformer to both X_train and X_test
transformer.transform(X_train) or transformer.transform(X_test)


Temporal difference learning with a neural network Suppose I would like to train a value network $v$ via TD(0).

So my TD target for a time step $t$ equals:

$$R_{t+1} + \gamma v(s_{t+1})$$

If I understand correctly, I just need to use mean squared error, so that $v(s_t)$ becomes closer to this target. However, my network outputs values between $(-1; 1)$ and rewards are from this interval also, so the TD target lies between $(-2; 2)$. Should I scale it before apply learning? What are the consequences of not doing this i.e. training a neural network with target values from a broader interval that it's output? Can we say anything about it from theoretical point of view?

Should I scale it before apply learning?

Definitely, you need to either:

Scale your network output so that it lies in the range (-2, 2), or,
Scale your training data so that it lies in the range (-1, 1)

The latter is marginally preferable because it will only need to be computed once, rather than on every pass through your network.

  Can we say anything about it from theoretical point of view?

This is a very standard design, and is simply a consequence of the fact that most commonly chosen activation functions have bounds of [-1, 1], [0, 1], [0, inf) or (-inf, inf). In particular for finite bounded activation functions (i.e. those with outputs bounded by [-1, 1] or [0, 1]), you will need to make sure that the output space of the network is capable of representing the desired target space. For example, for binary classification problems, you will often see people choosing labels of either 0/1 or -1/1 depending on whether the final output is activated by a tanh or sigmoid function.


Loss function for classifying when more than one output can be 1 at a time My desired output is not 1-hot encoding, but
like a 10 D vector: [1, 0, 1, 0, 1, 0, 0, 1, 1, 1]
and the input is like the normal MNIST data set.

I want to use TensorFlow to build a model to learn this,
then which loss function should I choose?

If your classes arre not mutually exlcusive, then you just have multiple sigmoid outputs (instead of softmax function as seen in example MNIST classifiers). Each output will be a separate probability that the network assigns to membership in that class.

For a matching loss function, in TensorFlow, you could use the built-in tf.nn.sigmoid_cross_entropy_with_logits - note that it works on the logits - the inputs to the sigmoid function - for efficiency. The link explains the maths involved.

You will still want a sigmoid function on the output layer too, for when you read off the predictions, but you apply the loss function above to the input of the sigmoid function. Note this is not a requirement of your problem, you can easily write a loss function that works from the sigmoid outputs, just the TensorFlow built-in has been written differently to get a small speed boost.


Variational Autoencoder TIme Series Can anyone suggest a blog where Variational Autoencoder has been used for time series forecasting?

Maybe look at this paper and references therein
Variational Encoding of Complex Dynamics


Weighted linear regression with a DNN (in Keras) I have a fairly small dataset of 225 points. I have a target (labelled numeric), a feature (normalised numeric) and a quality index with set of normalised weights that describe the likelihood that the observation will contribute to a linear relationship between target and feature. Good data for weighted linear regression.

The dataset is currently at only 225 points but it's growing so I'd like to explore DNNs to improve on the performance of the weighted linear approach.  Can Keras do this?

What I am curious about is the possibility that the weight optimization process in the DNN training could itself eliminate points with a low quality index, i.e. those points that do not contribute to a linear fit and result in a high loss function return.

But I am very new to deep learning so my understanding could be wrong.

This is a suggestion. I don't know if this will work. Maybe you could adapt a feature selection problem to a data point selection problem. For a standard linear regression, The LASSO technique can be used to do both the regression and the feature selection. The idea is to penalize the use of too much features by adding a term to the loss.

Where beta are the coefficients to be learned of your linear regression.

With only one feature x_i and with your quality index q_i (they should sum to 1) your loss can be written as :

I see 2 ways of doing this :

Minimize the loss above. beta_1 and beta_0 are the only coefficients that can be learned. The process will not eliminate the points with low quality but they will contribute just a little to the loss.
Consider the q_i as learnable parameters. You don't use the quality index that you already have. And then you can add a penalization term on the q_i. Those who end up to 0 will not contribute to the loss. The loss will look like this :

I think there are many details to solve. Like adding a constraint to the q_i like 0 < q_i < 1. Maybe this can not be solved easily. I suggest you to use Pytorch. This will be way much easier to implement.


Is there an intuitive explanation why some neural networks have more than one fully connected layers? I have searched online, but am still not satisfied with answers like this and this.

My intuition is that fully connected layers are completely linear. That means no matter how many FC layers are used, the expressiveness is always limited to linear combinations of previous layer. But mathematically, one FC layer should already be able to learn the weights to produce the exactly the same behavior. Then why do we need more? Did I miss something here?

There is a nonlinear activation function inbetween these fully connected layers. Thus the resulting function is not simply a linear combination of the nodes in the previous layer.


Machine Learning, Imputing values that should be blank Sometimes data sets contain variables that indicate the presence of an event and the value that represented the event.

As an example say a teacher wants to predict the grades of his students. Some of the students may have been in his class last year and he can use that grade as a variable. However maybe only 20% of the students were in his class so the rest of the 80% will have a Null value. Most ML algorithms cannot accept Null values so the variable would have to somehow be imputed.

I cannot think of an imputation method that would make sense here, the standard mean/mode would imply that all students were in the class and since the variable is pretty unbalance and 80% of the values would be imputed I don't imagine it would hold any valuable information.

Are there any methods to deal with this scenario?

Well, it seems that you are dealing with sparse data, however imputation is a difficult and often an attempt of imputation can add trivial amount of difference. You may look out on for this link for some approaches like Gharamani and Jordan.
These are variants of SVM, focused with Sparse nature.


Hosmer Lemeshow Test, variable lengths differ Doing a logistic regression in R version 3.1.2 and I want to conduct a Hosmer Lemeshow test

model <- glm(y ~ a + b + c + d, data = data, family = binomial)

summary(model)

Call:
glm(formula = y ~ a + b + c + d, family = binomial,
    data = data)

Deviance Residuals:
    Min       1Q   Median       3Q      Max
-2.5607  -0.7642  -0.4456   0.8998   2.3812

Coefficients:
             Estimate Std. Error z value Pr(>|z|)
(Intercept) -7.827805   1.169216  -6.695 2.16e-11

a            0.582214   0.134207   4.338 1.44e-05

b           -0.697126   0.462983  -1.506  0.13214

c            0.027485   0.009669   2.843  0.00448

d            0.996241   0.163006   6.112 9.86e-10

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 504.53  on 373  degrees of freedom
Residual deviance: 381.91  on 369  degrees of freedom
  (3 observations deleted due to missingness)
AIC: 391.91

Number of Fisher Scoring iterations: 5

Using the function hoslem.test from package "Resource Selection"

hoslem.test(model$y, fitted(model), g = 10)

I get the following error

Error in model.frame.default(formula = cbind(y0 = 1 - y, y1 = y) ~ cutyhat) :
  variable lengths differ (found for 'cutyhat')

Looking at the code in the hoslem.test function, cutyhat is derived only from fitted(model) values, which I've verified contain no nas and are all integers, so why would they differ?

Pls help.

I think it could be that the model is being used and not the data, try:

hoslem.test(data$y, fitted(model), g = 10)


How are images (roughly) clustered when using KNN When classifying data by KNN, the classes are calculated by using the distance between datapoints. For example, the case of real estate, where the x-axis is the price and the y-axis the size of real estate. But what are the values of axis when instead of numbers there are pictures? By which category are the distances calculated in this case?

Lets say its a gray scale image. And of size 224*224 than you have 50176 possible pixel values. So you can plot on x axis widt=224 and on y heigth=224.

Lets say they are coloured. Than you have more chanels=dimensions. In that case you perform clustering in higher dimensional space


Precision and Recall if not binary I have to calculate precision and recall for a university project to measure the quality of the classification output (with sklearn).
Say this would be my results:

y_true = [0, 1, 2, 1, 1]
y_pred = [0, 2, 1, 2, 1]

confusion matrix:
[1 0 0]
[0 1 2]
[0 1 0]

I have read about it and the definition makes sense for me in a binary setting but with 3 labels I find it hard to interpret precision/recall.
If I use sklearn.metrics.precision/recall_score it gives me 0.4 for both (average = micro)

Now for the precision this makes somewhat sense because 2 out of 5 are correctly classified. But I am having problems interpreting the 0.4 result for recall.

sklearn.metrics.classification_report provides precision and recall for all classes along with F-score and support. It might prove to be helpful in your case of 3 classes.


Extracting sections from document based on list of keywords - Python I am new to NLP and I would like to ask how can I extract sentences from the text based on keywords that I have using Python. I created a list of keywords which will be used to extract sentences from the document.

If this will be a simple tokenization problem in which you will loop the list through the tokens, how can I capture synonyms or related words?

For example:

Keyword: Internal business

Sentence: You can only use this software for your business only.

Keyword: Confidentiality

Sentence: Information will be kept as secure as possible.

I actually implemented text categorization using TF-IDF, but with small dataset and large number of keywords. I don't think this will work to.
Thanks in advance.

Is it possible to apply pre-trained models like word2vec?

Is it also possible to create a custom model that will fit my concerns?

The ideal way to get the related sentences would be to try to get a sentence vector for the sentences you want to categorise and then compare the vectors of your predefined keywords with the obtained sentence vectors .
You can get the sentence vectors by just averaging the word vectors of the words present in the sentences . Once the sentence vectors are obtained , you can use cosine similarity to compare the keyword vectors and the sentence vectors . The one with the max cosine similarity will give you the result .


How to Deploy your trained ML model in client VM without they getting access to code I am new to deployment and have a basic doubt about deploying my ML code on client's vm.

So I have built a python project which collects data from client site, processing, predicts and displays the result in dashboard. I have to use client VMs for deployment. Is there a way for me to hide the code or do something to it so that client cannot see my code and reuse my code for other purposes.

Might sound trivial but really stuck at this. Thanks in advance.

You can transform your python code into a standalone binary file, which could for example take as input some data and return the predicted labels. If you only put this file in the client's VM and not the .py files, the client could never see the code, but just execute it.
Several libraries exist for this purpose like pyinstaller


What's the difference between keras api augmentation and data augmentation definition? The augmentation definition is increasing the number of images by using rotation, crop and flip to avoid overfitting. The keras API apply augmentation but no increasing the number of image. What keras augmentation does in images? Is API augmentation such as preprocessing of images? Is augmentation replace the original image with new augmented images?

The Augmentation api apply different augment defined by you (horizontalflip, verticalflip, rotation, zcawhitening etc) to your original image rather than making a copy of it. It apply different augmentation on the same image on different epoch which reduce the generalization error.


Use Python to Scape Data on website ends with aspx https://blue.kingcounty.com/Assessor/eRealProperty/default.aspx

This is the website I want to scrape.

For example, I input 500 4TH AVE
I just want to get Parcel Number and Lot size.

I was trying to edit the link

https://blue.kingcounty.com/Assessor/eRealProperty/search=5004THAVE.aspx

I think it did not work, please demonstrate. Thank you.

You can use selenium for this crawl. When you hit that the link first time, it will ask to accept the terms and conditions then you need to click on the checkbox. After that, it will redirect to page, just pass the input to your current URL and read the HTML element using XPath or regex.


Are there some research papers about text-to-set generation? I have googled but find no results.

Text-to-(word)set generation or sequence-to-(token)set generation.

For example, input a text and then output the tags for this text:

'Peter is studying English' --> {'good behavior','person','doing something'}

Thank you!

Check out these papers below and google keywords: Multi-label Classification.

X-BERT: eXtreme Multi-label Text Classification with BERT
HAXMLNet: Hierarchical Attention Network for Extreme Multi-Label Text Classification
SGM: Sequence Generation Model for Multi-label Classification
Ranking-Based Autoencoder for Extreme Multi-label Classification
AttentionXML: Label Tree-based Attention-Aware Deep Model for High-Performance Extreme Multi-Label Text Classification

Also check out the code below:

https://github.com/chenyuntc/PyTorchText


Cross-validation average score I am using Repeated K-folds (RepeatedKFold(n_splits=10, n_repeats=10, random_state=999) from sklearn) to provide reliable scores for a linear regression on my dataset.

The dataset has some outliers which should stay and also similar cases can be seen in future observations. When a trained data in a fold tries to predict such observations, I get negative scores (at least, this is my interpretation)

Question: the main question is what should I do with one (or a few) bad score(s) out of many? How should I report them and how useful would that be?
Using 10 splits and 10 repeats for a data of size ~3000 observations, I will get 100 r-squared scores which are all in a good range (0.97 to 0.99). There is only one guy ruining the game and the score is so bad (-11535) that I cannot even get an average!

[ 9.87345591e-01  9.73912516e-01  ... -1.15353090e+04 ...  9.72986827e-01]

What shall I do in this case? how to report it and/or how to cure it?

Your result is really a bit strange (it’s the R2, right, so the score makes no sense as R2 should be in a range between 0,1). When you do 10fold cv, each bit of your data will be used in one of the folds. So when 9/10 runs are okay, but one of the 10 scores in a 10th run is very bad, it could be coincidental clustering of your outliers in this one case.

For me this raises the question of robustness. So I would run „a lot“ of 10fold cv over my data to see if the problem occurs again (and to what extent). This is cheap in your case but gives you good arguments.

Also this gives you a good idea of how sensitive your model actually is to potential „strange“ outliers which might be part of real world problems when I understand you correctly.

If true, I would also check where outliers are particularly problematic if possible. I don‘t know the scale of your model, but having a look at predicted values vs actual y, may give you an idea.


Extract names from email address Say I have two email addresses and I would like to see if it is likely that they belong to the same person. For example, Cameron_M_Thompson@company.com and cthompson1024@personal.com is likely to be from the same person (it doesn't have to be certain, providing the likeliness would be sufficient).
I had two directions in mind to achieve this, one is a string comparison between the two email addresses and the other is to first extract the names from the email addresses then compare if they might be the same person. Like in the example above, the names extracted should be Cameron M Thompson and c thompson.
I am also wondering if given that one of the email addresses is guaranteed to contain the full name (usually company email addresses have the full name), would that help the extraction of name in the other email address (personal email addresses might not always contain the full name), or would that help on the comparison of the two email addresses.
I have had a hard time trying to figure out if any of the above two directions would be feasible. Especially when email addresses might not have separators and names can vary a lot that a listing might not be sufficient to find a match.
How should I proceed in solving this problem? Would machine learning / deep learning help or I should go with something else simple like regex and fuzzy string match?
UPDATE:
I have a dataset that has two columns, email address and name, and about 2k rows there. I believe this could be used for the second direction (name extraction). For the first direction (string comparison similarity), I am thinking of modifying the dataset to three columns (email address 1, email address 2, label of whether they are the same person), which should give about 1k rows of data.

Before talking about the solution, why don't you focus on the content instead? I think it would be more helpful to solve your problem, considering that most of the email addresses end with the sender's sign, Name Surname. Also, the probability of failing to obtain this information from an email address is much higher than the probability of failing to get it from the content. Especially, this is the case with company email addresses which might not contain the whole name in the email address (first letter of name and surname e.g. John Travolta - jtravolta@company.com), but it must contain the author's full name (at least the name) at the end. Furthermore, consider that plenty of email addresses will contain only name or surname or neither of them, but substitutive words like superboy122133@+++.com :D. But most of the email apps contain a default sign that includes name and surname. In addition, you can combine these two techniques. That is, combine the email address data with email content data so that, if it is infeasible or impracticable to obtain data from one of these, then you can use another one.
However, if let's say you have to do it with nothing but an email address I think using Machine Learning techniques would be overrating or overestimating the problem. Also, using non-machine learning techniques does not mean you are simplifying the solution, all these techniques give the best outcome when they are applied in the correct context. Let's imagine a simple situation: if you know or can easily infer that [tax] = 0.2 * [salary] + 20 $, why would you find (or fit) this equation using Machine Learning?
Unless you have data in the format of |email address, fullname|, you shouldn't start with using Machine Learning. (If you would have |email address, fullname| data, as an option, you would train a model to learn the general relationship between the email address and full name, thus you would identify similar email addresses).
However, in this current situation, one approach would be finding all possible patterns in the email addresses. Which can be

first letter of the name and surname jtravolta@+++.com
name, special characters and surname john_travolta@+++.com
name and numbers john1954@+++.com
...

Then these features that are extracted from email addresses using identified patterns can be compared with other emails either hashing or using string distance algorithms.
One alternative approach would be having a hashed dictionary of all available names and surnames, then you can cut pieces(substrings) from the email address then hash them to find the names and surnames from the address (Of course, vice versa would be highly inefficient). The email addresses that have the most similar, properties would be matched.
Another solution would be, using the above-mentioned patterns, you can generate a bunch of artificial email addresses. Considering that it is highly probable that there is not a dataset that includes the name and surname of people and their one or more email addresses, data augmentation is the first order of business. (I am not sure whether the data augmentation term fits this situation. If it does not then let's say data generation). So your input would be Name Surname (you can include middle name, number, etc.), and output would be randomly generated email addresses based on the pre-defined patterns. The number of emails that are generated for a single input should be randomly selected also, but be careful about the generation of the same email address more than once. E.g. input -> John Travolta -> output -> j_travolta12@+++.com, john.t.99@+++.com, john.travolta@+++.com (Lets suppose for this example we randomly choose 3 emails to be generated).
Then after you created, email addresses with all possible (almost) patterns you can get help from Machine Learning techniques. So the model might give you a probability with the relevant name and surname. (Also, you can configure the output so that it would give you top n name and surnames which have higher probability)
Another thing that is needed to be considered is the possibility of two different persons having the same name and surname. Lastly, independently from using which approach your solution cannot be perfect because, for example, it is not possible to understand whether the character 'j' stands for John or Jake in the email address. Thus, if you can integrate the email content into your solution, that will increase the performance drastically.
Update Accordingly: Check this answer which does not exactly answer your problem, but the context is the same.


Is pandas now faster than data.table? Here is the GitHub link to the most recent data.table benchmark.
The data.table benchmarks has not been updated since 2014. I heard somewhere that Pandas is now faster than data.table. Is this true? Has anyone done any benchmarks? I have never used Python before but would consider switching if pandas can beat data.table?

Has anyone done any benchmarks?

Yes, the 2014's benchmark in question has turned into foundation for db-benchmark project. Initial step was to reproduce 2014's benchmark on recent version of software, then to make it a continuous benchmark, so it runs routinely and automatically upgrades software before each run. Over time many things have been added. Below is high level diff of the 2014's benchmark comparing to db-benchmark project.
New:

continuous benchmark: runs routinely, upgrades software, re-run benchmarking script
more software solutions: spark, python datatable, cuda dataframes, julia dataframes, clickhouse, dask
more data cases

two new smaller data sizes: 0.5GB (1e7 rows) and 5GB (1e8 rows)
two more cardinality factors: unbalanced, heavily unbalanced
sortedness
data having NA

advanced groupby questions

median, sd
range v1-v2: max(v1)-min(v2)
top 2 rows: order(.); head(.,2)
regression: cor(v1, v2)^2
count and grouping by 6 columns

benchmark task: join

Changes (see groupby2014 task for 2014 fully compliant benchmark script):

using categorical/factor instead of character
cardinality of v2 and v3 measures increased
function calls are NA-aware
aggregated columns are named
order of groups is irrelevant whenever possible
extra call to dim()/.shape is included in timings to force lazy evaluation
machine that runs benchmark has 128GB mem (not 244GB mem)
no 100GB (2e9 rows) data size

We are planning to add even more software solutions and benchmark tasks in future.
Feedback is very welcome, feel invited to our issue tracker at https://github.com/h2oai/db-benchmark/issues.

Is pandas now faster than data.table?

According the our results pandas is not faster than data.table.
I am pasting medium size data 5GB (1e8 rows) groupby benchmark plot taken from the report at h2oai.github.io/db-benchmark as of 20210312. Consult the h2oai.github.io/db-benchmark#explore-more-data-cases for other data sizes (1e7, 1e9), data cases (cardinality, NAs, sorted), questions groups (advanced), or tasks (join).
For up-to-date timings please visit https://h2oai.github.io/db-benchmark.


Extract sentences from beginning of news in single document summarization I am working on Single Document Summarization task on News datasets. I do some experiments in this task. A simple experiment that I make and has a good result is extracting sentences just from beginning of news. Now I want to find any paper or research result about this type of sentence selection.

Is there any research to show how good is to choose sentences just from beginning of text without any reordering?

Keselman, Schubert

Computational models for text summarization

The paper deals with methods (models) for text summarization. The reference (base) model was "first sentence model":

  As a baseline for our models we used a trivial model that repeats the
  first sentence of the input document.

Then, various experiments and results are presented, like this one: (notice that "first sentence model" is always present as "baseline")

Moreover, one of the datasets for training and evaluation of models in this paper is DUC, which may be interesting to you.

Steinberger (doctoral Thesis, 2005)

Text Summarization within the LSA Framework

In section 2.1, the author discusses document summarization approaches based on sentence extraction. He identifies five approaches:

Surface Level Approaches
Corpus-based Approaches
Cohesion-based Approaches
Rhetoric-based Approaches
Graph-based Approaches

(The "First Sentence Approach" belongs to the *Surface Level Approaches") The author further describes these approaches and compares them.

Khodra, Widyantoro, Aziz, Trilaksono (Journal of ICT Research and Applications, 2011)

Free Model of Sentence Classifier for Automatic Extraction of Topic Sentences

The author identifies and tests methods for identifying the most important sentences in a text (see list of 58 items below, called features). Surprisingly, in the conclusion, it is said that position of the sentence is a dominant feature, meaning that including all other features into consideration leads only to small improvement.

position
sentence length
number of words before a main verb
adjective incidence
existential there incidence
incidence of 3rd person singular grammatical form
anaphora incidence
coordinators incidence
cardinal number incidence
incidence of past tense endings
Hypernymy
Polysemy
concreteness index
affect_formulai
bad_formulaic
comparison_formulaic
continue_formulaic
contrast_formulaic
detail_formulaic
future_formulaic
gap_formulaic
good_formulaic
here_formulaic
in_order_to_formulaic
method_formulaic
no_textstructure_formulaic
similarity_formulaic
them_formulaic
textstructure_formulaic
tradition_formulaic
us_previous_formulaic
affect
argumentation
better_solution
change
comparison
continue
contrast
interest
need
presentation
problem
research
solution
textstructure
use
copula
aim_ref_agent
gap_agent
general_agent
problem_agent
ref_agent
ref_us_agent
solution_agent
textstructure_agent
them_agent
them_pronoun_agent
us_agent

For you, the most important part of the paper may be table 5:

Read carefully the explanation of the table in the paper, and the whole Section 4.3.

Other papers worth examining:

Luhn (1958)

The Automatic Creation of Literature Abstracts

Kupiec, Pedersen, Chen (1995)

A Trainable Document Summarizer

Yang, Pedersen (1997)

A Comparative Study on Feature Selection in Text Categorization

Sebastiani (2002)

Machine Learning in Automated Text Categorization


SGDClassifier : Get the weights and bias for each training step I would like to get the coef_ (weight vector) and the intercept_ (bias) for each iteration or training step for the sklearn.linear_model.SGDClassifier.

However, I cannot find anywhere such a functionality? Is this even possible with sklearn?

In tensorflow I could add a callback. Is there something similar available for SGDClassifier?

Thanks :)

You can use the partial_fit method to get the results of a single training epoch for sklearn.linear_model.SGDClassifier.


Create a new column based on two columns from two different dataframes I have one column in the first dataframe called 'id' and another column in the second dataframe called 'first_id' which refers to the id from the first dataframe. I need to create a new column which has value 1  if the id and first_id match, otherwise it is 0.  I tried this but I am getting an error ValueError: Length of values does not match length of index.

df2['new'] = np.where(df2[df2.first_id.isin(df1.id.values)], 1, 0)

I understand why is it happening, because df2 and df2[df2.first_id.isin(df1.id.values)] are of different lengths but I can't make them the same. Any ideas?

You were almost there!

Sample DFs:

In [387]: df1
Out[387]:
   id
0   1
1   2
2   3
3   4
4   5

In [388]: df2
Out[388]:
   first_id
0         7
1         6
2         5
3         1
4         3

Solution:

In [389]: df2['new'] = df2.first_id.isin(df1.id).astype(np.int8)

Result:

In [390]: df2
Out[390]:
   first_id  new
0         7    0
1         6    0
2         5    1
3         1    1
4         3    1


Anomaly Detection for Large Time Series Data I am working on detecting anomalies within a large time series data set. It is updated on a regular basis and consists of more than 30 parameters. I am using R as a reference language.

It is a first for me working on this type of projects and I am unfamiliar with most of the techniques. I have 6 weeks to implement a
good analytical toolbox to enhance the quality of the control checks on the production line.

I have found a couple of potential methods to analyze it including statistical machine learning, deep learning using auto-encoded neural networks or clustering approaches. The purpose of the chosen method is to detect the anomalies/outliers by itself. It doesn't really need to be real-time analysis.
What approach would you recommend to implement for the scope of the project, given the structure of the data?

Following J.Tukey, you should plot, draw graphs, visualize, etc... until you have a solid pack of examples.

Then make Tukey' fences on each of the 30 parameters. Let $q_1$ and $q_3$ be the 1st and 3rd quartiles, $d=q_3-q_1$ the inter-quartile distance, and define as outlier any observation outside the interval $q_1-k\cdot d < x < q_3+k\cdot d$, where $k$ is a constant. Traditionally, $k=1.5$ indicates an outlier and $k=3$ indicates the data is far out. However, the real value of $k$ should be tested against your examples.

Then make a cluster analysis (with a k-nearest neighbor) and define as outlier any point isolated in one cluster. Again, use your example to test various values of $k$.


Distribution of error values in linear regression vs logistic regression Why do error values in linear regression have to be normally distributed and why not in logistic regression?

Lets clarify with a simpler question

  Why points on a circle must be equally distanced from center, but
  not points on a square?

Because of how circle and square are "defined". The same goes for linear and logistic regression, we cannot pose a "why?" question when errors are "defined" in a certain way (or a better word here "assumed"). We may design a new version of linear regression by replacing Normal distribution with some other distribution, and then proceed to derive a formula or algorithm for estimating the parameters. Then use it in a real-world scenario to see how it works, and so on.

Also here is a list of good posts on stats.stackexchange.com related to characteristics of those errors:

Error distribution for linear and logistic regression,
Logistic Regression - Error Term and its Distribution.


Use of negative correlation coefficient in pearson correlation algorithm for recommender systems I'm new in recommender systems and I try to find similar users of a base users for user-based collaborative filtering.
When I calculated the similarity score now between two users (based on there ratings with pearson algorithm [or resnick's weighted pearson algorithm]) I get a similarity score from -1 to 1.

Is it a good idea to normalize this values to 0 to 1 (-1 would become 0 and 1 would be 1) to make it comparable to other algorithms?

In fact I try to build recommendations and with a negative similarity score of a user the calculated/predicted rating could be negative as well which make no sense.

Should I normalize/scale "-1 to 1" to "0 to 1" or cut off all users with similarity score below 0?

(maybe the question also could be: "Which users should be taken as mentor to recommend new items on a similarity score from -1 to 1? Or should I take the top n users with highest similarity score?")

One thing you can do is to separate the contributions of:

a) who have a positive correlation with you
b) who have a negative correlation with you

Then you can:

Predict the rating using only users in a) $\to$ those will have positive correlation, this positive weights. Call this $\hat{r}_a$
Predict the rating using only users in b) $\to$ in this case, consider the weights as positive (even in they are negative correlation). Call this $\hat{r}_b$
The final predicted rating is $\hat{r} = \frac{\hat{r}_a - \hat{r}_b}{2}$


Missing value Imputation in dataset I have two separate files for Testing and Training.
In the training data, I am dropping rows that contain too many missing values .
But , In the test data , I cannot afford to drop the rows so I have chosen to impute the missing values using KNN approach .
My question is , to impute missing values in the test data using KNN , is it enough to consider only the test data ? As in , neighbors in the test data alone ?

As a general rule of thumb you should avoid doing different things between your train and test dataset. As a second general rule of thumb you rarely want to use knn for missing value imputation.
One efficient way to deal with missing value in your case would be to use a model that can handle missing values, like a tree model. (decision tree, random forest, xgboost...).


Mapping sequences of different lengths to fixed vector - Python I am trying to make a chatbot using a deep neural network in python using keras. The problem I am having is that for the deep neural network to work, the input dimension has to be fixed. So my question is, how can I map a sequence of words (sentence of different lengths) to a fixed vector so I am able to feed this through a deep neural network?

A recurrent neural network (RNN) is probably what you want to look into. It has exactly that capability of dealing with variable length input sequence. In fact, currently some of the most successful applications of RNN are in the NLP area, where the inputs are sentences of varying size. Just google recurrent neural network LSTM and you will find lots of tutorials.


Can I retrain my keras model on small dataset with some empty folders I am working on a vehicle classification problem.

I trained my model with 605 labels(folders) and 300k photos.

After testing my trained model, I checked my wrong answers' training and validation photos. I realized some of my labels training or validation photos are wrong or not enough different from another.

I can prepare a small dataset that only contains these wrong learnt labels photos and keep other labels folders empty.

If i retrain my model, will I get better accuracy? Or should I train from scratch?

What you are suggesting is to fine tune the network giving it a portion of the starting data set?

If yes, well you can try, but I am not sure what the outcome would be. But remember to keep a train and test set separated!


What's wrong with my code? I'm making a model that predicts delays in flight takeoff times using the day of the week and month. Each data entry is of the form [0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0], with the first 12 digits being for the month and the last 7 for the day of the week. The labels are integers, either positive or negative, that represent the number of hours delayed.

This is my code:

#
#
# This model uses day and month as a predictor of delay.
# https://www.kaggle.com/usdot/flight-delays#flights.csv
#
#

# TensorFlow
import tensorflow as tf
from tensorflow import keras
from keras.models import load_model
from keras.models import Sequential
from keras.layers import Dense, Activation
from sklearn.preprocessing import minmax_scale

# Helper libraries
import numpy as np
import csv

# Get features and labels
train_features = np.load("train_features.npy")[:10000, :]
train_labels = np.load("train_labels.npy")[:10000]
print(train_labels)
test_features = np.load("test_features.npy")[:1000, :]
test_labels = np.load("test_labels.npy")[:1000]

# Risk Assessment Model
model = Sequential()
model.add(Dense(6, activation="relu", input_dim=19))
model.add(Dense(1, activation="sigmoid"))

model.compile(optimizer="rmsprop", loss="mean_squared_logarithmic_error", metrics=["accuracy"])

model.fit(
    train_features,
    train_labels,
    epochs=100,
    verbose=1,
    validation_data=(test_features, test_labels),
)

model.save("time_model.h5")

The model has terrible accuracy, and I'm having trouble understanding why. I don't know what kind of activation functions I should be using, and I'm feeling completely lost. Thanks in advance for your help.

I'm not sure it is a good problem setting for input to a feed-forward network (especially such a small network with only one activation over 6 neurons).

What is there even to predict here if you are just using the day of the week and the month in the year? You could simply take the average for each of the pairwise combinations and use that.

If you want to adjust for local trends over your dataset's timespan, then perhaps look int temporal decomposition. For example, you can use the seasonal_decompose method available in the statsmodels package for Python.

What this does is to find:

a global trend over the range of your data
parts of your data that vary on a repeating pattern (seasonally),
any remaining noise, or residuals not fitting parts 1. and 2.

It works by applying convolutions over the data to find a repeating frequency. You can think of it like finding correlations in time, compared to the ConvNets, which look for correlations over pixels in an image, for example.

Here is an example, using a dataset taken of total number of international airline passengers over time. I am using the statsmodels package to load this well-known R dataset as a Pandas DataFrame:

In [1]: import statsmodels.api as sm

In [2]: import matplotlib.pyplot as plt

In [3]: ap = sm.datasets.get_rdataset("AirPassengers")       # ap = AirPassengers

The actual dataframe is stored on the data attribute of ap and has two columns:

In [4]: ap.data.columns
Out[4]: Index(['time', 'value'], dtype='object')

We can see how the data looks, where there is clearly an upward trend as well as a repeating seasonality (more people fly over the summer time):

In [5]: ap.data.plot("time", "value");

In [6]: plt.show()

Now we can perform the seasonal decomposition of the data (freq=12, because the data is monthly):

In [7]: decomp = sm.tsa.seasonal_decompose(ap.data.value.values, freq=12)

This decomp result has a few attributes, but most helpful is the plot... We can see the nicely separated seasonlity and the trend, along with some noise

In [8]: decomp.plot();

In [9]: plt.show()

We see it isn't perfect, as the seasonality is best fitted to the middle section, not the start or end phases - the residuals are largest at the beginning and end.

You can access each of the values: observed, trend, seasonal, resid and nobs on the decmp object.

Now if this analysis helps you discover seasonality, you might want to look into Seasnal ARIMA models (also available in the statsmodels package).

Here is a blog post I just found that looks pretty detailed and would help you on this path.

If you are doing this excercise to learn about Keras and deep learning, it might be worth looking into LSTM models that are better for time-series analysis. You will also need to re-think how you feed data into the model for training, not to mix up the temporal information or create biases (letting the model cheat).


Using a neural network to learn regression in image processing I have a camera system with some special optics that warp the field of view of the camera, dependent on two variables, $\theta_1$ and $\theta_2$. Given a specific configuration of these two variables, each pixel on my camera (which is 500x600 resolution) will see a specific coordinate on a screen in front of the camera. I can calculate this for each pixel, but it requires too many computations and is too slow. So, I want to learn a model that fits this function, but computes much faster.

I have plenty of input/output data that I have generated, mapping the 500x600 input points to the 500x600 output points for different $\theta_i$ values, and I have already used some 2D polynomial least squares regression to learn these functions. They perform adequately, but I was wondering if a neural net could be used to learn a better function.

My question comes down to this: can a neural network learn what basically amounts to a regression problem trying to learn $f_{\theta_1,\theta_2}(px_1,px_2)=(a_1,b_1)$?

I know that neural nets excel in classification problems, which this is not, but I have also heard that a neural network can "learn arbitrary functions."

Yes - neural networks can be used to predict regression problems. The output layer will be a set of nodes that make continuous numeric predictions.
In your problem, it would be two nodes that learn the weights for $\theta_1$ and $\theta_2$.


Classify text labels in to a similar category I'm trying to classify same kind of text labels in to one category. For example, if I have labels like qty, quantity, qty_no all of them should direct to Quantity. Since I'm new to data science, what's the best way to start this kind of thing?

One way to do it could be with fuzzy string search. Levenshtein distance algorithm is what you may use for it.

  ... the Levenshtein distance is a string metric for measuring the difference between two sequences. Informally, the Levenshtein distance between two words is the minimum number of single-character edits (insertions, deletions or substitutions) required to change one word into the other.  From wikipedia

More on Levenshtein distance -

https://stackoverflow.com/a/5859823/5406448


XGBoost multiclassification interpreting predicted probabilities Let's consider an example. I have patient level data, their symptoms, reading from various medical tests. Based on that, I have built a binary classifier given patient data to classify if they are likely to have a disease, and if so, I'd want to manually run medical tests on them. For that case, we look at the disease class and rank order the predicted probabilities and select the riskiest patients to have the disease.

A few months later, say the disease evolves into a different strain. So we already have an imbalanced class of healthy vs infected patients. Not the new strain will be still a smaller population. Apart from that, the symptoms, various test readings will also be different. So translating this to a multi-classification problem makes sense. When you build a multi-classifier, and you have three probabilities, how would you go about rank ordering the same set of patients?

If you're interested in the risk of patients likely to be infected, taking the max of the either of the strain of disease a good way to proceed or is there a method to choose them based on their individual probabilities?

Edit: To bring a little more clarity.
The multi:softprob gives the probability for each of the classes. If you have three classes, it will give three probabilities for each class summing up to 1. More details are here.

Given that we will get multiple probabilities for each row/patient from the example above, how do you go about choosing the final probability to rank order risky patients.

Multiclassification with xgboost is normally done One vs All. Meaning that first you try to predict if it belongs to class one or not. Then two second class or not...

You can understand as the probability of belongin to one class and not to the rest.

Let's put an example for better understanding: say that you have a ton of patients that have 3 illnes: Corona Virus, Sars and HIV.

You build your multiclassification model with xgboost or whatever and for a new patient you do a prediction and check the probabilities:

Corona [0.8]
Sars [0.1]
HIV [0.01]
Not sick [0.09]

What you do from here is select the most probable cause (corona) and that is your final class.

If you do multilabel you can get something like this too
Corona [0.9]
Sars [0.20]
HIV [0.999]

Meaning that even if it has a high probability of getting corona it also has a higher probability of having HIV. In multiclassification is different from multilabel where you can have a patient that has both of the sickness

From the sklearn documentation:

  The one-vs-the-rest meta-classifier also implements a predict_proba method,
  so long as such a method is implemented by the base classifier. This method
  returns probabilities of class membership in both the single label and
  multilabel case.  Note that in the multilabel case, probabilities are the
  marginal probability that a given sample falls in the given class. As such, in
  the multilabel case the sum of these probabilities over all possible labels
  for a given sample will not sum to unity, as they do in the single label
  case.

In summary, if sickness are independent it will predict the probability of the patient having one sickness, there has to be a class not sick just if there is the possibility for the patient not being sick

Note: In NN is different


Converting ML Azure random forest to Python - what is "Number of random splits per node"? I want to convert a random forest model from ML Azure to Python (using SKlearn.ensemble.RandomForestRegressor. However there is a hyperparameter called "Number of random splits per node". There is no equivalent in SKlearn's random forest implementation.

Do you have any idea what it might refer to in SKlearn's random forest?

From the Notes section in your link above: "The features are always randomly permuted at each split."

So using the max_features parameter should be analogous to "number of random splits per node".


Predicting probability of occurrence using Historical Time Series Data If you are given 1000 two digits numbers in a row.
You have to find 5 most probable numbers that would appear at 1001th place by using historical pattern.
(If we can find probability of occurrence also, it would be really great)

Without having more context this is how I would approach this -
Assuming there is not meaning to being 'close' in the prediction. I.E, if real 1001 number was 42, then predicting 41 and 99 is just as wrong.
If we assume no stochastic structure to the series, I.E previous number dosnt affect current number I would use the top 5 most frequent numbers as predictions.
If we assume there is a stochastic structure to the series, under our assumption that difference between numbers is not a factor. We can treat this problem as a sequence prediction problem. This is common in text analysis and there are various models that can be used.
Assuming there is a meaning of being 'close' to the prediction.
IF no stochastic nature, use the 5 percentiles based on 1000 values series as predictions.
If yes stochastic nature, use time series modelling. AR(IMA) modelling perhaps.


Predicting what time will a John go to the gym tomorrow? I have a data set which contains the below columns:

Name    Day      Start_Gym_Time   End_Gym_Time
John    Monday   16:00            17:30
John    Tuesday  14:00            14:42
John    Tuesday  15:55            17:40

I need some advice on how to predict when will John go to the gym next? I am planning to use scikit-learn for trying out various algorithms? Any help would be highly appreciated.

I would start by plotting some of the raw data. You might look at histograms of start times/stop times to see what the range of values are. You could also plot the start times as a time series to see if there are any long range patterns (i.e maybe as the week goes on John starts getting tired and showing up at the gym later).


naive bayes classifier for non-binary feature values Given a training set $\{{(x^{(i)},y^{(i)});i=\{1,...,m}\}\}$ where $x^{(i)}\in\{1,2,...s\}^n$ and $y^{(i)}\in{0,1}$. We model the label as a biased coin with $\theta_0=P(y^{(i)}=0)$ and $1-\theta_0=P(y^{(i)}=1)$. We model each non-binary feature value $x_j^{(i)}$ (an element of $x^{(i)}$) as a biased dice for each class. This is parametrized by:

$$P(x_j=k|y=0) = \theta_{j,k|y=0}, k = 1, \ldots, s-1;$$

$$P(x_j=s|y=0) = \theta_{j,s|y=0}=1-\sum_{k=1}^{s-1}\theta_{j,k|y=0};$$

$$P(x_j=k|y=1) = \theta_{j,k|y=1}, k = 1, \ldots, s-1;$$

$$P(x_j=s|y=1) = \theta_{j,s|y=1}=1-\sum_{k=1}^{s-1}\theta_{j,k|y=1};$$

[![enter image description here][1]][1]

(1) Using the Naive Bayes assumption, write down the joint probability of the data:
$$P(x^{(1)},\ldots,x^{(m)},y^{(1)},\ldots,y^{(m)})$$
in terms of the parameters $\theta_0, \theta_{j,k|y=0},$ and $\theta_{j,k|y=1},$ Using the indicator function 1(.) may be useful

(2) Maximizing the joint probability you get in (1) with respect to $\theta_0, \theta_{j,k|y=0},$ and $\theta_{j,k|y=1},$, write down your resulting $\theta_0, \theta_{j,k|y=0},$ and $\theta_{j,k|y=1},$ and show intermediate steps. Comment on the meaning of your results.

My attempt:

I know that $P(Y,X_1,X_2,\ldots,X_d)=P(Y)\cdot \Pi_{i=1}^{d}{P(X_i|Y)}$. To expand this to non-binary, I'm not sure how to proceed. Any help is appreciated.

\begin{align}
&P(x^{(1)}, \ldots, x^{(m)}, y^{(1)}, \ldots, y^{(m)})\\
&=\prod_{i=1}^m P(x^{(i)},y^{(i)})\\
&= \prod_{i=1}^m P(y^{(i)})P(x^{(i)}|y^{(i)})\\
&= \prod_{i=1}^m \left[\textbf{1}(y^{(i)}=1)(1-\theta_0)\prod_{j=1}^nP(x^{(i)}_j|y^{(i)}=1)+\textbf{1}(y^{(i)}=0)\theta_0\prod_{j=1}^nP(x^{(i)}_j|y^{(i)}=1)\right]\\
&=(1-\theta_0)^{\sum_{i=1}^m\textbf1(y^{(i)}=1)}\theta_0^{\sum_{i=1}^m\textbf1(y^{(i)}=0)}\prod_{j=1}^n\prod_{k=1}^s \theta_{j,k|y=1}^{\sum_{i=1}^m\textbf1(x_j^{(i)}=k, y^{(i)}=1)}\theta_{j,k|y=0}^{\sum_{i=1}^m\textbf1(x_j^{(i)}=k, y^{(i)}=0)}
\end{align}

Let's take the logarithm of the expression above and we want to maximize the following

\begin{align}&\sum_{i=1}^m \textbf1(y^{(i)}=1)\log(1-\theta_0)+\sum_{i=1}^m \textbf1(y^{(i)}=0)\log(\theta_0)+\\& \sum_{j=1}^n \sum_{k=1}^s \sum_{i=1}^m\left[ \textbf1(x_j^{(i)}=k, y^{(i)}=1) \log \theta_{j,k|y=1} + \textbf1(x_j^{(i)}=k, y^{(i)}=0) \log \theta_{j,k|y=0}\right]\end{align}

subject to the condition that

$$\sum_{k=1}^s \theta_{j,k|y=1}=1$$
$$\sum_{k=1}^s \theta_{j,k|y=0}=1$$

as well as the nonnegative constraints.

Let the Langrange multiplier for the sum to $1$ constraint be $\lambda_{j,1}$ and $\lambda_{j,0}$.

The Langrangian that we want to optimize is

\begin{align}&\sum_{i=1}^m \textbf1(y^{(i)}=1)\log(1-\theta_0)+\sum_{i=1}^m \textbf1(y^{(i)}=0)\log(\theta_0)+\\& \sum_{j=1}^n \sum_{k=1}^s \sum_{i=1}^m\left[ \textbf1(x_j^{(i)}=k, y^{(i)}=1) \log \theta_{j,k|y=1} + \textbf1(x_j^{(i)}=k, y^{(i)}=0) \log \theta_{j,k|y=0}\right]\\
&-\sum_{j=1}^n \lambda_{j,1}\left(\sum_{k=1}^s \theta_{j,k|y=1}-1\right)-\sum_{j=1}^n \lambda_{j,0}\left(\sum_{k=1}^s \theta_{j,k|y=0}-1\right)\end{align}

Differentiate with respect to $\theta_0$ and equate it to zero, we get

$$-\frac{\sum_{i=1}^m\textbf1(y^{(i)}=1)}{1-\theta_0}+\frac{\sum_{i=1}^m\textbf1(y^{(i)}=0)}{\theta_0}=0$$

Solving it which is just solving a linear equation, we can check that $\theta_0=\frac{\sum_{i=1}^m\textbf1(y^{(i)}=0)}{m}$.

We can also differentiate with respect to $\theta_{j,k|y=1}$,

$$ \frac{\sum_{i=1}^m\textbf{1}(x_j^{(i)}=k, y^{(i)}=1)}{\theta_{j,k|y=1}}=\lambda_{j,1}$$

That is fixing $j$,$\theta_{j,k|y=1}$ is proportional to   $ \sum_{i=1}^m\textbf{1}(x_j^{(i)}=k, y^{(i)}=1)$. With the sum to $1$ constraint, we have

$$\theta_{j,k|y=1}=\frac{\sum_{i=1}^m\textbf{1}(x_j^{(i)}=k, y^{(i)}=1)}{\sum_{k=1}^s\sum_{i=1}^m\textbf{1}(x_j^{(i)}=k, y^{(i)}=1)}$$

I will leave the last part as an exercise.


Muti-hot encoding vs Label-Encoding I am learning about different input-vector representations for Neural Networks

One of the alternatives to sparse One-Hot encoded vector is the Multi-Hot encoding.

Do I understand correctly that a traditional binary approach to counting numbers is exactly what the Multi-Hot is?  We can imagine a byte as a vector of 8 components, and each entry is either 0 or 1.0

Second part of the question:

I would like to use Multi-Hot encoding to express up to 255 possible input values. If I use the binary-approach, will this be identical to Label-encoding? In a sense that network will figure out that 00000010 is "superior" to 00000001, that there is a "strong correlation and precedence"?

Or is it less exaggerated? For example, in Label-encoding, I could merely use 1 input neuron, just with a varied strength of 0 to 255, much like an enum. There the effect would be really exaggerated, as it presents a really obvious precedence of value $A$ over value $A-1$

This means I can't use it for 255 distinct categories (or for relatively unrelated categories)

Is the effect as bad in Multi-Hot encoding, in particular in a binary approach?

If it's the same as Label-Encoding (which only uses 1 input neuron), why would people ever consider Multi-hot, bloating the dimension of the input-vector?

This post points out we can use multi-hot when the input should contain $N$ concatenated one-hot vectors. For example to represent $N$ entities each of which can belong to $Z$ distinct categories.  Is there also another use?

You can think of binary encoding as a compromise between label encoding and one-hot encoding. For distinct categories, label encoding introduces a false linear order that brings a lot of noise into the model (category 1 < category 2 < category 3....) . Binary encoding introduces false additive relationships between the categories (e.g. category 4 + category 1 = category 5 or 100 + 001 = 101) but fewer of them.

Therefore, binary will usually work better than label encoding, however only one-hot encoding will usually preserve the full information in the data.

Unless your algorithm (or computing power) is limited in the number of categories it can handle, one-hot encoding will be preferred over other encoding schemes. If you are limited, mean encoding is a powerful alternative because it transform a categorical feature into a numeric one (giving you the minimal number of inputs) while preserving the most important information in the data.

[Mean encoding replaces every category with its target mean. The mean encodings need to be constructed carefully on a separate dataset to avoid data leakage. If you want to reduce your input dimensions as much as possible, this can be helpful. Mean encoding can also be helpful as an additional feature and is very popular on Kaggle to squeeze out some extra performance. This is also known as target encoding or likelihood encoding.]


Kernel Density in Scikit Learn I'm trying to understand how does the KernelDensity class in scikit-learn work. Consider the following two cases which build a kernel from two different arrays (a). I'm wondering why the result of scoring on the array b in both cases is the same? Shouldn't a kernel from 10 same points be different from that from 2 points? The one from the 10 points should indicate more density. So, why the final scoring result is the same in both cases?
case A:
a = np.array([[1],[1],[1],[1],[1],[1],[1],[1],[1],[1]])

kde = KernelDensity(bandwidth=0.1)

kde.fit(a)

b = np.array([[1]])

log_dens = kde.score_samples(b)

print('Probability is: {}'.format(np.exp(log_dens)))

Probability is: [3.9894228]
case B:
a = np.array([[1],[1]])

kde = KernelDensity(bandwidth=0.1)

kde.fit(a)

b = np.array([[1]])

log_dens = kde.score_samples(b)

print('Probability is: {}'.format(np.exp(log_dens)))

Probability is: [3.9894228]

First of all, the score_samples function of SKLearn's Kernel Density object returns the log of the probability density, not of probability. Therefore, its exponent isn't exactly probability - e.g. in your example you have a probability above 1, which can't be.
Second, apparently the log-probability-density is normalized by the number of points the kernel was trained on (see line 210 here, where log_density -= np.log(N).
This effectively changes the expression $\rho_K(y) = \sum_{i=1}^{N} K(y - x_i; h)$ (from the User Guide) to $\rho_K(y) = {1 \over N} \sum_{i=1}^{N} K(y - x_i; h)$, i.e. turning the sum across points in the train set to an average across the train set.
By the way, I think that this is what they meant to say in this line in the function's description:

This is normalized to be a probability density, so the value will be low for high-dimensional data.

But I agree that it's a bit unclear.


Correlation between examples I'm training a Neural Network for pattern recognition. I have a matrix of examples of size ($N$x$4$) with $N$ examples and $4$ variables.

When I train the Network the Number of examples used for training vs. Cross-entropy, the performance of the Network it's not improving when adding new examples.

I suspect that the examples are highly correlated between them and adding new examples is not giving new information to the network, but I don't figure how to measure that correlation.

Each row of the matrix can be labeled $\mathbf{X}_{i}$. Suppose zero mean, the covariance matrix between examples would be $\mathbf{R}_{ij} = E[\mathbf{X}_{i} \mathbf{X}_{j}^T]$.
I can also calculate the covariance matrix of $\mathbf{X}$, whose elements are $\mathbf{R}_{ij}$ but in that case I would have a matrix of $\sim 300000$x$300000$.

Is there a way of calculate some average correlation or an other measure of correlation between examples?.

Thanks.

EDIT: I don't want to know the correlation matrix of variables, I want to know the correlation between examples.

EDIT2: This is the learning curve:

When I train the Network the Number of examples used for training vs. Cross-entropy, the performance of the Network it's not improving when adding new examples.

You are assuming improving new samples will always improve the performance. This is not true in many cases. Did you plot a learning curve to see how the performance is improving with respect to increasing number of samples? You probably should try to do this before delving deep into sample correlation.


When it is time to use Hadoop? Hadoop is a buzzword now. A lot of start-ups use it (or just say, that they use it), a lot of widely known companies use it. But when and what is the border? When person can say: "Better to solve it without Hadoop"?

It's an economic calculation, really.  When you have a computing "problem" (in the most general possible sense) that you can't solve with one computer, it makes sense to use a cluster of commodity machines when doing so A. allows you to solve the problem, and B. is cheaper than a forklift upgrade to a bigger computer, or upgrading to specialized hardware.

When those things are true, and you are going the "commodity cluster" route, Hadoop makes a lot of sense, especially if the nature of the problem maps (no pun intended) well to MapReduce.  If it doesn't, one shouldn't be scared to consider "older" cluster approaches like a Beowulf cluster using MPI or OpenMP.

That said, the newer YARN based Hadoop does support a form of MPI, so those worlds are starting to move closer together.


What is the intuition of using clustering for performing feature engineering in machine learning tasks? I am trying to implement the research paper Combining Boosted Trees with Metafeature Engineering for Predictive Maintenance. The paper has a section called meta feature engineering where they have used hierarchical clustering to create features. The paper says:

The third method we used to analyze the outliers in the dataset is based on an
hierarchical Agglomerative Clustering algorithm [5].
Hierarchical Agglomerative Clustering starts with Z groups (Z being the
number of observations), each initially containing one object, and then at each
step it merges the two most similar groups until there is only one single group,
containing all data.
The rationale for this method is that the last observation that are merged
might still be significantly different from the group they are merged into. By
definition outliers are different cases and will typically not fit well into a cluster,
unless that cluster is comprised by other outliers itself. Yet again, since these
are not ordinary data points, we do not expect them to form large groups.

I am unable to understand the authors' intuition behind doing this.
The problem I am trying to solve and the paper is related to is the IDA-2016 competition dataset. You can find more about the competition here

Overall the paper is not very clear so there are a few uncertainties, but the general approach is this:

Their main idea is to create new features which represent the "outlyingness" of the instance. They use several different methods in order to detect outliers, however they do not explain how exactly are the new features created.
One of the methods they use to detect outliers is based on hierarchical clustering: the result of such clustering is a binary tree in which the most distant clusters/instances are connected last, i.e. close to the root of the tree. Their assumption is that an outlier tends not to be close to any other instance or cluster, therefore they are connected last. While the method makes sense, it's not clear in the paper whether they only retain the very last instance as outlier or the last few instances (or any other variant).

So the clustering and the feature creation are only indirectly related:

The clustering is used to detect one or several outliers in the instances, and several other methods are used for the same purpose.
Based on the detection of these outliers, one or several new features are created which contain a value describing the "outlier status" of the instance. The simplest option would be to create a single boolean feature which is true if the instance was detected as outlier by any of the methods, but one can imagine more advanced options. For example, based on the hierarchical clustering one can obtain the order in which the instances are connected, and the rank of every instance can be used as a feature.