How to aggregate data where instances occur over different time intervals I am working on a problem in which I have several instances that have predictors that have activity over various different time periods (i.e. <3 months to well over 20 months.) Originally I attempted to use knowledge I have about this problem (it is an opportunity to sale conversion model) and learned that the average time for a deal to close is about 9 months, so I broke my predictors up into three month intervals. However, I took another look at the lengths of these deals and see that there are a variety of instances that have durations that are not even close to 9 months so this idea does not make sense.

The only idea I have gotten is just creating a duration column where I subtract the start and the stop date and then just do the summation for each predictor. However, I feel that the instances might get incorrectly labeled because some might have an overwhelmingly higher amount of activity than another due to the duration of the deal. Has anyone else encountered such a problem. Not sure if this is a common problem but a quick glance at google/reddit did not come up with anything (I could be asking the problem wrong.)

Comment

I had the same problem. you can use aggregation functions. For example use Max, Min, Avg, count, std or some calculation like the slope of the line. In cases which you have a different period, you can divide your value to days of each period.


Complex-Valued input to CNN I want to train a CNN. However, my input is images of size 100*100 with complex numbers. I have runned the model, but it failed and the loss didn't decrease. Then I found out that my because my inputs are complex, they are not able to train very good. Actually, I think the activation functions and the convolutions and the optimizer, backpropagation, cost should be changed. I wonder how I can Implement my CNN with Complex-Valued input. I would be thankful if anyone could help me.

There has been some research in building very specialized deep learning models to use complex numbers using complex algebra, but these tools are not yet user-friendly. So unless you want to go down the rabbit hole and try to understand their implementations and then try to write your own implementation, i would avoid this.

In the past, I worked with MRI data which is a 3D complex-valued matrix. I inspired myself from the work done in color images, where 3 channels are used to represent red (R), green (G) and blue (B). I this separated the real and imaginary part, and treated them as separate channels into the deep learning model all as real numbers.

1D example:

From a complex vector, we get 2 channels where the imaginary part can then be treated as a real number.

$[1+1j, 2+2j, 3+3j] = [1, 2, 3], [1, 2, 3]$


How to transform dictionary data into a string vector? I have key,value data where each record is in a Python string.
An example record looks like this:

record = {
    'first_name': 'john',
    'last_name': 'doe',
    'age': '50',
}

To encode this into a neural net, I would like to firs have this data as a string vector. A sample output is like this:

"first_name john last_name doe age 50"

How can I make this transformation?

If you want output in any order, do this-

>>> string_vector = ''
>>> for k,v in record.items():
        string_vector += k + ' ' + v + ' '

>>> string_vector = string_vector.rstrip(' ')
>>> string_vector
'first_name john last_name doe age 50'


OCR / Text Recognition and Recovery Problem I am working on a research project that deals with American military casualties during WWII. Specifically, I am attempting to construct a count of casualties for each service at the county level. There are two sources of data here, each presenting their own challenges.

1. Army and Air Force data. The National Archives hosts lists of Army and Air Force servicemen killed in action by state and county. There are .gif images of the report available online. Here is a sample for several counties in Texas.

I DO NOT need to recover the names or any other information. I simply need to count the number of names (each on its own line, and listed in groups of five) under each County. There are hundreds of these images (50 states - 30-100 for each state).

I have been unable to find an OCR program that can tackle this problem adequately. How would you suggest I approach this challenge? (I have some programming expertise in Python and Java, but would prefer to use any off-the-shelf solutions that may exist).

2. Navy and Marine Core data. This data is organized differently. Each state has long lists of casualties with the address of their next of kin. Here is a sample for Texas again. For these images, I need to BOTH count the number of dead and recover their hometown, which is typically the last word in each entry. I can then match these hometowns to counties and merge with database 1.

Again, the usual OCR programs have proved inadequate. Any help on this (admittedly more difficult) problem would be very much appreciated.

Thank you in advance experts!

The answer to both data sets is an OCR application with some post-processing, but a more specialized program than a generic low-quality or an open source OCR. Essentially the harder the problem, the more capable and advanced tools need to be used to solve it.

There will be two major stages in this task: digitizing the data (image to text, i.e. OCR) and processing the data (performing the actual count). Look at them separately in order to select the best method for each stage.

The main challenges in these images and generic OCR are:

a) images have low resolution. For example the # 1 image has resolution of about 72 dpi. Suggested resolution for such text quality is to scan at 300 to 400 dpi, but it is clear that re-scanning or controlling scan resolution is not applicable now. That’s why one option is to clean and increase the size using image pre-processing tools. This is what the original #1 image snippet looks like after adaptive binarization and zoomed at 300%. It is clear that each character has too few pixels and characters can be easily misread.

b) GIF format in #1 is not supported by many OCR applications. Images need to be batch-converted to a different format, such as PNG or TIF.

c) in these scans the backgrounds and bleed-through (shadow from the text on the other side of the paper) is visible. Good binarization needs to be used to remove background and bleed-through, but not remove vital parts of actual characters.

After implementing specific pre-processing solutions for the items listed above, and then using a high quality OCR system, such as www.ocr-it.com API, highest possible results can be achieved.  Result is far from perfect, but it is as high accuracy as it could be achieved with a modern OCR engine on these images.

Luckily for this project, the data needs to be counted, so the second stage has all necessary data for reliable data post-processing analysis. Contrary to other basic OCR engines, the OCR provided by www.ocr-it.com API, which I used to produce the above recognition.  OCR-IT API is free to develop with, and very inexpensive to use per page, so that may be a very economical but powerful solution for this project.  It returns formatted text layout, including preserving line breaks and overall format structure.  This makes text post-processing easier.

A simple algorithm can be run to count the number of lines, resulting in the necessary for the research count.

The above describes a two-stage approach: getting best possible OCR result, and using an applicable method to process data for the required task

Bat wait, there is more…

There is a second option to use an even more specialized OCR application called FlexiCapture with FlexiLayout technology.  This powerful and intelligent data capture technology has built-in high-accuracy OCR, and it has a powerful rules and data analytics engine to perform very specialized user-defined chains of actions and tasks.

The implementation of this method using FlexiCapture with FlexiLayout takes the following logical steps.

First, full page OCR is performed and all objects are extracted, including characters, noise, black horizontal and vertical lines, white gaps, and objects (which could be pictures, logos, handwriting, etc.).  This produces objects upon which we can apply our search criteria.

Next, the following constraints are applied to the post-OCR data analysis and search criteria: separate image into three vertical columns and run the following logic per column, use line-start as individual count, skip header/footer/indented lines (county names), assume each name to have at least three characters, find recursively every name starting from top to bottom in every column, exclude previously found lines.

While the above logic sounds complex to setup, the actual setup takes just a few minutes and requires minimal work through user interface (UI) environment.  No coding or programming is necessary.   The following search elements and criteria have been created.

RepeatingGroup consisting of a CharacterString search object.

This setup produces the following search result for the first column of data:

As the last step, FlexiCapture is instructed to return the number of total found elements that fit our search criteria, effectively producing the necessary data for the research task.

There are other logic alternatives that can be setup in FlexiCapture, such as finding the number of white spaces between lines, or searching for the fixed-length fixed-placement 3-letter combinations at the end of every column linen.

In conclusion, there are several options (which is always nice) how this task can be achieved with relative ease in effort and high quality, but the success depends on the quality of tools used and necessary knowledge how to use them.

If you believe some of these tools and processes can be beneficial to your project, please contact my directly.  I specialize in these workflows.  Ilya @ WiseTREND.  My company may be able to help with the setup or guidance.  We have participated in various research initiatives, some through donations to a good cause.


Clustering without information about identifier I have a data-set with different products and binary value if it was sold in a store or not. I looks like:

product_id  store_1 store_2 store_3 store_4 store_5 store_6
0   A   1   0   0   1   0   1
1   B   1   1   0   0   1   0

Is there any way to cluster these products with any information about the products itself?

One thought I had was to generate distance between products and then cluster the product X product matrix.
Is this problem sort of similar to market basket analysis?

Any guidance will help.

1) About clustering the products with no product information
Clustering is always possible but not always what you expected
You can cluster the products based on your dataset using the Hamming distance (if values can only be 0 or 1). Two products separated by a small distance are products that are likely to be sold in the same shops. However, there is no assurance that these products will be similar in terms of their characteristics.
Example

expensive ham (A) & cheap ham (B)
expensive cheese (C) & cheap cheese (D)

It is likely that shops located in wealthy neighborhoods (Shop1) will sell expensive ham and expensive cheese. It is also likely that shops in modest neighborhoods (Shop2) will sell more cheap cheese and cheap ham.
  Shop1 Shop2
A   1     0      (Exp. Ham)
B   0     1      (Cheap Ham)
C   1     0      (Exp. Cheese)
D   0     1      (Cheap Cheese)

Based on this :

A and C are very similar (distance = 0)
B and D are very similar (distance = 0)

So the clustering algorithm will (in my example) cluster products, not on their descriptions (ham vs cheese) but based on the prices. (because of shops location in this case)
I have highlighted the price-shop relation, but it is possible that some shops are clothing-only stores and thus products sold in those will likely be clothing.
Conclusion: you will discover clusters of products that are sold in the same stores but it will be difficult to understand why without further information on the shops.
2) About the (Product x Product) distance matrix clustering
The points on the resulting matrix are product duo. AB, BC, DA, etc.
The value of the points is the distance between the products of the duo. So clustering this would result in clusters of duos that have the same similarity. I don't think this makes sense.
Hope this will help.


Seeking advice on knowledge discovery Background Information

I work for a fire department in Florida and the fire chief posed a question to me; At any given moment in time during the calendar year 2018, how many fire trucks are busy,  how many are available, how many are standing by and how many are out of service? Using the SQL Server database I was able to distill this information into tabular data.

Software available to me.

Also, because the data is in SQL Server I can parse out the date time stamp by hour, weekday, month, etc. Then Using Python or Excel I can generate statistical tests and univariate statistics and charts. I have modest experience with R. Therefore my tools are SQL Server, R, Python and Excel.

The data

For the first record with the primary key 21546912 at the time of january 1, 2018 at 00:02:31.800 there was 1 fire truck working, 9 fire trucks are available for work and 0 are standing by or out of service. For the entire year I have 104,179 observations showing at exactly that time stamp how many are working, available or standing by or out of service.
I have enclosed 30 records as a comma separated values file below my question for you to see a sample.

The Question !

So what do I do with this data? I have already generated histograms showing the frequency of FireTrucksWorking, etc. Is there a statistical test such as regression that I can perform on this data? How do I bring meaning to this dataset? Are there any general patterns to the dataset that I can discover? Can I create a probability model out of this data such as; at 08:00 during the month of July there is x% probability of N fire trucks working ? Would you use queueing theory on this dataset?

I am open to all suggestions. At this point I have a large dataset but it is really just a jumble of numbers, how do I generate meaning out of these numbers?

This is cross-posted in Mathematics Exchange. But here I make the following revision.
I am looking for some type of knowledge discovery that will show me something about my data set the naked eye cannot see or the human brain cannot discern. For example: "there is a much higher likelihood of working during the hours of 08:00 to 12:00 than the rest of the day". "Are there patterns in my data I do not recognize"?
Do you have any thoughts on this?

Thirty sample records of data

PrimaryKey,DateTimeStamp,FireTrucksWorking,FireTrucksAvailableForWork,FireTrucksStandingBy,FireTrucksOutOfService
21546912,2018-01-01 00:02:31.800,1,9,0,0
21546921,2018-01-01 00:04:46.720,1,9,0,0
21546950,2018-01-01 00:09:39.400,1,9,0,0
21546951,2018-01-01 00:09:47.320,2,8,0,0
21546955,2018-01-01 00:11:16.780,3,7,0,0
21546959,2018-01-01 00:12:04.840,2,8,0,0
21546962,2018-01-01 00:12:09.030,3,7,0,0
21546963,2018-01-01 00:12:14.470,3,7,0,0
21546966,2018-01-01 00:12:17.790,3,7,0,0
21546967,2018-01-01 00:12:21.240,2,8,0,0
21546970,2018-01-01 00:12:40.240,2,8,0,0
21546973,2018-01-01 00:12:46.720,2,8,0,0
21546990,2018-01-01 00:14:24.610,2,8,0,0
21547002,2018-01-01 00:16:03.130,2,8,0,0
21547036,2018-01-01 00:19:59.450,1,9,0,0
21547043,2018-01-01 00:21:21.950,0,10,0,0
21547064,2018-01-01 00:24:50.470,0,10,0,0
21547065,2018-01-01 00:25:13.000,0,10,0,0
21547165,2018-01-01 00:43:31.130,0,10,0,0
21547344,2018-01-01 01:15:00.980,1,9,0,0
21547361,2018-01-01 01:16:58.320,1,9,0,0
21547383,2018-01-01 01:21:38.130,1,9,0,0
21547421,2018-01-01 01:30:42.250,0,10,0,0
21547436,2018-01-01 01:33:30.320,1,9,0,0
21547442,2018-01-01 01:33:48.470,1,9,0,0
21547449,2018-01-01 01:33:58.780,1,9,0,0
21547474,2018-01-01 01:37:30.550,1,9,0,0
21547484,2018-01-01 01:39:12.350,1,9,0,0
21547487,2018-01-01 01:40:41.290,0,10,0,0
21547510,2018-01-01 01:47:02.420,0,10,0,0

There are a lot of different possibilities. It is a bit hard to get the message through here in the forum. But I try and focus on one thing that is relatively easy to implement:

One of the easiest things you can do is to do a linear regression. You dependent variable (y) would be „fire trucks working“ (or the share of working ones). You can estimate this dependent on time. So you can (for example) generate indicator variables (=1 if something is true vs =0 if not) which indicate things like weekday, hour of the day. Use this indicators as explanatory (x) variables in the regression.

The result will be, that each indicator shows you the average truck use at some point in time. Say indicator A is „monday“, B is „16:00“, you will see how many trucks are on average dispatched in this time window. You can also easily see if the difference is significant or not.

In the past such models were used to estimate the number of callers in callcenters. The thing is called „dummy variable regression“ (at least in some statistic fields). https://en.m.wikipedia.org/wiki/Dummy_variable_(statistics)

Note that you observe effects from each „dummy“ variable in contrast to a baseline. Imagine you have truck use (y) and only 1 indicator/dummy, which is „monday“. Say the intercept of the model (the result with all dummies =0) is 5, meaning on average 5 trucks are employed. If the effect/coefficient of your dummy is 2, it means there are 2 more trucks employed (on average) on mondays. So each dummy/indicator (viz in case it is 1) shows you an effect in contrast to the base category (where all the dummies are 0).

Here is some script in Python which I guess refers to a similar problem, maybe it helps for orientation: https://stackoverflow.com/questions/50733014/linear-regression-with-dummy-categorical-variables

Also this may help: https://songhuiming.github.io/pages/2017/01/21/linear-regression-in-python-chapter-3-regression-with-categorical-predictors/

Another note: In principle I would try to get many different dummies/indicators from the timestamp, e.g weekday, hour of day, special days or seasons (Christmas, winter etc). You need to try combinations which work for you and the model. But don‘t plug too many indicators into the model. Keep things easy in a first step and try to improve the model later.

In order to check the „quality“ of your model you can look at R2 (a measure that shows how well you are able to explain the data which is between 0,1). You can also randomly (!) set aside some 10-20% of the data (do not use them to estimate the model in the first place) to check how well you model can explain real truck use at some point in time.

I would use R for such a task (I find it more intuitive than Python). Here is some example code for R: https://stats.idre.ucla.edu/r/modules/coding-for-categorical-variables-in-regression-models/

Hope this helps: Good luck with your interesting project!


Losses of keras CNN model is not decreasing I am working on Street view house numbers dataset using CNN in Keras on tensorflow backend. I have queries regarding why loss of network is not decreasing, I have doubt whether I am using correct loss function or not. First I preprocess dataset so  my train and test dataset shapes are:

('Training set and labels', (26721,32,32,1), (26721, 6) )
('Validation set and labels ', (6680,32,32,1), (6680,6) )
('Test set and labels', (13068,32,32,1), (13068,6) )

Reason why labels array has 6 columns, because maximum digits in one image is 6. For suppose, if image has 2 digits "1 and 2", then labels array is considered as [1,2,10,10,10,10] , where 10 represents no digits.

Now this is my model definition:

def modelCNN(input_shape, num_classes):

  model = Sequential()

  model.add(Convolution2D(16, 7, 4, border_mode='same',
                    input_shape=input_shape))
  model.add(PReLU())
  model.add(MaxPooling2D(pool_size=(3, 3), strides=(2, 2), border_mode='same'))
  model.add(BatchNormalization())

  model.add(Convolution2D(32, 5, 3, border_mode='same'))
  model.add(PReLU())
  model.add(MaxPooling2D(pool_size=(3, 3), strides=(2, 2), border_mode='same'))
  model.add(BatchNormalization())

  model.add(Convolution2D(64, 3, 3, border_mode='same'))
  model.add(PReLU())
  model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2), border_mode='same'))

  model.add(Flatten())
  model.add(Dense(512))
  model.add(PReLU())
  model.add(Dropout(0.5))
  model.add(Dense(num_classes))
  model.add(Activation('softmax'))

return model

I tried with different customs model, but nothing seems working. I do compilation, loss and optimization like this:

model.compile(optimizer='adam', loss='categorical_crossentropy',
                                    metrics=['accuracy'])

print(model.summary())
csv_logger = CSVLogger('training.log')
early_stop = EarlyStopping('val_acc', patience=200, verbose=1)
model_checkpoint = ModelCheckpoint(model_save_path,
                                    'val_acc', verbose=0,
                                    save_best_only=True)

model_callbacks = [early_stop, model_checkpoint, csv_logger]
K.get_session().run(tf.global_variables_initializer())

model.fit_generator(train,
          samples_per_epoch=np.ceil(len(train_dataset)/batch_size),
          epochs=num_epochs,
          verbose=1,
          validation_data=valid,
      validation_steps=batch_size,
          callbacks=model_callbacks)

Now, What I think that categorical_crossentropy loss needs one hot vector representation, but I am not sure how should i convert my labels array. Because unlike Mnist dataset, I have multiple labels in each image.

Is there any loss function that I can use? or Any suggestion to make things work?

Note: I am using CNN just for classification only, not for detection.

Thanks very much in advance

you can try to remove some of the irrelevant features that may be the reason or you could try different solvers and activation methods.I previously answered similar question earlier in the following linklink


Micro Average vs Macro average Performance in a Multiclass classification setting I am trying out a multiclass classification setting with 3 classes. The class distribution is skewed with most of the data falling in 1 of the 3 classes. (class labels being 1,2,3, with 67.28% of the data falling in class label 1, 11.99% data in class 2, and remaining in class 3)

I am training a multiclass classifier on this dataset and I am getting the following performance:

                    Precision           Recall           F1-Score
Micro Average       0.731               0.731            0.731
Macro Average       0.679               0.529            0.565

I am not sure why all Micro average performances are equal and also Macro average performances are low compared to Micro average.

Micro- and macro-averages (for whatever metric) will compute slightly different things, and thus their interpretation differs. A macro-average will compute the metric independently for each class and then take the average (hence treating all classes equally), whereas a micro-average will aggregate the contributions of all classes to compute the average metric. In a multi-class classification setup, micro-average is preferable if you suspect there might be class imbalance (i.e you may have many more examples of one class than of other classes).

To illustrate why, take for example precision $Pr=\frac{TP}{(TP+FP)}$. Let's imagine you have a One-vs-All (there is only one correct class output per example) multi-class classification system with four classes and the following numbers when tested:

Class A: 1 TP and 1 FP
Class B: 10 TP and 90 FP
Class C: 1 TP and 1 FP
Class D: 1 TP and 1 FP

You can see easily that $Pr_A = Pr_C = Pr_D = 0.5$, whereas $Pr_B=0.1$.

A macro-average will then compute: $Pr=\frac{0.5+0.1+0.5+0.5}{4}=0.4$
A micro-average will compute: $Pr=\frac{1+10+1+1}{2+100+2+2}=0.123$

These are quite different values for precision. Intuitively, in the macro-average the "good" precision (0.5) of classes A, C and D is contributing to maintain a "decent" overall precision (0.4). While this is technically true (across classes, the average precision is 0.4), it is a bit misleading, since a large number of examples are not properly classified. These examples predominantly correspond to class B, so they only contribute 1/4 towards the average in spite of constituting 94.3% of your test data. The micro-average will adequately capture this class imbalance, and bring the overall precision average down to 0.123 (more in line with the precision of the dominating class B (0.1)).

For computational reasons, it may sometimes be more convenient to compute class averages and then macro-average them. If class imbalance is known to be an issue, there are several ways around it. One is to report not only the macro-average, but also its standard deviation (for 3 or more classes). Another is to compute a weighted macro-average, in which each class contribution to the average is weighted by the relative number of examples available for it. In the above scenario, we obtain:

$Pr_{macro-mean}={0.25·0.5+0.25·0.1+0.25·0.5+0.25·0.5}=0.4$
$Pr_{macro-stdev}=0.173$

$Pr_{macro-weighted}={0.0189·0.5+0.943·0.1+0.0189·0.5+0.0189·0.5}={0.009+0.094+0.009+0.009}=0.123$

The large standard deviation (0.173) already tells us that the 0.4 average does not stem from a uniform precision among classes, but it might be just easier to compute the weighted macro-average, which in essence is another way of computing the micro-average.


What are some good models to complement XGBOOST in stacking? What are some good models to complement XGBOOST via stacking in typical Kaggle datascience competition? I realize XGBoost with well-tuned hyperparamters are generally quite good already.

It really depends on how other models behave on your particular dataset, but generally, you should check 2 things:

Predictions from ensembled models are not highly correlated and other models also perform well on your dataset.
It's better to use a model from another family. So if you are using XGboost, its better to add some model like SVM than a catboost (because you want a model with different approach in solving the data, rather then a model with similar approach).

I think this article may give you additional insights about the topic
https://machinelearningmastery.com/stacking-ensemble-machine-learning-with-python/


How should i build a Knowledge Graph for a custom dataset? I'm new to machine learning and i'm trying to create a small Knowledge Graph for search purposes similar to google for a class project.

Okay, so i have been searching on this topic for few days and this is what i have found from the web and research papers.

Create RDF triples or use already existing database like Freebase, Wikidata etc.
Then train model using some algorithm like ComplEx, TransE etc.
And finally use it for the queries.

My problem is i don't want to use an already existing database. I have a set of documents with me. Is there any good library for making triples from custom data?
Also after training my model, which database should i use to store my model and how to query it back for answers.

Thank you.

You can build a Knowledge Graph from existing data sources too using
RML.
RML stands for RDF Mapping Language and allows you to transform heterogeneous
data sources into RDF and extends R2RML.
I created an example with JSON data as a data source, but other formats are
possible such as CSV, XML, relational databases, etc.

RML (RDF Mapping Language)

Turning your existing data sources into RDF goes as followed with RML:

Write RML mapping rules which instruct an RML processor how the data should
be transformed into RDF.
You can find many example in the RML documentation how
to write your mapping rules for your data sources.

RML rules consist of Triples Maps
which on its own turn have the following parts:

Logical Source: How the data source
should be accessed.
Subject Maps: How a subject of a triple
should be created.
Predicate-Object Maps: How a predicate
and its object should be created.

Logical Source

rml:logicalSource [
    rml:source "people.json" ;
    rml:referenceFormulation ql:JSONPath ;
    rml:iterator "$.people.[*]" ;
] ;

The data source people.json is accessed using JSONPath expressions defined as
$.people.[*].
The expression allows the RML processor to iterate over the JSON data.

Subject Maps

rr:subjectMap [
    rr:template "http://ex.com/Person/{firstname}_{lastname}" ;
    rr:class foaf:Person ;
] ;

Every subject created by this SubjectMap will look like
http://ex.com/Person/{firstname}_{lastname} where firstname and lastname
are replaced by the corresponding JSON values during the execution of an RML
processor. The subject has the class foaf:Person.

Predicate-Object Maps

rr:predicateObjectMap [
        rr:predicate foaf:givenName ;
        rr:objectMap [
            rml:reference "firstname" ;
        ]
    ] ;

This map generates a predicate foaf:givenName where the object will receive
the JSON value firstname during the mapping process.

Execute your mapping rules with an RML processor.
An example of an RML processor is the
RML Mapper or the
RML Streamer.
Other RML processors can be used too, if they comply with the
RML specification.

An RML processor will generate the following triples from the JSON data based on
the mapping rules shown previously:

<http://ex.com/Person/John_Doe> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://xmlns.com/foaf/0.1/Person>.
<http://ex.com/Person/John_Doe> <http://xmlns.com/foaf/0.1/givenName> "John".
<http://ex.com/Person/John_Doe> <http://xmlns.com/foaf/0.1/familyName> "Doe".
<http://ex.com/Person/Jane_Smith> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://xmlns.com/foaf/0.1/Person>.
<http://ex.com/Person/Jane_Smith> <http://xmlns.com/foaf/0.1/givenName> "Jane".
<http://ex.com/Person/Jane_Smith> <http://xmlns.com/foaf/0.1/familyName> "Smith".
<http://ex.com/Person/Sarah_Bladinck> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://xmlns.com/foaf/0.1/Person>.
<http://ex.com/Person/Sarah_Bladinck> <http://xmlns.com/foaf/0.1/givenName> "Sarah".
<http://ex.com/Person/Sarah_Bladinck> <http://xmlns.com/foaf/0.1/familyName> "Bladinck".

Full example

I created a small demo to create a FOAF Person with a first and last name:

RML Mapping rules

@base <http://example.com> .
@prefix rml: <http://semweb.mmlab.be/ns/rml#> .
@prefix rr: <http://www.w3.org/ns/r2rml#> .
@prefix ql: <http://semweb.mmlab.be/ns/ql#> .
@prefix foaf: <http://xmlns.com/foaf/0.1/> .

<#PersonMapping>
    a rr:TriplesMap ;
    rml:logicalSource [
        rml:source "people.json" ;
        rml:referenceFormulation ql:JSONPath ;
        rml:iterator "$.people.[*]" ;
    ] ;

    rr:subjectMap [
        rr:template "http://ex.com/Person/{firstname}_{lastname}" ;
        rr:class foaf:Person ;
    ] ;

    rr:predicateObjectMap [
        rr:predicate foaf:givenName ;
        rr:objectMap [
            rml:reference "firstname" ;
        ]
    ] ;

    rr:predicateObjectMap [
        rr:predicate foaf:familyName ;
        rr:objectMap [
            rml:reference "lastname" ;
        ]
    ] .

JSON data

{
    "people": [
        {
            "firstname": "John",
            "lastname": "Doe"
        },
        {
            "firstname": "Jane",
            "lastname": "Smith"
        },
        {
            "firstname": "Sarah",
            "lastname": "Bladinck"
        }
    ]
}

Note: I contribute to RML and its technologies.


Traversing trees in SQL: JOINs vs imperative algorithm I have a table representing posts on a message board. Posts may or may not have parents.

What is the most common way to get all posts starting from a given post, or to find the root post of any given post?

I can think of using JOINs to join parents and children, but how can I know how many joins I need before I stop?

There are several models for holding hierarchical data in a relational database. There's a good summary here. In summary they are

Adjacency list: each row contains a foreign key to its parent. Fast for structure changes. Requires recursive processing to find the base ancestor.
Path enumeration: each node stores its ancestry. An update requires all children to be updated. For a known node reaching the root requires no further work.
Nested sets: additional columns hold the ancestry. The cost of structure changes depends on how big the moving subtree is. The root node is fast to find.
Closure table: store all ancestor-descendant pairs. Structure changes can be expensive. Root node expensive to identify.

From the given information a path enumeration sounds like your best option.


How to do multivariable regression in Orange3? Orange3 contains a number of regression widgets, but they all seem to be univariable i.e. one independent variable that correlates to one dependent variable. When I have more independent variables that might influence a dependent variable, how to handle this in Orange3?

The regression models are multivariable. You can go to the training videos of Orange to learn how to do regression.


I am a programmer, how do I get into field of Data Science? First of all this term sounds so obscure.

Anyways..I am a software programmer. One of the languages I can code is Python. Speaking of Data I can use SQL and can do Data Scraping. What I figured out so far after reading soo many articles that Data Science is all about good at:

1- Stats

2- Algebra

3- Data Analysis

4- Visualisation.

5- Machine Learning.

What I know so far:

1- Python Programming
2- Data scrapping in Python

Can you experts guide me or suggest a roadmap to brush up both theory and practical? I have given around 8 months of time frame to myself.

Focus less on gaining skills and more on gaining experience. Try to actually solve some problems and post your work on github. You'll learn more in the process and be able to demonstrate knowledge and experience to employers, which is much more valuable than having a supposedly deep understanding of a topic or theory.

Data Science is a pretty loaded field these days so I'm not sure what kind of work you specifically want to do, but assuming that machine learning is a component of it then kaggle.com is a good place to start. In terms of goals, if you're able to work with the data in pandas/numpy/scipy, build models in sci-kit learn and make some pretty graphs in seaborn, ggplot or even matplotlib then you won't have a problem getting a job from a skills perspective -- especially if you have code samples and examples to demonstrate your abilities. If you get stuck then stackexchange will either have the answer or you can post a questions and you'll have an answer shortly. Once you're doing the work for a living then you'll learn even more, likely from a senior team member who mentors you.

Best of luck.


What are the differences between biased and unbiased learners? I'm new in machine learning and I am willing to know better what is the difference between biased and unbiased learners?
Anyone can provide some examples?

Biased in the context that you are speaking means, that your model overfits the training data and can not generalize well. It means your model performs very well on your training data, but can not do well on cross-validation and test data. It is customary to say that biased learners memorize the training data which is really true. Biased learners don't learn the data, they fit the data. For understanding the other usages of bias take a look at this question.

There is something that may be worth mentioning. You may have heard people saying that your model has a high-bias problem. It just means that your model can not learn the training data, whilst the biased learners overfits the training data, means fits the training data. The latter can not generalize well because it has fitted the training data, memorized it, the former can not generalize because it has not learnt even the training data so it has not learnt so much and can not generalize.


Neural networks, optimization math intuition When I look into the following partial derivative, I see it as being the key element of any optimization algorithm out there. Correct me if I'm wrong, but this gets us the slope of the loss function, so we can go opposite to that slope, therefore minimizing the loss.

$$\frac{\partial \theta}{\partial \mathcal{L}}$$

where: $\theta$ is the weights, and the $\mathcal{L}$ is the loss;

Does that make sense? Is there any other calculation step that is arguably more fundamental to the optimization of neural networks other than this derivative?

This topic is specially important for me right now, because I was thinking of tattoing this derivative, as a cool A.I. tattoo, and I want it to be fundamental and simple.

Note that $\frac{\partial L}{\partial \theta}$ is different from $\frac{\partial \theta}{\partial L}$. What you tried to describe seems to be $\frac{\partial L}{\partial \theta}$ where $\theta$ is a variable.  If $\theta$ is high dimensional, sometimes we just use the $\nabla$ notation.

Gradient descent is

$$\theta_{n+1}=\theta_n-\gamma \nabla L(\theta_n)$$

Not everything is differentiable and gradient might not be well defined for some optimization problem.
In the event that there are constraints, $L$ might need to take the role of Langragian rather than the objective function.
Gradient descent is just a means to find the parameters for a model. Gradient based approach seems to be the norm for now but things can change.

What you proposed to tattoo is just "gradient" or "slope". Not objection but just want to let you know what you are doing.


Estimate correlation in Python I have a dataset with labels and usernames:
Labels   Usernames
1         Londonderry
1         Londoncalling
1          Steveonder43
0         Maryclare_re
1         Patent107391
0         Anonymous
1         _24londonqr
...

It seems that the usernames containing the word London are very frequent in having assigned label=1.
Do you have any idea on how I could proof it?

You could create a second label for your usernames according to whether they contain london or not (pseudocode below):
for idx, username in df['Usernames']:
    if 'London' in username:
        df['London'].iloc[idx] = 1
    else:
        df['London'].iloc[idx] = 0

Consequently given you want to go with correlation and that you are comparing binary variables, the metric to go with is Pearson correlation coefficient (pseudocode below):
import scipy.stats.pearsonr as rho
corr = rho(df['labels'], df['London'])


ground truth fit is worse than cross validated fit on noisy data? I am having these weird results when playing around with cross validation that I would greatly appreciate to have any comments.
Briefly, I have a lower mean squared error (MSE) when doing regression (least-squares) using cross-valitation (CV), than when using the "ground truth weights" that I used to generate the data.
Note however, that I compute the MSE on noisy data (generated data + noise), so MSE of 0 would not be expected for noise levels above 0.
Weirdly, for high noise conditions, I get lower MSE with cross validated least squares than with the "ground" truth weights used to generate the clean data - to which I then add different levels of noise to the input (X). Instead, if I add guassian noise to the output (y) the "ground truth weights" perform better.
More details below.
Simulation of data
I am generating beta from a guassian and X from a uniform distribution. I then compute the to-be-regressed y as y = beta * X.
python 3 code:
def generate_data(noise_frac):
  X = np.random.rand(ntrials,nneurons)
  X = np.random.normal(size=(ntrials,nneurons))

  beta = np.random.randn(nneurons)
  y = X @ beta

  # not very important how I generated noise here
  noise_x = np.random.multivariate_normal(mean=zeros(nneurons), cov=diag(np.random.rand(nneurons)), size=ntrials)

  X_noise = X + noise_x*noise_frac

  return X_noise, y, beta

As you can see I also add noise to X.
Regression
I then project this noised data X_noise for different values of noise onto beta:
y_hat = (X_noise) @ beta

And compute the MSE:
mse = mean((y_hat - y)**2)

As expected, MSE increases with noise (blue line in the figure).

However, I get lower MSE if I use cross validated least-squares! This is now orange line in the figure.
To do CV, I split X_noise in random 100 train and test sets. In broad terms, This is how I do CV in python:
beta_lsq = pinv(X_train) @ y_train
y_hat_lsq = (X_test) @ beta_lsq
mse = mean((y_hat_lsq - y_test)**2)

On the other hand, if I add noise to y, instead of X, then everything makes sense:

Thank you very much in advance!
PS: This is a crosspost from stack overflow

The main problem in your test is that your X has the same scale as your noise level (0-1), as a result, adding a noise changes your data distribution significantly.
This is your data distribution before and after adding noise.

It is like the noise is 50-200% more than your initial data. That's why you get a better result with CV than the "ground truth weights". Because the new fitted model by CV is tuned for an X with a new distribution.
However, if you increase your X values scale, for example 100*np.random.rand(ntrials,nneurons), the your data have same distribution before and after adding noise.

Now if you run your code, you will get what you expected.


Python how to add a condition to the groupby while calculating the median of a field? I need to add a new condition to the following line:

train["Age"].fillna(train.groupby("Ticket")["Age"].transform("median"), inplace=True)

Currently, its taking the median of the age based on tickets class.

I need to add to this groupby("Ticket") another condition, where train["PrSb"]==0.

This should work:

train["Age"].fillna(train.loc[train["PrSb"] == 0].groupby("Ticket")["Age"].transform("median"), inplace=True)

This adds an extra rowwise filter to the dataframe that is used in the groupby operation, which only selects rows where the PrSb colum is equal to 0.


RL - Weighthing negative rewards Let's consider that I give an agent a reward of -1 (minimum reward) every time it performs an action which leads to the premature end of the episode (i.e., the agent dies). Besides, I also give a negative reward in the [-1, 0) interval when the agent performs an action that I want to avoid to repeat too much. However, these actions do not terminate an episode.

Is it possible that the agent will learn to take the "-1" action given that it is possible to collect many negative rewards which in the end will sum less than -1? This is, the agent will prefer to commit suicide rather than experience many negative rewards.

PD: the agent also receives positive rewards in the (0, 1] interval, which tend to be of higher magnitude than the negative rewards but the agent receives them less often when it is exploring.

Yes, it is possible. In my case, with a specific reward function, my agent (snake) preferred to suicide instead of trying to reach the target, because of the "live penalty" (the agent receives a penalty each step, to speed up the exploitation phase over the exploration). Check this blog posts, they go through all of this in detail and they're so far the best ones I have found:
https://mpatacchiola.github.io/blog/2017/01/29/dissecting-reinforcement-learning-3.html

Also, since the author has a background in Cognitive Neuroscience, each algorithm is explained from a neuroscience point of view.


Is it a good practice to always apply `ReduceLROnPlateau()`, given that models benefit from reducing learning rate once learning stagnates? The rationale behind the keras function ReduceLROnPlateau() is that models benefit from reducing learning rate once learning stagnates.

Is it a good practice to always apply ReduceLROnPlateau()? What are some situations, if any, to not apply ReduceLROnPlateau()?

I believe it is, in theory, always a good idea to use this method. I say in theory because the theory of gradient descent points to the fact that a minimum can only be reached when the learning rate approaches 0. Otherwise, with a permanent large learning rate, the model's performance (i.e. the loss metric) will bounce around the minimum – always overshooting it.

That being said, it is important to think practically and be aware of the dangers of using this method. There can be phases of training where you don't improve your metric by much, and so the learning rate is reduced. But this can happen prematurely - depending on the plateau-parameters you selected - and so actually impede/prevent the model from ever getting far enough down the loss curve! If you set a minimum learning rate, you may always get down to a minimum (local or otherwise), but it could take a lot longer if your learning rate was reduced prematurely.

In the end, trying out as many combinations as possible is likely your best best.

Alternatively, you could try combing this with simulated annealing, whereby the learning rate is boosted upwards again, according to a predefined rule, such as after a number of epochs or a %-reduction in the learning rate itself.


Functions to do MLE and MSE calculations in R Are there libraries available or does one have to write these functions from scratch.

Have you looked at the Metrics package for R?


Apriori gives 0 rules I am trying to use apriori() on this dataset.
After cleaning it, I made all attributes categorical with as.factor(). Then I uset these instruction:

chocolate_rules <- apriori(chocolateApriori, parameter=list(minlen=3, supp=0.1, conf=0.7), appearance=list(default="lhs", rhs=c("RatFactor=1", "RatFactor=2", "RatFactor=3", "RatFactor=4", "RatFactor=5")))

where RatFactor is categorial rounded "Rating" from the original dataset (and has only 5 possible values).

It seems to work, but when I call "chocolate-rules" the result is set of 0 rules.
Can you explain me why? Or can you help me getting another result?

"Apriori" algorithm is used for "Association Rules" learning.

In very simple terms its trying to determine that if people who buy chocolates ,do they buy roses also with that? or do they buy chocolate with ice-cream more? or its chocolate+roses+ice-cream always together? or any combination of it.

So, the data which contains these purchase transactions are analysed and then "frequent item sets" are determined.

From this "rules" are derived , eg. here a rule can be, chocolates are bought with roses, or roses are bought with chocolates.

Each rule has confidence and support.

Now in you dataset, do you don't seem to have transactions, rather it has attributes regarding chocolates and its rating.which doesn't suit associative learning since the objective is different.

For the "learning" purpose, if its predicting that there are NO rules, means there are NO rules to determine.

Here something to get you started-

https://en.wikipedia.org/wiki/Apriori_algorithm

https://en.wikipedia.org/wiki/Association_rule_learning


Binary classification and numerical labels I am trying to create a sentiment analysis model using a dataset that have ~50000  positive tweets that i labeled as 1, ~50000 negative tweets that i have labeled as 0. Also i have acquired ~10000 tweets that are neutral.
Due to the low number of neutral tweets my thinking is to label neutral with 0.5 and train the model using binary crossentropy as loss function. My output layer is 1 neuron with sigmoid activation function so prediction value would be between (0,1) .
Is my thinking right or it will mess the accuracy?

If you’re going to have more than two labels, you need to go with a softmax activation and a loss for multi class classification, ie cross entropy loss.
Also, be cautious for multi-class versus multi-label (below).

Multi-class
One-of-many classification. Each sample can belong to ONE of $C$ classes. The model will have $C$ output neurons that can be gathered in a vector $s$ (Scores). The target (ground truth) vector $t$ will be a one-hot vector with a positive class and $C−1$ negative classes. This task is treated as a single classification problem of samples in one of $C$ classes.
Multi-label
Each sample can belong to more than one class. The model will have as well
$C$ output neurons. The target vector $t$ can have more than a positive class, so it will be a vector of 0s and 1s with $C$ dimensionality. This task is treated as $C$ different binary $(C′=2,t′=0 \ or \ t′=1)$ and independent classification problems, where each output neuron decides if a sample belongs to a class or not.
source: https://gombru.github.io/2018/05/23/cross_entropy_loss/


Pig script code error? While running the below pig script I am getting an error in line4:
If it is GROUP then I am getting error.
If I change from 'GROUP' TO 'group' in line4, then the script is running.

What is the difference between group and GROUP?

LINES = LOAD '/user/cloudera/datapeople.csv' USING PigStorage(',') AS ( firstname:chararray, lastname:chararray, address:chararray, city:chararray, state:chararray, zip:chararray );

WORDS = FOREACH LINES GENERATE FLATTEN(TOKENIZE(zip)) AS ZIPS;

WORDSGROUPED = GROUP WORDS BY ZIPS;

WORDBYCOUNT = FOREACH WORDSGROUPED GENERATE GROUP AS ZIPS, COUNT(WORDS);

WORDSSORT = ORDER WORDBYCOUNT BY $1 DESC;

DUMP WORDSSORT;

'group' in strictly lower case in the FOREACH is the thing you are looping/grouping over.

http://squarecog.wordpress.com/2010/05/11/group-operator-in-apache-pig/ says:

  When you group a relation, the result is a new relation with two
  columns: “group” and the name of the original relation.

Column names are case sensitive, so you have to use lower-case 'group' in your FOREACH.

'GROUP' in upper case is the grouping operator. You can't mix them. So don't do that.


PCA before K-mean clustering If I applied PCA on feature vectors and then I do clustering, such like following:

reduced_data = PCA(n_components=2).fit_transform(data)
kmeans = KMeans(init='k-means++', n_clusters=n_digits, n_init=10)
kmeans.fit(reduced_data)

The reduced data will be the in terms of PCA components, so after
clustering in kmean, you can get a label for each point
(reduced_data), how to know which one from the origin data?
how to play with a number of PCA components regarding the number of
clusters? Thanks.

PCA reduces dimensionality. It does not change the number of observations you have. Nor does it change the order of the data. The n-th observation in your original dataset will still be the n-th observation post-PCA.
Choosing the number of components in PCA and choosing the number of clusters in K-Means are independent of each other. Both K-Means and PCA seek to "simplify/summarize" the data, but their mechanisms are deeply different. PCA looks to find a low-dimensional representation of the observation that explains a good fraction of the variance. K-Means looks to find homogeneous subgroups among the observations.

For PCA, the optimal number of components is determined visually through the scree plot or mathematically using Kaiser's criterion (drop all components with eigenvalue <1). From my experience the two don't always give the same results, but the difference is negligible.

Screeplot (source)

It does not make much sense to go beyond the 4th component. Law of diminishing marginal returns.

For K-Means: Since increasing the number of clusters will always reduce the distance from centroid to data points, increasing K will always decrease this metric, to the extreme of reaching zero when K is the same as the number of data points. If we plot the average within cluster distance to centroids vs number of clusters, we will find the "elbow" to be a good place to stop.

source


Suggestion of a model for these type of data? I've got a data set that looks like this

physical_data1   physical_data2  switch1   switch2   state
400              500             1         0         Normal
400              500             1         1         Normal
500              650             0         0         Normal
600              700             1         0         Normal
1000             300             1         1         Anomaly!

where physical_data are data that are from 0 to 1000 and switch are switches that are binary (1 means on 0 means off).

I am still relatively new to machine-learning and so I am wondering what kind of machine-learning algorithm is best suited for these kinds of data to detect anomalies, where my data has mixed features of physical and binary quantities.

Things that i have done are, normalizing to [0,1], however i am not quite sure if applying PCA to this kind of data will result in a loss of detection rate because it requires all features to determine if a particular reading is normal or an anomaly.

One other question I have is that what if in my data set i do not have the Anomaly data, instead, only have normal data. In that case what kind of model can i use?

Any kind of classification algorithm should be alright for your data.

Logistic Regression is probably the first thing you will learn in a ML tutorial.

https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html


Choice of weights for the Laplacian Eigenmaps algorithm In his thesis (section 2.3.3) Belkin uses the heat equation to derive an approximation for $\mathcal{L}f$:

$$\mathcal{L}f(x_i)\approx \frac{1}{t}\Big(f(x_i)-\alpha \sum_{x_j, ||x_i-x_j||<\epsilon}e^{-\frac{||x_i-x_j||^2}{4t}}f(x_j)\Big)$$
where $$\alpha=\Big(\sum_{x_j, ||x_i-x_j||<\epsilon}e^{-\frac{||x_i-x_j||^2}{4t}}\Big)^{-1}$$.

However, I'm not sure how these considerations lead to this choice of weights for the weight matrix (which will be used to construct the Laplacian):

$$W_{ij} =
\begin{cases}
e^{-\frac{||x_i-x_j||^2}{4t}} & if\ ||x_i-x_j||<\epsilon \\
0 & otherwise
\end{cases}$$

A very vague idea of mine was that the factors $\alpha$ and $\frac{1}{t}$ don't change for a given $x_i$ so if one choses the weights like above the resulting discrete Laplacian would (let aside those two constants) converge to the continuous version.

Any ideas or tips what I'd have to read up to in order to get a better understanding?

Recall the definition he makes for the graph Laplacian earlier, $L = D -W $. Now consider the map in the RHS parentheses which I'll call $L^*$,
$$ L^*f(x_i) := f(x_i) - \alpha \sum_{x_j, ||x_i-x_j||<\epsilon}e^{-\frac{||x_i-x_j||^2}{4t}}f(x_j).$$
The suggested weight matrix definition is natural because it lets us write
$$ L^* := I - D^{-1}W. $$ Here's a reference to a related paper with some easy to read exposition. Hope this helps!


What does sklearn's pairwise_distances with metric='correlation' do? I've put different values into this function and observed the output. But I can't find a predictable pattern in what is being outputed.

Then I tried digging through the function itself, but its confusing because it can do a number of different calculations.

Docs

According to the docs:

Compute the distance matrix from a vector array X and optional Y.

I see it returns a matrix of height and width equal to the number of nested lists inputted, implying that it is comparing each one.

But otherwise I'm having a tough time understanding what its doing and where the values are coming from.

Examples I've tried:

pairwise_distances([[1]], metric='correlation')
>>> array([[0.]])

pairwise_distances([[1], [1]], metric='correlation')
>>> array([[ 0., nan],
>>>        [nan,  0.]])

# returns same as last input although input values differ
pairwise_distances([[1], [2]], metric='correlation')
>>> array([[ 0., nan],
>>>        [nan,  0.]])

pairwise_distances([[1,2], [1,2]], metric='correlation')
>>> array([[0.00000000e+00, 2.22044605e-16],
>>>        [2.22044605e-16, 0.00000000e+00]])

# returns same as last input although input values differ
# I incorrectly expected more distance because input values differ more
pairwise_distances([[1,2], [1,3]], metric='correlation')
>>> array([[0.00000000e+00, 2.22044605e-16],
>>>       [2.22044605e-16, 0.00000000e+00]])

Computing correlation distance with Scipy

I don't understand where the sklearn 2.22044605e-16 value is coming from if scipy returns 0.0 for the same inputs.

# Scipy
import scipy
scipy.spatial.distance.correlation([1,2], [1,2])
>>> 0.0

# Sklearn
pairwise_distances([[1,2], [1,2]], metric='correlation')
>>> array([[0.00000000e+00, 2.22044605e-16],
>>>        [2.22044605e-16, 0.00000000e+00]])

I'm not looking for a high level explanation but an example of how the numbers are calculated.

Y = pdist(X, 'correlation')

Computes the correlation distance between vectors u and v. This is

$$ 1-\frac{(u-\overline{u})\dot{}(v-\overline{v})}{||u-\overline{u}||_2*||v-\overline{v}||_2} $$

where $\overline{u}$ is the mean of the elements of vector $u$, and $x\dot{}y$ is the dot product of $x$ and $y$.

The correlation between any vector which have ONLY TWO entries is always 0 (or nearly zero: $2*10^-16$), why? Because correlation distance measures the distance as the linearity between the data.

When I have [1,2] and [1,2] the equation $y=x$ fits perfectly, when I have [1,2] and [1,3] the equation $y=x+1$ also fits perfectly. The correlation distance says wheter a equation can be drawn for the data, in both cases the equation is perfect.

If you want to try getting a different result, try putting 2 vectors of three elements, and you will see changes.

Try with: [[1,2],[2,3],[3,4]] and [[1,4],[3,8],[-5,6]]. But first, plot them and you will comprehen what 'correlation' measures.


Remove Outliers from Dataframe using pandas in Python I would like to remove outliers from my dataset. It looks like this:

                  time       Gbps
0  2018-11-20 00:00:00  29.821748
1  2018-11-20 01:00:00  38.620987
2  2018-11-20 02:00:00   0.010000
3  2018-11-20 00:00:00  29.821748
4  2018-11-20 01:00:00  38.620987
5  2018-11-20 02:00:00   0.010000

As you take a look at this table, you can see that number 5 and 2 are the outliers. I wrote a interquartile range (IQR) method to remove them. However, it does not work. I don't know if I do something wrong in Pandas/Python, or it's the fact I do something wrong in statistics. Any ideas? The result from this function is the same frame as I had at the beginning.

def IQR(data):
    q1 = data['Gbps'].quantile(0.25)
    q3 = data['Gbps'].quantile(0.75)
    iqr = q3 - q1
    fence_low = q1 - 1.5 * iqr
    fence_high = q3 + 1.5 * iqr
    cleaned_data = data.loc[(data['Gbps'] > fence_low) & (data['Gbps'] < fence_high)]
    return cleaned_data

data = {
    'time': ['2018-11-20 00:00:00', '2018-11-20 01:00:00', '2018-11-20 02:00:00', '2018-11-20 00:00:00', '2018-11-20 01:00:00', '2018-11-20 02:00:00'],
  'Gbps': [29.8217476333333333, 38.6209872666666667, 0.01, 29.8217476333333333, 38.6209872666666667, 0.01]
}

df1 = pd.DataFrame(data, columns = ['time', 'Gbps']
cleaned1 = IQR(df1)
print(cleaned1)

You just don't have enough data in your dataset.

More accurately - your outliers are not affected by your filter function.

fence_low is equal to -35.974423375

fence_high is equal to 79.858537625

So the values of 0.01 are lying within this range.

I created an example notebook for you to show the difference you can check it out here:
https://github.com/OzmundSedler/learning/blob/master/IQR%20example.ipynb

Long story short - here is your data:

And here is the data, where I have added 15 more values in the range of [20:40]. In that case, 0.01 points are classified as outliers and filtered correctly.


What kind of analysis can I do in this dataset? I have a dataset of clients (their city, name, age, gender, number of children) and another dataset about the products that they have bought. i have been asked to do:

extract knowledge about client profiles.

I don't know what knowledge should I extract.

we are studying clustering and classification so they should connect to the question. what i thought about is to make clusters of clients. but i don't know what criterisas should I depend on.

should I just use a clustering algorithm like k means and let it give me the clusters ?

any suggestion would be appreciated

Update

These are the dataset that I have:

Card: CardID, City, Region, PostalCode, CardStartDate, Gender, DateOfBirth, MaritalStatus, HasChildren, NumChildren, YoungestChild
Item: ItemCode, ItemDescription, CategoryCode, SubCategoryCode, BrandCode, UpmarketFlag
Transaction: Store, Date, Time, TransactionID, CardID, PaymentMethod
Category: CategoryCode, CategDescription
Transaction_Item: Store, Date, Time, TransactionID, ItemNumber, ItemCode, Amount
SubCategory: SubCategoryCode, SubCategDescrip

The teacher said that we should categories the clients and then indicate which products best suit for each customer.

You may go through two different approaches:
Unsupervised Learning (Clustering)
You can choose attributes who make someone's profile and try to cluster it (e.g. using k-means). If you look at clusters according to different attributes it may give you some insight about the data. Do not forget to exclude names as they are meaningless for analysis.
Supervised Learning (Classification)
You can use relevant attributes which may affect an output e.g. age and city may affect gender or Nr of children so if u put these as targets you can classify different users according to their age and city.
The point is that your data is not suitable for more sophisticated analysis as your features are not many and too complicated (i.e. answering the questions like age distribution among a certain gender or the relation between cities, age and number of children or something like them are the only questions can be answered)
I hope it helped and if there are more questions please comment here.
UPDATE 1
You can look at your data from different point of view so first try to choose one. For example clients can be clustered via their transactions (when they bought, what they bought, how expensive they bought, etc) or by their personal info (gender, age, payment method, etc). It gives you a first impression of your client data. Then go through clustering using all those information i.e. put personal and transactional features together. It gives you an overview of customers in general and might result in some significant clusters (categories).
After that you can look at the problem from product point of view meaning you look at the transactional features and try to see different distributions and histograms to get an overview of what's happening to the product e.g. you can set time as the x axis and try to extract different time series like how many product are sold at this point of time?, what kind of people (age, gender, etc) bought at this point of time? and so on.
For general analysis a dimensionality reduction (e.g. PCA) might reveal some information and give you insight. Please note that for any kind of analysis ONLY use relevant features e.g. IDs are not informative but categories.
The most important point for recommending product to a customer (or predicting what they buy) is to use your input/output pair properly so the customer info (age, city, when does he/she usually buy something?, etc) are inputs and products (their categories, their type, whatever u know about them) are outputs.
Good Luck!
PS1: From experience of doing an industrial version of exactly the same project I would say payment method is not much informative.
PS2: StackExchange provides an upvote and an accept button for good answers!


Policy-based RL method - how do continuous actions look like? I've read several times that Policy-based RL methods can work with continuous action space (move left 5 meters, move right 5.5312 meters), rather than with discrete actions, like Value-based methods (Q-learning)
If Policy-based methods produce probability of taking an action from a current state $S_t$, how can such an action be continuous? We might get 4 probabities for our actions, but we still have to choose a single action from from four:
$A_1:  10\%$
$A_2:  35\%$
$A_3:  5\%$
$A_4:  50\%$
Thus, it's not obvious how can my action be something continuous like:  "turn +19.2345 angles clockwise". Such an action must have already been pre-defined to the "19.2345" value, right?

The main requirement of on-policy policy gradient methods is that they use a parametric policy $\pi(a|s, \theta)$ that is differentiable with respect to the parameters $\theta$.

This is not restricted to only describing discrete probability distribution functions (e.g. softmax output layer of neural network). A description of any probability distribution function that is differentiable and possible to sample from is all that is required. This is true for the Normal distribution for instance, so one relatively common solution in continuous spaces is for a neural network to output the mean and standard deviation for the distribution of each component of the action vector that accepts continuous values.

Typically the neural network does not perform the sampling to choose an action. This is also true for a softmax output - it is only additional code, outside of the NN, that interprets the values and selects the action. In addition, and unlike softmax, the NN does not need to directly represent the probability distribution function, just enough data to drive the sampling process. However, the nature of the distribution function does need to be taken into account when calculating the gradient in policy gradient methods.

  Thus, it's not obvious how can my action be something continuous like: "turn +19.2345 angles clockwise". Such an action must have already been pre-defined to the "19.2345" value, right?

What the policy might output here, is the two parameters of the distribution $\mathcal{N}(\mu, \sigma)$, which you then must sample to get an action like "turn x degrees clockwise". So for example, the neural network could output $(25, 7)$ and then additional code will interpret those values as describing the distribution and take a sample. If you got a mean 25, standard deviation 7, then at the point you select the action you could get "turn +19.2345 degrees clockwise" amongst a range of other values. The value 19.2345 does not need to be pre-defined or represented in the neural network in order to do that.


Boruta feature selection in R with custom importance (xgboost feature importance) According to the documentation - CRAN

  Boruta is an all relevant feature selection wrapper algorithm, capable of working with any classification method that output variable importance measure (VIM); by default, Boruta uses Random
  Forest.

Source: Boruta documentation

The package has one parameter getImp that defines the importance to be used, which by default is random forests from the Ranger package. So theoretically one could use xgboost's xgb.importance() function as the source of feature importance but I cannot find an example of how to use such parameter in practice because so far I've been unlucky. Is it possible at all to do this with the Boruta package or do I have implement from scratch the Boruta algorithm again using xgboost? And if it's possible how?

Boruta is by universal reputation dog-slow and not very good. Boruta runs take many hours or days. VIF feature-selection algorithm is not objective, anyway.
You can program your own feature-selection that runs faster. I ran Boruta a few times on various datasets and it wasted 4 days of my time, and the result was inconclusive.

Here's the fast-and-dirty not-100%-scientific approach, culled from consensus in a lot of Kaggle competitions and suchlike. It's 10-1000x faster than Boruta and probably more accurate.

Run some fast exploratory trees (RF/XGB), i.e. not very deep, not too many trees. Extract feature importances. Repeat for several random seeds and look at average or absmax feature importances.
For now, throw out very-high-cardinality features (e.g. user-ID, zipcode etc.), since trees won't tend to split on them until very deep and small nodes, so initially exclude them. Let's define "Very-high-cardinality" as >= ~0.3 distinct values per record, or less in a larger dataset.
Throw out all near-zero features, zero- and very-low-importance features (< 0.02-0.05 f.i. or lower is a rule of thumb).
Measure the matrix of which features are highly correlated (< 0.4) to each other. Corrplot is a nice way to visualize these, and order correlated features near each other). Pick a sensible color-scale. Identify subgroups of correlated features. Arbitrarily select one feature from those groups (you can revisit this selection later). Or you can decide if this warrants using PCA.
Rerun, your CV accuracy should have gone way up, and your training time will be way faster (since tree training is quadratic to no. of features). Revisit choices of borderline-low-important or correlated feature, and again use multiple random seeds.
For exploratory, 5-fold CV is fine, but if training time is an issue, 3-fold works. 10-fold is unnecessary.
For the subset of remaining features, use a 'respectable' method. See the excellent posts on CrossValidated in feature-selection.

Somebody with a PhD in Stats will probably wag their finger at me for this :)


Learning to rank: construct absolute ranking using pair-wise ranking approach I am learning about the "pairwise approach" for learning to rank. As far as I understood, the training output is a partial ranking function $r$ that:

given given some query $q$ and two document $d_i$ and $d_j$
predicts whether $$r(d_i)>r(d_j)$$ or not

However, for a IR system to work, the ranking should be absolute.

The natural question next is how to construct the absolute rank using only the partial ranks output by $r$?

But partial ranks does not guarantee absolute rank. For example

$$r(x)>r(y), r(y)>r(z), r(z)>r(x)$$ gives a cycle.

I guess I am misunderstanding about how pairwise approach works for IR system. Can anyone correct me?

Creating a total ranking from pairwise comparisons that don’t necessarily follow the axioms or rational preferences would certainly require some optimization, and you would need to compute a quadratic number of such comparisons in order to produce a ranking.

There is however one trick to do this easily: if the function that classifies whether one element is ranked above the other is linear in all the features, you can, intuitively, compare each element to a null element (i.e. just apply the decision function to the element alone), get a score assigned to them, and sort them by this score.

So, intuitively, your linear classifier would be something like this: element 1 is above element 2 if:
F(el1 – el2)>0 [function has to be symmetrical, thus there can't be a non-zero threshold] --> F(el1) – F(el2) >0 (only when F is linear!!!)
Which implies that el1 > el2 iff F(el1)>F(el2)


LSTM - Incorporate word embedding in layer with multiple records in same date I have a time series data having more than one record in a single date. Number of records in a single date is not consistent.

I have 3 input features namely phrase, cost and weight. My goal is to predict 'Cost'.
I have used Keras texts_to_sequences to deal with the text input 'phrase' column.

I have following concerns

how to incorporate Phrase column in the LSTM model
how to deal with multiple records in a single date
what will be the shape of input data
How do I structure the model like 1st layer to be Embedding layer, 2nd to Dense and so on.

Here is the sample of the data:

Here is the processed data:

Kindly assist me.

Your phrases are very short, you will be fine with a bag of words encoding instead of sequence.

2-3. If you want total cost over day prediction, you can just sum all the features including BoW per date and then do time series prediction on the resulting data.

Try just LSTM predicting next date from the sequence of past dates


Using local outlier factor score to detect outliers at run time I am using LOF ( local Outlier factor) to detect outliers in my data. I get LOF score as outlier distance. this unsupervised learning doesnt help to detect outliers at run time. So I want to use my data points and LOF score to have a supervised model [regression /classification].

My question is which should be approach out of
 1) classification ( taking a cutoff of LOF score and after having binary variable , build classification model) and   use this classification model to predict outliers at run time.

How does any clustering/anomaly detection( using clustering)  is even used at run time ?? Any guidance will help.

You should find an appropriate threshold of the LOF score, above which the point will be considered as an outlier. Typically, a point with score below 1 is considered as normal (non-outlier), but the threshold above 1 at which to consider a point as outlier depends on the dataset.

If you have a dataset with outlier points labeled, then you can use that to fine tune the value of the threshold, such that you get an acceptable tradeoff between false-positive rate (out of points labeled as outliers, fraction of normal points), and recall (out of all actual outliers, fraction which are labeled as outliers).

At run time, if new points are getting added to your dataset, then the local densities and LOF scores across points could change, and these should be recomputed. However, the threshold of LOF score that you determined as optimal should continue to apply, and requires revisiting only over a larger time period.


Different Test Set and Training Set Distribution I am working on a data science competition for which the distribution of my test set is different from the training set. I want to subsample observations from training set which closely resembles test set.

How can I do this?

Great question, this is what is known in Machine Learning paradigm as either
  "Covariate Shift", or "Model Drift" or "Nonstationarity" and so on.

One of the critical assumption one would make to build a machine learning model for future prediction is that unseen data (test) comes from the same distribution as training data! However, in reality this rather simple assumption breaks easily and upcoming data (its distribution) changes over time for many reasons. For those who may not be familiar with this very important problem, I encourage looking here or post!

To me, your question falls into the same category. Although I do not have the perfect solution (an implementation to offer), but I think you may look:

This blog post gives you a simple way to handle the subsampling of training data with code provided in Python!
Check this research paper. They propose to solve the problem by reweighting the training data so that the distribution of training is closer to the distribution of test using Kullback-Leibler Importance Estimation Procedure base on "Kullback-Leibler divergence" theorem. I do not know if they provide an implementation or it can be implemented easily, but I think it might worth digging as it sounds a professional way to deal the distribution mismatch.

QUICK update (a good solution): I found a Python implementation of KLIEP algorithm of that research paper (last point) to find those weights. It rather seems easy to use! Basically it resamples the training by putting weights (via the KLIEP algorithm) so that the assumption of having a similar distribution of train and test holds true as much as possible.


Imbalanced dataset in MLP classifier in python I am dealing with imbalanced dataset and I try to make a predictive model using MLP classifier. Unfortunately the algorithm classifies all the observations from test set to class "1" and hence the f1 score and recall values in classification report are 0. Does anyone know how to deal with it?

model= MLPClassifier(solver='lbfgs', activation='tanh')
model.fit(X_train, y_train)
score=accuracy_score(y_test, model.predict(X_test), )
fpr, tpr, thresholds = roc_curve(y_test, model.predict_proba(X_test)[:,1])
roc=roc_auc_score(y_test, model.predict_proba(X_test)[:,1])
cr=classification_report(y_test, model.predict(X_test))

You can try using data re-sampling techniques. They can be divided in four categories: undersampling the majority class, oversampling the minority class, combining over and under sampling, and creating an ensemble of balanced datasets.

The above methods and more are implemented in the imbalanced-learn library in Python that interfaces with scikit-learn. See ipython notebook for an example.


About building database for persian speech to text project In connection with the persian speech to text project on github, I am trying to collect the database from the audio and text available on the ganjoor site and Youtube Persian videos with Persian subtitles (also, Maybe compare to others database via internet like this)
But since I am new to this fields, your help can be very It worked.
If it is possible, please help in how to properly save the audio and text by suggesting proper format for saving data and pre filtering for better sorting data like said here. (Any python sample codes, would be very helpful.)
Project github site (Forked):
https://bit.ly/3hRIb1V
Ganjoor scraper page (text and audio collection):
https://bit.ly/3kGlijQ
Question related ( Ganjoor scraper codes):
https://bit.ly/3cnR8zd
Thanks.

You may want to consider using Persian songs which have the lyrics (i.e. متن) readily available as well. There are also Islamic eulogies, known as noheh (نوحه), which are essentially poems recited somewhat similar to songs and sometimes include the written poem (i.e. نسخه).
Some examples:
https://bit.ly/2FRnI0g
https://bit.ly/3mKpj8W


accuracy at a false positive rate of 1% I need to calculate the accuracy but at a false positive rate of 1%.
I am not sure if it is the normal accuracy that we can calculate with sklearn or I need a customized formula?

Calculate this by finding the threshold at which the false positive rate is 1%. Your model outputs a probability, not a category. You get a category by seeing on which side of some threshold, probably 0.5, the probability lies. Test some other thresholds. You can use the ROC curve for this.
Once you have that threshold, calculate the accuracy.
However, be cautious about accuracy (or any threshold-based scoring rule, such as F1 score) as a scoring rule.
https://stats.stackexchange.com/questions/312780/why-is-accuracy-not-the-best-measure-for-assessing-classification-models/312787#312787
https://stats.stackexchange.com/questions/464636/proper-scoring-rule-when-there-is-a-decision-to-make-e-g-spam-vs-ham-email
https://stats.stackexchange.com/questions/494023/proper-scoring-rule-in-optical-character-recognition
(Frank Harrell’s blog that gets linked is a good read, too.)


The use of feature scaling in scikit learn I'm studing machine learning from here and the course uses 'Scikit Learn'  for regression - https://www.udemy.com/machinelearning/

I can see that for some training regression algorithms, the author uses feature scaling and for some he doesn't because some 'Scikit Learn' regression algorithms take care of feature scaling by themselves.

If I apply feature scaling to an algorithm which already has feature scaling internally, will it affect in any way ?
Can I always apply feature scaling, no matter whether the algorithm I'm using has a system of feature scaling internally or not ?

Short answer: if you apply the same scaling method as what the algorithm does, it won't change anything. If you choose a different scaling method, then you'll end up performing two transforms on the original data, which may not be a bad thing depending on the context.

This implies that you have to know what kind of scaling the algorithm performs.

As a takeaway, it is good practice to scale the data beforehand yourself, so that you are sure what you input the algorithm with. An scaling rarely arms your model results.

As a note (I know this is not your question, but it may help), you have to understand why you scale the data. Taking the simple case of a linear regression, if your input features are house size and number of rooms and say you want to build a model of the house price from those features, number of rooms and size will have very different ranges. Scaling will help to:

make the algorithm converge more rapidly
allow you to answer questions such as "what is the main contributor to the price" by comparing the coefficients (this would be incorrect w/o scaling beforehand)


Two questions on hyper-parameter tuning Question 1:

In the example of logistic regression, I often see the regularization constant and penalty methods being tuned by a grid search. However, it seems like there are a lot more options for tuning:

classifier_os.get_params()
gives:
{'C': 1.0,
 'class_weight': None,
 'dual': False,
 'fit_intercept': True,
...
and many more!

So my question is: Are these other parameters typically not worth tuning, or are they left out in examples for another reason?
For example, I changed to solver='liblinear' and got sub-par results compared to lbfgs. Is that a coincidence, or are the other parameters already at their best by default?

Question 2:

Does performing grid search on hyper-parameters guarantee improved performance when tested on the same data set?

I ask because my intuition was "yes", however I got slightly lower scores after tuning my regularization constant:

p_grid = {
    'C': np.logspace(-3, 3, 7)
}
clf = model_selection.GridSearchCV(classifier_os, p_grid, scoring='accuracy')
clf.fit(x_train, y_train)
y_pred = clf.predict(x_test)
metrics.classification_report(y_pred, y_test, output_dict=True)

Gives me the following scores (C=10):

accuracy :  0.8218181818181818
 macro avg:
     precision :  0.8210875331564986
     recall :  0.8213603058298822
     f1-score :  0.8212129655428624
     support :  275

As compared to before tuning (C=1):

accuracy :  0.8290909090909091
 macro avg:
     precision :  0.8287798408488063
     recall :  0.8285358354537744
     f1-score :  0.8286468069310212

The only thing that the tuning changed was to make the regularization constant C=10 instead of the default C=1

Grid search grows geometrically with the number of hyperparameters and values for each hyperparameter. It is best practice to minimize the search space to fewer options.
Increased performance on training dataset is no guarantee of increased performance on test set.


L1 & L2 Regularization in Light GBM This question pertains to L1 & L2 regularization parameters in Light GBM. As per official documentation:

reg_alpha (float, optional (default=0.)) – L1 regularization term on weights.

reg_lambda (float, optional (default=0.)) – L2 regularization term on weights

I have seen data scientists using both of these parameters at the same time, ideally either you use L1 or L2 not both together.

While reading about tuning LGBM parameters I cam across one such case: Kaggle official GBDT Specification and Optimization Workshop in Paris where Instructors are ML experts. And these experts have used positive values of both L1 & L2 params in LGBM model. Link below (Ctrl+F 'search_spaces' to directly reach parameter grid in this long kernel)

http://www.kaggle.com/lucamassaron/kaggle-days-paris-gbdt-workshop

I have seen same in XGBoost implementations.

My question is why use both at the same time in LGBM/XGBoost.

Thanks.

First, note that in logistic regression, using both an L1 and an L2 penalty is common enough to have its own name: ElasticNet.  (Perhaps see https://stats.stackexchange.com/q/184029/232706 .)  So using both isn't unprecedented.

Second, XGBoost and LightGBM have quite a number of hyperparameters that overlap in their purpose.  Tree complexity can be controlled by maximum depth, or maximum number of leaves, or minimum sample (count or weight) per leaf, or minimum criterion gain.  Any combination of these might be optimal for some problem.  Overfitting can also be combatted with the learning rate vs. number of trees (and early stopping), subsampling rates, and either of the regularization penalties.

Finally, since L1 regularization in GBDTs is applied to leaf scores rather than directly to features as in logistic regression, it actually serves to reduce the depth of trees.  This in turn will tend to reduce the impact of less-predictive features, but it isn't so dramatic as essentially removing the feature, as happens in logistic regression.  You might think of L1 regularization as more aggressive against less-predictive features than L2 regularization.  But then it might make sense to use both: some L1 to punish the less-predictive features, but then also some L2 to further punish large leaf scores without being so harsh on the less-predictive features.

Toy example: https://github.com/bmreiniger/datascience.stackexchange/blob/master/trees_L1_reg.ipynb

Possibly useful:
https://github.com/dmlc/xgboost/blob/release_0.90/src/tree/split_evaluator.cc#L118
https://xgboost.readthedocs.io/en/latest/tutorials/model.html#model-complexity


Best Way to tackle to time series classification problem? I have a dataset where the input is a dataset for ICU patients where each ICU stay has 40 features (20 vitals, 20 lab values) and multiple time steps (the stays' length is between 6 and 19-time steps). The output is the 20 lab values (represented as binary numbers where 1 means the lab value is in range and 0 is out of range). The task is to predict the future output for an input ICU stay.
I have tried to make it a sequence to sequence problem where the output is a sequence shifted by a one-time step. However, the results were not good enough.
The question is there a way to better tackle this problem? (like maybe windowing ?)

Since I don't see the data and I cannot fully understand the data format, I am just giving some general advice. But first, I should mention that it doesn't make sense to consider each output (each 20-element array) as timeseries sequence unless there is a priority among them (by priority, I mean if you know a particular lab value is obtained before others)
So, if you have 20-element array of binary numbers as output, you might find out there is no two ouput exactly the same or there might be only a few observations with exactly similar output (because you can have $2^{20}$ combinations!). So, let's say if you have N observations, and you may see N distinct output array as well. Therefore, you may first cluster the labels into k groups and assign new labels to each group (the label of group i-th is i). Then, find the closest distance between your query and each group, so you can then find the appropriate class. Then, you can predict the ouput in array format (e.g. let's say you realize your query belongs to a group of 20 members). So, you can say, the output is expected value of output arrays in element-wise manner. So, e.g. your first element of 20-element output array might be 0.9. Or, you can be more precise and after finding the appropriate group, define a weight based on the distance between your query and members and find the weighted average of output arrays of that group!


Input and output Dimension of LSTM RNN I am fairly new to RNNs and Im having trouble setting up the desired output from RNN using Keras library. Each datapoint in my dataset consist of a pattern of labels and timestamp of occurrence of each label and based on the pattern of label I want to predict what the next label in the pattern be. I have developed the model which takes series of label as input and outputs the next label in the series but now I want to input labels as well as their timestamp and receive output the next label as well as the timestamp of its occurrence. How can I achieve this with a single model?

You can convert input to vectors of labels + day number . E.g:

(First column is day-number, rest of the columns indicate presence/absence of a label)

[[ 1 0 1 0 0 0 0 .......]
 [ 2 1 0 0 1 0 0 .......]
 [ 3 1 1 1 1 0 0 .......]
 [ 4 1 0 1 1 0 0 .......]
 [ 5 1 1 0 1 0 0 .......]]

(5000, 201)

Output should be a vector of probabilities for individual labels.

This article has an example :

https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/


Feature selection using feature importances in random forests with scikit-learn I have plotted the feature importances in random forests with scikit-learn. In order to improve the prediction using random forests, how can I use the plot information to remove features? I.e. how to spot whether a feature is useless or even worse decrease of the random forests performance, based on the plot information? The plot is based on the attribute feature_importances_ and I use the classifier sklearn.ensemble.RandomForestClassifier.

I am aware that there exist other techniques for feature selection, but in this question I want to focus on how to use feature feature_importances_.

Examples of such feature importance plots:

You can simply use the feature_importances_ attribute to select the features with the highest importance score. So for example you could use the following function to select the K best features according to importance.

def selectKImportance(model, X, k=5):
     return X[:,model.feature_importances_.argsort()[::-1][:k]]

Or if you're using a pipeline the following class

class ImportanceSelect(BaseEstimator, TransformerMixin):
    def __init__(self, model, n=1):
         self.model = model
         self.n = n
    def fit(self, *args, **kwargs):
         self.model.fit(*args, **kwargs)
         return self
    def transform(self, X):
         return X[:,self.model.feature_importances_.argsort()[::-1][:self.n]]

So for example:

>>> from sklearn.datasets import load_iris
>>> from sklearn.ensemble import RandomForestClassifier
>>> iris = load_iris()
>>> X = iris.data
>>> y = iris.target
>>>
>>> model = RandomForestClassifier()
>>> model.fit(X,y)
RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',
            max_depth=None, max_features='auto', max_leaf_nodes=None,
            min_samples_leaf=1, min_samples_split=2,
            min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=1,
            oob_score=False, random_state=None, verbose=0,
            warm_start=False)
>>>
>>> newX = selectKImportance(model,X,2)
>>> newX.shape
(150, 2)
>>> X.shape
(150, 4)

And clearly if you wanted to selected based on some other criteria than "top k features" then you can just adjust the functions accordingly.


Machine learning technique to calculate weighted average weights? I'm just starting to investigate machine learning concepts, so I'm sorry if this question is very naive, but I'm hoping that it will be an easy one to answer!

I have a document matching algorithm that individually calculates a match for each field (0-1, with 0 = no match, 1 = 100% match), and applies a separate weight to each field match to be used in calculating an overall weighted average relevance "score".

E.g., given a document of 3 fields (d1-3) and an input query against each of the fields (q1-3), field matches are calculated for each pair (m1-3) and then weights (w1-3) are applied using a weighted average for a final relevance score: s = sum(mi x wi)/sum(wi).

For this contrived example, perhaps we can simply say that a document is considered relevant if the score is above 0.5. I.e., there is either a "relevant" (0.5-1.0) or "not relevant" (0-0.5) outcome. But I don't want every field to have equal weight in determining the outcome.

So, my question is simply: What type of machine learning technique is "best" used to calculate the appropriate weights (w1-n), based on past, known results? Is this even an appropriate use of machine learning?

And secondly, if instead of a simple outcome of relevant and non-relevant, I actually want to rank the documents by relevancy, can this also be achieved using a machine learning technique?

Yes it is definitely possible to calculate optimised weightings provided you have some training examples where you know the document fields, the query, and either the outcome (relevant/not-relevant) or the desired score.

I think your training feature set should be the query score in range [0.0,1.0] for each field of each example. The training label should be either relevance 0 or 1 for each example, or the relevance score that the example has.

If you have a target score for each example

You want to determine the weights $W_i$ to use for each field $i$. Your calculated relevance score would be $\hat{y} = \sum_{i=1}^{N_{fields}} W_i * X_i$ where the caret signifies this is the estimate from your function and $N_{fields}$ is the number of field. Note I am ignoring your original idea of dividing by the sum of all $W_i$, because it makes things more complex. You can either add that term or force the sum to be equal to 1.0 if you wish (I am not going to show you how though, as this answer would get too long, and it probably won't help you much)

With a target score and training data, the simplest approach is to find the weights which cause the lowest error when used with the training data. This is a very common goal in supervised learning. You need a loss function. Having a target scalar value means you can use difference from target and a very common loss function for this kind of regression problem is the mean squared error:

$$E = \frac{1}{N_{examples}} \sum_{j=1}^{N_{examples}} (\hat{y}_j - y_j)^2$$

Where $\hat{y}_j$ is your calculated relevance score for example $j$ and $y_j$ is your training label for the same example.

There are a few different ways to solve for lowest $E$ with this loss function, and it is one of the simplest to solve. If you express your weights as a vector $W$ length $N_{fields}$ your example features as a matrix $X$ size $N_{examples} \times N_{fields}$ and the labels as a vector $Y$ length $N_{examples}$ then you can get an exact solution to minimise loss using the linear least squares equation

$$W = (X^TX)^{-1}X^TY$$

There are other approaches that work too - gradient descent or other function optimisers. You can look these up and see which you would prefer to use for your problem. Most programming languages will have a library with this already implemented.

Note that you will likely get scores greater than 1.0 or less than 0.0 from some document/query pairs.

You will have to use a adjust the technique if you want to divide by total of all weights or want sum of all weights equal to 1 in your scoring system.

If you have a relevance 0 or 1 for each example

You have a classification problem, relevant or not are your two classes. This can still be made to work, but you will want to change how you calculate your weighted score and use logistic regression.

Your weighted score under logistic regression would be:

$$\hat{y} = \frac{1}{1 + e^{-(b + \sum_{i=1}^{N_{fields}} W_i * X_i)}}$$

Where $b$ is a bias term. This looks complicated, but really it is just the same as before but mapped by a sigmoid function to better represent class probabilities - the result is always between 0 and 1.

You can look up solvers for logistic regression, and most stats or ML libraries will have functions ready to use.

Caveats

You have made a starting assumption that a simple combined relevance score will lead to a useful end result for your users performing search. This has led to simple linear models looking like a good solution. However, this may not be the case in practice, and you may need to re-visit that assumption.


How does Pooling Layer in CNN introduce invariance to other transformations besides translation Here is a quote from deeplearningbook which I am trying to process. I am not sure what do they mean by this quote, can someone help me understand please?

  Pooling over spatial regions produces invariance to translation, but
  if we poolover the outputs of separately parametrized convolutions,
  the features can learnwhich transformations to become invariant to
  (see ﬁgure 9.9)

This will become clearer if you try to visualize the convolutional filter functioning.

Let's say you have a filter that is extracting horizontal lines from the image. When the filter extracts the dominance of horizontal lines in the image and assign a value in the generated matrix. Now we have the weights of the horizontal lines after the operation of the filter. Among all the horizontal weights, the most significant will be the one that has maximum weights so, we apply pooling (Max-pooling in this case) and discard all the other weights to select only that one having the maximum impact. By doing so, the filter selects only the features having maximum wights and discard the remaining. As the significance of the required object is from the line having maximum weight, the transformation is invariant. This also makes to tensor smaller in size and the process becomes faster too.

Perhaps, you can also refer the CS231 for a better understanding of the explaination.


What is the intuition behind using Monte Carlo to solve a differential equation Conceptually, I understand how a numerical method like Monte Carlo is used to solve a definite integral. Because integral of a function is the area bounded by the curve, the ratio of random points that land inside the curve to the total number of points is the value of the integral.

Conceptually, can someone explain for a non math person, how we can solve a PDE/ODE using Monte Carlo?

The trick is to convert ODE/PDE into an integral equation and then Monte Carlo comes to play.
Here are some examples.
http://jotterbach.github.io/2018/08/08/MonteCarloODE/


Puterman or Sutton Barto? I wonder which of these two books is better to read for a beginner in RL and which are the pros and cons of them.

Also, if you know any book that in your opinion is better for a beginner in RL, feel free to suggest.

Sutton and Barto (2nd edition is out!). Also David's Silver slides for the RL course at UCL plus the youtube videos from his classes. You can start with the lecture videos and the slides and going back to the book for more details and exercises. Putterman is quite hard for a beginner, heavy on math and focusing on Dynamic Programming mostly.


It depends, generally speaking it might be okay as those columns only seems to holds very few information. However there are some case, notably when the outcome is unbalanced, where some skewed features are usefull. Say you are trying to predict something that happen 1% of the time, a feature with only 2% of non-zero value but causally leading to the outcome, might be very predictive for your target.
So when removing those it is important to check (before) there is no clear dependency to the outcome and (after) the performance doesn’t drop too much when removing them.