how can i collect data set from social networks like instagram? My name is Reza, a Master Student at Golestan University of Technical and engineering, Iran and my M.Sc. thesis is about detect fake account on social networks. so i need data set. how can i collect data set from social networks like instagram????

I am not aware about Instagram, or if there is a ready-to-use dataset for your purposes, but in general in order to get data from (almost) any online source you should a method called "web scraping" (This might help you while you google).

It might also be useful to you, if you decide what kind of tools you are going to use for your analysis.

I have tried something simple, but it was for Twitter though

Here is a rather simple piece of code that gets twitter 500 tweets concerning Barbie using R

require(twitteR)

setwd("C:/Users/slin/desktop/informatie")

ck <- "4bKJU1Z4B60SzbPPIva7OknnQ"
cs <- "cCxAPYDN3d0APvlurEBoXMNCKl0DKgrneV0w6KQCQenNpmM7fa"
at <- "715872051559575552-qMYaugTPHJeRg9fh70ABNWCjdl0Lp6T"
as <- "zxn6lOyCw2dcPyllkiHJq4zk0wHwGQ4tuKa9IpR96TNvO"

setup_twitter_oauth(ck, cs, access_token = at, access_secret = as)

t_stream <- searchTwitter('barbie', resultType="recent", n=500)

df <- do.call("rbind", lapply(t_stream, as.data.frame))

The ck, cs, at and as can be obtained as it is explained here in step 2

Also in that site you can see a guide of how to web scrap data from Twitter using Python.

You might also find useful this (Python), this (Python) and this (R)


Help reducing a set of features I am trying to do some clustering. I have a dataset that is very
sparse - with the majority of features only occurring in a single vector.

Here is a list of our features:
https://gist.github.com/scrooloose/5963725dc88e5d15d74dcae522bebf82

I am looking for any suggestions/hints/pointers as to how we can merge some of
these isolated features together. This should hopefully make my clustering
experiments more successful.

For example, from a manual inspection of the data, I can see this group of
features that could be all merged into a feature like "health" or perhaps
"mental health" + "general health" or similar.

618: Mental Health Research
619: Mental disorder
1616: mental health
1617: mental illness
1618: men’s health
410: Genital wart
402: Genomic Medicine
476: Hygiene

Another example is this set of features that could be merged into something
like "education":

536: Kiir Primary School
591: Makonzi Boarding School
609: Mathematics
670: New York University
300: Education
301: Educational psychology
349: Female education

Any thoughts would be very welcome, thanks :)

Side note: These features are keywords as returned from alchemy
(http://www.alchemyapi.com/). Resulting from keyword searches for a set of
URLS. The intention is to cluster the URLs (and hence they companies they
represent) by these keywords.

If I understand correctly you want to cluster urls by using the keywords extracted as features. As these features are really sparse, you can try to use dimensionality reductions methods to help you.

One way is to treat each URL keywords as a document. Then you can use document embeddings algorithms such as LDA or doc2vec that learn denser representations of your documents.

If you want to group keywords, you can try to use word embeddings methods that learn representations of words. Using this you can then measure the similarity between words and groups of words.
An example is the well-known word2vec. Recent methods like FastText can be alternative that take the morphology into account.


I managed to fine some stuff

Dataset of vector digits on Kaggle
Microsoft now offers an API for the OneNote handwriting recognition (using InkML)
Another data set on Kaggle using InkML

I then found out about 'CROHME' (Competition on Recognition of Online Handwritten Mathematical Expressions) this also uses InkML.


Time required to train a model based on features shape and CPU capacity? How to estimate the time required time to train a model, given feature shape, CPU/GPU sepcs and type of model

There is no answer to that question because besides model parameters, hardware specs and dataset size, there is also the question of the difficulty of the problem, which is not quantifiable.

An easily-solvable problem can require fewer epochs for the model to sufficiently converge. Take for example two image classification tasks:

The first is ILVRC which contains around 1m images.
Another dataset that contains 1m red and green images (just pure red and pure green).

Both datasets have the same image dimensions, the same number of images and will be used to train the same model on the same computer. Even though these tasks seem identical, the latter is much easier to solve and will require only a few iterations (i.e. a few seconds). The former is harder and could require 2-3 weeks.

Due to this huge difference caused, solely, on a non quantifiable characteristic of the dataset, there is currently no way to tell how long a model will require to train.

Furthermore, due to the stocahasticity of the training process, two identical models might take different times to converge, even on the same dataset (due to different initial conditions).

Note: what you can estimate is the time it takes for each epoch if you know the model (number of parameters, choice of certain hyperparameters), the dataset (features, samples) and the hardware.


How do Bahdanau - Luong Attentions use Query, Value, Key vectors? In the latest TensorFlow 2.1, the tensorflow.keras.layers submodule contains AdditiveAttention() and Attention() layers, implementing Bahdanau and Luong's attentions, respectively. (docs here and here.)

These new type of layers require query, value and key inputs (the latest is optional though). However, Query, Value, Key vectors are something I've always read referred to Transformer architectures.

What do these vectors represent, when it comes to Bahdanau and Luong attention? For example, if I want to train an RNN model for a common task (let's say time series forecast), what would these inputs represent?

EDIT:
I'm thinking about a seq2seq to make forecasts. The input would be a series of given length, and a series of external variables. The output would be the series shifted forward of n steps.

To answer your specific questions:

AdditiveAttention() and Attention() layers, are (loosely but not exactly) based on Bahdanau and Luong's attentions, respectively.

They use post-2018 semantics of queries, values and keys. To map the semantics to the Bahdanau or Luong's paper, you can consider the 'query' to be the last decoder hidden state. The 'values' will be the set of the encoder outputs - all the hidden states of the encoder. The 'query' 'attends' to all the 'values'

If you run thru' the library code you will see that the query is first expanded over the time axis and then there is a dense layer that determines the weights of w1, w2. These weights are applied to the expanded query and the values and then they are added and another weight 'v' is finally applied. A softmax of this is taken to return the attention weights and then these attention weights are multiplied with the 'values' and added to return the context. This is Bahdanau's additive logic

However, while analysing tf.keras.layers.Attention Github code to better understand how to use the same, the first line I could come across was - "This class is suitable for Dense or CNN networks, and not for RNN networks". Since you are using RNN, I would use this layer with caution.  In general, all these ready-to-use layers are mostly for self-attention and if you want to create a transformer like model where you are doing away with the RNN altogether and want too use only attention to represent the sequence, you can consider these classes.

If you still want to use the same, you can go ahead and try the following:

    ##Input 1 = the last decoder hidden state: stminus1
    ##Input 2 = All hidden states of the encoder: lstm_out
    ##Apply Bahdanau additive attention and give me the
    ##output = context
    context = tf.keras.layers.AdditiveAttention()([stminus1, lstm_out])

You can now use the context additionally to strengthen the predictions.
However I would strongly recommend you write your own attention layer in less than half a dozen lines of code. See for e.g.: https://stackoverflow.com/questions/63060083/create-an-lstm-layer-with-attention-in-keras-for-multi-label-text-classification/64853996#64853996


Overlaying a line graph and an area graph: adding recession bars to a time series I am trying to reconstruct a time series graphs from FRED: https://fred.stlouisfed.org/series/LABSHPUSA156NRUG using Python. However, I am unable to get a clean figure where the time series is overlaid with the area graph representing the recession bars.
I am quite new to python so it's been frustrating to find a simple reason why my attempts are not working.
I have attempted this two ways.
My first attempt looks like this:
fig, ax = plt.subplots()
ls_data.plot.line(ax=ax, figsize=(8,5), x='Date', color=blue)
rec_data.plot.area(ax=ax, figsize=(8,5), x='Date', alpha=0.5, color=gray)
plt.ylim(0.59,0.65)

And produces the following perplexing mess:
My second attempt goes for the "ax2 = ax1.twinx()" route
fig, ax1 = plt.subplots()
ax2 = ax1.twinx()

ls_data.plot.line(figsize=(8,5), x='Date', color=blue)
rec_data.plot.area(figsize=(8,5), x='Date', alpha=0.5, color=gray)

However this produces two separate figures instead.

One possible issue is that the x-values for the labor share are plotted in individual years whereas the x-values for recession are plotted in months. But based on the second attempt, it looks like the graphs should overlay smoothly but then I'm liable to get the same result as in my first attempt. Would transforming the recession data provide a fix?
Edit: The data for recession dates are available here. https://fred.stlouisfed.org/series/USREC#:~:text=For%20daily%20data%2C%20the%20recession,the%20month%20of%20the%20trough.

Using the data from FRED for both labour data and recession data and slight adjustments to your code I think I get the result you want:
import pandas as pd
import matplotlib.pyplot as plt

# read in data
rec_data = pd.read_csv("USREC.csv")
ls_data = pd.read_csv("LABSHPUSA156NRUG.csv")

# make sure date columns are actual dates
rec_data["DATE"] = pd.to_datetime(rec_data["DATE"])
ls_data["DATE"] = pd.to_datetime(ls_data["DATE"])

# create plot
fig, ax = plt.subplots()
ls_data.plot.line(ax=ax, figsize=(8, 5), x='DATE', color="blue")
rec_data.plot.area(ax=ax, figsize=(8, 5), x='DATE', alpha=0.5, color="gray")
plt.xlim("1950-01-01", "2020-01-01")
plt.ylim(0.58, 0.65)


Analysis of passed/not passed students by enrolled year I am analysing some data from my University about successful students through years. The dataset looks like as follows
Student     Passed        Enrolled_Year
    A             0                2016
    B             1                2008
    C             1                2008
    D             0                2012
    E             1                2007
    F             1                2019
    G             1                2006
    H             1                2006
    I             0                2012
    L             1                2019
    M             0                2018
    N             1                2008

I already plotted the frequency of passed/not passed through years, to see when students, who passed exams.
I would like to know if it makes sense to look at a possible correlation between Passed and Enrolled_Year, when the number of Passed students is small compared to those who not passed.
I would like to get some insights from my dataset:
https://drive.google.com/file/d/1tUgHoFdzq2vSLHaPzAiNQC5Xkrk2SmKX/view?usp=sharing and one of this was looking at the possible correlation.
I would appreciated it if you could let me know if it makes sense a such analysis with big difference among frequency.

To start with, you could use a simple thresholding.
If you have the dataset $S$ where an element has the form $(x,y,c) \in S$, $x$ denotes the year,  $y$ is a binary value (exam passed or not), and $c$ is the student id.
you can obtain a classifier by using
$\{(x,y,c) \in S \mid x \leq \theta\}$ and $\{(x,y,c) \in S \mid x > \theta\}$.
Now you can check all plausible $\theta$ values (..,2008,2009,..,2020) to see how good your data can be seperated. For example you could use Matthews correlation coefficient to evalute which $\theta$ is best.
But If you want to see if there is some linear correlation, yes you could compute the correlation coefficient, but I would assume that there is no linear correlation. Essentially you are computing the Point-biserial correlation coefficient.
Also with the thresholding method, in case you see that all $\theta$ perform more or less the same, and badly, you can see that there is no linear correlation.
You should have a look at histograms to see how your data is distributed.


adding a feature as "generic" I am using sklearn and python, to build a "malicious" login identifier. Reading some documents and examples, I chose the RandomForest classifier, then I decided to use the following features:
Timestamp (hour)
IP (this is expanded in feature creation adding geolocation info like Coordinates)
Username (hashed)

In order to train the model, I got some data from a real log. I'm assuming here that these are GOOD data. Then I need to train the BAD (malicious) login.
For this I got some IPs from some public blacklist and then I create a BAD training file with the same format.
But since I don't know the username for BAD data (since these are not real login), I decided to set it to 0 and than train the model.
I think that this is messing everything up. The model does not work very well and it is not able to detect BAD login, unless I try to query it with a '0' login name. In other words, if I query the model with a  BAD IP and a valid username, I always get a "GOOD" result.
May be this is completely expected in this case, but I'd like to understand some things:

is this way of creating the BAD list actually wrong?
do I have a way to consider a feature like a "wildcard" without replicating the line in the training for any good username?
is it possible to set a "priority" in features evaluation, so that some are more important than other? But this should be done by the algorithm itself....or not?

If an information is never available for one of the classes, it's not a usable indication. So it seems to me that the login name is simply irrelevant for the task, so it shouldn't be included as a feature.
Mixing different sources of data can be ok in some cases, but only if the different datasets provide consistent features and are generally representative of the distribution you would expect (another important problem). It looks like you have no username nor timestamp in the "bad" one, do you? if so it's impossible to use these features.
No you cannot give priority to any feature. This wouldn't make sense, since the point of ML is to let the algorithm find patterns in the data. If you already know how the decision should be made then you should simply write a regular deterministic program (this is called a heuristic).
Don't forget that you can decide to add any feature that you think could be relevant:

you could have a boolean feature indicating whether a login name is known or unknown, instead of setting the value to 0. But be careful that by definition in your external dataset they are all unknown whereas they are all known in your "good" dataset, that's an clear construction bias.
if you have a timestamp you could think of many other features, for instance number of repeated attempts in the past N seconds, regular time patterns for login, etc.


How to explain a relationship between Accuracy and F1 Score / F-Measure? I am building a CNN model for pitch estimation using a song recording. Pitch estimation is done by inputting spectrogram to CNN model and make the CNN predict pitch sequence (250 pitch values per recording) from that spectrogram. For the evaluation metrics, I am using Accuracy and F1 Score. Sample of overall test result are given below using mean measurement.

Some notes:

Val-Acc is the validation accuracy. I am using this to see how well the model analyze new data that is not given during training.
Delta acc is the difference value between accuracy and val-acc.

Right now, I am wondering how can I explain the relationship between Accuracy and F1 Score. My supervisor said to me that accuracy is measured to get how accurate the model performs, and F1 is how well the model performs. Is the relationship really like that? May I get some insight on how to explain the relationship between them?

Saying that accuracy is measured to get how accurate the model performs, and F1 is how well the model performs

This doesn't mean anything, it's obviously too vague.
The first things to check in order to understand this relationship are the definitions of accuracy and F1-score.
Wikipedia has a good page which explains how different classification evaluation measures are related.
Observations on your results:

The accuracy and F1-score are almost identical everywhere. This suggests that your data is probably quite well balanced, i.e. the difference in the number of positive vs. negative instances isn't very big. Why? Because if the data was imbalanced then the model would over-predict the majority class, and this would cause the F1-score to be much lower than the accuracy: assuming the majority class is the negative class, the recall would be somewhat low but the accuracy could still be high because most instances (majority class) would be correctly predicted.
As a consequence, there's no insight to gain from analyzing the relationship between accuracy and F1-score since they're virtually identical. The small differences might be due to the geometrical mean between precision and recall. F1-score is more informative in case of imbalanced data, but this is not the case here.
The F1-score is calculated only on the training data. It would be more useful to calculate it on the validation data.
There's some serious overfitting happening especially with the high learning rates, but with the low learning rates the fact that difference between training and validation accuracy increases is also worrying. Maybe the model is too complex or there are not enough instances in the data. Ideally the two accuracy values should converge.


Mututal Information in sklearn I expected sklearn's mutual_info_classif to give a value of 1 for the mutual information of a series of values with itself but instead I'm seeing results ranging between about 1.0 and 1.5.  What am I doing wrong?
This video on mutual information (from 4:56 to 6:53) says that when one variable perfectly predicts another then the mutual information score should be log_2(2) = 1.  However I do not get that result:
import pandas as pd
from sklearn.metrics import confusion_matrix

y = [1,1,1,1,1,0,0,0,0,0]
print("Confusion matrix:")
print(confusion_matrix(y,y))

print("Mutual information:")
result = mutual_info_classif(pd.DataFrame(y), y)
print(result)

which gives:
Confusion matrix:
[[5 0]
 [0 5]]
Mutual information:
[1.28730159]

When the two variables are independent, I do however see the expected value of zero:
x = [1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1]
y = [1,1,1,1,1,0,0,0,0,0,1,1,1,1,1,0,0,0,0,0]

print("Confusion matrix:")
print(confusion_matrix(x,y))

print("Mutual information:")
result = mutual_info_classif(pd.DataFrame(x), y)
print(result)

which gives:
Confusion matrix:
[[5 5]
 [5 5]]
Mutual information:
[0]

Why am I not seeing a value of 1 for the first case?

Sklearn has different objects dealing with mutual information score.
What you are looking for is the normalized_mutual_info_score.
The mutual_info_score and the mutual_info_classif they both take into account (even if in a different way, the first as a denominator, the second as a numerator) the integration volume over the space of samples.
Your code becomes
import pandas as pd
from sklearn.metrics import confusion_matrix
from sklearn.metrics import normalized_mutual_info_score, mutual_info_score

y = [1,1,1,1,1,0,0,0,0,0,1,1,1,1,1,0,0,0,0,0]
print("Confusion matrix:")
print(confusion_matrix(y,y))

print("Mutual information:")
result = normalized_mutual_info_score(y, y)
result_not_normalised = mutual_info_score(y,y)
print('norm: ', result)
print('not-norm: ', result_not_normalised)

giving,
Confusion matrix:
[[10  0]
 [ 0 10]]
Mutual information:
norm:  1.0
not-norm:  0.6920129648318737

Note also that $ln(2) \simeq 0.6931$.


Extract Pattern using Short Text Processing We are facing issue in our project. We have a data set of around 25000 rows, we have a column name title, it contains text data and we have a score column in the data set.We want to use Machine Learning techniques to know what are the factors that make the label High, by factor i mean what are the key words etc.., as use can see we have a short text so what is the most proper technique to extract the maximum knowledge related to the label from this short text,I did some pre processing on the text by cleaning it then extracting ngram features using tf-idf weighting function, then i tried using decision trees algorithm for classification of text
For example:

   Score Label                       Tactic Title
1.  High                        Opportunity Movement.
2.  Low                         Partner Launch.
3.  High                        Implement Mix panel stories.
4.  Low                         Improve app performance and reduce multiple API Calls.
5.  High                        Review Comments.
6.  Low                         Support SimpleStarta Team Goal.

Problem 1: "Tactic Titles" Are Really TEXT

i.e. they are not standard "tags". I say this because they seem so structured and tag-like. For instance in text analysis using NLP techniques you mostly deal with texts in which "Review Comments" and "Reviewing Comments" and "Comment Review" are supposed to be considered the same.

Actually the way you already tried is a valid way to do it.

Another idea could be using information theoretic approaches. It means using something like Tf-IDF approach but this time taking classes into account. Here you are going to look for words/n-grams which appear a lot in desired class (High) and not a lot in the other class. A simple formulation could be:

$$Score_{w_i} = \frac{N_{high}}{N_{low}+1}$$
where $N_{high}$ is the number of times the word $w_i$ appeared in class High and $N_{low}$ is the number of times the word $w_i$ appeared in class Low. The constant $1$ is just a smoother to avoid a zero denominator.

If you use it be careful about overfitting as you are learning features in a very implicit and hand-crafted way. To avoid overfitting you may prune your found words based on a min_count or max_count limitation.

Of course this is a simple score and you can improve it by modification (e.g. i did not consider the normalization which is better to be considered, etc.)

For prediction you can also simply use naive Bayes. Count words/phrases based on Bayesian approach (just count the frequency of words happening in each class) and normalize them to probabilities. Now you have most likely data given classes and the probability of classes. A new data comes and $P(C|D)=P(D|C)\times P(C)$ is calculated easily.

Problem 2: The "Tactic Titles" Are Just Fixed Tags (Maybe irrelevant to score)

Means that semantics are not correlated with classes i.e. two titles with similar terms or meanings might be in two different classes.

Then it's not a learning problem. Either you have the label of the new data in the dictionary so you just extract the class, or you don't have it so you don't know about it!

How would you want to know that "Partner Launch" is low priority but "Review Comments" is high if you have no meta-information form the organization behind it? How can you predict that based on this most probably "Write Comments" is High or Low? .If you argue that Machine Learning is supposed to learn that latent variable behind, then I would say yes! but please note that maybe that latent phenomenon is not correlated with Text. For instance if you get meta-data about which department is doing any of this tasks, you may end up with a more correlated/causal set of features. The long story short: Maybe the information is not in TEXT but some other aspects.


Free API for historical weather US data? I am trying to retrieve a free R-Python API that provides historical weather data in US.

In fact wunderground API is no longer available.

Any suggestion?

Maybe you can use NCEI Data

And use "httr" package to access the API in r.


Events prediction with time series of continuous variables as features We have the feeling that behavior of a device in terms of continuous variables (fans speeds, temperatures, voltages, ...) has influence on rare events happening (components failures).

I now have to build a predictive model for that, to proof the influence.

Those continuous features are given as time series, and events are punctual.

I've made a model based on descriptive statistics of those variable (see this question) with decision tree, random forest, adaboost, and clustering but it doesn't works. I will still improve by balancing classes, but I'm convinced it is not the best approach.

I'm pretty sure that there are nicer algorithms for such predictions (this is quite common problem), but I don't find anything.

Do you have ideas?

Thanks a lot

PS: I'm working with Python and cython

First of all, you will not be able to prove anything with a model, you will have false positives/negatives. With a good model you may be able show what variables may be an indicator of component failure.

In problems like this feature generation can have the most important influence on the accuracy of the model. The time stamps can be used for aggregation. For example, you may aggregate metrics per device per hour. The metrics/features that you might create for input into the model might be average/max temperature or fan speed, rate of change in temperature or fan speed, number of seconds device was above some threshold temperature or fan speed, boolean indicator of voltage spike, etc. There could be any number of features you may create. You can then find which features are not strong predictors and remove these columns to reduce noise, if need be.


python print values seasonal_decomposition I am totally beginner in Python and after using seasonal_decompose for time series decomposition result=seasonal_decompose(series, model='additive', freq=365) I got plotted results with commands result.plot() and pyplot.show(), but I cannot understand how to print this results values on screen or how to see decomposed time series values?
I have plot, but I need to have values in console or some data file.

Complete code:

from pandas import read_csv
series=read_csv('B.csv', header=0, parse_dates=[0], index_col=0, squeeze=True)
from random import randrange
from matplotlib import pyplot
from statsmodels.tsa.seasonal import seasonal_decompose
result=seasonal_decompose(series, model='additive', freq=365)
result.plot()
pyplot.show()

seasonal_decompose returns an 'object with seasonal, trend, and resid attributes.'  We can access the data by calling the object:

res = seasonal_decompose(series, model='additive', freq=365)
residual = res.resid
seasonal = res.seasonal
trend = res.trend
print trend

etc...


Customer-Product Analytics I am new to Data Science and I want to make Customer Product Analytics for my company(bank). I can have a data of customers, their income, daily transactions, average balance etc and what product(saving certificates etc) they have taken according to their account balance. Can i have a prediction for new customers or existing customers that what product will be suitable for them according to their average balance , income etc? Can the machine learning algorithm predict each product to a particular customer?
I got to know cluster analysis and predictive analysis can be useful for such task. But i want to recommend a particular product to a particular customer. Which algorithm can be useful? And from where I have to begin?

If you have historic data of earlier purchase by customers. Try to build any classification algo ( Decision tree/ RF) to build rule associated with customer and product.
   Now you can suggest product to a new customer associating his properties to rules which are already created. Just goole classification algos in R/Python

Clustering ( K means/Hiearchical etc)  would be good for customer segmentation, not directly useful for prediction.

https://machinelearningstories.blogspot.com/2017/09/hierarchical-clustering-bottom-up.html

You can also use market Basket analysis for recommending similar products to existing customer.

https://machinelearningstories.blogspot.com/2016/11/recommendation-engine-market-basket.html

http://courseprojects.souravsengupta.com/cds2016/product-recommendations-a-multi-classification-problem/


Weighted loss functions vs weighted sampling? For image classification tasks, is there a practical difference between using weighted loss functions vs. using weighted sampling? (I would appreciate theoretical arguments, experience or published papers, anything really.)

Some details:

By "weighted sampling", I mean attributing different sampling probabilities for each sample in the training set.

By "weighted loss functions", I mean weighting error terms differently depending on the sample considered.

First off, I would point out that both concepts can very well coexist. Let's take the following example:

Image classification including 2 classes and samples covering 2 domains. The classes are imbalanced and one domain is "harder" than the other. You could use weighted sampling to sample more examples from the "harder" domain while simultaneously use a weighted loss to counter the class imbalance.

Now let's look at some pros and cons:

Weighted sampling

Weighing each sample implies that you have knowledge of all samples and score them all. From a practical perspective, this is not always possible or feasible. Indeed, in cases where training samples are streamed, you don't control which samples come your way and weighted sampling is impossible. Or in cases where you continually update the weights of each samples.

On the other hand, it is much more practical to update the weights of your samples than to modify your loss throughout training.

Weighted loss

On the plus side, a weighted loss isn't dependent on how the instances are sampled, which can be more practical. On the down side, if you sample your data in such a way that most instances have low weights, your model will not update rapidly.


Is a new datapoint similar to my reference set? I have a large reference data set and everyday I get new data points.
I would like to know how can I test if those new data points are similar to the reference set?

Thanks in advance.

Maria

If you have many features, likely meaning many columns in your table of data, then you could try clustering. Something as simple as k-Nearest Neighbours could work nicely.

You would first fit a model using your available data, then look at the resulting clusters. Each cluster will represent combinations of your features.

Next you can put new data points into the model and it will tell you to which cluster it would best fit.

This would actually be a fully fledged predictive model! It is unsupervised learning, because you are not using and pre-existing labels (for each cluster in this case).

If you are working with Python, the Sci-kit Learn documentation is a good place to start. There are many other clustering algorithms, which might work better for you, depending on your type of data (if it is spatial, DBSCAN is good, for example).


order of features importance after make_column_transformer and pipeline I have a data preparation and model fitting pipeline that takes a dataframe (X_trn) and uses the ‘make_column_transformer’ and ‘Pipeline’ functions in sklearn to prepare the data and fit XGBRegressor.
The code looks something like this

 xgb = XGBRegressor()

 preprocessor = make_column_transformer(
        ( Fun1(),List1),
        ( Fun2(),List2),
            remainder='passthrough',
    )

    model_pipeline = Pipeline([
        ('preprocessing', preprocessor),
        ('classifier', xgb )
    ])

   model_pipeline.fit(X_trn, Y_trn)

Therefore, the training data which inputted into the XGBRegressor have no labels and resorted due to the make_column_transformer function. Given this, how do I extract the features importance using XGBRegressor.get_booster().get_score() method?

Currently, the output of get_score() is a dictinary that looks like this:
{‘f0’: 123 ,
‘f10’: 222,
‘f100’: 334,
‘f101’: 34,
…
‘f99’:12}

Can I assume that the order of the features provided by get_score() is identical to the order of features after make_column_transformer function (aka, I have to incorporate the feature sorting) such that 'f0' == 1st feature after make_column_transformer, 'f1' ==2nd  feature after make_column_transformer, etc.?

Your assumption is correct. Usually after column Transformation columns lose their names and get default values corresponding to their orders.

Additional Info:
You may try Eli5

from eli5 import show_weights,show_prediction
show_weights(model)
show_prediction(model,data_point)

The later function shows the impact of every features for predicting a data_point.


How to convert night image to day image? I have set of night images which I will be using for self driving. But I want to convert those images into day images. I have developed algorithm based on day image but it is not good for night images , so I want to convert night images to day images then feed into the network. As far as I have explored image colourization techniques of grey scale image ( converting night image to black and white and then coloring it) ànd night to day style transfer using GANs. Is it possible to achieve the goal with the above two mentioned techniques or you can also suggest any other way.

Example link. :-  https://goo.gl/images/kTwnHB

One problem with this is that dark images simply contain less information. Anyone with a background in photography will tell you it’s easier to decrease exposure on a bright image than increase exposure on a dark one (you can’t create what’s not there but you can throw away information you have). If you want all images on the same playing field, perhaps do the reverse. Or, try to extract only edges or high contrast regions from all images. Or, model them differently.


Approximating density of test set in train I am looking for a method to approximate how similar a test set (i.e., test set  features) to a train set. For example, something like, for each row in test: is there a similar enough data point in train? I've been thinking about using a mixture model approach, but I haven't been able to find a good reference on this. Can anyone suggest a good approach, or provide good references for how to use mixture models for this application?

The approach that comes to mind, is to calculate the kullback-leibler divergence between the kernel density estimations of your train dataset and of your test dataset.

The kernel density estimation of each of your datasets will give you an approximation to the pdf's of your datasets. The kullback-leibler divergence will give you a number that will represent the divergence in bits from one distribution to another (if you use base 2 for your logarithm). Below are some references I think you would fine useful.

https://en.wikipedia.org/wiki/Kernel_density_estimation
https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence
https://jakevdp.github.io/blog/2013/12/01/kernel-density-estimation/

If you would like me to show the math behind this method. Feel free to ask.

EDIT: Added math as asked by author of question

Let $\hat x_1, \hat x_2,\hat x_3...,\hat x_n$ be your training dataset while $x_1,x_2,x_3...,x_n$ is your testing dataset, where both $\hat x_i$ and $x_i$ belong to $\mathbb{R}^d$.
$$\hat f(x;H)=\frac{1}{n} \sum_{i=1}^{n}K(x-\hat x_i;H)$$
$$f(x;H)=\frac{1}{n} \sum_{i=1}^{n}K(x-x_i;H)$$
$\hat f$and $f$ represent the kernel density estimation for training set and testing set respectively. The parameter $H$ represents the bandwidth parameter and is a symmetric positive definite $d \times d$ matrix. $K(u;H)$ can be rewritten as $$|H|^{-\frac{1}{2}} K(H^{\frac{1}{2}} u)$$ where K can be any kernel function. I would recommend for simplicity purposes the standard multivariate normal kernel. Okay so now that we have the kernel density estimations of both our training and our testing dataset, we can use the kullback-leibler divergence in order to estimate the difference between the two.
Optimally we would like to calculate the kullback-leibler divergence with respect to every point in our space. Mathematically speaking.

$$\int_X f(x;H) \ log_2(\frac{f(x;H)}{\hat f(x;H)})dx$$

But this is computationally unpractical to compute. We can approximate this integral by sampling a set of points from the $f(x;H)$ and then computing the discrete sum.
$$\sum_{x \in X} f(x;H) \ log_2(\frac{f(x;H)}{\hat f(x;H)})$$

Quick Note: To sample from a kernel density estimator, uniformly randomly select a point from the respective dataset. Then sample from the kernel of choice centered around the point chosen.


SVM with multiple features import numpy as np
import matplotlib.pyplot as plt
from matplotlib import style
style.use("ggplot")
from sklearn import svm

X=[[1,0,0,0,0],
           [0,1,0,0,0],
           [0,0,1,0,0],
           [1,0,0,1,0],
           [1,0,0,0,1]]

y=[0,1,1,1,0]

model=svm.SVC()
model.fit(X,y)
print(model.predict([1,0,1,0,0]))

I was working on this but Iam getting an error as

"if it contains a single sample.".format(array))
ValueError: Expected 2D array, got 1D array instead:
array=[1. 0. 1. 0. 0.].
Reshape your data either using array.reshape(-1, 1) if your data has a single fe
ature or array.reshape(1, -1) if it contains a single sample.

Iam new to this can u guys help me out?

After having performed

X=np.array([[1,0,0,0,0],
           [0,1,0,0,0],
           [0,0,1,0,0],
           [1,0,0,1,0],
           [1,0,0,0,1]])

y=np.array([0,1,1,1,0])

you have to do the following:

y = y.reshape(1, -1)

model=svm.SVC()
model.fit(X,y)
test = np.array([1,0,1,0,0])
test = test.reshape(1,-1)
print(model.predict(test))

In future you have to scale your dataset. You can use either Standard Scaler (suggested) or MinMax Scaler.


Verifying business ideas based on keywords derived from a description of the business idea I want to find keywords for a business idea. The problem is that I have a hard time summarizing the business idea to one to three terms that would describe it.

Is there a (machine learning) tool that takes a description as an input and outputs - intelligently - categories and/or explicit search terms that fit the business idea most likely?

You need keyword extraction technics. Basically they try to find the most important words in the text. There are many types of such methods.


Canny edge detection not working on Gaussian blurred images I am trying to detect edges on this lane image. First blurred the image using Gaussian filter and applied Canny edge detection but it gives only blank image without detecting edges.

I have done like this:
#imports
import matplotlib.pyplot as plt
import numpy as np
import cv2
import matplotlib.image as mpimg
image= mpimg.imread("Screenshot from Lane Detection Test Video 01.mp4.png")
image = image[:,:,:3]
image_g = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)

image_blurred = cv2.GaussianBlur(image_g, (3, 3), 0)
threshold_low = 50
threshold_high = 100
image_blurred = image_blurred.astype(np.uint8)
image_canny = cv2.Canny(image_blurred, threshold_low, threshold_high)
plt.imshow(image_canny,cmap='gray')

You don't have to blur your image before passing it to Canny.
OpenCv's implementation already includes a blurring step. So by passing a blurred image, you're effectively blurring the image twice.  That will suppresses edges.
Read about openCv's implementation here.


High Cross Validation Score on Training Set, High Score on Test Set, But Low Score on Kaggle? I've been trying to complete this regression task on Kaggle. As usual they gave a train.csv(with response variable) and a test.csv (without response variable) file for us to train the model and compute our predictions, respectively.
I further split the train.csv file into a train_set and test_set. I use this subsequent train_set to train a list of models which I will then shortlist to one model only based on 10-fold cross validation scores (RMSLE) and after hyperparameter tuning.  Now I have one best model, which is Random Forest (with best hyperparameters) with an average RMSLE score of 0.55. At this point I have NOT touched the test_set.
Consequently, when I train the same exact model on train_set data, but evaluate its result on test_set (in order to avoid overfitting the hyperparameters I have tuned), it yields an RMSLE score of 0.54. This is when I get suspicious, because my score on test_set are slightly better than the average score of the train_set (test_set results are supposed to be slightly worse, since the model hasn't seen the test_set data, right?).
Finally, I proceed to submit my results using the same model but with the test.csv file (without response variable). But then Kaggle gave me an RMSLE score of 0.77, which is considerably worse than my cross validation scores and my test_set scores!
I am very frustrated and confused as to why this would happen, since I believe I've taken every measure to anticipate overfitting my model. Please give a detailed but simple explanation, I'm still a beginner so I might not understand overly technical terms.

This "train split" you named train_set and test_set are not guarantee to be clean or even balanced.
When your test set has better performance than your training set that might mean that you have data leakage (some examples in the test set are equal to the training set) or just mean your test set is slightly easier than the training set.


Ideas for next step of Machine Learning I have completely followed the machine learning course on coursera Machine Learning by professor Andrew Ng

Now I want to put my knowledge to action. Some ideas that I have include :

-Voice synthesis
 -Voice recognition

But since the course did not focus specifically on application of machine learning in these domains, could some one point me to some other course or books that can get me started.

I would suggest you to check this excellent presentation by Li Deng (Microsoft Research). Many of the slides contain references to relevant research papers and even several interesting books on the topics of interest (it should be pretty easy to find). It might be also helpful to check references, listed in this research paper by Prof. Andrew Ng and his colleagues at Baidu Research. Finally, a focused Internet search will provide you with comprehensive list of resources for further research.


SARIMAX with seasonality greater than one year I would use Statsmodels SARIMAX with data having seasonality greater than one year.
In my case, I have seasonality equal 4 years, therefore I have tried:

smt.SARIMAX(endog, order=(p, d, q), seasonal_order=(P,D,Q,4))

Is it correct set seasonality = 4 when the seasonality is equal 4 years?

From the documentation of statsmodels:

s is an integer giving the periodicity (number of periods in season), often it is 4 for quarterly data or 12 for monthly data. Default is no seasonal effect.

So it depends on how you have your data recorded, if each entry is a year, your seasonality would be 4, however if you have monthly data, then it would be 36.


Predictive modeling based on RFM scoring indicators RFM - is a ranking model when all customers are ranked according to their purchasing F requency, R recency and M monetary value. This indicator is highly used by marketing departments of various organizations to segment customers into groups according to customer value.

The question is following: are there any substantial models based on RFM scoring (or related to) which have solid predictive power?

Update:

predicting which customer will most likely spend more
who is going to upgrade/renew subscribtion/refund etc

Update2:

I understand, this is simple problem with three independent variable and one classifier. My guess and experience say these pure three factors do not predict future customer value. But they can be used together with another data or can be an additional input into some model.
Please share which methodologies worked for you personally and are likely to have high predictive ability. What kind of data you used together with RFM indicators and it worked well?

A Google search leads to many relevant resources that answer your question:

USING RFM DATA TO OPTIMIZE DIRECT MARKETING CAMPAIGNS: A LINEAR PROGRAMMING APPROACH http://www.thefreelibrary.com/Using+RFM+data+to+optimize+direct+marketing+campaigns%3A+a+linear...-a0272246211
Data Mining using RFM Analysis http://cdn.intechweb.org/pdfs/13162.pdf
Libby on Recency, Frequency, Monetary Value (book) http://www.amazon.com/Recency-Frequency-Monetary-Century-Library/dp/1882222067

From a data science point of view, there is nothing very special or unique about this problem. You have three independent variables and one dependent variable. Regression, clustering, and classification methods can be applied.


Unbalanced class: class_weight for ML algorithms in Spark MLLib In python sklearn, there are multiple algorithms (e.g. regression, random forest ... etc.) that have the class_weight parameter to handle unbalanced data.

However, I do not find such parameter for the MLLib algorithms. Is there a plan of implementing class_weight for some MLLib algorithm? Or is there any approach in MLLib for unbalanced data? Or we actually have to handle all the up/downsampling ourselves in MLLib?

Thanks!

Algorithms in MLLib are always used as baseline in production scenario , and they indeed can not handle some industrial problems , such as label imbalance . So if you want to use them , you have to balance your instances .

Besides , mechanism of BSP in Spark , you can simply see as data parallel ,  might be the main reason why Spark does not cover that problem . It might be hard for Spark to dispatch instances to all nodes in cluster , while the partial instances of each node share the same label distribution as the whole .

At last , you only have to weight the loss value for every minor labeled instance during your iteration process if you want to implement it .

Hopes this will help you , good luck -)


Feature extraction of images in Python In my class I have to create an application using two classifiers to decide whether an object in an image is an example of phylum porifera (seasponge) or some other object.

However, I am completely lost when it comes to feature extraction techniques in python. My advisor convinced me to use images which haven't been covered in class.

Can anyone direct me towards meaningful documentation or reading or suggest methods to consider?

In images, some frequently used techniques for feature extraction are binarizing and blurring

Binarizing: converts the image array into 1s and 0s. This is done while converting the image to a 2D image. Even gray-scaling can also be used. It gives you a numerical matrix of the image. Grayscale takes much lesser space when stored on Disc.

This is how you do it in Python:

from PIL import Image

%matplotlib inline

#Import an image
image = Image.open("xyz.jpg")

image

Example Image:

Now, convert into gray-scale:

im = image.convert('L')

im

will return you this image:

And the matrix can be seen by running this:

array(im)

The array would look something like this:

array([[213, 213, 213, ..., 176, 176, 176],
       [213, 213, 213, ..., 176, 176, 176],
       [213, 213, 213, ..., 175, 175, 175],
       ...,
       [173, 173, 173, ..., 204, 204, 204],
       [173, 173, 173, ..., 205, 205, 204],
       [173, 173, 173, ..., 205, 205, 205]], dtype=uint8)

Now, use a histogram plot and/or a contour plot to have a look at the image features:

from pylab import *

# create a new figure
figure()
gray()
# show contours with origin upper left corner
contour(im, origin='image')
axis('equal')
axis('off')

figure()

hist(im_array.flatten(), 128)

show()

This would return you a plot, which looks something like this:

Blurring: Blurring algorithm takes weighted average of neighbouring pixels to incorporate surroundings color into every pixel. It enhances the contours better and helps in understanding the features and their importance better.

And this is how you do it in Python:

from PIL import *

figure()
p = image.convert("L").filter(ImageFilter.GaussianBlur(radius = 2))
p.show()

And the blurred image is:

So, these are some ways in which you can do feature engineering. And for advanced methods, you have to understand the basics of Computer Vision and neural networks, and also the different types of filters and their significance and the math behind them.


How can conclusions be drawn from recommendation systems evaluation? From my research, a recommendation system are a subclass of information filtering system that seek to predict the "rating" or "preference" that a user would give to an item. And basically exists many type of recommendation systems, such as collaborative filtering and content-based.

One important aspect, is that evaluation is important in assessing the effectiveness of recommendation algorithms. The commonly used metrics are the mean squared error and root mean squared error, and also there exists others such as precision an recall.

My question is after I developed a recommendation system, and after evaluating using those metrics, how can I consider that my system provides good quality recommendations, or in others words, what should be the tresholds or MSE or RMSE values to be considered a good recommendation system.

Thanks. Any suggestions are welcomed.

"Good", I think, is based on the state of the art at the moment.  So I would look at respected models from industry leaders and use their  reported accuracies as a base line for what is "good": since it comes down to what is possible.


One hot encoding error "sort.list(y)..." I'm trying to do one hot encoding on a data set containing 4 categorical features in R. Unique levels per feature:  400, 60, 6, 5, respectively. I get the following error during the first call to the ohe function (acm.disjonctif):

Error in sort.list(y) : 'x' must be atomic for 'sort.list'
Have you called 'sort' on a list?

I can't do sorting on a per column basis because it will remove the data relationship between columns.  What am I missing in understanding?  I searched on the web but couldn't find anything helpful.

Here is the code I'm running in RStudio:

ohe_feats = c('RSN_CD', 'RSN_CAT', 'ELIG_SYS_CD', 'PROD_CD')
for (f in ohe_feats) {
    df_all_dummy = acm.disjonctif(as.factor(nodups[f]))   # package = ade4
    nodups[f] = NULL
    nodups = cbind(nodups, df_all_dummy)
}

I may be missing the point, but you don't need to do this one variable at a time in a loop and then bind the columns together - that's not the R way.  Here's an alternative:

# create some simulated data
n <- 10000
X <- data.frame(f_a_ = factor(sample(1:400, n, replace = TRUE)),
                f_b_ = factor(sample(1:60, n, replace = TRUE)),
                f_c_ = factor(sample(1:6, n, replace = TRUE)),
                f_d_ = factor(sample(1:5, n, replace = TRUE)),
                y = rep(1, n))

# use R's inbuilt understanding of formulas to do the "one hot encoding" for you
x_onehot <- model.matrix(y ~ ., data = X)


Improve k-means accuracy Our weapons:

I am experimenting with k-means and Hadoop, where I am chained to these options for various reasons (e.g. Help me win this war!).

The battlefield:

I have articles, which belong to c categories, where c is fixed. I am vectorizing the contents of the articles to TF-IDF features. Now I am running a naive k-means algorithm, which takes c centroids to begin with and starts, iteratively, grouping articles (i.e. rows of the TF-IDF matrix, where you can see here how I built it), until converenge occurs.

Special notes:

Initial centroids: Tried with random from within each category or
with the mean of all the articles from each category.
Distance function: Euclidean.

Question(s):

The accuracy is poor, as expected, can I do any better, by making another choice for the initial centroids, or/and pick another distance function?

print "Hello Data Science site!" :)

Great question, @gsamaras! The way you've set up this experiment makes a lot of sense to me, from a design point of view, but I think there are a couple aspects you can still examine.

First, it's possible that uninformative features are distracting your classifier, leading to poorer results. In text analytics, we often talk about stop word filtering, which is just the process of removing such text (e.g., the, and, or, etc.). There are standard stop word lists you can easily find online (e.g., this one), but they can sometimes be heavy-handed. The best approach is to build a table relating feature frequency to class, as this will get at domain-specific features that you won't likely find in such look-up tables. There is varying evidence as to the efficacy of stop word removal in the literature, but I think these findings mostly have to do with classifier-specific (for example, support vector machines tend to be less affected by uninformative features than does a naive bayes classifier. I suspect k-means falls into the latter category).

Second, you might consider a different feature modeling approach, rather than tf-idf. Nothing against tf-idf--it works fine for many problems--but I like to start with binary feature modeling, unless I have experimental evidence showing a more complex approach leads to better results. That said, it's possible that k-means could respond strangely to the switch from a floating-point feature space to a binary one. It's certainly an easily-testable hypothesis!

Finally, you might look at the expected class distribution in your data set. Are all classes equally likely? If not, you may get better results from either a sampling approach, or using a different distance metric. k-means is known to respond poorly in skewed class situations, so this is something to consider as well! There is probably research available in your specific domain describing how others have handled this situation.


How do you visualize neural network architectures? When writing a paper / making a presentation about a topic which is about neural networks, one usually visualizes the networks architecture.

What are good / simple ways to visualize common architectures automatically?

I recently created a tool for drawing NN architectures and exporting SVG, called NN-SVG


Making Sense of this Error Message I am using a book and a video to learn how to use KNN method to classify movies according to their genres.This is my code:

import numpy as np
import pandas as pd

r_cols = ['user_id', 'movie_id', 'rating']

ratings = pd.read_csv('C:/Users/dell/Downloads/DataScience/DataScience-Python3/ml-100k/u.data', sep='\t', engine='python', names=r_cols, usecols=range(3))                         # The file is u.data from MovieLens
print(ratings.head())

movieProperties = ratings.groupby('movie_id').agg({'rating': [np.size, np.mean]})
print(movieProperties.head())

movieNumRatings = pd.DataFrame(movieProperties['rating']['size'])
movieNormalizedNumRatings = movieNumRatings.apply(lambda x: (x - np.min(x)) / (np.max(x) - np.min(x)))
print(movieNormalizedNumRatings.head())

movieDict = {}
with open('C:/Users/dell/Downloads/DataScience/DataScience-Python3/ml-100k/u.item') as f:                       # The file is u.item from MovieLens
    temp = ''
    for line in f:
        fields = line.rstrip('\n').split('|')
        movieID = int(fields[0])
        name = fields[1]
        genres = fields[5:25]
        genres = map(int, genres)
        movieDict[movieID] = (name, genres,
movieNormalizedNumRatings.loc[movieID].get('size'),           movieProperties.loc[movieID].rating.get('mean'))

print(movieDict[1])

from scipy import spatial
def ComputeDistance(a, b):
    genresA = a[1]
    genresB = b[1]
    genreDistance = spatial.distance.cosine(genresA, genresB)
    popularityA = a[2]
    popularityB = b[2]
    popularityDistance = abs(popularityA - popularityB)
    return genreDistance + popularityDistance        # Everything seems fine up to this point. But the problem seems to be the next line of code.

print(ComputeDistance(movieDict[2], movieDict[4]))

Note: This code can be found here: https://hendra-herviawan.github.io/Movie-Recommendation-based-on-KNN-K-Nearest-Neighbors.html

Note: This code can also be found on pg 245-250, chp7, Hands-On Data Science and Python Machine Learning by Frank Kane. I have an e-copy and I can send it on request.

Note: I don't know how to share the csv files here. I will appreciate it if somebody can tell me how I can do that here.

I think I have bumped into what works. I am posting the correct code since somebody may need this someday:

import numpy as np

import pandas as pd

r_cols = ['user_id', 'movie_id', 'rating']

ratings = pd.read_csv('C:/Users/dell/Downloads/DataScience/DataScience-Python3/ml-100k/u.data', sep='\t', engine='python', names=r_cols, usecols=range(3))  # please enter your file path here. The file is u.data

print(ratings.head())

movieProperties = ratings.groupby('movie_id').agg({'rating': [np.size, np.mean]})

print(movieProperties.head())

movieNumRatings = pd.DataFrame(movieProperties['rating']['size'])

movieNormalizedNumRatings = movieNumRatings.apply(lambda x: (x - np.min(x)) / (np.max(x) - np.min(x)))

print(movieNormalizedNumRatings.head())

movieDict = {}

with open('C:/Users/dell/Downloads/DataScience/DataScience-Python3/ml-100k/u.item') as f:     # The file is u.item

    temp = ''

    for line in f:

        fields = line.rstrip('\n').split('|')

        movieID = int(fields[0])

        name = fields[1]

        genres = fields[5:25]

        genres = map(int, genres)

        movieDict[movieID] = (name, genres, movieNormalizedNumRatings.loc[movieID].get('size'), movieProperties.loc[movieID].rating.get('mean'))

print(movieDict[1])

from scipy import spatial

def ComputeDistance(a, b):

    genresA = np.array(list(a[1]))

    genresB = np.array(list(b[1]))

    genreDistance = spatial.distance.cosine(genresA, genresB)

    popularityA = np.array(a[2])

    popularityB = np.array(b[2])

    popularityDistance = abs(popularityA - popularityB)

    return genreDistance + popularityDistance

print(ComputeDistance(movieDict[2], movieDict[4]))

I am using this opportunity to thank those who gave me one or two suggestions.


How to fill an missing values in a column based on another column I have a dataframe called shoes

Brand   Comment
Ugg       NaN
Prada     NaN
Clarks    NaN
Ugg       NaN
Clark     NaN
Prada     Made from horse leather
Prada     Made from pig leather
Prada     NaN
Ugg       Made from Australian cow leather
...

and another dataframe df_mode which was obtained by taking the mode of the comments for each shoe brand in the shoes dataframe for nonnull values

Brand  Comment
Ugg    Made from sheep
Prada  Made from pig leather
Clarks Made from Cow leather

How can I assign the missing values for each shoe brand in the shoes dataframe with its respective mode response shown in the df_mode dataframe.

This is basically what I'm trying to achieve

Brand   Comment
Ugg       Made from sheep
Prada     Made from pig leather
Clarks    Made from Cow leather
Ugg       Made from sheep
Clark     Made from Cow leather
Prada     Made from horse leather
Prada     Made from pig leather
Prada     Made from pig leather
Ugg       Made from Australian cow leather

import pandas as pd
import numpy as np

shoes = pd.DataFrame({'Brand':['Ugg', 'Prada', 'Clark', 'Ugg', 'Clark'],
                     'Comment':[np.NaN, np.NaN, np.NaN, np.NaN, np.NaN]})

df_shoes = pd.DataFrame({('Ugg','Made from sheep'),
                     ('Prada', 'Made from pig leather'),
                     ('Clark', 'Made from Cow leather')}, columns=['Brand', 'Comment'])

shoes.merge(df_shoes, on=['Brand'], how='left', suffixes=('_x', '_y'))

The result will show like:

You can then drop the null columns.

EDIT:
As discussed in the comments, in case for the edited question, you can do that:

shoes[shoes.Comment.isnull()].merge(df_shoes,on=['Brand'], how='left',suffixes=('', '_notnull'))

shoes.Comment.fillna(value=temp.Comment_notnull)


How to find the best fitting parametric distribution for an empirical dataset (stock returns)? Given some real-valued empirical data (time series), I could convert it to a histogram to have an (non-parametric) empirical distribution of the data, but histograms are blocky and jagged.
Instead, I would like to identify the best-fitting parametric distribution from the scipy or scipy.stats libraries of distribution functions, so that I can artificially generate a parametric distribution that closely fits the empirical distribution of my real data and is continuous.
If the empirical data are monthly returns of empirical AAPL stock returns, for example, I know that the parametric Johnson-SU distribution resembles, and can mimic, stock return distributions because of its customizable skew. However, the Johnson SU distribution in scipy requires four input parameters to be calibrated. How can I search for the best parameter settings of this parametric distribution from scipy that fits to the empirical distribution of my sample of AAPL returns?

First of all, if you want to find the best distribution that fits your data you just iteratively fit your data to the longlist of distributions. Scipy supports most of them. After fitting, you can either use KS-test to find which distribution fitted best or you can use fit error to decide. This solution does what you want, also other solutions in that post are very good approaches to your problem.
Also, if you are sure about your distribution's being Johnson-SU distribution, to find the parameters, use Scipy's fit function, which will return you the parameters that best represent your data.
a, b, loc, scale = scipy.stats.johnsonsu.fit(data)


Cannot fig out error in my gradient function implementation in python Im trying to implement following gradient descent function in Python for logistic regression:

$∇θ(−logL)=−X^T (y−e^{Xθ})$

This is my python implementation:

def gradient(X, y, theta):
    dtheta = -(np.dot(X.T,y - np.exp(X * theta)))
    return dtheta

X is a dataframe of size: (2458, 31),
y is a dataframe of size: (2458, 1)
theta is dataframe of size: (2458,1)

when i pass values to my gradient descent function, it returns a dtheta parameter with size (31,31) due to which i cannot update my theta to pass it to cost function, i cannot fig out where im going wrong.
any help will be appreciated.

Error i keep getting is: ValueError: operands could not be broadcast together with shapes (2458,1) (31,31)

and this is how im implementing the algorithm:

theta = np.random.uniform(low=-0.1,high=0.1, size=(2458,1))
# Iterate and update theta by using the gradient of the negative log-likelihood
max_iter = 100
learning_rate = 1e-3
for i in range(max_iter):
    # Calculate the gradient
    dtheta = gradient(X,y,theta)

    # Update theta

    theta = (theta - learning_rate) * dtheta

    # Calculate the value of the log-likelihood
    cost = negative_loglikelihood(X,y,theta)

    # Print iteration
    print("Iteration %d, cost function %.3f" % (i+1,cost))

Check your theta dimensions.

Most likely, your X dimensions indicate you have 2458 training samples per iteration with each having 31 features. Hence, your theta should be a matrix of shape (31, 1).

With X having shape (2458, 31) and if theta has shape (31, 1), X*theta will have dimensions (2458, 1), same as y and as expected. Now, y-theta has the same dimensions as that of y or theta. And so does exp(y-theta)

X_T has shape (31, 2458) and hence, d_theta = - X_T*exp(y-theta) will have shape (31, 1), same as our initial assumed theta shape and now, you can subtract d_theta from theta.


How to deal with an attribute including a lot of character or factor values when using caret package in R I am studying caret package in R. I know in this package, all the data are assumed to be numeric. But I always have some data that has a lot of strings or factors. For example, a product category could include hundreds of thundred of strings indicating all the different products. A worker ID list may includes a lot of numbers. They look like numeric, but you cannot do any calculation because it does not make any sense. They are actually supposed to be characters or strings. So, we cannot use model.matrix or dummyVars to transfer them to numeric because 1). It should not be transferred like this; or 2). The matrix will be so huge after transferring that we cannot deal with. So my question is:
How can we deal with an attribute with a lot of characters or a lot of so-called numbers when using caret package in R?

Yeah, some (not all) libraries have that numeric input limitation.

What you're looking for is called "one-hot encoding" which can basically convert an N level factor like your product categories to N binary columns. Here is a post describing a few ways of doing that:

https://discuss.analyticsvidhya.com/t/how-to-do-one-hot-encoding-in-r/6982/5


How Orange makes decision about declaration of a meta attribute I wonder how Orange decides which attribute is declared as a string? I have few discrete attributes in Excel, which I can't use as Orange does not recognize them as discrete. Have tried export excel to csv format as well, no difference. Where in Orange I can enforce change from meta to discrete attribute?

Many thanks in advance,
Marko

As far as I know, Orange supports advanced annotation with a three-line header in both CSV and Excel files. So if you have a discrete attribute that gets marked as string, you should annote it as discrete on the second header line, or prefix the attribute name on the first line with D#.

When no advanced annotation is provided, it seems Orange's heuristic treats attributes containing general strings (as opposed to numbers) as discrete only if they contain fewer than approximately $\sqrt{|data|}$ unique values.


Normalizing / standardizing training and validation data Say I split my data to 80% training and 20% test/validation and I want to standardize it, I think I'm right in saying I shouldn't standardize across 100% of the data, and then do the split, because then the validation has some insight into the training data?

I'm not sure if I should either

1) Generate the mean and standard deviation stats on the 80% of training data and then apply the same mean/standard deviation to standardize the validation data.

Or 2) Standardize the training data, and then standardize the validation data, i.e mean and SD is derived from the 80% for the training data, and then mean/SD is derived separately on the 20% of validation data?

Many thanks

Yes, the test data might contain something important that you didn't learn from the test set. No, you're not allowed to use the test set for learning.

The purpose of the test set is to understand how the system would behave on new data not seen during training. Fitting your scaler in the test set would defeat that purpose.

  1) Generate the mean and standard deviation stats on the 80% of training data and then apply the same mean/standard deviation to standardize the validation data.

That's what you should do.

  Or 2) Standardize the training data, and then standardize the validation data, i.e mean and SD is derived from the 80% for the training data, and then mean/SD is derived separately on the 20% of validation data?

This is valid, if your production system makes it like that. So if you get data in batches and normalize the batches it might be fine as well.


Initial embeddings for unknown, padding? Last time I've been passing pretrained word embeddings into LSTM to solve text classification problems. Usually, there are additional <pad>, <unk> replacements for padding and unknown types. Of course, there are no pretrained vectors for them.

Solutions I've come up with are

Fill them with random values
Fill them with zeros

Which approach is better/wrong? What is the common practice?

Note: I use pytorch+torchtext if it matters.

In my experience, what works well is :

For padding, fill a zero vector embedding (as pixel intensity in image data padding) is the only and best solution.
For words that don't have a pre-trained embedding, you should try to train them: as you do, fill them with random values when initializing, but set them to trainable.


Ensemble modelling using model's probabilities In a classification project, on the training sets, I ran a selection of classifiers. These give me about 20-30% accuracy at best. For each sample, I generate probabilities of each class. I want to make a new model which takes these probabilities and gives a weighted average of the probabilities which is accurate.
For this, I tried averaging probabilities. I also tried taking the best probabilities from each model (For instance, assuming Class A has better precision/recall for model1 output, I take model1's outputs for class A and so on). This also hasn't improved the accuracy much. Can you suggest some ensembling techniques?

Let's say you fit n models on given dataset, to put them together, you have in fact few options:

Take the average (I'd play with weighted or geometric)
Voting (soft or hard). Basically you predict a class instead of probability and take the one which is most often predicted (hard) or take argmax or probabilities (soft). You havent mentioned what language you are using, but scikit has nice documentation for that.
Fit a metaclassifier on these predictions. You have n models, hence n predictions for each record, so you fit another model on these predictions. This is called stacking as mentioned in the comment.
You can go further for blending, boosting or bagging, there is really nice article about these methods here.


How to fake data based on the condition and weight I'm trying to fake data for the coffee shop. I've two features age and menu. Menu includes various type of drinks such as coffee [latte, espresso, mocca, etc], tea [milktea, lemontea], milk [freshmilk, matchamilk, etc]. What I'm trying to do is to fake menu based on the age like if the age is higher than 15, 80% of people who has the age higher than 15 will mostly order coffee randomly from the list of coffee [latte, espresso, mocca, etc], 20% will order milk and tea. Any idea on how to do this in the python?

There are many Python packages to generate synthetic data. One example is Faker.


Important CNN architectures What are some of the important CNN architectures one should know about? What CNN architectures did well on the ImageNet ILSVRC challenge? What CNN architectures are good candidates for transfer learning?

Here is a list of several important CNN architectures:

LeNet-5 the first successful application of CNNs to digit recognition,   developed by Yann LeCun in 1990. It consists of a sequence of Convolutional, ReLU, Max Pooling layers followed by a Fully Connected layer.
Reference: Y. LeCun et. al., "Gradient-Based Learning Applied to Document Recognition", Proceedings of the IEEE, 1998
AlexNet popularized CNNs in computer vision, did really well on the ImageNet ILSVRC challenge in 2012 showing significant gains in performance.
The network has similar architecture to LeNet but is deeper and bigger and features convolutional layers stacked on top of each other. Reference: A. Krizhevsky, et. al. "ImageNet Classification with Deep Convolutional
Neural Networks", NIPS, 2012
VGG16 demonstrated the importance of depth as a critical component to good performance, it was a runner-up in ILSVRC 2014. The architecture consists of a stacked convolutional and max pooling layers with increasing depth and
it uses a large number of parameters due to the final fully connected layers. Reference: K. Simonyan, "Very Deep Convolutional Networks for Large-Scale Image Recognition", ICLR, 2015
Inception original version called GoogLeNet from Google won the ILSVRC 2014 challenge the architecture consists of inception modules that dramatically reduce the number of parameters, it uses multi-scale 3x3, 5x5 convolutional filters including 1x1 convolutions for dimensionality reduction. Reference: C. Szegedy et al., "Going Deeper with Convolutions", CVPR, 2015
ResNet50 residual network was the winner of ILSVRC 2015, it introduces skip connections for easier training that enable very deep architectures
and makes use of batch normalization. Reference: K. He et al., "Deep Residual Learning for Image Recognition", CoRR, 2015


Data aggregation and split train test samples I'm working on a data science project where the goal is to predict daily electricity consumption of a building based on some of its characteristics (e.g., size, location, etc.) and weather features (e.g., temperature, humidity, wind, rain, sun radiation).

Some of my weather features (temperature, humidity, wind) are in hourly intervals and others in daily intervals (rain, sun radiation).
My target (daily electricity consumption) is also in hourly intervals.

My goal is to predict daily consumption so I need to aggregate my input data at a daily interval (sum of 24 values for consumption, average of 24 temperatures, ....)

My question is: Do I have to aggregate the data before split train/test of after ?

If I do aggregation first, I will consider only those days with 24 values (one for each hour) and drop others in order to not introduce bias
Then I will split train/test. So basically, I will clean my data before splitting.

Am I missing something?

If you are training the model based on the aggregated values anyway then it wouldn't make any difference on the final dataset that is fed to the model.

You might consider aggregating earlier in your pipeline as to reduce the amount of work that has to be done afterwards (whatever steps you go through).  Aggregating means less data points, so following steps will run faster.


Meaning of variance in machine learning models I know that high variance cause overfitting, and high variance is that the model is sensitive to outliers.

But can I say Variance is that when the predicted points are too prolonged lead to high variance (overfitting) and vice versa.

Variance actually measures the variability of the model prediction (say, for simplification, for a particular sample instance) if we would retrain the model multiple times (on different subsets of the data).

To gain an intuitive feeling of variance, suppose you have 100 training examples in your dataset (150 examples in total where 50 examples are reserved for a 50:50 split between validation and test sets).

Pick a specific example from this set of 150 examples, say ${x}_{i}$. You are required to classify this in any one of the classes: ${k}_{1}, {k}_{2}, ..., {k}_{n}$.

Now randomly pick 100 samples from your original set of 150 samples (excluding your ${x}_{i}$). Now suppose you choose a hypothesis that has many higher degree terms, for instance: ${a}_{1}{x}^8 + {a}_{2}{x}^{7} + ... {a}_{0} = 0$ and do not regularise it, it will fit the model nicely and you get a low error on your training set.

You now run this model on ${x}_{i}$ and it predicts wrongly. You repeat the above process of having 100 samples five times and each time you get a different result, say ${k}_{a},    {k}_{b}, {k}_{c}, {k}_{d}$, and  ${k}_{e}$.

This variability in your prediction (coupled with the low error on your training set) implies your model is overfitting the data it is given. That's why it is not able to follow the general pattern in your data and is being affected by outliers more than it should.

I guess that answers your question about declaring a model as highly variant when

  the predicted points are too prolonged


Why are character level models considered less effective than word level models? I have read that character level models need more computation power than word embeddings, and this is one of the major reasons for their less effectiveness, but i got curious because the word embeddings need a huge vocabulary while character level models need very less vocabulary, so why is it not considered?

You are absolutely right about vocabulary size. I am actually conducting research on making character-level more effective.

Here is why word-level tokens are often favoured despite, characters requiring a much smaller vocabulary size.

Bag-of-word

In a bag-of-words scenario, it is pretty obvious. First, the name. Second, if you receive a word cloud of most common words, you may be able to know what the document is about. If you receive a character cloud, you would likely be completely lost. So would your computer.

Character tokens contain much less information than word tokens.

Sequential approaches

Whether you use RNNs or Transformers, dealing with text as a sequence is often where the difference between words and characters is less obvious.

Words will create sequences containing larger vectors since each vector needs to encode more information. However, it is quite rare to use a one-hot encoding approach, which would require a vector to be the size of the vocabulary (usually 30,000). Instead, word embedding are used, which are usually less than 1024-dimensional, often between 100 and 300. So, practically, those vectors are not huge.

Characters can require much smaller embeddings. Based on my research, embeddings can be as low as 32-dimensional. But, on average, at least for western languages, each word contains 7 characters. Meaning that your models will need to deal with a sequence that is 7 times bigger than if you were dealing with words.

So in terms of actual size, if you use 32-dimensional embeddings for characters, it would take as much "space" to use 224-dimensional word embeddings.

Now, where is the difference? Ultimately it comes down to this:

Characters allow you to virtually not have any "out-of-vocabulary characters" (~99.99% using a vocabulary of 300 tokens) whereas words usually only cover 50% of a language if you use 30,000 tokens. Preprocessing techniques (stemming, amortization) are there to help reduce vocabulary size, but one can argue that it removes valuable information from the input.
Words are more informative than characters and word embeddings have shown to embed information really well. Basically, performance is often good enough, and computational constraints not too big for people to worry.

In summary, when using character-level tokens, your model has to do more work. The input itself is more raw and contains less information. Plus, sequential models have shown to struggle with long-range dependencies.


How to identify new job descriptions/postings from a set of documents when I have a set of already labeled job descriptions/postings Suppose I have a set of already labeled documents -- some of them are job descriptions/postings (these are documents of interest), and some of them are not. I wonder what kind of method would allow me to build a model that can generalize to new data, specifically new job descriptions that may be very different from the already labeled job descriptions.

What I have done so far (in Python):

(1) Trial 1: I tried the classic bag-of-words + TfidfVectorizer + binary classification (LogisticRegression specifcally) approach. With parameter tuning based on the train + valid sets, precision and recall on the test set could reach over 98%. However, after I built the model, I collected a new set of documents (again, some are job postings and others are not), and asked annotators to label these new data. I then used the model to classify these new docs. Perhaps expectedly, the precision and recall dropped pretty drastically to, ~65% and ~40%.

(2) Trial 2: I thought perhaps using tfidf alone might cause some overfitting, so I tried applying TruncatedSVD (with 100 components) on the TfidfVectorizer (bag-of-words + TfidfVectorizer + TruncatedSVD + binary classification).  Again, I tuned the parameters using the train and valid sets, and the precision and recall were similarly high on the test set. When I applied this new model to the newly collected data, the recall improved a bit, from 40% to 50%, but the precision dropped a bit, to 58%.

So my question is: is there a way to leverage the initial set of coded documents to build a kind of model that can generalize better to new data set. My specific interest is identifying job descriptions/postings. It doesn't have to be a binary classification task since the non-job-descriptions can be much more varied than job descriptions/postings

You should at least try pre-trained embedding vectors. TfidVectorizer is particularly sensitive to out-of-vocabulary words, which are likely to appear if you're trying transfer learning to a new domain. The GloVe embeddings [1] have a dictionary of 400k vectors, unless you're working with documents from a technical domain they should provide some improvement. Also, consider to perform some extra feature engineering, words by themselves do not provide much information. Why not trying adding n-grams & dependencies extraction, entity recognition, etc. as preprocessing steps?

[1] http://nlp.stanford.edu/projects/glove/


Autoencoder doesn't learn to reduce dimesions I coded a neural network from scratch in Python. I tried it with the XOR problem and it learned correctly. So I tried to encode an Autoencoder with 3 inputs (and therefore with also 3 outputs) to reduce a color (r, g, b) in one dimension. I have normalized the data from 0 to 1 so I can use activation functions like sigmoid, relu etc. I have tried many different activation functions and learning rates, but the Autoencoder error(calculated with the mean squared error) is high (the lowest I got is 0.1), although I have trained it for more than 30,000 iterations. Did I miss something? (I think so, but is it possible to reduce a color to a size with good accuracy in the first place?)

Thank you all

What kind of color are you trying to learn? If it's a random color, then yep, tough luck, you can't reduce the dimensionality of a 3D Euclidian manifold to less than 3D.

To be more precise, this is the case for an infinite manifold, for non-infinite one, you could get a 1D curve that goes through a 3D cube (in philosophy, it would be close to a Kohonen map, despite having just one node instead of a set of discrete nodes), but you would need a very big set of intermediate layers to make it happen.


sklearn package with AttributeError: 'MissingValues' object has no attribute 'to_list' I am currently trying to reproduce this tutorial on building a CNN based time series classifier for human activity recognition.
My setup is:

Windows 10, Pycharm IDE with a new project for this tutorial, Python3.6, freshly installed the needed packages.

For reproducing, you need to download the activity data here and place it in the project directory under ./Data
The code executes the graphs well until this position:
df[LABEL] = le.fit_transform(df["activity"].values.ravel())

and throws following error:
Traceback (most recent call last):
  File "C:/Users/bobin/PycharmProjects/Mussel/cnn_musseltest.py", line 226, in <module>
    df[LABEL] = le.fit_transform(df["activity"].values.ravel())
  File "C:\Users\bobin\PycharmProjects\Mussel\venv\lib\site-packages\sklearn\preprocessing\_label.py", line 117, in fit_transform
    self.classes_, y = _unique(y, return_inverse=True)
  File "C:\Users\bobin\PycharmProjects\Mussel\venv\lib\site-packages\sklearn\utils\_encode.py", line 31, in _unique
    return _unique_python(values, return_inverse=return_inverse)
  File "C:\Users\bobin\PycharmProjects\Mussel\venv\lib\site-packages\sklearn\utils\_encode.py", line 133, in _unique_python
    uniques.extend(missing_values.to_list())
AttributeError: 'MissingValues' object has no attribute 'to_list'

Related threads that have not helped me so far:
Link1
Link2

The problem was with the pandas version. I've used Python3.6 as interpreter and pycharm would only let me install version 0.25 pandas. Now I am on Python3.8 and pandas 1.2.3 and the minimum code example provided by @Oxbowerce works now. I'll be more careful with checking the dependencies in future.
The sketch goes into the epochs now, there was just a minor labelling error:
Traceback (most recent call last):
  File "C:/Users/bobin/PycharmProjects/Mussel/cnn_musseltest.py", line 348, in <module>
    plt.plot(history.history['acc'], "g--", label="Accuracy of training data")
KeyError: 'acc'

Renaming 'acc' to 'accuracy' and val_acc to val_accuracy helped, as stated in the warning log.
WARNING:tensorflow:Early stopping conditioned on metric `acc` which is not available. Available metrics are: loss,accuracy,val_loss,val_accuracy


Which is better: Out of Bag (OOB) or Cross-Validation (CV) error estimates? I have seen other posts in this forum but didn't find any convincing answer.

Random Forest has an another way of tuning hyperparameter via OOB by design. OOB and CV are not the same as OOB error is calculated based on a portion of trees in Forest rather by full Forest.

So what are the advantages and disadvantages of using OOB instead of a CV?

Is getting to train on more data by using OOB correct to say?

OOB samples are a very efficient way to obtain error estimates for random forests. From a computational perspective, OOB are definitely preferred over CV.

Also, it holds that if the number of bootstrap samples is large enough, CV and OOB samples will produce the same (or very similar) error estimates. Thus, if you perform many bootstrap samples, I would recommend performing cross-validation along the way with OOB samples until the OOB error converges.


How is the loss between actual and predicted labels being calculated? Quoting from this example of Predicting movie review with BERT and Tensorflow,

with tf.variable_scope("loss"):
        # Dropout helps prevent overfitting
        output_layer = tf.nn.dropout(output_layer, keep_prob=0.9)

        logits = tf.matmul(output_layer, output_weights, transpose_b=True)
        logits = tf.nn.bias_add(logits, output_bias)
        log_probs = tf.nn.log_softmax(logits, axis=-1)

        # Convert labels into one-hot encoding
        one_hot_labels = tf.one_hot(labels, depth=num_labels, dtype=tf.float32)

        predicted_labels = tf.squeeze(tf.argmax(log_probs, axis=-1, output_type=tf.int32))
        # If we're predicting, we want predicted labels and the probabiltiies.
        if is_predicting:
            return (predicted_labels, log_probs)

        # If we're train/eval, compute loss between predicted and actual label
        per_example_loss = -tf.reduce_sum(one_hot_labels * log_probs, axis=-1)
        loss = tf.reduce_mean(per_example_loss)
        return (loss, predicted_labels, log_probs)

How is the loss between actual and predicted labels being calculated?

The above example id for multi-class, single-label setting. If I want to calculate the loss for a multi-class, multi-label problem, how can I do the necessary computation in Tensorflow?

The per example loss is calculated by multiplying the one hot encoded result which will be 1 for the true class and 0 for the others with the softmax output which outputs a probability for each class.
Essentially we are picking the prediction for the relevant class.
In the next step we reduce the negative of this value which essentially is maximizing the probability of the class of interest.

The above approach should work fine for multi label as well. However using sigmoid instead of softmax to arrive at the probabilities for the multiple classes doesn't clip probabilities of other classes


Classify future performance of customer I have a dataset with monthly revenue per customer. I want to build a model that can try to predict if the customer will exceed $10,000 3 months out (yes/no).

While this seems like a traditional ML problem I have an important questions

Should I build my dataset with one row per customer id and let the label be the revenue 3 months out
Should I instead have one row per month per customer and let the label by the revenue 3 months out

Thanks

I'm not sure it will matter very much whether you have one row per user or if you have one row for per month per user. The important part is that the data you have is accurate for that user for a particular month. You might construct your training data like this:

--------|-----------------|--------------------|
cust. id| time on website | profit over 10,000?|
--------|-----------------|--------------------|
 3         30               0
--------|-----------------|--------------------|
 3         80               1
--------|-----------------|--------------------|
 5         100              1
--------|-----------------|--------------------|
 7         5                0
--------|-----------------|--------------------|

The important thing to notice is that even though customer 3 is in the dataset more than once, he/she has different values for their data features on which to predict, and different from how they were represented in the previous month. This is assuming that you are aggregating the data by month per customer.

This blog predicts customer churn, but you might be able to use the general strategy for your problem.


Transfer learning for a regression problem if my understanding is correct, in case of image classification and NLP, if I have a pre-trained model, to train on new data, I can reshape the data according to the pre-trained model. So there is no problem even if the new data is slightly different from the previous data.  I am trying to use transfer learning for a regression problem. Consider I train a base model with 15 parameters and 1 million rows. I train a model. Now if I want to use this model for a similar problem case but I have only 14 parameters, one parameter is missing. Will the pre-trained model be of any use. Is there a way I can use transfer learning in such cases?

Thank you.

The question is if you can provide the trained model with the data it expects to provide a good output.

In your example, for the model to provide an output, you will have to provide 15 inputs. If you have only 14, can you put a value for input 15 that will make sense to the model? If feature 15 is a person's age, can you make an estimate? If you trained a model on 16x16 images (256 features) but now you have images that are 4x4 (16 features), you may be able to just provide 256 - 16 = 240 black pixels and the model may work fine. Or you could reformat your data to multiply each pixel into 4x4 patch, and get a 16x16 image that way.

What you cannot do, is simply reshape the input so that it's 15 columns wide. If you do that, the first example you provide to your model will contain 14 features from the first real example, plus feature number 1 from the next case. That way, the model is using data from two different examples to make a prediction and it's getting a value for feature 1 on the input node where it expects a value feature 15, and feature 1 and feature 15 can have very different meaning.


Handling categorical data with more over 100 unique classes I am working with a pure categorical data set. And some classes have more than 100 unique values.
I could not find any appropriate encoding possibility. So I created a SQL table, where each value got its ID. Then I extracted the IDs and used it in ML Classification.
However the results are poor.
So anyone has an idea how to encode such values better?

Even though you have 100 unique values - Do you see a pattern where perhaps 80 of those unique values appear only once or twice - and the remainder 20 occur much more often.
In that case - you may want to encode the 80 with a special "Other" value. By doing this you lose information - by not being able to distinguish between the 80 categories - but perhaps it doesn't matter to your problem.
The other option is - You can convert your categorical data into an integer (perhaps the ID approach you are talking about ?) and run a Decision Tree model (e.g. Random Forests) on it. My experience with Random Forests is that they work pretty well on such data.


Excel file merge with different headers but same data I need to merge data from 1000s of excel files provided by different operations managers on productivity and other reports. The excel files have similarity of data but the headers are all custom since being different manager and different clients.

For example manager A will have a.xlx and manager b will have a.xlx, but the headers for each will be different though the data inside will be usually the same. Each day 100 different excel files are updated by all team members via new files e.g a_todays_date.xlx and manager b using /a.xlx.

Is this something that can be handled via python ML libraries.

What is the best way to merge all these data and save to the DB on a per day basis. The average data would per day would be around 15GB. The end goal is to create dashboard.

I suggest SSIS (SQL Server Integration Services) .
It is designed for handling collecting data  from different sources and export  to database.
You can design a simple data flow and run it from SQL serevr on schedule (daily in your case)


Is PCA considered a machine learning algorithm I've understood that principal component analysis is a dimensionality reduction technique i.e. given 10 input features, it will produce a smaller number of independent features that are orthogonal and linear transformation of original features.

Is PCA by itself considered as a learning algorithm or it is a data pre-processing step.

PCA is actually just a rotation. Seriously, that's all: it's a clever way to spin the data around onto a new basis. This basis has properties that make it useful as a pre-processing step for several procedures.

The basis is orthonormal. This is incredibly useful if your features exhibit multicolinearity (two or more features are linearly dependent): applying PCA is guaranteed to give you a basis where this is no longer a problem. This procedure is known as principal component regression
The basis vectors are meaningful with respect to the spread of the data: they are the eigenvectors of the covariance matrix. This second property gives rise to PCA's famous utility as a dimensionality reduction technique: after rotating the data, projecting the data onto a subset of the basis vectors associated with a significant portion of the total variance yields a lower dimensional representation that (often) retains (most of) the (interesting) structural properties of the data.

So: is it a learning algorithm? This is sort of a philosophical question. What makes something a learning algorithm? Certainly PCA isn't a "supervised" learning algorithm since we can do it with or without a target variable, and we generally associate "unsupervised" techniques with clustering.

Yes, PCA is a preprocessing procedure. But before you write it off completely as not "learning" something, I'd like you to consider the following: PCA can be calculated by literally taking the eigenvectors of the covariance matrix, but this is not how it's generally done in practice. A numerically equivalent and more computationally efficient procedure is to just take the SVD of the data. Therefore, PCA is just a specific application of SVD, so asking if PCA is a learning algorithm is really asking if SVD is a learning algorithm.

Now, although you may feel comfortable writing off PCA as not a learning algorithm, here's why you should be less comfortable doing the same with SVD: it is a surprisingly powerful method for topic modeling and collaborative filtering. The properties of SVD that make it useful for these applications are exactly the same properties that make it useful for dimensionality reduction (i.e. PCA).

SVD is a generalization of the eigendecomposition, and that too is extremely powerful even as a constrained version of SVD. You can perform community detection on a graph by looking at the eigenvectors of the adjacency matrix, or determine the steady-state probabilities of a markov model by looking at the eigenvectors of the transition matrix, which coincidentally is also essentially how PageRank is calculated.

Under the hood, PCA is performing a simple linear algebra operation. But, this is exactly the same operation that underlies a lot of applications that most people wouldn't question applying the label "machine learning" to. This class of algorithms is called Matrix Factorization, and even extends to sophisticated techniques like word2vec: indeed, you can actually get word2vec-like results by literally just applying PCA to a word co-occrrence matrix. Generalizing again, another word for the results of PCA is an embedding. Word2vec is probably the most famous example of an embedding, but constructing embeddings (as intermediaries) is also an important component of the encoder-decoder architecture used in RNNs and GANs, which are the bleeding edge of ML research right now.

So back to your question: is PCA a "machine learning algorithm?" Well, if it's not, you should be prepared to say the same about collaborative filtering, topic modeling, community detection, network centrality, and embedding models as well.

Just because it's simple linear algebra doesn't mean it isn't also magic.


How should I choose n_features in FeatureHasher in sklearn? How should I choose n_features for FeatureHasher in scikit-learn? Assume that I have 1000 categories in feature "case" and I would like to hash them.

As mentioned in its documentation, it is advisable to use a power of 2 as the number of features; otherwise, the features will not be mapped evenly to the columns. Also, it is suggested to leave the number of features as its default value of 2 ** 20 for a real-world setting. Select a lower value such as 2 ** 18 when memory or downstream model size is an issue Ref.
Consider in mind that, as also stated in the documentation, small numbers of features are likely to cause hash collisions, but large numbers will cause larger coefficient dimensions in linear learners Ref
My overall suggestion is to use a power of 2 as the number of features. If you have a small number of features, you can treat the number of features (e.g., 30 as you mentioned in the comment below) as a hyperparameter and find the optimal value using cross-validation. For example, you can test different powers of 2 such as 2, 4, 8, 16, etc depending on the size of your data and use the cross-validation to find the optimal value. That is the best solution. But please note that the hashing method is efficient when the number of input features is very large. In your case, I would go with the other methods available in the technical literature.