Converting a speech recognition model from CNNs to GRUs I am trying to convert the simple audio recognition example from TensorFlow to use GRUs instead of CNNs.
The idea is to classify an audio clip onto a set of 8 labels: ['go', 'down', 'up', 'stop', 'yes', 'left', 'right', 'no']
The original code builds a model as follows:
norm_layer = preprocessing.Normalization()
norm_layer.adapt(spectrogram_ds.map(lambda x, _: x))

model = models.Sequential([
    layers.Input(shape=input_shape),
    preprocessing.Resizing(32, 32),
    norm_layer,
    layers.Conv2D(32, 3, activation='relu'),
    layers.Conv2D(64, 3, activation='relu'),
    layers.MaxPooling2D(),
    layers.Dropout(0.25),
    layers.Flatten(),
    layers.Dense(128, activation='relu'),
    layers.Dropout(0.5),
    layers.Dense(num_labels),
])

The input shape is (124, 129, 1) - single channel spectrograms with 124 time steps and 129 frequency bins. The X data is the spectrogram and Y data is an integer label which is an index into the labels array.
I have tried converting the above into using GRUs as follows:
for spectrogram, _ in spectrogram_ds.take(1):
  input_shape = spectrogram.shape #(None, spectrogram.shape[0], spectrogram.shape[1])
print('Input shape:', input_shape)
num_labels = len(commands)

model = models.Sequential([
    layers.Input(shape=input_shape),

    # Step 1: CONV layer (≈4 lines)
    layers.Conv1D(196, 15, strides=2, input_shape=input_shape),
    layers.BatchNormalization(),
    layers.Activation('relu'),
    layers.Dropout(0.8),

    # Step 2: First GRU Layer (≈4 lines)
    layers.GRU(128, return_sequences=True, input_shape=input_shape, reset_after=True),
    layers.Dropout(0.8),
    layers.BatchNormalization(),

    # Step 3: Second GRU Layer (≈4 lines)
    layers.GRU(128, return_sequences=True, reset_after=True),
    layers.Dropout(0.8),
    layers.BatchNormalization(),
    layers.Dropout(0.8),

    # Step 4: Time-distributed dense layer (see given code in instructions) (≈1 line)
    layers.TimeDistributed(layers.Dense(num_labels, activation = "sigmoid"))

])

The input shape in this case is (m, 124, 129). The Conv1D layer reduces the time steps from 124 to 55.
In this case, the X data is still the spectrogram data. For Y data, I had to replicate the label index over 55 time steps. So Y has shape: (m, 55, 1).
The training is done as follows:
model.compile(
    optimizer=tf.keras.optimizers.Adam(),
    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    metrics=['accuracy'],
)

EPOCHS = 10
history = model.fit(
    train_ds,
    validation_data=val_ds,
    epochs=EPOCHS,
    callbacks=tf.keras.callbacks.EarlyStopping(verbose=1, patience=2),
)

The issue is that I am getting a very low accuracy for the GRU model compared to the CNNs. The GRU training is also very slow. My feeling is that I have not set up the model correctly - especially I am not sure if I have the Y data setup correctly for the GRU.
I'd appreciate any insights in setting up this model correctly. Thanks!

I think your network's design is fine but need change the way you feed training data.
In essence you try convert a classic CNN (having "image" data, but spectrograms) to a sequential time-domain type of network (recurrent one, having GRU or LSTM at the heart).
To be able to do that you have to reconsider the way you feed the input data:

Input data should not be the "image" snapshots of spectra (as expression of audio spectra in a time-window) but instead have to feed evolving sequences like text ones (the .wav audio samples in sequential way).

If still would like to feed with spectrograms and not raw sequential wave samples, try feed the spectrograms by cloning each of them into time evolving variants having offset by t, t+1, t+2, t+N variants of same audio snapshot (recompute FFT as many times) thus recurrent GRU unit may have chance to capture sequence information out if it.

Also:

BatchNorm & Relu was added, a good way to improve generalization and overfit, at the price that might requires more data to see benefits.
Pooling & Flatten summarisation was replaced by GRU recurrent unit, but these units learns only when capturing from time evolving data.
Very Conv2d in front of GRUs helps generalization but Conv2d x times x GRU also means lots of combinatoric space opened up, implict might require more input data or prolonged training.

Anyway is a fun experiment worth trying !


Creating a metric based on some features I want to create a new metric based on some features but dont know how to start. I basically want to create a "job satisfaction level" metric based on some features. The features could be work hours, shift, If working on weekend and so on. I dont know how to start. In ideal world, I want to comp up with weights for each of these features and compute a final value and then put the final value in a job satisfaction level bucket. I then want to use this metric in my training model. Is there any methodology to do so? Lets assume I have different warehouses with different values for those features and I want to compute a "job content" or "job satisfaction" metric based on the features I mentioned above for all of these locations. Then I want to use this new computed metric with my other features for an employee resignation prediction. Any help is appreciated.

Thanks

Indeed, there are methodologies that have been tested elsewhere, some with greater and less success.

I will propose one of them to build a prediction of job satisfaction, which you can then enter as an explanatory variable in a supervised model of employee resignation, whose methodology you can review in this tutorial with Python code that I did some time ago: HR analytics MVP

Methodology to generate a satisfaction level prediction: Deduce the importance of the variables from a score that represents the satisfaction declared by a subset of the members of your company

I think the best way to start doing a good MVP (minimum viable product) with which you can deliver relatively fast results having a result that incorporates elements of your company is one in which you derive the importance of the features from a dataset in which have your explanatory variables and a target with a declarative satisfaction survey made to the workers from which the score that is the variable explained was calculated. For this you must follow the following steps:

1.-You design a satisfaction survey that will be answered by the workers and that will allow you to calculate a Score from it. Here the important thing is that the design of the survey is as complete as possible, that the number of respondents allows you to draw conclusions at a statistical level and, most importantly, that of those who answer the survey have how to extract the raw data that allows you later deduce which are the most relevant variables. Here are some resources that can give you some ideas of how to generate the satisfaction level index

2.-Then, using that dataset generated in step 1, you can make a feature engineer and establish which variables have the greatest impact on the satisfaction declared by the workers.

3.-Solved the point 2 you can generate predictions on the score and apply your model to the future and with other workers of the same company.

Important: Whenever you run the prediction for the next period you should do a few satisfaction surveys in each iteration to confirm that the model is still valid and to use that data as a permanent retraining. In general, the model should be useful as long as the context of the company does not undergo major changes (mergers, significant deterioration of the work environment due to massive dismissals, etc.), since in such cases you should try to capture the short and long term effects of these shock

Although this methodology is a good starting point, it is omitting many things that are difficult to detect for a company because it corresponds to exogenous variables to it, such as:

a.- That the person changes his interests and / or goals in terms of career. Example: a software developer who wants to change the focus of his career towards a more commercial facet or another specialty such as Data Science or Data Engineer

b.-That the person change their objectives and / or prioritize them in their life. Example: A person who wants to begin to dedicate more time to his personal life because he went through a crisis with his partner

Here is an example of where they used that methodology: Mining the drivers of job satisfaction using algorithmic variable importance measures

PD: There are other lines of research that avoid extracting the satisfaction index from the direct query to the employee and occupy other variables such as equivalent income or time spent in the company as equivalent metric. It is not my favorite line, but here I leave an example of that: Using equivalent income as metric


CASE sentence for SQL query I have a question abut MySQL

I would like to handle a column with characters without changing the original DB. Below is the levels of the column named 'Maker'.

기아, 현대, BMW, VolksWagen, Porsche
So there are about 10 기아s, 20 현대s, 30 BMWs and so on.

As you can see, some of them are written in Korean and others are in English.
I do not want to touch English-written words. I only want to change Korean words into English.

I tried to you CASE and IF sentences like below.

SELECT
    CASE
        WHEN Maker='기아' THEN 'KIA'
        WHEN Maker='현대' THEN 'HyunDai'
    END AS new_Maker
FROM table_name

It successfully changes KIA and HyunDai but BMW, VolksWagen, and Porsche shows as NULL.
I would appreciate any advice. Thank you.

You haven't specified what happens for values other than '기아' ('KIA') and '현대' ('HyunDai').
Try

SELECT
CASE
    WHEN Maker='기아' THEN 'KIA'
    WHEN Maker='현대' THEN 'HyunDai'
    ELSE Maker
END AS new_Maker

FROM table_name


Regression model performance with noisy dependent variable I'm doing a support vector regression with the dependent variable representing measurements from an uncalibrated sensor (measurement error between 2% and 20%) and I want to study the effect of this error on the model performance.
What's the best method to inject error on the dependent variable samples ? is it sufficient to do this by adding white noise?
Any help or suggestion is appreciated

Adding white noise is simple enough and should work, alternatively you could permute the values of your variable, which is another commonly used method.


Can features passed into Conv1d layers be randomized? If I have input time series data shape as such, X.shape = (batch_size, 50, 5), it means that the data has 5 "features", each having 50 time steps. Does passing data like this into a Conv1d layer mean that the 5 features have some kind of spatial relationship with each other, like how a pixel in an image is more closely related to the pixels nearby around it than further away? Is there a way to remove this feature relationship when dealing with time series data like this? As in, even if the order of the features were randomized, the network would still learn from the patterns seen in 1d space of each feature?

Typically in a Conv1D layer for a time series, the features can be different measurements taken or recorded at the same time period. So they can be related to each other. For example, if you are trying to predict time=4 below, the question is whether there is a relationship between your features, meas1 and meas2? If so, you want to keep the features together to give you a better result.
| time | meas1 | meas2 |
|------|-------|-------|
| 1    | 4     | 6     |
| 2    | 3     | 7     |
| 3    | 2     | 8     |
| 4    | ?     | ?     |

You can look at cross-correlation in your dataset to answer this. Since this is time series data, you can use Pearson's correlation for this.


K-Means Algorithm - Feature Selection Suppose I've this dataset:

Employee_ID  Store_ID  Company_ID  Stock_ID  Product_Value
    1            1          1          2           3.7
    4            1          4          2            8
                           ...

Where:

Employee_ID: Is the unique number for a employee that sold the product
Store_ID: Unique number for Store Chain
Company_ID: Unique number for product supplier
Stock_ID: Unique number for the product purchased
Product_Value: Product Value

And I want to make a segmentation over my dataset using K-Means... Its make sense use all the variables for my dataset?

So long as you want to use those variables to define closeness then yes, you just have to encode them differently. You should use a one-hot-encoding for the discrete variables (e.g., Employee_ID, Store_ID, etc.) and just the Product_Value as is. I answered a similar question about K-means for just categorical variables (like Employee_ID) so I've copied the code below that gives a quick demo for using the Clusters as a feature for predicting something after you use K-means. As I said in my old answer, in general, this framework isn't optimal but it's okay for a simulation.

library(glmnet)
library(Matrix)
n <- 1e5
nclusters <- 5
set.seed(420)
ls <- data.frame(sample(letters, n, replace=TRUE))
xs <- sparse.model.matrix(~.,data=ls)
print(head(xs))
# Now let's run k-means
out <- kmeans(xs, centers=nclusters)
bs <- rep(1, dim(xs)[2])
# Let's run k-means on the different categories
clusterpred <- data.frame(out[[1]])
ys <- xs %*% bs + rnorm(n)
print(table(clusterpred))
# Now let's use a clustered data set to predict some outcome
cxs <- sparse.model.matrix(~.,data=clusterpred)
model <- glmnet(y=ys, x=xs, alpha=0)
cmodel <- glmnet(y=ys, x=cxs, alpha=0)

# Predictions
yhat <- predict(model, xs)
yhatc <- predict(cmodel, cxs)
# Looking at the difference RMSEs
print(sqrt( sum( (ys-yhat)**2 )))
print(sqrt( sum( (ys-yhatc)**2 )))


why the RFE to select features is changing when the os or computer is changed I used the the feature selection method: RFE to select feature for features. Now I want to select 20 features(there are about 50 features or more) with LGBM model as following code: But I found that the features(sel.ranking_) are different when the os or os version, or computer is changed. I don't know what caused the changing. More, how to solve it that features selection is fixed. Thanks!

gbm = lgb.LGBMClassifier(
    boosting_type='gbdt',
    objective='binary',
    learning_rate=0.01,
    colsample_bytree=0.9,
    subsample=0.8,
    random_state=21,
    n_estimators=200,
    num_leaves=18)
sel = RFE(gbm, step=1, n_features_to_select=20, verbose=1)
sel.fit(X_train, y_train)
print (sel.ranking_)

Since you do not provide any information about the creation of X_train and y_train I cannot be sure that this is the issue, but I would guess that it has to do with some change in the order of the features. Three-based algorithms' results are affected when the order of the input features changes, so make sure that this order does not change every time you run your code (e.g. you can sort them alphabetically before RFE).

You might find this discussion interesting.


Converting RDD to spark data frames in python and then accessing a particular values of columns I want to access values of a particular column from a data sets that I've read from a csv file. The datasets are stored in pyspark RDD which I want to be converted into the DataFrame.
I am using the below code :

from pyspark.sql import SQLContext
sqlc=SQLContext(sc)
df=sc.textFile(r'D:\Home\train.csv')
df=sqlc.createDataFrame(df)

but it is showing error:

Can not infer schema for type: <class 'str'>

First 2 rows of df are :

['"id","product_uid","product_title","search_term","relevance"',
 '2,100001,"Simpson Strong-Tie 12-Gauge Angle","angle bracket",3']

I think the first row is creating this problem. Moreover I want to create data frame which stores the values from 2nd row to last.(Not the first row because it will be the header). How can I achieve this ? I've searched for it but could not find any solution. Thanks in advance.

To read a csv file to spark dataframe you should use spark-csv.
https://github.com/databricks/spark-csv

df = sqlContext.read.format('com.databricks.spark.csv').options(header='true', inferschema='true').load('cars.csv')

How to use spark csv
If you are using pyspark directly from the terminal. Instead of calling

$SPARKHOME/bin/pyspark

You have to use

$SPARKHOME/bin/pyspark --packages com.databricks:spark-csv_2.11:1.4.0

and then use the code above.

If you are using ipython + findspark, you'll have to modify your PYSPARK_SUBMIT_ARGS (before starting ipython)

export PYSPARK_SUBMIT_ARGS=--master local[4] --packages "com.databricks:spark-csv_2.11:1.4.0" pyspark-shell


Evaluate Keras model with scikit-learn metrics How does Keras calculate accuracy, precision, recall, and AUC?
I've created a model for categorical classification (i.e., multiple classes) by using keras.losses.CategoricalCrossentropy() as loss function, and in the model.compile() method I've specified the following metrics:

METRICS = [
    keras.metrics.CategoricalAccuracy(name='acc'),
    keras.metrics.Precision(name='precision'),
    keras.metrics.Recall(name='recall'),
    keras.metrics.AUC(name='auc'),
]

When I evaluate the model with model.evaluate(X_test, y_test, verbose=2) the result is the following:

  147/147 - 1s - loss: 0.5127 - acc: 0.8073 - precision: 0.8437 - recall: 0.7696 - auc: 0.9537

Then, I want to compare its performance with the results obtained with other algorithms (e.g., DecisionTreeClassifier() implemented in scikit-learn).

To evaluate the other classifiers, I'm using the following:

from sklearn.metrics import balanced_accuracy_score
from sklearn.metrics import roc_auc_score
from sklearn.metrics import precision_score
from sklearn.metrics import recall_score

print("AUC: %2.3f" % roc_auc_score(y_test, pred_bin))
print("ACC: %2.3f" % balanced_accuracy_score(lb.inverse_transform(y_test), pred))
print("PRE: %2.3f" % precision_score(lb.inverse_transform(y_test), pred, average='weighted'))
print("REC: %2.3f" % recall_score(lb.inverse_transform(y_test), pred, average='weighted'))

However, when I test the performance of my NN with this piece of code, I obtain the following results:

  ACC: 0.715 PRE: 0.801 REC: 0.807 AUC: 0.920

As you can note, the result differ from those obtained with model.evaluate() of Keras.
How can I fairly compare the different methods?

You could use class KerasClassifier from keras.wrappers.scikit_learn, which wraps a Keras model in a scikit-learn interface, so that it can be used like other scikit-learn models and then you could evaluate it with scikit-learn's scoring functions, e.g.:

from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.metrics import roc_curve, auc

keras_model = ...
classifier = KerasClassifier(keras_model, batch_size=32)
y_pred_keras = classifier.predict(X_test).ravel()
fpr, tpr, thresholds = roc_curve(y_test, y_pred_keras)
auc_score = auc(fpr, tpr)

You can find more info on the Keras docs, or googling around (e.g. this and this).


How do I approach learning Data Science/ML the 'rightest' way? First of all, I am not sure if this is the right place to post this so please do let me know if it isn't and tell me where it should be. I really just don't know where to go with this question.

Some backstory: I am going through my 2nd semester of my 2nd year of Software Engineering. I love math, I love computers, I love data and I love image processing so the most logical place for me to be is Data Science because I seem to be gravitating towards ML/CV a lot.

Thing is, I want to start learning Data Science/ML but I don't know where to start. Everyone keeps recommending online courses such as Andrew's coursera course but I have become somewhat skeptical of most online courses because they seem to love simplifying info too much that I feel like it just passes the threshold of it being useful info.

My assumption is: learning Data Science/ML/CV requires a very strong and rigorous foundation. I shouldn't start learning it with high level tools and little understanding of what makes it what it is. This is my problem. I don't know where to begin learning that way.

I searched for books and the two books I found people talking about were:

Pattern Recognition and Machine Learning, by Christopher M. Bishop. This book seemed to be very rigorous with a strong focus on the mathematics and intuition of things (looking at its index). Not much skepticism towards this one.
Hands-On Machine Learning with Scikit & TensorFlow, by Aurelien Geron. This one seems to be extremely popular and almost unanimously praised/liked/recommended by people but just looking at its name I felt a bit skeptical that it might have less focus on building strong foundation/intuition and more focus on using high-level tools.

I am open for other suggestions if you think some 3rd book/resource is more adequate for my need. I'd also greatly appreciate it if you briefly explain why you think it is more adequate.

Also it's worth noting that I have no problem with high-level tools and I know that I will eventually have to use them. I just feel like I'd be significantly better at what I want to do and valuable if I start from there.

Welcome to the community!

I normally don't answer questions like this, they're pretty broad and there are many others just like it. It seems like you're not actually asking how to get started, you already seem set on reading some books (which is great!), so let me answer that question, rather than a very broad 'how to get started'.

First of all, I'd like to challenge your assumption about learning from the bottom up, and offer an alternative opinion (yes, this part is just an opinion). I'd instead encourage you to get your hands on the high level tools and start using them right away. If everything we ever did required us to forego abstractions and learn everything from the beginning, we'd still be reinventing the wheel over and over. Start experimenting with the existing tools and supplement that with your reading. You'll develop an excellent understanding this way.

So to answer your question... I'd recommend the second book you quoted: Hands-On ML with Scikit Learn and Tensorflow. This book will teach you good habits right from the beginning, give you a good understanding, and get you using those tools right away.

Best of luck.


Different learning rate for each of the layers? I noticed that some popular deep learning frameworks like Keras or Pytorch allow you to set different learning rate for each layer.

What are the benefits of that approach?

In trivial update rules like gradient descent, the learning rate is important and it somehow specifies the speed you go downhill. In popular papers like Adam optimisation technique, and in non-paperised(!) popular solution namely RMSProp the authors cared that the slope of different features may vary differently and in a direction you may need to go faster due to its slope. Consequently, They decided to set the learning rate and update each parameter based on its own slope and this learning rate is somehow affected by the slope of each direction independently to the other dimensions. The motivation is this. As far as I know, you just need to set the learning rate for your optimisation and it will be adapted by itself.


How can I improve my model on a very very small dataset? I am starting as a PhD student and we want to find appropriate materials (with certain qualities) from basic chemical properties like charge, etc. There are a lot of models and datasets in similar works, but since our work is pretty novel, we have to make and test each data sample ourselves. This makes the data acquisition very very slow and very expensive. Our estimated samples will be 10-15 samples for some time, until we can expand it.

Now I want to use this samples to make a basic predictive model, but with as much 'good generalization' as possible. I will use this model to screen other possible candidates from a large pool of properties to find best probable materials, and will then proceed to make them for testing.

Now I clearly don't expect performances anywhere near 95% or so, but I want a working model with predictive capability that will actually help me to find some of the best probable material candidates, so we can expand our work. I am uncertain if I can (or rather should) use some of regular ML methods like dataset splitting and cross-validation. So I would appreciate your thoughts.

Since our data size is minuscule, I have been searching for ways to improve its robustness. These our my ideas:

1- Use an ensemble model to avoid overfitting and avoid skewed biases (using algos like elasticnet, SVM, random forests, etc).

2- Setting heavy regularization to avoid certain biases that can arise from small data.

3- Using algos that arrive at the minimum periphery faster.

I would appreciate any suggestions on how I can improve this model as much as possible, to reach the best generalization performance.

I have also thought about synthetic data generation a lot. Do you have any suggestions on how I can go about it?

From what you say, I think you should start with checking three options:

I) Ordinary least squares (OLS): Just run a „normal“ linear regression. This will not yield great predictions, but you could view the model as a causal one, if you can assume a linear relation between $y$ and $x$. When you have five predictors and 35 observations, you have a total of 29 degrees of freedom which is „okay“. When you estimate the model in „levels“, so just values as they are, you can directly interprete the estimated coefficients as marginal effects. E.g. a model $y=\beta_0+\beta_1 x + u$, tells you that when $x$ increases by one unit, $y$ changes by $\beta_1$ units, just like a linear function.

II) You can use Lasso/Ridge/Elastic Net: All of them are linear-like models with a penalty term to „shrink“ $x$ variables if they are „not useful“. This works like automatic feature selection if you like to say so. There is a great package by Hastie et al. for R. You can find it here. It is also available for Python.

III) Maybe (!) boosting could be an option as well: You would (likely) need to do some feature selection/engineering on your own. But Boosting is able to work with a small number of observations, with highly correlated features, and it often works well with highly non-linear problems. There are LightGBM or Catboost as possible Python packages. Find some minimal examples here.

With II) and III) you will find that you are not really able to „set aside“ a number of observations to check if your models work (because you don’t have much data). You could use cross validation (Ch. 5 in ISL, link below), but you need to see how it works. Instead of going for a predictive model, I tend to say that you might be better off starting with a „causal-like“ OLS model. With OLS you do not really need a „test-set“. OLS is very robust.

Since you seem to be new to statistical modeling, you might benefit from having a look at „Introduction to Statistical Learning“ (Chapters 3 and 6 in particular). The PDF is online and there is code for the Labs in Python and R. The advanced book would be „Elements of Statistical Learning“.

Good luck with your project!


Are there any examples of neural networks that take two samples as input with a label of {same class, different class}? There must be examples of this, though I haven't been able to find any. Maybe I don't know what to search for.

Yes. This is how a face recognition algorithm might work for example, where two pictures might be of the same person or different person.

To build such a system, just pair up your data in a training set, double the input vector space and run a binary classifier that outputs "true" if the two items are the same. Any ML classifier could be adapted to this problem.

In practice, you may want more control over classification, and be robust against new classes that your algorithm has not seen before. A big problem in face recognition is the large number of potential classes, including classes not seen in training data, coupled with an equally large variance of images in the wild. This means that a naive approach as above will have poor performance in practice.

There is a more sophisticated approach: Triplet Loss. This requires that you train with three inputs for each example. Unlike the naive version:

The input is one image at a time. You train in triplets - an "anchor", and a postive match, and a negative match, in order to calculate one loss value for backprop.
The output is not a same/different class, but a vector description of the object. Similarity of objects is the distance between vectors. You do not need a label for this vector, even though this is supervised learning, thanks to how the loss function works.
The loss function is based on difference of distance from anchor example to positive example compared with a desired higher distance to negative example. This encourages learning key features of the inputs by making distance between vectors low (ideally zero) when they represent the same class, and as high as possible when they represent different classes.

In both the naive approach and triplet loss approach, you need to be careful about selecting training data. You want to make the learning algorithm work hard to learn key differences, otherwise it is too easy to get a good loss. So taking MNIST as an example, when looking at negative matches, you want to pair up more (1, 7), (3, 2), (3, 8), (4, 9) etc pairs than (0,1), (9, 5). There are ways to drive this selection based on feedback from previous training epoch.


Encoding with OrdinalEncoder : how to give levels as user input? I am trying to do ordinal encoding using:

from sklearn.preprocessing import OrdinalEncoder

I will try to explain my problem with a simple dataset.

X = pd.DataFrame({'animals':['low','med','low','high','low','high']})
enc = OrdinalEncoder()
enc.fit_transform(X.loc[:,['animals']])

array([[1.],
       [2.],
       [1.],
       [0.],
       [1.],
       [0.]])

It is labelling alphabetically, but if I try:

enc = OrdinalEncoder(categories=['low','med','high'])
enc.fit_transform(X.loc[:,['animals']])

Shape mismatch: if n_values is an array, it has to be of shape (n_features,).

Which I do not understand. I would like to be able to decide how the labelling is done.

I considered doing this:

level_mapping={'low':0,'med':1,'high':2}
X['animals']=data['animals'].replace(level_mapping)

However, I have large number of features in my dataset which have similar categories.

Thanks.

I'm not sure if you ever figured this out but I was trying to find answers on this exact same question and there aren't really any good answers in my opinion. I finally figured it out though.
OrdinalEncoder is capable of encoding multiple columns in a dataframe. So, when you instantiate OrdinalEncoder(), you give the categories parameter a list of lists:
enc = OrdinalEncoder(categories=[list_of_values_cat1, list_of_values_cat2, etc])

Specifically, in your example above, you would just put ['low', 'med', 'high'] inside another list:
end = OrdinalEncoder(categories=[['low', 'med', 'high']])
enc.fit_transform(X.loc[:,['animals']])
>>array([[0.],
         [1.],
         [0.],
         [2.],
         [0.],
         [2.]])
# Now 'low' is correctly mapped to 0, 'med' to 1, and 'high' to 2

To see how you can encode multiple columns with their own individual ordinal values, try this:
# Sample dataframe with 2 ordinal categorical columns: 'temp' and 'place'
categorical_df = pd.DataFrame({'my_id': ['101', '102', '103', '104'],
                               'temp': ['hot', 'warm', 'cool', 'cold'],
                               'place': ['third', 'second', 'first', 'second']})

# In the 'temp' column, I want 'cold' to be 0, 'cool' to be 1, 'warm' to be 2, and 'hot' to be 3
# In the 'place' column, I want 'first' to be 0, 'second' to be 1, and 'third' to be 2
temp_categories = ['cold', 'cool', 'warm', 'hot']
place_categories = ['first', 'second', 'third']

# Now, when you instantiate the encoder, both of these lists go in one big categories list:
encoder = OrdinalEncoder(categories=[temp_categories, place_categories])

encoder.fit_transform(categorical_df[['temp', 'place']])
>>array([[3., 2.],
         [2., 1.],
         [1., 0.],
         [0., 1.]])


Creating queries on the fly and general manipulation for dataset of half a million data records What would one use for manipulating data of the kind below ?

a) Data is bio-markers of different globs

GlobA 3 4 5 ....
GlobB 2 1 1 ....
GlobC 3 2 1 ....

b) Manipulations are queries like:
- show me the Globs where average of each Glob in file Efficiency is greater than 50%
- or show me sorted Glob list where first and second sort criterion are xx and yy)
- Construct a chart of globs that differ in criteria x by integer 3 (show me globs whose average over 5 runs for Structure and Efficiency differ from their nearest neighbors by 3)

Currently, this data is stored in a 100MB Excel file that is painfully slow to load on the speediest computer our lab can afford.

Ideally, there would be some open source program that accepts csv files of this data and has ability for user to construct queries that can be stored in a library for easy pulling up, charting abilities would be great too.
Here are 2 files (real data would be around 40 such files each file containing 20K rows):

Efficiency File:

Glob,Run1,Run2,Run3,Run4,Run5
SigX,6.2,4.8,2.4,4.32,5.59
SigY,8.44,8.16,5.99,0.98,9.6
SigZ,0.00,0.00,0.00,0.01,0.20

Structure File:

Glob,Run1,Run2,Run3,Run4,Run5
SigX,3.2,3.8,2.4,7.32,6.32
SigY,2.4,5.16,6.99,0.98,9.6
SigZ,1.02,0.00,2.23,0.01,0.20

Checkout the R programming language, specifically the dplyr and ggplot2 packages.


Remove correlated features before or after splitting test and training set? I want to remove highly correlated features before training my classifier. I am wondering if I should do this before or after splitting the test and training set. I don't immediately see how doing it before would leak test information into the training set, but I could very well be missing something. I'm also concerned that the split of the data will impact the analysis of correlation between features, though maybe in practice the likelihood of this is low. Anyone have any insight? Thanks!

By principle in supervised ML any decision which affects the model should be made using only the training set, in order to avoid data leakage. Following this principle requires the training/test split to always be done first.
Quite often the decision seems minor and has little chance to cause data leakage, like in your case. It's tempting to make our life easier and use the whole data for some pre-processing like this. However it's a good idea to follow this principle as strictly as possible not because the effect of data leakage would be huge, but because it would be impossible to detect it if it happens: when it happens, data leakage means that the performance on the test set is biased (overestimated), and there's simply no way to know by how much (except by collecting a new test set, of course).
Your concern that the split of the data could impact the analysis of correlation is a good example of the problem: suppose there would be a difference indeed. This means that there is data leakage, since the features are different depending directly on the test data. In this case the evaluation on the test set is likely to be biased.
Another way to explain the same idea is this: if something happens or doesn't happen in the training set by chance, then it's important to see if the model is able to focus on the real patterns and ignore the noise. The only way to obtain a reliable answer to this question is if nothing in the model depends on the test set.


What model should I use to find a common pattern for a specific user group based on the other dimensions? I have a big .CSV database of 25k users with various attributes of the last user's activity and events during the past 6 weeks

This is an example of the data:

username        (B)      (C)             (D)        (E)
nicole          524      329             203        787
asteria         197      186             286        120

I want to create a common behavior pattern based on the values of the attributes of each user and run an algorithm to find a common pattern that defines this group's behavior and to find out if there is any correlation in the dimensions values and which dimension define this list of users.
I am fully aware that correlation does not necessarily equal causation.

Now I see several challenges in front of me and would greatly appreciate some input from others, or some good resources to find further information.

What is the model for this problem ?
What kind of Algorithm is the best to deal with this situation?
What is the tools that you recommend to use of the project ?

Any ideas would be great.

The most common approach is to create business rules handmade, based on the univariate and multivariate analysis of the variable.

Basically, do some frequency count, see if you could isolate some subset of your data just looking at one or two variables.

Then when you have your labels, create a linear or so model with this new variable as output. For exemple, a linear discriminant analysis. The analysis will supply you with new insights on your group.

If you want to rely on an algorithm, two solutions:

As you don't seems to have a lot of variables, a non-supervised segmentation could do the job. For exemple, a k-Nearest Neighbor or a decision tree are basic and good aproachs.

With a few more variables, I like is to do a principal component analysis then a non-supervised classification to define your group on the result of the PCA.
Note that a PCA + handmade rules based on the analysis of your PCA result may be enough.

Each time, in the end, a discriminant analysis and a profile of your groups to asses the quality of your results.


Why Deep Reinforcement Learning fails to learn how to play Asteroids? Deep Q-learning, A3C, policies evolved with genetic algorithms, they all fail to learn Asteroids, or at least perform way worse than human. From the hardest Atari games according to RL most of the focus is on Montezuma's revenge, which clearly suffers from sparse rewards. However I don't think this is the case of Asteroids (video), since for every asteroid shot a reward is provided. Why DRL performs that bad then?

Here are some papers that report a bad result on Asteroids (some articles refer to each other):

Human-level control through deep reinforcement
learning
Massively Parallel Methods for Deep Reinforcement Learning
Deep Reinforcement Learning with Double Q-learning
Dueling Network Architectures for Deep Reinforcement Learning
Prioritized Experience Replay
Rainbow: Combining Improvements in Deep Reinforcement Learning
Evolution Strategies as a Scalable Alternative to Reinforcement Learning
A Neuroevolution Approach to General Atari Game Playing

I suspect a bug, or some subtle implementation detail.
In many ways, Asteroids is as near an ideal environment as one can get without custom design:

Unambiguous, simple reward system (get points, don't die)
limited options (move or shoot a single weapon type)
all enemies are very similar
The player has perfect information.

Compare this to DOTA 2, which has not been mastered, but with a moderate reduction in complexity (1v1 instead of 5v5), OpenAI was able to achieve some impressive results, despite being magnitudes of order more complex than asteroids.
There are certain compromises made in the 2015 DQN paper, for example:

"Following previous approaches to playing Atari2600 games,we also use
a simple frame-skipping technique (15). More precisely, the agent sees
and selects actions on every kth frame instead of every frame, and its
last action is repeated on skipped frames. Because running the
emulator forward for one step requires much less computation than
having the agent select an action, this technique allows the agent to
play roughly k times more games without significantly increasing the
runtime.
We use k - 4 for all games"

As a counter-example, top level human players often make single frame level decisions / actions in modern fighting games (rendered both visually and in game logic at 60 FPS), so we know this approach will not work well with all video games.
I suspect that even though these were effective generalized approaches, one of them severely failed with asteroids.
This is a good question, as diagnosing why asteroids is causing difficulty could give some strong insight into testing procedures or algorithm design, as clearly something is happening. I think to get an accurate answer would require fixing the problem.
Sources


Simplest way to build a semantic analyzer I want to build a semantic analyzer i.e., to find how similar the meaning of two sentences are.
For example-
English: Birdie is washing itself in the water basin.
English Paraphrase: The bird is bathing in the sink.
Similarity Score: 5 ( The two sentences are completely equivalent, as they mean the same thing.)

I have to find the similarity between the meaning of those sentences.

Here is a github repo of what I want to implement.
https://github.com/anantm95/Semantic-Textual-Similarity

Is there any simpler approach?

Is there any simpler approach?

Very unlikely, semantic similarity is a very complex problem related to Natural Language Understanding (NLU). You could look at the techniques used for textual entailment, Question Answering and summarization.

Simple methods like the baseline system proposed in the github link, but they don't really try to analyze the semantics.


Modeling pixel intensity with the normal distribution I was reading a machine learning book with applications in computer vision. In it, it mentioned that "in vision, it is common to ignore the fact that the intensity of a pixel is quantized and model it with continuous normal distribution." So my question is why do we choose to do that?

The first thing to consider is that, even continuous measurements are actually discrete. For instance, temperature is always considered as continuous but every time you process temperature data, these are discrete data to the precision of the sensor used to measure them. Same is true for every physical, measurable quantity (including intensity). That being said, with our modern physics models, it makes no sense to consider that temperatures follow discrete mass probabilities with several thousands possible output and everyone studies them under continuous assumptions.

So where do we set the boundary? When do we consider that there are "too many" possible outputs to work with probability mass functions and choose to work with continuous distributions? This is an open question! :)

Here is the thing, we do not lose information when processing discrete data in a continuous way so there is no restriction for doing so!

Now consider the pixel intensities of an image, most of the time, we would need to consider discrete mass probabilities with 256 possible outputs, meaning 256 parameters (or weights) to be estimated or tuned. If we model the intensity by a continuous distribution, the number of parameters is drastically reduced! For a Gaussian for instance: 2 parameters. Even using more complex distributions or mixture models, we will not need to tune hundreds of parameters. This is the main advantage! Compact representation!

About the choice of the Gaussian specifically, let say that it has been empirically proven to work great. The Central Limit Theorem also supports this common choice. And now, on the top of this, many algorithms are readily available under the assumption that the data follows Gaussian distributions making them even more used by people in the field. Doesn't mean that nothing else could be used but this is the reality of things.


Why is Random Forest feature importance biased towards high cadinality features? I understand how a random forest algorithm works but could someone tell me the rationale behind Random Forest feature selection being biased towards high cardinality features?

Random forest (as almost any other algorithm) is prone to selecting variables which can lead to a one-to-one relationship with the $Y$ variable.

Why? Because you are leading them to be overfitted. If your variables have high cardinality, it means they form little groups (in the leaf nodes) and then your model is "learning" the individuals, not generalizing them. The more "cardinal" the variable, the more overfitted is the model.

For example, if you have social security number as variable (biggest cardinality possible), this variable will for sure have the biggest feature importance. Because in the leaf nodes you will find every individual with his social sec. number and his output. But the capacity of generalization of the model is zero.

Almost every task in data science looks that this doesn't happen.


Should Discriminator Loss increase or decrease? This question is purely based on the theoretical aspect of GANs.
So, when training a GAN how should the discriminator loss look like?

Should the loss of discriminator increase (as the generator is successfully fooled discriminator)

Or should the loss of discriminator decrease?

Can someone please help me in understanding this?

Discriminator consist of two loss parts (1st: detect real image as real; 2nd detect fake image as fake). 'Full discriminator loss' is sum of these two parts.
The loss should be as small as possible for both the generator and the discriminator. But there is a catch: the smaller the discriminator loss becomes, the more the generator loss increases and vice versa.
Discriminator loss: Ideally the full discriminator's loss should be around 0.5 for one instance, which would mean the discriminator is GUESSING whether the image is real or fake (e.g. the same as coin toss: you try to guess is it a tail or a head).
Generator loss: Ultimately it should decrease over the next epoch (important: we should choose the optimal number of epoch so as not to overfit our a neural network).
I could recommend this article to understand it better.


Xgboost Parameter Tuning Are there methods to tune and train an xgboost model in an optimized time - when I tune paramaters and train the model it takes around 12 hours to execute?

I would like to run the solution 100 times with 100 seeds; my machine has 8 GB RAM and I can't buy a cloud solution.

Typically XGBoost does not require many parameters to be tuned to get good performances.

I would start with optimizing the n_estimators, max_depth and min_child_weight parameters only. This should already bring you close enough.

Another thing you can do to speed-up the process is to prefer Random Search over Grid Search, since in most cases is as or more effective.

You could also have a look at the LightGBM implementation, which is faster (and needs less memory) than XGBoost.


Machine Learning applied to database design Is it possible to apply machine learning to database design given a data source with (pseudo) relationships?

This is such an interesting question. I suppose that it is possible but you would have to answer some more questions before you can actually get help with modeling something.

Are you looking for it to learn SQL or NoSQL?
You'd have to make a distinction between something that can learn relational database design versus something that learns how to be a DBA and work in a particular language. For example, relational databases are based on theory (and relatively straightforward) but how you implement certain things in Oracle or SQL Server (as examples) will vary greatly. Or maybe you're looking for a particular type of design like data warehousing (star patterns, etc). Whichever approach you choose would have a profound effect on the type of model you are going to build.
There are some pitfalls that you would have to account for. A relationship based on text columns is acceptable design, but a relationship based on integer hashes of those same text columns is much better. How a model would account for something like this is unknown to me.
Relating to the item above, you would have to come up with some metric for the success of your model. Is it the levels of relational design that it can reach? Is it some hardware performance benchmark? Is it some level of cognition that your model can reach for extremely complex designs?

I think that once you answer these types of questions you will be in a much better position to start model development.


Can I fine tune the xgboost model instead of re-training it? I am using the xgboost library. My system runs a cronjob each night, where it pulls the data from the database and trains the model. However, I would like to remove the re-training of the model again and again, and just fine-tune it with any new data that came in the database. In sklearn's implemantation (http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html) one could use warm_start option, what about xgboost ?

I see that in the current version of python wrapper of xgboost you can specify file name or existing xgboost model (class Booster) in train function.


How to understand Bionomial Theroem and the Recursion Rule? In this video from EDX, the instructor explains the binomial theorem as:
Binomial Theorem:
When you calculate $(a + b)^n = a^n + C(1)a^{(n-1)b} + C(2)a^{(n-2)b^2} + ... + C(n-1)ab^{(n-1) + b^n}$
The coefficients = $C1, C2, ..., C(n-1)$
Coefficient of $(a + b)^4 = 1, 4, 6, 4, 1$
From the formula above, you get after some manipulation:
$(a + b)^n = a^n + (n choose 1) a^{(n-1) b^1} + (n choose 2) a^{(n-2) b^2} + ... + (n choose n-2) a^2 b^{(n-2)} + (n choose n-1) a^1 n^{(n-1)} + b^n$
The coefficients will be the same as pascal's triangle
Is it necessary to know this? I still do not understand how he went from $(a + b)^n$ formula to plugging in $(n choose 1) (n choose 2)...$
Additionally, for the recursion theory:
Recursion Rule:
If we have 52 cards and we select 5 cards we have a collection of 5 cards taken from 52 cards
The order the cards are dealt does not matter
To get the recursion, we will split the hands into hands that contain the ace of spaces with hands that do not contain the ace of spades
For the hand that does not contain the ace of spades, there are 51 cards to choose from
So it is 51 choose 5
For the hand that does contain the ace of spades, then that hand is made from
4 cards in the remaining 51 cards in the deck
So it is 51 choose 4
So since every hand either has ace of spades or does not
We get our recursion
52 choose 5 is = 51 choose 5 + 51 choose 4
But if you actually calculate 52 choose 5 you get 311875200
And if you calculate 51 choose 5 + 51 choose 4 you get 287884800
To me it seems mathematically inaccurate?

Firstly, this is probably best to ask on the math stackexchange, but I'll have a crack at it anyway.

Let's start by establishing something you already know: $(a+b)^n$ will expands to a sequence of terms with general form $c_ia^xb^y$ such that $n=x+y$. We also know that the expansion process hits every positive integer combination of $x$ and $y$ that sum to $n$ (of course assuming $n$ is a positive integer).
I'd also like to mention that you can notate the expansion entirely with $choose$ coefficients: the first and last terms have coefficients given by $n\choose{0}$ and $n\choose{n}$. Below, we'll see why this is important.

Let's do an example where $n=4$.
Let's start by solving for the first term where $x=n$ and $y=0$. We can ask ourselves: Given the first $(a+b)$ term, how many paths can we take, by multiplying out the rest of the terms $(a+b)(a+b)(a+b)(a+b)$, to reach a term with $x=n$ and $y=0$?

For the first term, it is of course one path: you need to multiply the first $a$ with the second and third and fourth: $aaaa$. Given only one path, the first coefficient is $c_1=1$.

Now let's solve for the second term where $x=n-1$ and $y=1$. We can ask ourselves the previous question with the modified values of $x$ and $y$. We find, stepping from one $(a+b)$ to the next, that starting with $a$, there are three ways to get to $a^3b$ and starting with $b$, there is only one way to get to $a^3b$. Namely, starting with $a$, we could take the following paths: $abaa$,$aaba$, or $aaab$. And starting with $b$, we can only take one path: $baaa$. So $c_2=4$.

Solving for the third term where $x=n-2$ and $y=2$, we can find three paths starting from $a$ and three paths from $b$. Namely: $aabb$,$abab$, and $abba$ for starting with $a$ and $bbaa$,$baba$, and $baab$ for starting with $b$. So in total, $c_3=6$.

We can see a pattern of solving for $c_i$ whereby we ask ourselves How many ways can we construct a sequence of $a$'s and $b$'s such that the number of $a$'s is $x$ and the number of $b$'s is $y$?
This question is precisely solve via the choose operator!
The first coefficient, $c_1$, can be solved by asking, How many ways can we construct $4$ elements consisting of $a$'s and $b$'s such that the number of $a$'s is $4$? This pretty much analogous to the choose operator's definition: ${4\choose4}=1$. Similarly, $c_2$ asks how many ways we can construct the sequence such that there are $3$ $a$'s and $1$ $b$, which is, by definition, ${4\choose3}=4$.

I hope this answers your question about the binomial expansion theorem. Unfortunately, I do not understand what you are trying to ask in your second question, but again, it probably should be asked on the math stack exchange instead. Good luck!


Algorithm to construct similarity structure from hash lookup table I have constructed a lookup table using locality-sensitive hashing for comparing nearly similar documents/records. If two records (columns) have the same hash value in a row, they are considered to be similar. For example, the structure is

   R1 R2 R3 R4 R5
b1 a1 a2 a3 a2 a5 .....
b2 a2 a4 a1 a4 a4 .....
b3 a3 a5 a3 a7 a4....

Since each band holds similar records, similar record sets will be

S1 = {R2,R4}
S2 = {R2, R4, R5}
S3 = {R1,R3}

And the merged similarity structures will be

S1' = {R2,R4,R5}
S2' = {R1,R3}

I want to span the matrix and map all similarity structures so I have buckets of similar records. The dimensionality of the hash table is large, hence a naive approach is not likely to work. What types of algorithms should I consider to achieve this efficiently?

Edit 1:
Question updated to clarify more on the objective

The algorithm you are asking about is very straightforward.

What you do, you are looking for connected components in a certain graph where edges are determined by matching hash values. You can implement this with a modification of the disjoint-set data structure.

Your particular variation would be that apart from keeping track of vertices in each component, you have to keep track of m (number of rows) sets of hash values that have been found for each component.


Can we consider high correlation to be a good predictor? The problem of predicting the daily number of COVID-19 cases is indeed challenging and many (external) factors should be taken into account to come up with a reasonable predictor. However, we have studied Twitter for a specific country (not English) for the period Mar-Nov 2020, and found out that the volume of daily tweets related to symptom X is highly correlated with the number of confirmed cases in that country (pearson correlation 0.84 with p-value 0.00031).
In the field of data science, would this suffice, at least partially, to say daily tweets of X is a good predictor for the number of COVID-19 cases?

In this case, I would fit a linear model (lm(tweets ~ cases)) and then use bootstrap or cross-validation to test the model.


Pattern Recognition - Kernel Density estimators 2.5.1 Please refer page 122-123 of Pattern recognition and Machine Learning - Bishop. A few equations:

  Density estimate: $$ p(\mathbf{x}) = \frac{K}{NV} \tag{2.246}$$ where
  $K$ = #points in: $N$ regions of volume $V$ each.

  Kernel function: Number $K$ of points falling within a hypercube
  centered at origin: $$\begin{equation}k(\mathbf{u}) =\begin{cases} 1, \quad |u_i| \leq \frac{1}{2} \qquad i = 1,\dots,D,\\ 0, \quad otherwise  \end{cases} \tag {2.247} \end{equation}$$

  From 2.247, the quantity $k((\mathbf{x} - \mathbf{x_n})/h)$ will be
  one if the data point $\mathbf{x_n}$ lies inside a cube of side $h$
  centered on $\mathbf{x}$, and zero otherwise. The total number of data
  points lying inside this cube will therefore be: $$K = \sum_{n=1}^{N} k \Big(\frac{\mathbf{x} - \mathbf{x_n}}{h}\Big) \tag{2.248}$$
  Substituting this expression into (2.246) then gives the following
  result for the estimated density at $\mathbf{x}$ $$p(\mathbf{x}) = \frac{1}{N}\sum_{n=1}^{N} \frac{1}{h{^D}}k\Big( \frac{\mathbf{x} - \mathbf{x_n}}{h}\Big) \tag{2.249}$$

  where we have used $V$ = $h^D$ for the volume of hypercube of side $h$
  in $D$ dimensions. Using the symmetry function $k(\mathbf{u})$, we can
  now re-interpret this equation, not as a single cube centered on
  $\mathbf{x}$ but as the sum over $N$ cubes centered on $N$ data points
  $\mathbf{x_n}$

I am struggling to follow explanation of last paragraph.
Each point $\mathbf{x_n}$ is at the center of (respective) cube - total $N$ cubes. If so, then concept of boundary/ distance from a fixed point seems to be missing - ref. 2.247, 2.248 resulting all 1's. Which points will be zero, if any.

Just a rough sketch:

Left sketch represents 2.248 - imagine few points outside the cube (=0); points inside cube=1. Right sketch is my understanding of 2.249's explanation: 3 points, all at center of respective cube. On what basis will the data points be classified as 0 or 1?

Pls. refer the equations above. In case of eq 2.248 (left box image) we're calculating density at point $x$, only. There are total of 3 points, only 2 qualify as per kernel function and hence probability at $x$, $p(x) = \frac{2}{3}$.

The right box diagram shows that we take the points, one by one, and calculate the probability at each point using kernel function. Thus we get probability at each of point of data set. And effectively, we get a probability distribution for the entire data set.

The bar chart shows the PMF for 2 cases - left: probability based on (eq 2.248) and right: for (eq. 2.249). I use PMF to easily communicate the understanding.

*Note:

1- With x_1 centered cube 2 points: x_1 & x_2 lie inside the cube; same for x_2 centered cube. However for x_1 only one point x_1 lies in the cube - out of total of 3 points.

2- d|a, b| gives distance between points a, b.

3- for a large collection of data points and cubes of small sides (h), we get a PDF distribution curve.


Clustering vs Non Clustering problems? I'm just getting started with Andrew Ng's Machine Learning wherein he explained the example of the cocktail party problem vs the gene clustering problem in order to explain the difference between clustering and non clustering problems within unsupervised learning itself. However I still don't understand the difference between the two. Can someone please help clarify.

Both of the examples are clustering examples. Clustering is about grouping of similar dataset when one is not given the data.

In the gene problem,

One possible setting is you are given the DNA micro-array data. Your task is to learn how many types of people are there. This is an unsupervised learning problem, we are not given the labels. We just group  people with similar type of genes together.

In the cocktail party problem:

There are two people in a room and there are microphones. We just record the audio and pass it to an algorithm and tell the algorithm, hey, learn the pattern if there is any. Each individual might have their own speech pattern/ language/accent. The algorithm pick it up on its own such pattern and detect that there are two people in the room and they can distinguish the speeches of the two people.

Let me give you an example of a non-clustering example.

For example, a question of interest is to detect anomaly. For example, what you are given could be thr normal operating state of a machine, say their sensor readings. From those readings, you have to learn what is normal for the machine and you have to figure out when you are given a new data point, should it be considered normal.


Understanding the algebra behind a specific partial derivative equation I am following this article about neural networks.
Given:

Until here I understand everything, but then he continues to:

I don't understand how he got to that conclusion. I think he skipped some algebra steps that would have made it easier for me to understand.


Is there a synset for phrasal verbs? Is there a database of phrasal verbs of similar semantics? Eg. one where querying ‘get in touch’ would return ‘get in contact with’, among other phrasal verbs of similar meanings?

If not, given a phrasal verb how would you obtain others of similar meanings?

Dataset
Here is a data set I created.
There are $143$ phrasal verbs. However, since some have multiple meanings, there are $181$ phrasal verbs pairs.
Examples
phrasal_verb | meaning
------------------------
black out    | faint
calm down    | become more relaxed
get off      | leave
get off      | remove
hold on      | wait
hold on      | grip tightly

Creation
I couldn't find any ready made, but I created the dataset by scraping this website.
I loaded the data set and for 'phrasal verb' I found the meaning in the 'Meaning' column by applying the following rules:
def tokenize_meaning(sentence: str) -> list:
    """
    Splits the string in the meaning column into a list of possible phrasal verbs.
    :param sentence: str from the meaning column
    :return: list of strings
    """
    meanings = re.split('\d\.\s+', sentence)
    meanings = flatten([re.split(', ', meaning) for meaning in meanings])
    meanings = [meaning for meaning in meanings if meaning[:3] == 'To ']
    meanings = flatten([re.split(' or ', meaning) for meaning in meanings])
    meanings = [meaning for meaning in meanings if not meaning == '']
    meanings = [re.sub('[\(|\/].*', '', meaning) for meaning in meanings]
    meanings = [re.sub('[.,?!]', '', meaning) for meaning in meanings]
    meanings = [re.sub('To ', '', meaning) for meaning in meanings]
    meanings = [meaning.strip() for meaning in meanings]

    return meanings


Problem extracting words from dataframe I have the following dataset which is a .json file:

and I would like to get the first word for every string inside lista_asm, so I would like to get: jmp,push,uncomisd,...etc
what I am doing to do this is the following:
dataFrame['opcodes'] = dataFrame['lista_asm'].apply(lambda x:[i.split()[0] for i in x])

but it gives me back the following error message:
   3589             else:
   3590                 values = self.astype(object).values
-> 3591                 mapped = lib.map_infer(values, f, convert=convert_dtype)
   3592
   3593         if len(mapped) and isinstance(mapped[0], Series):

pandas/_libs/lib.pyx in pandas._libs.lib.map_infer()

<ipython-input-18-5506b5721bf1> in <lambda>(x)
----> 1 dataFrame['opcodes'] = dataFrame['lista_asm'].apply(lambda x:[i.split()[0].strip() for i in x])

IndexError: list index out of range

I don't understand what is wrong. Can somebody please help me?
[EDIT]Trying the code:
dataFrame['opcodes'] = dataFrame['lista_asm'].apply(lambda x:x[0].split(" ",2)[0])

and adding :
df = dataFrame[["opcodes", "semantic"]].copy()
df

I get:

what I would like to get is a list of the type [push,mov,..] and this for every row.
It seems like when I do x[0] it does not return the first element of the list, but returns the pharentesis, which is weird. Am I doing something wrong which I don't see?
My objective is to pre-process this dataset in order to feed features to my model, but I haveing hard times in doing so.

The problem is that you say "apply(lambda x:[i.split()[0] for i in x])"
As soon as you say apply, x is your list. So you can say following
"apply(lambda x:x[0].split(" ", 2)[0])"
Meaning you say take first element in the list, than split on " " and in two parts. And than take the first word (part) with the last [0]


Manage x-axis using ggplot() Source: https://d396qusza40orc.cloudfront.net/exdata%2Fdata%2FNEI_data.zip

Here is my data prep.

NEI <- readRDS("summarySCC_PM25.rds")
Baltimore <- NEI[NEI$fips=="24510", ]
Baltimore$type <- as.factor(Baltimore$type)
Total_Emmisssions <- aggregate(Baltimore$Emissions,
                               by=list(Baltimore$year, Baltimore$type),
                               FUN=sum)
names(Total_Emmisssions) <- c("Year","Type","Emissions")

Plot code

library(ggplot2)
g <- ggplot(Total_Emmisssions,aes(x=Year, y=Emissions, colour=Type))
g1 <- g+geom_point()
g2 <- g1+facet_grid(. ~ Type)
g3 <- g2+geom_smooth(method="lm", se=FALSE)

However, in this plot, the year is always 2002-2008. How do I change the scale / ticks to show from 1999-2008 at an interval of 3 years. Using scale_x_discrete does not work and it messes the graph. Please help.

str(Total_Emmisssions)
'data.frame':   16 obs. of  3 variables:
 $ Year     : int  1999 2002 2005 2008 1999 2002 2005 2008 1999 2002 ...
 $ Type     : Factor w/ 4 levels "NON-ROAD","NONPOINT",..: 1 1 1 1 2 2 2 2 3 3 ...
 $ Emissions: num  522.9 240.8 248.9 55.8 2107.6 ...

Change Year to a factor and add group=1:

g <- ggplot(Total_Emmisssions,aes(x=factor(Year), y=Emissions, colour=Type, group=1))

you can leave the rest the same (you'll also prbly want to change the xlab).


Building a tag-based recommendation engine given a set of user tags? Basically, the idea is to have users following tags on the site, so each users has a set of tags they are following. And then there is a document collection where each document in the collection has a Title, Description, and a set of tags which are of relevance to the topic being discussed in the document as determined by the author. What is the best way of recommending documents to the user, given the information we have, which would also take into consideration the semantic relevance of the title and description of a document to the user's tags, whether that be a word embeddings solution or a tf-idf solution, or a mix, do tell. I still don't know what I'm going to do about tag synonyms, it might have to be a collaborative effort like on stackoverflow, but if there is a solution to this or a pseudo-solution, and I'm writing this in C# using the Lucene.NET library, if that is of any relevance to you.

One solution could be to train a single embedding space, StarSpace is one such implementation. That single embedding space would contain all users, documents, and tags. Then it is a nearest neighbor search to recommend any combination. Given a user, find the nearest documents. Given a tag, find the nearest tags …

For new entities (i.e., users, documents, or tags), split the individual entity into parts. For example, a document will have tokens or tag will be associated with documents that have tokens. Then find the average embedding of all the parts. That location in the embedding space is the approximately semantic meaning of new entities.

Overall, this is a complex, open-end problem, thus there are many possible ways to create useful solutions. The next best solutions depend on what has already been done and what the next feature that would add the most value to end-users.


what is the best approach to detect small objects with similar shape? I'm working a model which detect different products in supermarket shelf. In the training data, there are a lot of objects with similar shape placed very close to or stacked to each others.(eg: milks with different brands are stacked, placed on the same shelf, the model should be able to detect milk1, milk2). What is the best approach to this problem. I've tried to train a Faster RCNN, but the RPN isn't  working well. I've also tried feature matching, but it cannot detect partially visible objects. Any help will be appreciated!

The training images look like this

Link to FRCNN result when detect 2 type of milk and 1 type of yogurt

faster r-cnn detection result

If all objects are observed in the same distance and almost same angle, the relative height and width can be helpful features for recognizing objects with similar shape and different size. By this features different methods like GAN algorithms such as CoGAN and BiGAN may help you in this problem.

It should be noticed that for recognizing the size of the objects the features play more important role than the algorithms.


Lightgbm vs xgboost vs catboost I've seen that in Kaggle competitions people are using lightgbms where they used to use xgboost. My question is: when would you rather use xgboost instead of lightgbm? What about catboost?

On Kaggle, LightGBM is indeed the "meta" base learner of almost all of the competitions that have structured datasets right now. This is mostly because of LightGBM's implementation; it doesn't do exact searches for optimal splits like XGBoost does in it's default setting (XGBoost now has this functionality as well but it's still not as fast as LightGBM) but rather through histogram approximations. The result is a slight decrease in predictive performance for a much larger speed increase in training. This means more opportunity for feature engineering/experimentation/model tuning (all of which are key to winning Kaggle competitions) which inevitably yields larger increases in predictive performance (despite using histogram approximations).

CatBoost is not used as much because on average, it it found to be much slower than LightGBM. That being said, CatBoost is different in its implementation of gradient boosting which at times can give slightly more accurate predictions, in particular if you have large amounts of categorical features. I have never used CatBoost and so I encourage you to read that paper. Regardless, because rapid experimentation is vital in Kaggle competitions, LightGBM tends to be the go to algorithm when first creating strong base learners.

In general, it is important to note that a large amount of approaches I've seen involve combining all three boosting algorithms in a model stack (i.e. ensembling). LightGBM, CatBoost, and XGBoost might be thrown together as three base learners and then combined via. a GLM or neural network. This is done to really squeeze out decimal places on the leaderboard and so I doubt there is any theoretical (or practical) justification for it besides competitions.


Relationship between Sigmoid and Gaussing Distribution I was reading this article where I came across the following statement in the context of "Why do we use sigmoid activation function in Neural Nets?":

The assumption of a dependent variable to follow a sigmoid function inherently assumes a Gaussian distribution for the independent variable which is a general distribution we see for a lot of randomly occurring events and this is a good generic distribution to start with.

Could someone elaborate on this relationship between the two?

Yes sure, the core idea is that the sigmoid function approximate very nicely the cumulative density function of a Gaussian distribution. Therefore, if the input of a neuron is a continuous Gaussian distribution, the probability of the output being inferior to a certain value will follow (almost) the shape of the sigmoid function.

A detailed explanation can be found p16 of the following article:

https://www.researchgate.net/publication/220233479_Catastrophic_Forgetting_and_the_Pseudorehearsal_Solution_in_Hopfield-type_Networks


Can the number of epochs influence overfitting? I am using a convolution neural network ,CNN. At a specific epoch, I only save the best CNN model weights based on improved validation accuracy over previous epochs.

Does increasing the number of epochs also increase over-fitting for CNNs and deep learning in general?

Yes, it may. In machine-learning there is an approach called early stop. In that approach you plot the error rate on training and validation data. The horizontal axis is the number of epochs and the vertical axis is the error rate. You should stop training when the error rate of validation data is minimum. Consequently if you increase the number of epochs, you will have an over-fitted model.

In deep-learning era, it is not so much customary to have early stop. There are different reasons for that but one of them is that deep-learning approaches need so much data and plotting the mentioned graph would be so much wavy because these approach use stochastic-gradient-like optimizations. In deep-learning again you may have an over-fitted model if you train so much on the training data. To deal with this problem, another approaches are used for avoiding the problem. Adding noise to different parts of models, like drop out or somehow batch normalization with a moderated batch size, help these learning algorithms not to over-fit even after so many epochs.

In general too many epochs may cause your model to over-fit the training data. It means that your model does not learn the data, it memorizes the data. You have to find the accuracy of validation data for each epoch or maybe iteration to investigate whether it over-fits or not.


Books on time series and sequence classification Though I have been using traditional machine learning algorithms (Regression and Classification) , I have no experience of using Time series and would like to understand what is time series and different approaches(ex:ARIMA,SARIMA,SARIMAX, LSTM etc) used for time series analysis. I also see that people use LSTM's/RNN for time series/sequence classification. Can you people recommend me a book that discusses the above algorithms and how Time series is different from sequence classification etc?

There are numerous topics that you've mentioned but I will suggest those which I've read and are helpful.

For hidden markov models and markov processes I suggest reading Pattern Classification by Richard O. Duda. You can also take a look at Pattern Recognition and Machine Learning by Christopher Bishop. For better understanding Markov processes and their behaviour, you can also take the course stochastic process which is also known as random process. I suggest taking that course before reading any book, because you may need some help if you're not very familiar to the concepts.

For LSTMs, I highly suggest taking a look at the fifth course of deep learning on coursera by pr. Andrew Ng. If you do their homework you can realise how the inner operations exactly work and you'll have a very deep understanding of time series and what their nature is. After that, you can take a look at the Deep Learning book by Ian Goodfellow.


What is "data science"? In recent years, the term "data" seems to have become a term widely used without specific definition. Everyone seems to use the phrase. Even people as technology-impaired as my grandparents use the term and seem to understand words like "data breach." But I don't understand what makes "data science" a new discipline. Data has been the foundation of science for centuries. Without data, there would be no Mendel, no Schrödinger, etc. You can't have science without interpreting  and analyzing data.

But clearly it means something. Everyone is talking about it.  So what exactly do people mean by data when they use terms like "big data" and why has this become a discipline in itself? Also, if it is an emerging discipline, where can I find more serious/in-depth information so I can better educate myself?

Thanks!

I get asked this question all the time, so earlier this year I wrote an article (What is Data Science?) based on a presentation I've given a few times. Here's the gist...

First, a few definitions of data science offered by others:

Josh Wills from Cloudera says a data scientist is someone "who is better at statistics than any software engineer and better at software engineering than any statistician."

A frequently-heard joke is that a "Data Scientist" is a Data Analyst who lives in California.

According to Big Data Borat, Data Science is statistics on a Mac.

In Drew Conway's famous Data Science Venn Diagram, it's the intersection of Hacking Skills, Math & Statistics Knowledge, and Substantive Expertise.

Here's another good definition I found on the ITProPortal blog:

  "A data scientist is someone who understands the domains of programming, machine learning, data mining, statistics, and hacking"

Here's how we define Data Science at Altamira (my current employer):

The bottom four rows are the table stakes -- the cost of admission just to play the game. These are foundational skills that all aspiring data scientists must obtain. Every data scientist must be a competent programmer. He or she must also have a solid grasp of math, statistics, and analytic methodology. Data science and "big data" go hand-in-hand, so all data scientists need to be familiar with frameworks for distributed computing. Finally, data scientists must have a basic understanding of the domains in which they operate, as well as excellent communications skills and the ability to tell a good story with data.

With these basics covered, the next step is to develop deep expertise in one or more of the vertical areas. "Data Science" is really an umbrella term for a collection of interrelated techniques and approaches taken from a variety of disciplines, including mathematics, statistics, computer science, and software engineering. The goal of these diverse methods is to extract actionable intelligence from data of all kinds, enabling clients to make better data-driven decisions. No one person can ever possibly master all aspects of data science; doing so would require multiple lifetimes of training and experience. The best data scientists are therefore "T-shaped" individuals -- that is, they possess a breadth of knowledge across all areas of data science, along with deep expertise in at least one. Accordingly, the best data science teams bring together a set of individuals with complementary skillsets spanning the entire spectrum.


How filters are made in a CNN? I am new to Data Science and CNN.

My understanding of CNN is that:

An image's pixel data is convoluted over with filters which extract features like edges and their position.
This creates filter maps.
Then we apply max pooling which will down sample the data.
Then we feed this data to a neural network which learns to classify.

Now I know I can do things with convolution like blurring or get edges from image.

But lets say I am trying to make a "Hotdog or not" app, how do I make my filters? What will the matrix be?

First of all, feature maps are the output of the convolution after an activation function (e.g. ReLU or sigmoid) is applied, not the matrix that the image is convolved with. Usually, this is called a filter.

The magical thing about CNNs is that we don't know what the filters should look like for any given problem. The CNN works out what each filter should look like automatically. This is done through the backpropagation procedure. Without getting heavy into the math of it all, essentially every time a training example (or a batch of examples) goes through the network, the values inside each filter get updated by some small amount. This small amount is determined using the derivatives of a loss function. As each step of the training procedure is completed, the values inside each filter (if all goes well!) slowly converge towards a value that minimises the loss function, thereby producing the best quality predictions.

For more information on the math, I encourage you to read the chapter on backprop in the free online book Neural Networks and Deep Learning, available here.

A more simple explanation is also provided here


high root mean square error in regression model I am applying regression to a data of 110 rows and 7 columns ,each having targets. When I applied Lasso to the data and calculated the RMSE value ,the RMSE value is coming to be 13.11.I think the RMSE value should be close to zero.What is the permissible values of RMSE in a regression model.What could have gone wrong in the computation.

from sklearn import linear_model
reg = linear_model.Lasso(alpha = .00001)
reg.fit(Xt,Yt)
ans=reg.predict(Xts)
print(ans)
from sklearn.metrics import mean_squared_error
print(mean_squared_error(Yts, ans))

whereas when I try cross validation the MSE scores are way below 0.35

kfold = KFold(n_splits=10)
results = cross_val_score(reg, full_data, target, cv=kfold)
print("Results: %.2f (%.2f) MSE" % (results.mean(), results.std()))
results
Results: -0.13 (0.45) MSE

RMSE doesn't work that way. An RMSE of 13 might actually great, it completely depends on how your target variable is scaled. For example, if your target variable was in the range [0,1e9], than an RMSE of 13 is spectacular. On the other hand, if your target is in the range [0,1], an RMSE of 0.5 is terrible. If you want to try a metric that can be more readily interpretable as having a "good" or "bad" score, try Mean Average Percent Error (MAPE).

As far as why you get a lower MSE when you cross validate: you don't show us how you constructed your training and test sets, but my guess is that you basically just got unlucky and ended up with a training/test split that performed poorly on your holdout set. Your CV-MSE is clearly better than your single holdout MSE, but you should also check the spread of CV scores as well. In any event, for a dataset as small as yours I'd recommend using bootstrap cross validation instead of k-fold.


How to change a cell in Pandas dataframe with respective frequency of the cell in respective column I have a pandas dataframe with binary value columns. I would like to replace values in each cell with its frequency in rspective column in place. My question is how to keep track of the current column while using apply on the subset of columns like here: (to be applied from 8th columns to the end) :

train_data.ix[:,8:] = train_data.ix[:,8:].apply(x: what should come here?)

I know that train_data.ix[:,col_number].value_counts()[0] will return number of zeros in col_number but how can I use it inside apply function?

import pandas as pd
import numpy as np

df = pd.DataFrame(np.random.randint(2,size=(10, 4)), columns=list('ABCD'))
df

    A   B   C   D
0   0   1   1   1
1   1   0   1   0
2   1   0   0   1
3   0   1   1   1
4   0   0   1   1
5   0   1   1   1
6   0   0   0   0
7   1   1   1   0
8   1   1   1   0
9   1   0   1   0

values = df.apply(pd.value_counts)
values

    A   B   C   D
0   5   5   2   5
1   5   5   8   5

new_df = pd.DataFrame()
for x in df.columns:
    new_df[x] = df[x].apply(lambda row: values[x][row])
new_df

    A   B   C   D
0   5   5   8   5
1   5   5   8   5
2   5   5   2   5
3   5   5   8   5
4   5   5   8   5
5   5   5   8   5
6   5   5   2   5
7   5   5   8   5
8   5   5   8   5
9   5   5   8   5

I created a df with random integers 0 and 1.

Then count them by column in values df.

Then loop through each column and replace each cell with their respective count. Being random it is close to 5/5 split, but you can see the C column has 8/2 split.


How can access to modify feature_importances of Random Forest Classifier model? My goal is to extract the feature importances from already trained random forest classifier and transfer them to another classifier. How this can be done? and How can access to modify feature_importances of Random Forest Classifier model? using scikit learn.

I'm not sure it's possible to "transfer" the feature importances from model to model in a Random Forest Classifier. Although I think there are two work-arounds you may be able to use.
The first option you have is to retrain the second model on the exact same training data. This will have the effect of both models having close to the same feature importances, though I think doesn't guarantee they will be exactly the same.
Another way would be to try knowledge distillation (https://arxiv.org/abs/1503.02531). Here you're basically making a target, or student, model train to not only match the hard labels, but also a set of soft labels, generated from a teacher network. In this case, the model that you want to transfer the knowledge to would be the student network. However, I believe this will involve writing a custom loss function, which I don't think is possible in sklearn. The paper I linked is in the context of neural networks so I also don't know if there would be any gaps in the performance between model types, as presumably, you would doing this with a random forest.
Just to confirm, I ran this short script that trains two models on the same data and it seems to be doing what you want it to:
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=1000, n_features=4,
                           n_informative=2, n_redundant=0,
                            random_state=0, shuffle=False)

clf = RandomForestClassifier(max_depth=2, random_state=0)
clf.fit(X, y)
print('model 1 feature importances: ', clf.feature_importances_)

clf2 = RandomForestClassifier(max_depth=2, random_state=0)
clf2.fit(X, y)
print('model 2 feature importances: ', clf2.feature_importances_)

The results I got were:
model 1 feature importances:  [0.14205973 0.76664038 0.0282433  0.06305659]
model 2 feature importances:  [0.14205973 0.76664038 0.0282433  0.06305659]

Here you can see that the results were the exact same. I would imagine that the integrity of the transfer may change as the data changes. Make sure both models are initialized with the same random state if you want the feature importance values to match exactly.


ESC-50 Audio data for binary classifier I'm building a binary sound classifier using the ESC-50 dataset. I have taken one class, "dog bark", to be positive and the rest of the 49 classes to be negative. As the dataset is imbalanced I'm running into lot of training issues. I tried building a model but couldn't get a f1-score greater than 0.3.

I'm using mfcc and fft as features. I have tried used LR and SVM to train without much success. Can't use Deep learning models as it's a real-time system and can't have much delay.

How can I approach this problem?

On the github page of the ESC-50 dataset, there is a list of tried classifiers as well as links to the relevant papers. The best ones are all using some kind of deep learning, mostly CNNs, and currently the best score is 85.5%. A baseline Random forest (notebook available here) achived 44.3% and SVM (available here) achieved 39.6%.

I would recommend you take a look at the approaches linked above and see if you can improve upon them. However, for higher accuracy you probably need some kind of deep neural network.


Two-tower net does not learn when made deep I have been trying to train a relatively simple two-tower net for recommendation. I am using PyTorch and the implementation is the following - basically embeddings layers for users and items, optional feed-forward net for both towers, dot product between the user and items representations, and sigmoid.
class SimpleTwoTower(nn.Module):

    def __init__(self, n_items, n_users, ln):
        super(SimpleTwoTower, self).__init__()

        self.ln = ln
        self.item_emb = nn.Embedding(num_embeddings=n_items, embedding_dim=self.ln[0])
        self.user_emb = nn.Embedding(num_embeddings=n_users, embedding_dim=self.ln[0])

        self.item_layers = [] #nn.ModuleList()
        self.user_layers = [] #nn.ModuleList()

        for i, n in enumerate(ln[0:-1]):
            m = int(ln[i+1])
            self.item_layers.append(nn.Linear(n, m, bias=True))
            self.item_layers.append(nn.ReLU())

            self.user_layers.append(nn.Linear(n, m, bias=True))
            self.user_layers.append(nn.ReLU())

        self.item_layers = nn.Sequential(*self.item_layers)
        self.user_layers = nn.Sequential(*self.user_layers)

        self.dot = torch.matmul
        self.sigmoid = nn.Sigmoid()

    def forward(self, items, users):

        item_emb = self.item_emb(items)
        user_emb = self.user_emb(users)

        item_emb = self.item_layers(item_emb)
        user_emb = self.user_layers(user_emb)

        dp = self.dot(user_emb, item_emb.t())
        return self.sigmoid(dp)

I am trining with Binary cross entropy loss and Adam optimizer. When I am using only the embeddings, I see improvements from epoch to epoch (loss is decreasing and the evaluation metric are increasing). However, once I add even a single feed-forward layer, the network learns just a bit in the first epoch and then stagnates. I have tried to had code one linear layer with ReLU, to check if the issue is with the way I am creating the list of layers, but this did not change anything.
Has anybody else had a similar problem?
Edit: Here I have posted the question in the PyTorch forum and I have some replies.

I now have a working network. It turned out that the gradients were all zeros after only about 3000 update step. I tried two approaches to fix this - using Batch Normalization after each activation function in the feed-forward net and changing the activation function from ReLU to Leaky ReLU. Both worked, and I ended up using the Leaky ReLU without normalization.
For the full thread in the PyTorch discussion forum here


how to classify Iris flowers I'm working on a classification problem .I want to classify Iris flowers from the famous Iris data set using MLP. I know that I the number of neurons in output layer should be the same number of classes but can I use one neuron in output layer which output is the value (0 or 1 or -1) to refer to the three types or then it is considered as regression not classification ???? thanks

trin= [4.7 3.2 1.6 0.2;
   4.8 3.1 1.6 0.2;
   5.4 3.4 1.5 0.4;
   5.2 4.1 1.5 0.1;
   5.5 4.2 1.4 0.2;
  5.7 2.6 3.5 1;
  5.5 2.4 3.8 1.1;
  5.5 2.4 3.7 1;
  5.8 2.7 3.9 1.2;
  6 2.7 5.1 1.6;
  6.7 3.3 5.7 2.1;
  7.2 3.2 6 1.8;
  6.2 2.8 4.8 1.8;
  6.1 3 4.9 1.8;
  6.4 2.8 5.6 2.1
  ];
   trout=[-1;-1;-1;-1;-1;
          0;0;0;0;0;
         1;1;1;1;1];
         inp=size(trin,2);
         out=size(trout,2);
          hidden=2;

      x=[-0.8000,-1.520,-0.9400,-3.040,3.800,2,-2,3.790,-1,0,4.600,4.400,0];
 iw = reshape(x(1:hidden*inp),hidden,inp);
 b1 = reshape(x(hidden*inp+1:hidden*inp+hidden),hidden,1);
 lw = reshape(x(hidden*inp+hidden+1:hidden*inp+hidden+hidden*out),out,hidden);
 b2 = reshape(x(hidden*inp+hidden+hidden*out+1:hidden*inp+hidden+hidden*out+out),out,1);

 y =

tanh(tanh(trin*iw'+repmat(b1',size(trin,1),1))*lw'+repmat(b2',size(trin,1),1));
    e = gsubtract(trout,y);

is this classification or it is considered as regression . I mean should I make the out put 3 bits to be consedered as classification and how to do this if yes?

The iris dataset is meant to be used for classification. You have 3 separate classes of irises and attempting to solve it as a regression problem would be a mistake.

Think about your proposed solution, you want the output to be -1,0 or +1 (for classes a,b and c). But this implies that class a is more similar to class b than to c, and by the same principal that class c resembles class b more than a. You are adding a prior to the model that was not there before, and you should not do that (unless you are an iris specialist).

you need to take your class output labels and convert them to a one-hot-encoding representation:
class a = [1,0,0]
class b = [0,1,0]
class c = [0,0,1]

Then use categorical cross entropy for your loss function.


Separate business emails from personal emails I am working on a classification problem where I would like to separate business emails from personal emails to analyse their behaviours separately. I am thinking about using regex but after looking at the emails, I realised that the email addresses vary significantly. Although I could manually spot some emails with obvious business names, such as xxx@carsales.com.au, it is hard to filter these emails systematically. Are there some useful techniques I could use? Thx

Thank you for clarifying your question.
So, from what I have got, you want to predict user demographics from the e-mails (i.e. the text).
If you knew some of the demographics for a same group of emails, you could do a supervised task by:

Using a Recurrent Neural Network (RNN) or LSTM (https://towardsdatascience.com/understanding-rnn-and-lstm-f7cdf6dfc14e), you can encode the emails by feeding each email into the network as sequence of tokens. From this we get a "hidden representation". From this "hidden representation", we then decode this with a standard neural network, which then outputs demographics (whether it be age, socioeconomic status, etc.)

If you don't know demographics, then you could build customer profiles by agglomerative clustering:
Here, you use something like Doc2Vec (https://medium.com/wisio/a-gentle-introduction-to-doc2vec-db3e8c0cce5e) to encode the emails into vectors in semantic space and then use agglomerative clustering (https://www.datanovia.com/en/lessons/agglomerative-hierarchical-clustering/) over these vectors to find clusters of users.
The above is an unsupervised approach and you can also decide after looking at the resulting dendrogram, how many customer profiles you think are necessary.


Dense? or TimeDistributedDense? after LSTM layer in Keras Dense and TimeDistributedDense, which one is suitable after LSTM layer in Keras?

For example,

input = Input(shape=(12,N_indepen), dtype='float32', name='inci_input')
lstm = Bidirectional(LSTM(100, activation='elu',return_sequences=False))(input)

# Dense? or TimeDistributedDense?
dense = layers.Dense(30, activation='elu')(lstm)
dense = layers.TimeDistributedDense(30, activation='elu')(lstm)

m_y = layers.Dense(1, activation='linear')(dense)

I read some articles, most of them used Dense, but some used TimeDistributedDense. I hope to know what is proper theoretically.

When using the TimeDistributed, you need to have a sequence through time so that you can apply the same layer (in this case, Dense), to each time slice. In your code, you have return_sequences=False which does not produce a sequence, so you need to apply Dense.
Assuming you meant to put return_sequences=True, then Dense and TimeDistributedDense do the same task in this case.


K-fold-cross-validation if training dataset is much smaller than test dataset? I'm a beginner in machine learning and I have a special case in which I have only a small training dataset of about 500 images and a test dataset of 10,000 images.
Does it still make sense to do a 10-fold-cross-validation or Repeated-Cross-validation on the training data? Or would this be not necessary anymore due to the large test dataset?

Many thanks in advance

Generally speaking, when evaluating a model you either choose to do cross-validation or train-test splitting, but not both.  Your dataset appears to already be split between training and testing sets, so you seem to have implicitly chosen train-test splitting.

In your particular case, it may still make sense to run cross-validation if you have a reason to mistrust the results from the testing set.  Is the distribution of labels in the test data balanced or highly skewed?  Is the test data representative of examples you're likely to see in production?  If your test data is representative, then you'll likely get a better accuracy estimate evaluating on the test set rather than running cross-validation.

But I'm confused.  How did you come to have a training set of 500 images and testing set of 10,000 images?  If you 10,500 labeled examples available to you, then you can divide them however you like - right?  Why not run cross-validation with the entire set of 10,500 images?  That will give you the most robust accuracy estimate.


Error in SimpleImputer My code:
from sklearn.impute import SimpleImputer

fill_values=SimpleImputer(missing_values=0,strategy='mean')

X_train=fill_values.fit(X_train)
X_test=fill_values.fit(X_test)

imp_X_train=fill_values.transform(X_train)
imp_X_test=fill_values.trainsform(X_test)

Gives the error:
TypeError: float() argument must be a string or a number, not 'SimpleImputer'

Where did I go wrong?

If you want to transform a data array, for example, X_train, with a preprocessor like SimpleImputer you should use method .fit_transform.

For example: X_train = fill_values.fit_transform(X_train)

If you use only fit it will return an instance of type SimpleImputer which is your error.


How do I change the marker edge color of a seaborn relplot? I am trying to output a complex facet grid plot in the format of the following image:

But the problem is that I don't want the edge color of the markers to be white, I want it to be the face color.

In matplotlib, I can change the color of marker edges by calling the following:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

df = pd.DataFrame({"values_x": np.random.randn(100), "values_y": np.random.randn(100)})

plt.scatter(x=df["values_x"], y=df["values_y"], edgecolors="red")
plt.show()

This produces a plot that looks like:

Now, if I call the following:

sns.relplot(data=df, x="values_x", y="values_y", edgecolors="red")
plt.show()

I still get white marker edges like in the following:

So, how do I correctly change the color of the marker edges within a call to sns.relplot()?

*For reference, I create the top facet grid plot with the following:

sns.relplot(data=df,
            x="C",
            y=metric,
            row="penalty",
            col="solver",
            hue="fold")

The issue was with the parameter I was using. The parameter "edgecolors" is incorrect. The correct parameter is the non-plural "edgecolor". So the correct call to relplot is:

sns.relplot(data=df, x="values_x", y="values_y", edgecolor=None)

This would remove the marker edges completely, which is what I was trying to do originally.