Semantic segmentation in high-resolution images with high variance - cannot avoid underfitting I am working on a dataset of 2K images for a semantic segmentation problem.
I want to detect and localize small objects, with the smallest mask to be 5x5 pixels.
The images include 5 different textures, quite different from each other.
I am using Unet and EfficientNetb0-3 as backbones, but whatever I do, I get underfitting.
I've reached the conclusion that it is probably due to the fact that the resolution is too high and the variance as well.
Do you have any advice on how I could overcome this underfitting problem?

Given your mask is very small, you should look at reducing your convolutions to 2x2 since that will help aggregate more information from these smaller masks. EfficientNet has 3x3 and 5x5 convolutions which may not be suited for your purpose.
It is a better idea for you to train from scratch using smaller convolutions (2x2). Also, since you will lose the edge of transfer learning, look at self-training, if it is possible for your network.


Need some advice on approach to select only the informative emojis from the data set? I have a giant data set from a local elections, which contains hashtags, emojis, and comments. I wanted to make a network analysis using only emojis.

So far I have a network analysis graph made in R which looks like this:

Sorry, you may have to zoom in to see the nodes. So, basically my goal is to see what people are talking about as a whole group. Currently there are lot of nodes which don't really say anything concrete about the main context or the hook, yet also creating clutter. I had extracted the data with political hashtags. Therefore, nodes such as milk bottle, cows, joy faces, toffees, flags, etc. doesn't really give me anything between people's comments and the context. My goal is to see a collective sentiment of people via their use of emojis in their context. I don't know if I make sense.

I don't know how should I approach the problem of only selecting informative emojis. Should I focus on the hashtags, and make a list of hashtags I am interested in, and only extract their associated comments and emojis? Or should I look at the sentimental values associated with emojis, and focus on the extreme positive and negative ones only?

I am pretty lost, any direction which method/ algorithm should I use to de-clutter this graph a bit yet also keep it in the context of politics & elections?

One option is to reframe it as a word embedding problem. Emojis can be embedded in a vector space along with comments and hashtags. Then distance measures and clustering can be used to find the emojis that are associated with different sentiments.


how to compare between kmeans and hierarchical clustering results I am using 2 types of clustering algorithm
I apply hierarchical clustering the K-means clustering using python sklearn library
Now the results are a little bit different so how can I compare the results and which algorithm to use? because I want to write a conclusion for a set of unlabeled data.
Is there any benefit to use multiple algorithms and compare between them?

In general, even using k-means more than once will create slightly different clusters (that s if you do not set a random seed).
Ideally, the profiles of the clusters and the distribution of data points into them will be the similar. If this is the case then it shouldn't really matter which one you choose. If not then, I suggest you examine carefully what caused any major differences.
If you really want to chose, then:

you can use a metric like total within distance or silhouette (ex. select the clustering solution with the lowest total with distance or with maximum average silhouette),
you can use your judgement to select the clustering solution that makes more sense business wise.

As mentioned before, ideally, metrics like total within distance or silhouette should not differ much. Hence, it all comes down to using the second criterion.
As for the part of the question about any benefits from using multiple algorithms,

if you are referring to k-means and hierarchical clustering, you could first perform hierarchical clustering and use it to decide the number of clusters and then perform k-means. This is usually in the situation where the dataset is too big for hierarchical clustering in which case the first step is executed on a subset.
in general, since not all clustering algorithms are suitable for every case it is useful to use multiple algorithms. For example k-means and hierarchical clustering are good at detecting clusters that have a spherical/ball shape. They would perform poor on more complex shaped data. A problem that DBSCAN does not have (though DBSCAN has other disadvantages, see discussion at Wikipedia)


Adversarial attacks on non image data Reading the literature around deep learning adversarial attacks it appears to be wholly concentrated on attacks of image classification models. Are there papers that describe attacks on non image data ? Searching archive for deep learning adversarial attacks appears to contain results that are just related to image classification field.

Carlini and Wagner (2018), “Audio Adversarial Examples: Targeted Attacks on Speech-to-Text”


Predicting churn - deal with missing dates in time series and improve modelling result This is the follow up question for General approach on time series for customer retention/churn in retail.

I have a time series of data in the following form:

| purchase_date    |    cutomer_id  |   num_purchases | churned |
   2018-10-31            id1              39             0
   2018-11-31            id1              0              0
   2019-01-31            id1              6              0
                         ...
   2019-03-31            id1              88            1
   2019-03-31            id2              300            0
   2018-04-31            id2               2             1
   2019-02-31            id3               1             1
   2019-07-31            id4               100           0
     ...                 id5

I grouped the data by month and summed num_purchases by month. The churned column for user id1 for example represents in which month customer churned. So id1 in my case churned in March. Before this, to label who has churned or not, we sampled customers based on 2 months of inactivity  period from the churn date. I need to predict if a user is going to churn in a 2 months from now.

I am getting very bad prediction results using logistic regression for example and the churned column as a class column. I suspect this is because some users  like id3 and id4 appear only once (or very few number of times) and other users like id1 appear a lot. I am not sure how to approach imputation in this case because these users just didn't exit before or after and I am not sure if it would make sense to impute them. Does anyone have idea on how to improve my model result?  I am getting 0.85 for accuracy, and 0 for precision, recall and F1.

It would be interested to deal with it as a sequence classification problem. For instance, you could use HMM (Hidden Markov Model) or equivalent to classify the sequences. The data format would be:

ID:  sequence      label
id1: 39,0,6,...,88  1
id2: 300, 2         1
id3: 1              1
id4: 100            0

Some suggestions:

Create more samples to balance also your dataset (e.g id1   39, 0    0)
Possibly bin the variables (e.g. to the decimal 6 -> 10, 39 -> 40)


How to self-learn data science? I am a self-taught web developer and am interested in teaching myself data science, but I'm unsure of how to begin. In particular, I'm wondering:

What fields are there within data science? (e.g., Artificial Intelligence, machine learning, data analysis, etc.)
Are there online classes people can recommend?
Are there projects available out there that I can practice on (e.g., open datasets).
Are there certifications I can apply for or complete?

Welcome to the site, Martin! That's a pretty broad question, so you're probably going to get a variety of answers. Here's my take.

Data Science is an interdisciplinary field generally thought to combine classical statistics, machine learning, and computer science (again, this depends on who you ask, but other might include business intelligence here, and possible information visualization or knowledge discovery as well; for example, the wikipedia article on data science). A good data scientist is also skilled at picking up on the domain-specific characteristics of the domain in which they working, as well. For example, a data scientist working on analytics for hospital records is much more effective if they have a background in Biomedical Informatics.
There are many options here, depending on the type of analytics you're interested in. Andrew Ng's coursera course is the first resource mentioned by most, and rightly so. If you're interested in machine learning, that's a great starting place. If you want an in-depth exploration of the mathematics involved, Tibshirani's The Elements of Statistical Learning is excellent, but fairly advanced text. There are many online courses available on coursera in addition to Ng's, but you should select them with a mind for the type of analytics you want to focus on, and/or the domain in which you plan on working.
Kaggle. Start with kaggle, if you want to dive in on some real-world analytics problems. Depending on your level of expertise, it might be good to start of simpler, though. Project Euler is a great resource for one-off practice problems that I still use as warm-up work.
Again, this probably depends on the domain you wish to work in. However, I know Coursera offers a data science certificate, if you complete a series of data science-related courses. This is probably a good place to start.

Good luck! If you have any other specific questions, feel free to ask me in the comments, and I'll do my best to help!


Sensorfusion: Generate virtual sensor based on analysis of sensorsdata I have a steam engine which is equipped with the following sensors:

temperature sensor in the boiler room
temperature sensor in the heating room
pressure sensor in the boiler room
rotations-per-minute-sensor which measures rpm of the wheel, which is powered by the steam engine

What the steam engine does not measure is the current water level in the boiler. The time-series data is stored in a csv-file with the following structure:

| timestamp | temp_boiler | temp_heater | pressure_boiler | rpm |

With help of machine learning, I would like to predict the current water level in the boiler. As I am very new to Machine Learning, I do not really know how to accomplish this. I know that I cannot apply a linear regression, because it is a time-series data and there is a correlation between the current sensor values and the previous sensor values. Furthermore, I would need historic water level date in order to run the linear regression.

So what options are there?
Do I have to use a Neuronal Network because I do not have any water level values? (Keywords: Sensorfusion, Virtual Sensor, Softsensor)
Any help is greatly appreciated!

EDIT: As linear regression is not an option, I would like to know what the alternatives are. As I am a complete newbie to Machine Learning in general, I would like to know which algorithms or strategies I can use to solve this. I am not asking for code snippets or something like that. Just some input so that I can work this out!

The first major point here is that you do not have historical water level values. These are the labels that you'd use to teach/train any supervised model. Without labels, you are limited to using unsupervised methods. Have a look at the Wikipedia overview

You could try looking into something like clustering, with k-Means for example, which would be quite fast. Other clustering algorithms like DBSCAN are also very good, but your variables (temperature and pressure) might not necessarily cluster well on their density in the input space (the D in DBSCAN).

There are unsupervised neural network approaches, but these are a bit more involved. Autoencoders might be a good place to start.

Have a look here for some more ideas on specific models to try out, such as Denoising AutoEncoders, which can use recurrent components (such as LSTMs), which can perform well on time series data.


Clustering with custom criterion (minimum cluster weight) Edit: following comment from @anony-mousse, I'm changing the question to search for a general clustering approach that matches this criterion (minimum weight per cluster).

I am to use a clustering method on a set of $n$ weighted points:

---------------------------------------------
| id  | weight | feature_1| feature_2 | ... |
---------------------------------------------
| 1   | 4      | 0.2345   | -0.2345   | ... |
| 2   | 2      | 0.675    | 0.7433    | ... |
| 3   | 15     | -0.45    | 0.123     | ... |
| ... | ...    | ...      | ...       | ... |
---------------------------------------------

I have a custom criterion: some algorithms make sure there is a minimum number of points $n_{min}$ per cluster ; here I would like to make sure each cluster has a minimum weight (sum of point weights) $\sum w_i > s_{min}$.

Is there such a clustering method already implemented in Python?

This does not work, and it's not how hierarchical clustering works.

If you stop at $n_\min$, no cluster will be larger than $2n_\min-2$ but there will either be plenty of badly clustered points, or unclustered points.

Consider the data set 0 2 3 5 with nmin=2. The first merge is (2,3) and fulfills the stopping criterion. So either you cluster this as (0), (2,3), (5) or as (0,5), (2,3) neither of which is convincing: either nmin is not a minimum size, or the clusters can be arbitrarily bad (and still may be below the minimum size).

The same concern applies to a weighted version.


What is difference between one hot encoding and leave one out encoding? I am reading a presentation and it recommends not using leave one out encoding, but it is okay with one hot encoding. I thought they both were the same. Can anyone describe what the differences between them are?

They are probably using "leave one out encoding" to refer to Owen Zhang's strategy.
From here
The encoded column is not a conventional dummy variable, but instead is the mean response over all rows for this categorical level, excluding the row itself. This gives you the advantage of having a one-column representation of the categorical while avoiding direct response leakage
This picture expresses the idea well.


Combining Neural Network with Reinforcement Learning in a Continuous Space I'm trying to learn how to do reinforcement learning on my own and I am not sure how to implement a neural network for a specific problem.

The game goes on for roughly 1 million steps. At each step, I have 36 continuous features available (unknown rules), and 4 actions to choose from. Afterwards, the environment will tell me what the 36 features are for the next step, and what my score is. I've been creating and looking at the data, and it sometimes only becomes clear a move was good/bad a lot later (10 moves or so).

So there are 2 things involved.

1) I have to "learn" how, given an action, the world would change from pre_state to post_state, and

2) I have to learn to optimize the reward given the state and choosing from 4 actions.

So I was thinking to just record state_before, state_after, reward of a lot of random moves. I could perhaps just use 36 * 10 of the last moves as predictors. But then again, perhaps I should only be interested in the difference between states?

The problem lies in the fact that I've looked at Markov Decision Processes, but they assume a discrete search space (whereas here it is continuous).

Any help would be useful in trying to understand what layers should be involved to solve this, and perhaps what the most logical way would be to sample data.

I really hope people can point me in the right direction. I'm willing to use any neural networks framework in Python basically.

I might be a bit late for answering but hope it helps!I assume that you are familiar with RL so I will omit lots of details (please if you are still interested comment so I can help you).

Neural Networks and RL: You have two options. The first one is to use a network which you will have as input your feature vectors (states) and output probability of each action. This is called a policy network and you can find a very detailed tutorial with Python Code in order to implement it by A. Karpathy.
Your second option is to use the Q-Network approach. Your input will be again the same but the output will be values of your Q function for each action you have ($Q(a_i)$). You will use the Q-learning  equation $Q(s_t,a_t)=Q(s_t,a_t) +\alpha[r_t+\gamma \max _a'Q(s_{t+1},a')-Q(s_t,a_t)]$. The details of the implementation can be found in the paper of V. Mnih. Also do not worry about the delayed rewards as the discount factor $\gamma will "help" your agent to be affected by the future rewards.

In order to calculate your states, I would suggest you to create a simulation of the environment and a step function. You don't mention what kind of game you are dealing with but the general idea is that the step function will take as input your current state and current action and output the next state and reward (don't mind about the continuous space of your features as you can discretize it - you can use Kalman filters or other models to have a better state estimation as well).

My advice would be to choose your approach (Policy net or Q-net) and read the blog or the paper, create a simulation of the environment and a step function for your game. You can find tons of implementations of the Deep Q-net although I would suggest you to start with a very simple network so you don't get in trouble by tuning the Deep net.


Reframing multilabel classification with imbalance in "both" directions Consider the multilabel problem when asking "does the sample belong to this class" with, for example, a movie label dataset where almost every movie is labelled "drama" because of this label's vagueness. It also has a few rare labels such as "French late 60s noir".
To both boost recall, as well as maintain compatibility with most classifiers, we can both resample and train a binary classifier for each label individually (drama: yes/no, noir: yes/no).
But while "French late 60s noir" might be a 0.05:0.95 imbalance that requires serious effort for the model to guess, the "drama" category is so widely applied that the problem is reversed, i.e. 0.95:0.05.
How should one logically best proceed to maintain recall over precision, now that the "target class" is 95% of the labels (kind of breaks with the definition)? Upsample the non-target class? Downsample the target class? Reframe the problem as "not drama" to maintain underbalance for every label?

I don't think there's any obviously best option. I'd suggest trying a few reasonable solutions, evaluate on a development set and then pick the one which performs best. Don't forget to try with the original data as it is, resampling doesn't always work better.

Upsample the non-target class? Downsample the target class?

These two options are very likely to give the same result (assuming you use the same proportion).

Reframe the problem as "not drama" to maintain underbalance for every label?

For the classifier it would be exactly the same.


How to select multiple columns in a RDD with Spark (pySpark)? Lets say I have a RDD that has comma delimited data. Each comma delimited value represents the amount of hours slept in the day of a week.

So for i.e. [8,7,6,7,8,8,5]

How can I manipulate the RDD so it only has Monday, Wednesday, Friday values? There are no column names by the way. But the PySpark platform seems to have _co1,_co2,...,_coN as columns.

I dont know which version you are using but I recommend DataFrames since most of upgrades are coming for DataFrames. (I prefer spark 2.3.2)

First convert rdd to DataFrame:

df = rdd.toDF(["M","Tu","W","Th","F","Sa","Su"])

Then select days you want to work with:

df.select("M","W","F").show(3)

Or directly use map with lambda:

rdd.map(lambda x: [x[i] for i in [0,2,4])

Hope it helps!


Interpreting a precision recall curve I have plotted the above precision recall curve for a binary classification problem where the classes are imbalanced. Classes are either negative or positive There are 3018 observations and 550 of them are positive. This means that only about 18% of the data are positive results. I read from here that the baseline for the precision recall curve is the ratio of the positives to the number of observations (in a balanced dataset the baseline would be 0.5). This is differently from the AUC baseline which is always fixed.
The Precision Recall AUC score is only .44, but the ROC AUC score is .72. What exactly does this mean? I know the ROC AUC is misleading for imbalanced datasets, but a precision recall AUC score of only .44 seems very low. How do I interpret this? Is it actually a bad score? How can I understand this curve in reference to the baseline?

a precision recall AUC score of only .44 seems very low.

I agree, but your model has definitely learned something because you beat your naïve baseline (technical term not criticism!) Depending on what you’re trying to predict, that AUC might either be impressive or disappointing. By a similar token, the model you have trained here might be useful as it is, or might not have reached a minimum viable level of performance.

How do I interpret this?

When your model is able to retrieve half of the positive samples (~50% recall) it happens that about half of the samples it identifies as positive are actually negative (~50% precision). Beyond that, the precision decreases as we attempt to get greater recall, which is what we’d expect.

Is it actually a bad score?

Has anyone tried to predict the thing you’re trying to predict before? Maybe there is a more complex baseline you could compare yourself to. If not, you’ll need to do more work to establish whether you could do better. For an unbalanced dataset, gradient boosted trees are a good baseline beyond the naïve (no learning takes place) baseline. Try something like xgboost if you haven’t already. Take the model you just trained as a new baseline and try some other stuff.

How can I understand this curve in reference to the baseline?

The baseline you are using represents the precision you would see if you just always predicted that samples were positive. This would obviously result in 100% recall. If your model moves above that in terms of precision, anywhere on the graph, it may be learning something. It’s clear this is the case as it is doing consistently better.


Numerically stable hyperbolic tangent The hyperbolic tangent is commonly used as an activation function:
$$
tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}
$$
Although, it is unclear how this function is implemented to be numerically stable in frameworks like pytorch or numpy. Since $e^x$ will explode for any $x \gtrapprox 1000$, is the function being approximated in some way? Or are there static checks for big input numbers?

Commonly you divide the domain up into different accuracy,stability regimes and apply a different approximation within each.
For $|x|<1$ you can use a polynomial approximation, like Chebyshev. For $1<|x|\leq E$, $E$ depending on implementation of the formula, you can use a related formula to the one you described.
For $|x|>E$ you can use $\tanh(x) \approx sign(x).$
More details here from a documented implementation.


Does tensorflow pick samples from a dataset randomly or sequentially when training? I have a dataset which consists of more than 10000 images. But similar images are grouped together. I mean first 50 images are very alike then the next 50 images are different(not as much similar as the first ones. I am talking about guns specifically) than the first 50 but they are similar between each other. If I choose a batch size of 50, will it lead to worse results? Or It picks random subsets of the dataset to train at a time?

I am new at deep learning if the answer is obvious sorry.

Welcome to the site! My understanding is that it does it sequentially. It's usually a good idea to shuffle your dataframe as a pre-processing step and then train on that.


Concat two tensors of different dimensions I have two tensors. For example -
a = torch.randn((500, 200, 10))
b = torch.randn((500, 5))

I have to concat each of b tensor to all elements of corresponding a tensor i.e., each 200 tensors of a[0] should get concatenated with b[0] - final dimension should be (500, 200, 15).
Without using explicit for loop, how can I achieve this in Pytorch efficiently?

For that, you should repeat b 200 times in the appropriate dimension this way:
c = torch.cat([a, torch.unsqueeze(b, 1).repeat(1, 200, 1)], dim=2)
c.shape

As desired, the shape of the result is torch.Size([500, 200, 15])


Which Kind of Machine Learning should I use for an Optimization Problem? I have a problem where I would like to apply machine learning. For this purpose I have been learning about the topic and have come across supervised, unsupervised, semi-supervised machine learning techniques.

I know the problem that I want to solve, so unsupervised machine learning doesn't seem to be a fit, because from how I understood, it is used when we don't know what we are looking for.

But I don't have a pre-labelled data set either, because we ourselves don't know if we are operating at the optimum level.

Basically we want to optimize profit, and we have the option of tweaking certain parameters. As an example let's say we want to the ML Algorithm to find the price of a product that would optimize the profit.

What kind of ML can I apply here? Do you have any recommend frameworks? Any book that I can read to understand this (the books that I have referred to so far have all given examples of classification)?

Adding more context:- The actual problem is to balance a logistics network. There are multiple warehouses and delivery service providers(DSP) (DHL, GLS etc). We have some parameters that define to which warehouse/DSP we want to send the orders(to be packed and delivered). We want to find the optimum parameters that will yield profit. We have already processed data where we can train and validate the algorithm. But we don't know if the current parameters are correct, so I would assume that training the algorithm with the data that we have would not give us an optimum solution (because the labelled targets, i.e. the achieved profit might not be optimum)

With this specific algorithm we are trying to figure out the best warehouse/DSP combination for a given order. While we want to send a parcel to a cheapest warehouse/DSP option, it might be the case that the cheapest warehouse/DSP is out of capacity (limited by manpower at the warehouse or limited by the number of ordered trucks of a given DSP), in which case we might choose to send the order to a non cheapest warehouse, or to send the order the next day. With this solution we don't plan to optimize anything else (that is optimizing how stock is stored/storage costs etc.). The goal is to utilize the warehouse/truck capacity the best way we can. I don't need a full blown solution, but a few keywords/concepts/places to which I can refer would be enough.

PS:- Sorry for the half blown example in the original question. I just wanted to give a simple/similar example so that I can get some keywords, frameworks, books to further study.

You have no existing data where the parameters you mentioned have been tweaked and you then have the resulting profits?

If you don't have any pre-labeled data then you will find it very difficult because the NN will have nothing to learn from.  NNs learn from either the information you give; the inputs and corresponding outputs, or by experimenting with values and then using the result of using those outputs as labels / targets to learn from.


R Script to generate random dataset in 2d space I want to analyze the effectiveness and efficiency of kernel methods for which I would require 3 different data-set in 2 dimensional space for each of the following cases:

BAD_kmeans: The data set for which the kmeans clustering algorithm
will not perform well.
BAD_pca: The data set for which the Principal Component Analysis
(PCA) dimension reduction method upon projection of the original
points into 1-dimensional space (i.e., the first eigenvector) will
not perform well.
BAD_svm: The data set for which the linear Support Vector Machine
(SVM) supervised classification method using two classes of points
(positive and negative)  will not perform well.

Which packages can I use in R to generate the random 2d data-set for each of the above cases ? A sample script in R would help in understanding

None of the algorithms you mention are good with data that has uniform distribution.

size <- 20             #length of random number vectors
set.seed(1)
x <- runif(size)          # generate samples from uniform distribution (0.0, 1.0)
y <-runif(size)
df <-data.frame(x,y)

# other distributions: rpois, rmvnorm, rnbinom, rbinom, rbeta, rchisq, rexp, rgamma, rlogis, rstab, rt, rgeom, rhyper, rwilcox, rweibull.

See this page for tutorial on generating random samples from distributions.

For specific set of randomized data sets that are 'hard' for these methods (e.r. linearly inseparable n-classes XOR patterns), see this blog post (incl. R code): http://tjo-en.hatenablog.com/entry/2014/01/06/234155.


Where is the majority of Tensorflow memory usage? Trying to wrap my head around where Tensorflow starts to use a lot of memory.

If I have a batch_size=10 and my samples are 150MB each, then that’s only ~1GB at a time or less if the samples are processed 1 at a time.

Is it more about the accruing size of the neural net itself as I go through more epochs? If I try to train against 10,000 of these 150MB samples, am I more likely to run out of memory toward the end of the training?

Assuming your samples have a fixed size and you use constant batch size, you should never run out of memory - that is: as soon as you have managed to train a full batch (forward and backward propagation complete) you should be fine to assume it will train until completion!

Extra info / tips

The total memory requirement will be a product of the size of the model (number of parameters) along with the batch size. Each single sample that gets sent through the network will effectively influence each of the $N$ weights. A batch of size $B$ will mean that there needs to be roughly $N \times B$ memory available.

If you model is already fixed, you can start with a large batch size, decreasing it slowly until you no longer receive an outOfMemory error. You could also try computing exactly what your memory requirement are, but there are some other factors that might make that a little difficult, so it is faster to try the first approach.

If you are using a GPU, you ca look at the output of the terminal command nvidia-smi, you can see the available memory of the GPUs. You will notice it essentially becomes all used as soon as training begins. This is because Tensorflow, by default, will occupy all available memory. There are ways around that, so search for allow_growth in the Tensorflow docs for relevant information and configuration tactics.


How to properly resize input images for transfer learning I have to resize some images of different size to 224x224 before they can be passed as input for VGG19, and then apply transfer learning.

I have tried these methods: add patches, take a square of 224x224 from the images' center, automatically adjust height and width.

I would like to know, if there is one, which is the best method for resizing these images.

This can be accomplished using the PIL library in Python.

One thing to note - if you are resizing the images to 224x224, you might want to keep the proportion of your image, e.g. if the height and width is significantly different, then you might lose perspective by resizing in this way.

If proportion is not an issue, you can resize as follows:

from PIL import Image
baseheight = 224
img = Image.open('image.jpg')
width = 224
img = img.resize((width, baseheight), Image.ANTIALIAS)
img.save('resizedimage.jpg')

That said, if you wanted to keep the same proportions as the original image and only wanted to set height to 224 for instance while keeping a proportional width, you could also use PIL as follows:

from PIL import Image
baseheight = 224
img = Image.open('image.jpg')
hpercent = (baseheight / float(img.size[1]))
width = int((float(img.size[0]) * float(hpercent)))
img = img.resize((width, baseheight), Image.ANTIALIAS)
img.save('resizedimage.jpg')


Predict sinus with keras feed forward neural network I have a very simple feed forward neural network with keras that should learn a sinus. Why is the predictive power so bad and what is generally the best way to pinpoint issues with a network?

In the code below, I have one input neuron, 10 in the hidden layer, and one output. I would expect the network to perform much more accurately.

import numpy as np
from keras.layers import Dense, Activation
from keras.models import Sequential

x = np.arange(100)
y = np.sin(x)

model = Sequential([
    Dense(10, input_shape=(1,)),
    Activation('sigmoid'),
    Dense(1),
    Activation('sigmoid')
])

model.compile(loss='mean_squared_error', optimizer='SGD', metrics=['accuracy'])
model.fit(x, y, epochs=10, batch_size=1)

scores = model.evaluate(x, y, verbose=0)
print("Baseline Error: %.2f%%" % (100-scores[1]*100))

print(model.predict(np.array([.5])))

Output:

  1/100 [..............................] - ETA: 0s - loss: 1.2016 - acc: 0.0000e+00
 79/100 [======================>.......] - ETA: 0s - loss: 0.4665 - acc: 0.0127
100/100 [==============================] - 0s - loss: 0.5044 - acc: 0.0100
Baseline Error: 99.00%
[[ 0.35267803]]

Keep in mind that the Accuracy measure is measuring whether the values are Exactly The Same. I.e. it is a classification measure, whereas approximating a sin curve is much better suited for measurement as a regression problem.

That said:

In evaluating the network's performance, what is the network actually doing? Let's take the network and perform a little visual analysis on its performance:

import matplotlib.pyplot as plt
preds = model.predict(x)
plt.plot(x, y, 'b', x, preds, 'r--')
plt.ylabel('Y / Predicted Value')
plt.xlabel('X Value')
plt.show()

Hmm. The model seems to be minimizing error by simply getting closer and closer to guessing 0 for every value, rather than approximating the function. There are several hypotheses here to explain this. One is that the network is not complex enough to model the function. In order to test this, let's simplify the function- that is, let's bring the range down to one sine cycle:

x = np.arange(0, math.pi*2, 0.1)
y = np.sin(x)

And try to train the network again:

Not wonderful, but a better fit, certainly.

How about with 100 epochs instead of 10?

How about with 1000 epochs?

This is, of course, very interesting. After 1000 epochs, our networks is able to roughly approximate the downward curve from 1:0 ($\pi/2$: $\pi$) of the sine response, but not the initial upward curve 0:1 (0:$\pi/2$) or the region in which the function is negative ($\pi$:$2\pi$).

This result begs the question- what will it look like after 10000 epochs?

Not significantly better. It looks like we'll have to change the architecture of the network (more layers, more neurons, and/or different activation functions) to improve beyond this point.

To inform this architecture change, let's take a look at the sigmoid activation function:

Uh-oh. The value of this sigmoid function can only ever be in the range 0:1, and the range of the sin function is -1:1.

To correct this, let's just normalize the sin response between 0 and 1:

y = (np.sin(x)+1)/2

Now, the network performs much better than before after 1000 epochs:

And 10000:

After 100000 epochs, it's roughly perfect:

Even still, this advancement doesn't help much on the larger sin range (after 1000 epochs):

x = np.arange(0, 100, 1)
y = (np.sin(x)+1)/2

If we, however, take the model trained on a single sin curve and further train it on the larger range, we begin to see progress after 1000 epochs:

x = np.arange(0, 100, .1)
y = (np.sin(x)+1)/2

model_copy = model
model_copy.fit(x, y, epochs=1000, batch_size=8, verbose=0)
model_copy_preds = model_copy.predict(x)

And more so after 10000 epochs:

And more so, well into the third repetition after 100000 epochs:

So, with careful training, our network with a single layer of sigmoid activations appears to be learning to generalize the sin curve. Further investigation could find a limit to that generalization, certainly.

For reproduction:

import numpy as np
from keras.layers import Dense
from keras.models import Sequential
import matplotlib.pyplot as plt
import math

x = np.arange(0, math.pi*2, .1)
y = (np.sin(x)+1)/2

model = Sequential([
    Dense(10, input_shape=(1,)),
    Activation('sigmoid'),
    Dense(1)
])

model.compile(loss='mean_squared_error', optimizer='SGD', metrics=['mean_squared_error'])
model.fit(x, y, epochs=100000, batch_size=8, verbose=0)

preds = model.predict(x)

plt.plot(x, y, 'b', x, preds, 'r--')
plt.ylabel('Y / Predicted Value')
plt.xlabel('X Value')
plt.show()

x = np.arange(0, 100, .1)
y = (np.sin(x)+1)/2

model_copy = model
model_copy.fit(x, y, epochs=10000, batch_size=8, verbose=0)
model_copy_preds = model_copy.predict(x)

plt.plot(x, y, 'b', x, model_copy_preds, 'r--')
plt.ylabel('Y / Predicted Value')
plt.xlabel('X Value')
plt.show()


State of the art approaches for Information retrieval tasks based on deep learning I am interested in the state of the art approaches for information retrieval (IR) tasks, where you have a single query and a set of documents and the IR model will give you the best matched document.

I have worked on vector space models (tfidf-cosine similarirty ) and LSA.

I have also tried Wordnet, NER, fuzzy matching etc for improving the accuracy.

Now I would like to know how to improve accuracy of IR tasks by applying nueral networks, word embeddingstopic models etc by capturing more context/sematics information

Deep learning extends the ability of information retrieval (IR) systems.

Deep learning has proved to be a powerful tool for feature engineering. This improvement in feature engineering, over manual construction, has improved the quality of any machine learning system, including information retrieval.

Specifically, word embeddings create a dense, vectorized representation of words that encodes the semantic relationships based on co-occurrence. Both documents and queries can be projected into the same latent space and nearest-neighbors can be found. Word embeddings can be used to extend the posting index to by finding related terms. They can also be used for query-expansion / query rewriting.


How to understand features impact in a non linear case? I give a simple example:
I have a set of houses with different features (# rooms, perimeter, # neighbours, etc...), almost 15, and a price value for each house. The features are also quite correlated (i.e. perimeter is often correlated with #rooms).
I want to establish what are the main features (or non-linear combination of them) that determine the price.

In a linear case, for instance, I can compute a Lasso regression and see the importance of each feature through the coefficients.
In my case, every feature (or maybe combination of them) has a non linear impact.
For example, the # of neighbours can have a quadratic impact (increase the price if #neighbours < 10, and decrease the price if > 10).

I want to identify the main important relationship among the features and the prices.  I don't need a predictor. For example, at the end I will discover that the price depends principally by #rooms/perimeter and #neighbours^2.

I was thinking to apply Kernel methods, in combination with regression or PCA. But I don't know a lot about kernel methods.

Thank you in advance.

As far as I know, kernel methods cannot deal with categorical variables (don't now whether it is the case). In addition, you will have to use indirect methods to evaluate the variable importance. This could work, although I havent tested it yet:

Giam, X., Olden, J.D., 2015. A new R2-based metric to shed greater insight on variable importance in artificial neural networks. Ecol. Modell. 313, 307–313. http://dx.doi.org/10.1016/j.ecolmodel.2015.06.034

I would definitively go for a tree-based approach. Since you already know that there are correlated variables, I would advocate for conditional Random Forests (which solve many drawbacks of the standard random forests implementation). Check:

Strobl, C., Hothorn, Zeileis, A., 2009. Party on! R J. 1 (2), 14–17.

And references therein. At least in R there are complementary packages (https://cran.r-project.org/web/packages/pdp/index.html) that allow plotting the impact of each predictor variable over the target variable (house prices). That complements the variable importance ranking pretty well.

Good luck.


Scaling multiple time series data I am using crypto currency chart data and was wondering what would be the best process for scaling the time series data. The issue is that some of these currencies are highly volatile and can see extreme swings. The standard process that I see is something like:

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler
x_train, y_train, x_test, y_test = train_test_split(x,y, test_size=0.2)
scaler = MinMaxScaler()
scaler.fit(x_train)
x_train = scaler.transform(x_train)
x_test = scaler.transform(x_test)

By default MinMaxScaler scales between 0 and 1 but there are cases where x_test may contain a max that could be double or more what the max was when the scaler was fit. Will this cause model issues or is this acceptable? How do you handle scaling data that you don't know the min/max for? Is min/max scaling best for this?

Also, some currencies have drastically different prices. For example, one could be trading +10x or -10x or more per unit than another. Will having several time series on drastically different scales effect the models? Its seems to me that scaling each time series and each feature to 0-1 would be the best but I am not completely sure. Lets say I have n different time series that have drastically different values. Would something like the following be a decent approach?

#assume data is a list of padas DataFrames
#e.g. data = [pd.DataFrame(asset1), pd.DataFrame(asset2)]
scaled = []
for asset in data:
    fit_data = data.iloc[:int(len(data*0.2))]
    scaler = MinMaxScaler()
    scaler.fit(fit_data)
    scaled.append(scaler.transform(asset))

Rather than using MinMaxScaler, I think StandardScaler would be a better option if you have outliers that are quite different from most of the values, but are still valid data.

StandardScaler is used the same way as MinMaxScaler, but it will scale your values so they have mean equal to 0 and standard deviation equal to 1. Since those values will be found based on all the values in the series, it is much more robust against outliers.


Adaboost - Show that adjusting weights brings error of current iteration to 0.5 I'm trying to solve the following problem but I've gotten sort of stuck.

So for adaboost, $err_t = \frac{\sum_{i=1}^{N}w_i \Pi (h_t(x^{(i)}) \neq t^{(i)})}{\sum_{i=1}^{N}w_i}$

and $\alpha_t = \frac{1}{2}ln(\frac{1-err_t}{err_t})$

Weights for the next iteration are $w_i' = w_i exp(-\alpha_t t^{(i)} h_t(x^{(i)}))$
and this assumes $t$ and $h_t$ takes on a value of either $-1$ or $+1$.

I have to show that the error with respect to the new weights $w_i'$ is $\frac{1}{2}$.
i.e., $err_t' = \frac{\sum_{i=1}^{N}w_i' \Pi (h_t(x^{(i)}) \neq t^{(i)})}{\sum_{i=1}^{N}w_i'} = \frac{1}{2}$

i.e., we use the weak learner of iteration t and evaluate it according to the new weights, which will be used to learn the $t+1$-st weak learner.

I simplified it so that $w_i'=w_i \sqrt{\frac{err_t}{1-err_t}}$ if $w_i$ was correctly classified and $w_i'=w_i \sqrt{\frac{1-err_t}{err_t}}$ if $w_i$ was incorrectly classified. I then tried plugging this into the equation for $err_t'=\frac{1}{2}$ and got $\frac{err_t}{1-err_t} \frac{\sum_{i=1}^{N}w_i \Pi (h_t(x^{(i)}) = t^{(i)})}{\sum_{i=1}^{N}w_i \Pi (h_t(x^{(i)}) \neq t^{(i)})} = 1$ but at this point I sort of ran into a dead end and so I'm wondering how one might show the original question.

Thanks for any help!

You're nearly there.  The quantity $err_t/(1-err_t)$ is exactly what you need it to be.  It might be easier to see if you think about
$$\sum_{i=1}^N w_i \Pi(h_t(x^{(i)})=t^{(i)})$$
as
$$\sum_{i:\ x^{(i)}\text{ is correctly classified}} w_i$$
(just using the indicator function to reduce the summation range).


How to implement this CNN architecture in Keras I am trying to implement in Keras the CNN architecture used by Rajpurkar et al and illustrated below:

I am particularly confused about that max pool that is shown in the right side. Is that a fork from the main sequential line? How do I implement that?

Thanks.

In this kind of situation, you can not use sequential API which is generally used in the architecture where you to stack layers on each other.

For this kind of problem use the functional API of keras. Based on references image tried to create architecture & as the author has used RESNET architecture in his paper. So, I have tweaked network according to it rather than replicating architecture give in the image.

input_img = keras.Input(shape = (224, 224, 3))

x1 = keras.layers.Conv2D(32, (3, 3), padding = 'same')(input_img)
x1 = keras.layers.BatchNormalization(axis = 3)(x1)
x1 = keras.layers.Activation('relu')(x1)

x2 = keras.layers.Conv2D(32, (3, 3), padding = 'same')(x1)
x2 = keras.layers.BatchNormalization(axis = 3)(x2)
x2 = keras.layers.Activation('relu')(x2)
x2 = keras.layers.Dropout(0.2)(x2)
x2 = keras.layers.Conv2D(32, (3, 3), padding = 'same')(x2)

merge_x2 = keras.layers.Add()([x1, x2])
x3 = keras.layers.BatchNormalization(axis = 3)(merge_x2)
x3 = keras.layers.Activation('relu')(x3)
x3 = keras.layers.Dropout(0.2)(x3)
x3 = keras.layers.Conv2D(64, (3, 3), padding = 'same')(x3)
x3 = keras.layers.BatchNormalization(axis = 3)(x3)
x3 = keras.layers.Activation('relu')(x3)
x3 = keras.layers.Dropout(0.2)(x3)
x3 = keras.layers.Conv2D(64, (3, 3), padding = 'same')(x3)

conv_merge_x2 = keras.layers.Conv2D(64, (3, 3), padding = 'same')(merge_x2)
merge_x3 = keras.layers.Add()([conv_merge_x2, x3])
x4 = keras.layers.BatchNormalization(axis = 3)(merge_x3)
x4 = keras.layers.Activation('relu')(x4)
x4 = keras.layers.Flatten()(x4)
x4 = keras.layers.Dense(2)(x4)
final = keras.layers.Activation('softmax')(x4)

model = keras.models.Model(input_img, final)
model.compile(loss = "categorical_crossentropy", optimizer = "adam", metrics = ["accuracy"])


Bias-variance tradeoff and the uncertainty principle Bias variance tradeoff seems to behave like the uncertainty principle, is it just another name for the same principle?

No, the uncertainty principle describes a property that is specific to electrons. That electrons don't display their wave and particle properties simultaneously. Here from Wikibooks:

  The Heisenberg uncertainty principle states that the exact position
  and momentum of an electron cannot be simultaneously determined. This
  is because electrons simply don't have a definite position, and
  direction of motion, at the same time!

.

  This is due to the fact that electrons cannot exhibit both their wave
  and particle properties at the same time when being observed to
  interact with their surroundings. The momentum of an electron is
  proportional to its velocity, but based on its wave properties; its
  position is based on its particle position in space.

The bias-variance tradeoff describes a property of predictive models. That error from bias and variance usually exist in a tug-o-war where you cannot decrease one without increasing the other and that you have to find the perfect middle ground. You can observe it here from this Medium:

I agree that there are some similarities. There is a sort of tradeoff between measuring the momentum and position of electrons. But they are not the same thing.


Image features (produced by VGG19) do not properly train an ANN in Keras I've used a VGG16 network to extract features from an image dataset, creating a features dataset using the following code:

model = VGG16(weights='imagenet', include_top=False)
x = 0

for folder in os.listdir(os.fsencode('training_cnn_images/')):
    folder = str(folder)[2:-1]
    directory = os.fsencode('training_cnn_images/' + folder)
    dn = str(directory)[2:-1]
    for file in os.listdir(directory):
        fn = str(file)[2:-1]
        img = image.load_img(dn + '/' + fn, target_size=(224, 224))
        img_data = image.img_to_array(img)
        img_data = np.expand_dims(img_data, axis=0)
        img_data = preprocess_input(img_data)
        vgg16_feature = model.predict(img_data)
        with open(('training_cnn_features' + dn[19:] + '/' + fn)[:-4] + '.pkl', 'wb') as f:
            pkl.dump(vgg16_feature, f)
        x += 1
        printProgress(x, 1895, 'Generating Features:')

I simply feed an image to VGG16, and the resultant output is used as the extracted features.
Now that I have a .pkl dataset containing the features for all the images of the dataset, I wanted to create an ANN that takes these features as an input and classifies the images (technically their features) into two categories.

So, I wrote this code to do just that:

inney = [None] * 315
outey = [None] * 315

x = 0
for file in os.listdir(os.fsencode('training_cnn_features/n02085620-Chihuahua/')):
    fn = str(file)[2:-1]
    with open('training_cnn_features/n02085620-Chihuahua/' + fn, 'rb') as f:
       inney[x] = pkl.load(f).flatten()
       outey[x] = np.array([[0, 1]])
    x += 1
for file in os.listdir(os.fsencode('training_cnn_features/n02085782-Japanese_spaniel/')):
    fn = str(file)[2:-1]
    with open('training_cnn_features/n02085782-Japanese_spaniel/' + fn, 'rb') as f:
       inney[x] = pkl.load(f).flatten()
       outey[x] = np.array([[1, 0]])
    x += 1

inney = np.asarray(inney)
outey = np.asarray(outey)

model = Sequential()
model.add(Dense(1024, input_shape=(25088,), activation='relu'))
model.add(Dense(2, activation='softmax'))
model.compile(loss = "mean_squared_error", optimizer = optimizers.Adamax(lr=0.002, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0))
model.fit(inney, outey, epochs=10, batch_size=4, verbose=1)
model.save('test.h5')

I had also created a test dataset by partitioning the original dataset into a training and test set. When I attempted to test the network, the network always seems to output [0, 1] or [*some very small value*, 1]. When I try with sigmoid, it flips, i.e, it becomes [1, 0].

A similar issue arises (one output is always one or very close to it while the other is always zero or very close to it) when I use inceptionV3.

Nothing seems to work. I've attempted adding the extra layers directly to the inception or VGG16 network, and freezing the non-classifier layers, but I face the same issue over and over again. The neural network just won't classify.

I've attempted using both sigmoid and softmax, but face the same issue in both cases.

Help would be very much appreciated.
Thanks.

After doing a bunch of searching and trial & error, I found out that using the sigmoid activation function for all the layers and using a binary_crossentropy loss fixed the issue. The newly added hidden layers were too few and too small for the ReLU function to work.


Feature selection by overfitting a small sample size I am using a CNN based model to do sequence classification. Since training an entire dataset is very expensive, and I have a large set of features needed to try, its impossible for me to select features by full trainings.

Usually a good sanity check before training, is try to let model overfit a small set of training samples, to make sure the model is at least capable of remember a small sample size.

Borrowed from this idea, my question is, can I train a small subset of training data, and use its loss curve as a metric, to select best features? Each training is to test how quick would the training loss converges given a subset of selected features.

Congratulations! You have suggested independently the Wrapper method for feature selection. Yes, you can use this method.
However, consider that the wrapper method is slow since you have to train a model for each iteration. More than that, feature selection is a NP-complete problem so don't expect the optimal subset.


Is it possible to make tensorflow print out everything it see in a given image and not just the top five results? I'm working through the python API tutorials for Tensorflow and I'm seeing the results that are normally displayed, but it's always giving me the top five results.

I'm trying to discern all possibilities within a certain list of basic attributes, like if I'm given a picture of a forest, I want to ask tensorflow if the picture contains oak trees, pine trees, bushes, rivers, etc. I don't need to know if the image is a picture of a forest. Is this possible?

I'm not saying give me results it hasn't been trained to see, I'm saying I'm going to train the model with different types of trees/bushes/etc and I want to know if the given image contains any of those attributes (or the probability it thinks of for a given attribute).

Sounds like you want to see the predicted probabilities of a softmax function. You can assign the values to a list during training so you can see probabilities of each epoch if you'd like. As is written below, in_top_k will select the single top prediction of the softmax cross entropy function but if you have multiple targets in each picture you will want to change that "1" to the desired amount.

xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(logits = logits,
           labels = y)

logit = tf.nn.in_top_k(logits, y, 1)

y_one_prob = tf.sigmoid(logit)


Does high error rate in regression imply the data set is unpredictable? I have a data set of video watching records in a 3G network. In this data set, 2 different kind of features are included:

user-side information, e.g., age, gender, data plan and etc;
Video watching records of these users, each of  which associated with a download ratio and some detailed network condition metrics, say, download speed, RTT, and something similar.

Under the scenario of internet streaming, a video is divided into several chunks and downloaded to end device one by one, so we have download ratio = download bytes / file size in bytes

Now, Given this data set, I want to predict the download ratio of each video.

Since it is a regression problem, so I use gradient boosting regression tree as model and run 10-fold cross validation.

However, I have tried different model parameter configurations and even different models (linear regression, decision regress tree), the best root-mean-square error I can get is 0.3790, which is quite high, because if I don't use any complex models and just use the mean value of known labels as prediction values, then I can still have an RMSE of 0.3890. There is not obvious difference.

For this problem, I have some questions:

Does this high error rate imply that the label in data set is unpredictable?
Apart from the feature problem, is there any other possibilities? If yes, how can I validate them?

It's a little hasty to make too many conclusions about your data based on what you presented here. At the end of the day, all the information you have right now is that "GBT did not work well for this prediction problem and this metric", summed up by a single RMSE comparison. This isn't very much information - it could be that this is a bad dataset for GBT and some
other model would work, it could be that the label can't be predicted from these features with any model, or there could be some error in model setup/validation.

I'd recommend checking the following hypotheses:

1) Maybe, with your dataset size and the features you have, GBT isn't a very high-performance model. Try something completely different - maybe just a simple linear regression! Or a random forest. Or GBDT with very different parameter settings. Or something else. This will help you diagnose whether it's an issue with choice of models or with something else; if a few very different approaches give you roughly similar results, you'll know that it's not the model choice that is causing these results, and if one of those models behaves differently, then that gives you additional information to help diagnose the issue.

2) Maybe there's some issue with model setup and validation? I would recommend doing some exploration to get some intuition as to whether the RMSE you're getting is reasonable or whether you should expect better. Your post contained very little detail about what the data actually represents, what you know about the features and labels, etc. Perhaps you know those things but didn't include them here, but if not, you should go back and try to get additional understanding of the data before continuing. Look at some random data points, plot the columns against the target, look at the histograms of your features and labels, that sort of thing. There's no substitute for looking at the data.

3) Maybe there just aren't enough data points to justify complex models. When you have low numbers of data points (< 100), a simpler parametric model built with domain expertise and knowledge of what the features are may very well outperform a nonparametric model.


value of fit_generator parameters I am doing simple CNN based binary image classification on classifying types of waste as organic and recyclable using image augmentation. How ever I have a doubt.

In the third figure i have set the steps_per_epoch = 22564/32= 705 where 32 is batch size and validation_steps as 2513/32 = 79 as per documentation does that mean out of 22564 training and 2513 validation samples 705 and 79 are trained and validated ?

No, that is the steps per epoch. One step is taking 32 images(batch size) So, 709*32 images used in that epoch and validated with 79*32 images.


Where to start with Data Science I know next to nothing about Data Science. Where should I start? Where do you recommend I should take courses (hopefully free)? Or even a book.

Well, I started studying in Datacamp and I really recommend this Machine learning course it's very complete for new learners.

Udacity has this Deep-learning course and it's a complement for the Coursera Machine learning course.

It's very important to research a lot in other sources but there you can paint a path for your Data Science skills through these ones.

I'm working and studying in Machine learning and I store the test projects in this repository, but is very important to have a little bit of knowledge of python and octave

Already exist other languages and tools like R for test your data sets


Chat Bot Answering based on Data Corpus Self-Training I have created a very simple chat bot based on RASA NLU. In this case, I manually create some sample input text and create a model for using it against unknown source of input. It's fine for now.

As my next step of learning, I want to do use a big documnet as source for my chat bot.

What steps should I do to make the program train automatically on the documnet text corpus and able to answer based on the user query? I want to avoid the manual training.

For this NLU problem statement, what libraries can I use?

Almost all internet sources talk about SQuaD models. How they are related to a custom domain training?

Any blogs, tutorials, libs that can help me to do this will be useful.

Some other related questions but without solid answers:

Build knowledge bot using deep learning

It seems like your problem is some form of FAQ matching. The user asks a question, the NLU part matches this question to a relevant answer located in your files.

For the training aspect, you could do two things:

Similarity measure

You could simply use a string matching distance like Levenshtein between your input and the different outputs, and return the output that is the closest. This way you do not need to train your model per se. Since this is a form of a K-Nearest Neighbour with one neighbour, the learning happens during the inference. The advantage of this is that if your files change, or you have more answers, you won't have to change anything.

This is what you mean by self-learning.

This blog explains how to use fuzzywuzzy (a Python package to compute such distances): https://marcobonzanini.com/2015/02/25/fuzzy-string-matching-in-python/

Trained model

Obviously, simple distance measures might not scale well or might simply not perform great. Your other option is to treat this as a classification problem. Your classes are your different answers from your file, and your inputs are the user inputs. This will require the user input to be labelled.

This is what you refer to as manual training.

Question answering

Everything that was mentioned above was assuming that you have a known list of possible answers. If what you want is to have your bot "query" your text files and figure out the answer on its own, it will be much more complicated task. Question answering from open text is a complicated field of research. Just think of Google. It is able to sometimes answer simple questions like "Who?", "What time?" etc. but it's not very good yet at giving concise answers to complex queries.


BLEU_SCORE gives bad scores - what am I doing wrong? I want to calculate the BLEU_SCORE but it gives me bad results I don't know why?
For example, this is my reference and predicted sentences:
ref :
['electron', 'and', 'a', 'proton']
predicted :
['electron', 'and', 'a', 'proton']
ref :
['to', 'reach', 'the', 'nectar', 'at', 'the', 'bottom', 'of', 'flowers']
predicted :
['to', 'reach', 'the', 'nectar', 'at', 'the', 'bottom', 'of', 'flowers']
ref :
['during', 'the', 'summer', 'near', 'the', 'north', 'pole']
predicted :
['during', 'the', 'summer', 'near', 'the', 'north', 'pole']
ref :
['only', 'blue', 'light', 'is', 'reflected', 'by', 'the', 'block']
predicted :
['only', 'blue', 'light', 'is', 'reflected', 'by', 'the', 'block']
ref :
['between', '20', 'and', '40', 'degrees', 'latitude']
predicted :
['between', '20', 'and', '40', 'degrees', 'latitude']
ref :
['external', 'and', 'internal', 'combustion', 'engines']
predicted :
['external', 'and', 'internal', 'combustion', 'engines']
ref :
['cleaning', 'disinfecting', 'and', 'in', 'swimming', 'pools']
predicted :
['cleaning', 'disinfecting', 'and', 'in', 'swimming', 'pools']
ref :
['body', 'mass', 'index', 'bmi']
predicted :
['body', 'mass', 'index', 'bmi']
ref :
['they', 'put', 'nutrients', 'into', 'the', 'soil', 'that', 'plants', 'use', 'to', 'grow']
predicted :
['they', 'put', 'nutrients', 'into', 'the', 'soil', 'that', 'plants', 'use', 'to', 'grow']
ref :
['structure', 'of', 'earth', 'interior']
predicted :
['structure', 'of', 'earth', 'interior']

And here is the code I used to calculate the BLEU_SCORE :
from nltk.translate.bleu_score import corpus_bleu

print("Individual n-gram")
print("Individual 1-gram")
print('BLEU-1: %f' % corpus_bleu(ref, pre, weights=(1.0, 0, 0, 0)))
print("Individual 2-gram")
print('BLEU-2: %f' % corpus_bleu(ref, pre, weights=(0, 1.0, 0, 0)))
print("Individual 3-gram")
print('BLEU-3: %f' % corpus_bleu(ref, pre, weights=(0, 0, 1.0, 0)))
print("Individual 4-gram")
print('BLEU-4: %f' % corpus_bleu(ref, pre, weights=(0, 0, 0, 1.0)))

OUTPUT:
Individual n-gram
Individual 1-gram
BLEU-1: 0.015625
Individual 2-gram
BLEU-2: 0.000000
Individual 3-gram
BLEU-3: 0.000000
Individual 4-gram
BLEU-4: 0.000000

Anyone can help with this I don't know why it doesn't give me good results?

You must be getting a warning with this output. Warning pretty much tells you the reason why the scores are 0. Because there ARE NO 2-gram , 3 -grams in you example which are overlapping.

Here is the detailed explanation, I couldn't explain it better-
https://github.com/nltk/nltk/issues/1838

EDIT-
Solution-

Although the warning tells the reason, here is how you can fix this-

  Notice the ref and pre,

from nltk.translate.bleu_score import corpus_bleu
ref =[[['electron', 'and', 'a', 'proton']]]
pre =[['electron', 'and', 'a', 'proton']]

print("Individual n-gram")
print("Individual 1-gram")
print('BLEU-1: %f' % corpus_bleu(ref, pre, weights=(1.0, 0, 0, 0)))
print("Individual 2-gram")
print('BLEU-2: %f' % corpus_bleu(ref, pre, weights=(0, 1.0, 0, 0)))
print("Individual 3-gram")
print('BLEU-3: %f' % corpus_bleu(ref, pre, weights=(0, 0, 1.0, 0)))
print("Individual 4-gram")
print('BLEU-4: %f' % corpus_bleu(ref, pre, weights=(0, 0, 0, 1.0)))

You can refer help of python-


Unsupervised image segmentation I am trying to implement an algorithm where given an image with several objects on a plane table, desired is the output of segmentation masks for each object. Unlike in CNN's, the objective here is to detect objects in an unfamiliar environment. What are the best approaches to this problem? Also, are there any implementation examples available online?

Edit: I am sorry, question may have been a bit misleading. What I meant by "unfamiliar environment" is that objects may be unknown to the algorithm. The algorithm shouldn't need to understand what the object is, but should only detect the object. How should I approach this problem?

Fast answear
Mean Shift LSH which is an upgrade in $O(n)$ of the famous Mean Shift algorithm in $O(n^2)$ well know for its image segmentation ability
Some explanations
If you desire a true unsupervised approach to segment images, use clustering algorithms. The fact is that there is a lot of algorithms with different time complexity and specificity. Take the most famous one, the $K$-Means, it is in $O(n)$ so pretty fast but you have to specify how many cluster you want which is not what you intend by exploring an unknown image without any information about how many shapes are presents in it. Moreover even if you suppose that you know how many shape are present, we may suppose that there shapes are random which is another point where the $K$-Means fail because it is design to find elliptic clusters and NOT random shape ones.
At the opposite we have the Mean Shift that is able to find automatically the number of cluster -- which is useful when you don't know what you're looking for -- with random shapes.
Of course you replace the $K$ parameter of $K$-Means by others Mean Shift parameters which can be tricky to fine tuned but it doesn't exist a tool that allow you to do magic if you're not exercising to do magic.
An advice to image segmentation clustering
Transform your color space from RGB to LUV which is better for euclidean distance.
$K$-Means vs Mean Shift LSH time complexity

Mean Shift : $O(\alpha.n)$
K-Means : $O(\beta.n)$
$\alpha \gt \beta$

Mean Shift LSH is slower but it fits better with your needs. It stay still linear and is also scalable with the mentioned implementation.
PS : My profile picture is an application of the Mean Shift LSH on myself if it can help to figure out how it works.


Value of features is zero in Decision tree Classifier I used CountVectorizer and TfidfVectorizer seperately to vectorize text which is 100K reviews and passed the vector data to a Decision tree Classifier. Upon using _feature_importances__ attribute of Decision tree Classifier, the feature importance values for all my feature are just 0.0. But with the same dataset, I'm able to find feature importances for naives bayes and logisitic regression by using feature_prob for naive bayes and coef_ attributes for logisitic regression.

Other things I tried:
1. I tried changing ngram_range in countvectorizer
2. I tried limiting/not limiting min_df and max_feature parameters passed in countvectorizers
But couldn't make it work.

Any help is appreciated

Code:

positivereviews = df[df.Score == 1]
negativereviews = df[df.Score == 0]

countvect = CountVectorizer(stop_words='english')

positivebow = countvect.fit(positivereviews.CleanedText[0:100000])

pos_xtrain = positivebow.transform(positivereviews.CleanedText[100000:200000])

pos_y = positivereviews.Score[100000:200000]

clf = DecisionTreeClassifier(max_depth=3, min_samples_split=2)

clf.fit(pos_xtrain, pos_y)

def show_most_informative_features(vectorizer, clf, n=20):
    feature_names = vectorizer.get_feature_names()
    coefs_with_fns = sorted(zip(clf.feature_importances_, feature_names))
    top = coefs_with_fns[:-(n + 1):-1]
    for (coef_1, fn_1)in top:
        print ("\t%.4f\t%-15s" % (coef_1, fn_1))

Counter(clf.feature_importances_)
Counter({0.0: 63514})

1) positivereviews is the dataframe which has features CleanedText and Score
where CleanedText is text which is preprocessed and Score is either 1 in this set since I splitted positive and negative review using Score

2) I also searched online for this problem, but couldn't find any instance of this issue

EDIT1: can it be because since we are dealing with categorical feature in this problem, I might be overfitting badly and hence I do not see any value for any features?

Thanks in advance!!

I'm pretty sure that your feature importances are 0 because your classifier isn't doing any classifying. From the code, it looks like you're training only on positive examples, and giving the fit function a label vector that consists entirely of 1s. The classifier has no information; the decision rule is just "when given an example, predict 1".

There's no way to measure which features are most strongly associated with the label because they're all equally associated - there's only one label, so there's no way to associate the features with anything else.

Is there a reason you're not using the negative examples?  It seems like you have the dataframe available.

When you ran naive bayes and logistic regression, did you also give those models only the positive examples?


What's the difference between data products and intelligent systems? Basically, both are software systems that are based on data and algorithms.

This is a very vague question. However, I will try to make sense of it. Considering rules of logic as well as your statement that both entities are "software systems that are based on data and algorithms", it appears that data products are intelligent systems and intelligent systems are, to some degree, data products. Therefore, it can be argued that the difference between the terms "data products" and "intelligent systems" is purely in the focus (source of information or purpose of system dimensions) of each type of systems (data vs. intelligence/algorithms).


Advice and Ideas appreciated - Machine Learning one man project I have a project where I am supposed to start from scratch and learn how machine Learning works. So far everything is working out better than expected but I feel as I am offered to many ways to choose from.

My Project:
I have data with 700 rows and 108 columns as my features and I get pretty decent results when using a RandomForestClassifier.
By now I was using train_test_split to split my data but I was reading a lot of articles where it was recommended to split data into 3 Sets (train, dev, test).
Since I dont have that much Data I thought of using a Cross-Validation.

My Problem:
So I implemented it, but couldn't really find the difference between a CV and a train_test_split with shuffle.
Before doing so I thought I know the difference betweeen these to model-selection strategies but now I am a bit confused.

My Knowledge And Questions:
1. train_test_split has the problem that the sets can be unbalanced, so if I am unlucky I train my model only with positive or negative examples.
--> can't that be solved by using stratify=True?
2. train_test_split doesn't split the sets the same way all the time so the results aren't comparable
--> setting random_state=0 solves the problem?
3. When does it make sence for me to use CV?
4. How to understand the following picture and what benefit does it provide:

What would be a good way to proceed?

thanks a lot for all the help and time in advance!
Cheers!

train_test_split has the problem that the sets can be unbalanced, so if I am unlucky I train my model only with positive or negative examples.
--> can't that be solved by using stratify=True?
--> yes, that's what stratify=True is for. However you still only train on the data of your training set and test with the data from the test set
train_test_split doesn't split the sets the same way all the time so the results aren't comparable
--> setting random_state=0 solves the problem?
--> it does...keep in mind that random_state= will work fine as long as you keep the seed aka random-state the same

for 3. and 4. let me make sure you understand the difference between CV and TTS. TTS splits your data once, trains on the now "fixed" training set and tests on the "fixed" test set. However this introduces a kind of bias to your evaluation because you are not training or testing on all observations. By going for a CV you make sure that all observations are used for testing and training. This reduces the bias of setting a "fixed" training and test set.

Now for 3.:
When in doubt use CV. HOWEVER...sometimes this doesn't make sense. Think about time series data and how cv would work then and what the result would be. That being said...now think about how TTS with stratify or random would work and what those would mean. CV usually is a good way to go however you always have to think about if the evaluation method fits your problem. TTS has its merit for time series as well as for performance reasons using large datasets.

And 4.:
The picture visualizes TTS and CV. First you have CV which splits your training data so that each small data set is used as test data. The "Testing" set is then used to measure generalization of your model. This is called the holdout method

The diagram that says "Data Permitting" demonstrates TTS with an additional "Testing" set for measuring generalization. You train on "Training", validate the model on "Validation" and as with the holdout method you keep a "Testing" set for measuring how well your model generalizes on unknown data.

Last but not least...5.:
If you get good results and generalization by using TTS...go for it. If you get problems with over-/underfitting or generalization try CV and see if it helps. Of course as with everything in Data Science and Machine Learning there are multiple levels of complexity that you can add on top of TTS and CV as well. For example you could read about K-Fold or LOOCV etc. I don't think that is necessary though.


Co-joining multi-peak histograms I am analysing a bunch of data files which represent responsiveness of cells to addition of a drug. If a drug is not added, cell responds normally, if it is added, it shows abnormal patterns:
, .
We decided to analyse this using an amplitude histogram, in order to distinguish between a change in amplitude and in change of a probability of elliciting the binary response. What we get with file 1 is :

So we fit a pdf on it:

But the issue is: how should we quantify this change: an obvious answer is to do an integral, but as the addition of the drug may increase the 'noise' spectrum (first peak auc) it is not entirely obvious that this is what should be done. Furthermore, other data files do not necessarily contain the same ammount of ion channels (responsive units), so due to biological variability their pdfs may have more or less peaks. How does one summarise the characteristics of multiple peak gaussians should the number of peaks vary? I was thinking about integral over freq ranges, but that will certainly differ between a file with two gaussian peaks, and one with 5: how could one make a summary statistics of an array of gaussian distributions with unknown ammount of peaks?

Another analysis file does literally not contain the triple openings: , . What is a reasonable way of showing the difference between multiple peak gaussians in a condition a and condition b, if the number of peaks is unknwon?

I was thinking of integral, but that looses the potential increase in noise caused by treatment. I was thinking of integral/frequency delta, but I have never seen it published in literature, and I would be really happy if I could support my reasoning for choosing this analysis with some previously published work. Am I missing something really obvious, or is there no simple answer to this? Thanks, J.

One option is Kullback–Leibler divergence which is a measure of how one probability distribution is different from a second probability distribution.

Kullback–Leibler divergence compares the entire distribution (not just peaks) and does not make any assumptions of the functional form of the distribution (which is useful since your data is not very Gaussian).

In fact, Kullback–Leibler divergence can compare discrete distributions so you do not need to fit a Kernel Density Estimation (KDE) curve.


Discovering string "motifs" in python I have millions of strings from different sources that tend to exhibit some common patterns. Is there a way to extract these common motifs?

For example, in a list (of millions) that includes strings

['rs12346','rs1212122',...,'sxs-rs333',...,'kgp222']

..is there a way to extract the following patterns?

'rs' plus one or more digits
'sxs-rs' plus one or more digits
'kgp' plus one or more digits

Some other parameters:

cleaning up beforehand isn't an option
some manual tweaking would be possible (e.g. manually changing a pattern because of outside knowledge)
exceptions (i.e. classifying less than 100% of the strings) can be tolerated
an ideal solution would be using a built-in python library.

Assuming that letters are indicative of "motifs" and numbers are considered as digits and not exact numbers, this is what I would do:

First - transform numbers into a digit placeholder (#)

import re
s = re.sub("\d",s,"#")

Then I would transform a string into a bag-of-bigrams vector in the char level

from sklearn.feature_extraction.text import CountVectorizer
vecs = CountVectorizer(s, analyzer="char_wb", ngram_range=(1,3))

After these 2 steps, we got a sparse vector from any string:

'ab123' --> {" a":1,"ab":1,"b#","##":2,"# ":1}

Next we want to convert those vectors to a pairwise distance matrix and cluster by that distance.

from sklearn.metrics.pairwise import pairwise_distances
from scipy.cluster import  hierarchy
def link_cluster(X, threshold=0.1, metric="cosine", algo="average"):
    X = X.todense()
    Z = hierarchy.linkage(X, algo, metric=metric)
    C = hierarchy.fcluster(Z,threshold, criterion="distance")
    return C

Your end result would be

C = link_cluster(vecs)


Suggest an order to read data science books I'm an undergraduate student that is currently studing data science.
However, since we generally don't delve deeply onto things in class, I want to take some time to review the fundamentals and understand things deeply in order to advance to more complex notions. When I did some research I came across these books :

Data Mining: The Textbook, by Charu C. Aggarwal
The Elements of Statistical Learning, by Jerome H. Friedman, Robert Tibshirani, and Trevor Hastie
Pattern Recognition And Machine Learning, by Christopher M. Bishop
Machine Learning: A Probabilistic Perspective, by Kevin P. Murphy
Computer Age Statistical Inference: Algorithms, Evidence, and Data Science, by Bradley Efron and Trevor Hastie

If you know or came across these books, can you please help me choose the proper order to read these books, or tell me if these is a redundant book, or even suggest a book to add to the list

Any help is appreciated, thanks.

I guess its common to start with
 "An Introduction to Statistical Learning by Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani" and then, if you want a deeper picture, move to
 "The Elements of Statistical Learning, by Jerome H. Friedman, Robert Tibshirani, and Trevor Hastie".

I also really like the video course based on the first book, taught by Hastie and Tibshirani ( if im not mistaken) - a real gem in my mind.
From my perspective its a good way to get both practical and teoretical background for machine learning.
Link to the course published at R-bloggers:
https://www.r-bloggers.com/in-depth-introduction-to-machine-learning-in-15-hours-of-expert-videos/


What is difference between final episodes of training and test in DQN? What is difference between running in final episode of training mode and running in test mode in DQN?

Is there any difference more than after training and tune the hyper-parameters, we test for one episode and without any exploration?
This means that test mode is similar to training mode in episode n+1 without exploring (while we train for n episode) ?Is it correct?

Why in some test code of DQN, they test for multiple episodes?

What is difference between running in final episode of training mode and running in test mode in DQN?

As Q learning is an off-policy method, and DQN is based on Q learning, then the agent effectively has two different policies:

A behaviour policy that explores actions regardless of whether data so far suggests that they are optimal. The simplest and perhaps most commonly implemented behaviour policy is $\epsilon$-greedy with respect to action values $\hat{q}(s,a,\theta)$.
A target policy that is being learned, which is the best guess at an optimal policy so far. In Q learning that policy is fully greedy with respect to action values $\hat{q}(s,a,\theta)$.

When assessing how well the agent has learned the environment after training, you are interested in the target policy. So with Q learning you need to run the environment with the agent choosing actions greedily. That could be achieved by setting $\epsilon = 0$, or with a different action selection routine which doesn't even check $\epsilon$ (which to use is an implementation detail).

  Is there any difference more than after training and tune the hyper-parameters, we test for one episode and without any exploration?

Essentially yes to second part - without exploration. But . . .

  Why in some test code of DQN, they test for multiple episodes?

That is because one episode is often not enough to get reliable statistics.

If the environment is completely deterministic, including having only one start state, then one episode would be sufficient to assess an agent.

However, if the environment is stochastic, or you are interested in multiple possible start states, then multiple episodes are necessary in order to have low mean squared error bounds on the average total return, or any other metrics you might be interested in. Otherwise it would be hard to compare results between different hyperparameters.

How many episodes are enough? There is no fixed number, it depends on how much variance there is in the environment and how accurately you want to know the values. In some cases - e.g. OpenAI gym, there are suggested values. For instance, LunarLander suggests a target score over 100 episodes, so they are suggesting measurements are accurate enough over 100 episodes. However, the advice only applies to that problem. In general, you can measure the variance in the score, like any other statistical measure, and calculate error bounds on your metric.


The affect of bootstrap on Isolation Forest I've been using isolation forest for anomaly detection, and reviewing its parameters at scikit-learn (link). Looking at "bootstrap", I'm not quite clear what using bootstrap would cause. For supervised learning, this should reduce overfitting, but I'm not clear what the effect on anomaly detection should be.
I think it would require the trees to achieve more "consensus" about what the anomaly is, therefore, reducing the effect of any single feature. I.e, an anomalous observation would probably need to be anomalous consistently and over a number of features (?).
Is this a correct interpretation of this parameter?

This is well explained on the original paper Section 3.
As well as in the Supervised Random Forest, Isolation Forest makes use of sampling on both, features and instances, so the latter in this case helps alleviate 2 main problems:

Swamping

Swamping refers to wrongly identifying normal instances as anomalies. When normal instances are too close to anomalies, the number of partitions required to separate anomalies increases – which makes it harder to distinguish anomalies from normal in- stances.

Masking

Masking is the existence of too many anomalies concealing their own presence.

Contrary to existing methods where large sampling size is more desirable, isolation method works best when the sampling size is kept small. Large sampling size reduces iForest’s ability to isolate anomalies as normal instances can interfere with the isolation process and therefore reduces its ability to clearly isolate anomalies. Thus, sub-sampling provides a favourable environment for iForest to work well. Throughout this paper, sub-sampling is conducted by ran- dom selection of instances without replacement.


How do I prepare data in which each output row depends on multiple input rows? My goal is to predict the value of Y based on multiple values of X1 and X2 for each observation of Y.

In my example, I want to predict whether a customer will file for bankruptcy (table 1) based on limits and balances of their credit cards (table 2). The challenge is that customer 1 has two credit cards, whereas customer 2 has one credit card.

How do I map table 2 to table 1 in this scenario?

(Of course I can create summary statistics, but I do not want to bias the model.)

Table 1: Bankruptcy filing

customerId | Customer filed for bankruptcy (Y)
1            1
2            0

Table 2: Customer credit cards

customerId | creditCardId | Credit limit (X1) | Credit used (X2)
1            1              5                   5
1            2              9                   8
2            1              10                  1

A simple, practical approach would be to aggregate your data on each customer. The idea is that the repartition of credit usage / credit limit might not really matter for the overall bankruptcy. You might then want to build new features to avoid loss of information : the number of maxed out credit card, average interest on credit card. This is the general idea :
Table 1: Customer credit usage
customerId | nb_cc  | maxed-out_cc| Overal Credit limit| Overal Credit used | bankruptcy
1            2        1             14                   13                   1
2            1        0             10                   1                    0

The main alternative would be to try to guess on which credit card clients will default. Depending on the law and procedure you might want to build a target at credit card level (when someone goes bankrupt, do you lose the overall credit used or simply the credit used on the cc he failed to reimburse ?). Contrary to bankruptcy this is not a well defined (ie law defined and enforced target). This would be the general idea :
Table 2: Credit cards
cc_id | Limit | Used | nb_other_cc | other_limit | other_used |  bankruptcy (Y1) | cc_default (Y2)
1-1     5       5      1             9             8             1                 1
1-2     9       8      1             5             5             1                 0
2-1     10      1      0             0             0             0                 0

In general I don't think using Y1 here would really be beneficial compared to using it in the first approach. (It might even add some modelling problem as it would overweigth people with multiple credit cards)
Defining and mesuring Y2 might be difficult overall and it might require further modelling to aggregate predictions at a client level (if #1 default on cc #1 what does that mean for cc #2 ?).
Note : as you didn't mention it in your initial question I didn't deal with it, but generally speaking you'll have to deal with a time horizon. Basically you might want to have multiple observations over the life of your products (say each year) and a time horizon for your prediction (say bankruptcy over the next year). Even if that introduce some modelling problems it might help you have better business oriented metrics (how much will we lose next year ?).


how to take CSV file input in list of tuples I have a .txt(data.txt) file containing csv data like:

X         Class
15.0001   Yes
18.00     NO
17.07     Yes

I need to make a function to return a list of tuples of each samples. So far I did:

import csv
def readAllData(str):
    with open(str,'r') as f:
        f.readline()
        data=[tuple(line) for line in csv.reader(f)]
    return (data)

Output:

[('15.001\tYES',),
 ('18.00\tNo',),
 ('17.07\tYes',),]

I want:

[(15.001, Yes), (18.00, No), (17.07, Yes)]

Technically speaking, you do not have a CSV, you have a TSV. So I would take the time to split out each row accordingly and then build your tuple.

In the line:

data=[tuple(line) for line in csv.reader(f)] return (data)

You need to replace csv.reader to handle tsv instead.


Sample Importance (Training Weights) in Keras How do you add more importance to some samples than others (sample weights) in Keras?

I'm not looking for class_weightwhich is a fix for unbalanced datasets.

What I currently have is:

trainingWeights which is the desired importance I want to give to each sample.

epochs = 30
batchSize = 512

# Fit model with selected data
model.fit(trainingMatrix, trainingTargets,
          batch_size=batchSize, epochs=epochs,
          sample_weight=trainingWeights)

However the training error is much lower than before, and according to Keras' documentation:

  sample_weight: Optional Numpy array of weights for the training samples, used for weighting the loss function (during training only).

As I understand it, this option only calculates the loss function differently without training the model with weights (sample importance) so how do I train a Keras model with different importance (weights) for different samples.

PD. This is a similar question xgboost: give more importance to recent samples but I would like an applicable answer to Keras.

I'm not sure I completely understand what you want but it looks like you're trying to find an implementation for sample_weight. Well, I had tried something similar. Before I go into it, I want to mention that my error actually went up - I haven't had the time to go into why that happened.

For simplicity, lets say you know the weights you need for each class and can pass it as a dictionary. In my example I have my y_train as a one hot encoded vector. I'm using that fact to reverse engineer which class each row is pointing to and adding a weight for it. You essentially need to pass an array of weights mapping to each label (so the same length as your training data) when you fit the model.

def generate_sample_weights(training_data, class_weight_dictionary):
    sample_weights = [class_weight_dictionary[np.where(one_hot_row==1)[0][0]] for one_hot_row in training_data]
    return np.asarray(sample_weights)

And this is passed when you call fit on the model.

model.fit(x=X_train,
    y=y_train,
    batch_size = 64,
    validation_data=(X_val, y_val),
    shuffle=True,
    epochs=20,
    sample_weight = generate_sample_weights(y_train, class_weights_dict)
)


What's immediate superset? What's immediate superset?

Why is it called immediate?

They appear in the context of Apriori algorithm. Particularly the one enhanced with hashing.

Immediate often occurs simply to say that there are perhaps many supersets, e.g. when we have a tree like structure. There is also for example, the converse, an immediate subset.

Example

Set A has a superset B, and that superset has its own superset C. In this case, A must also be contained within C, but the set B lies within that realm and so is the immedaite superset of A.

Perhaps it is best visualised with a Venn diagram, where B is the immediate superset of A and equally the immediate subset of C:

There is some discussion here on Cross-Validated too, where the selected answer shows some further usage of the term (albeit in a slightly different context). Have a look at this source for some usage of the term, which leads you to my understanding.


k-means|| in PySpark I'm trying to apply k-means$\|$ clustering in PySpark.

According to this paper, there is an oversampling factor, $l$, that would affect the model's cost.

I couldn't find any parameter regarding oversampling factor in PySpark's k-means function. There is a parameter called initializationSteps but there is no proper definition for that.

Is there a way I could use oversampling factor in k-means function?

Based on the paper you set k but want to sample greater than k and apply the log calculation.

I would recommend using the foreach function. Where you could say foreach() apply further reach than k.

where the following aglo from the paper could be interpreted as follows.

Algorithm 2 k-means||(k, ) initialization.

1: C ← sample a point uniformly at random from X
2: ψ ← φX(C)
3: for O(log ψ) times do
4: C 0 ← sample each point x ∈ X independently with probability px =·d2(x,C) φX(C)
5: C ← C ∪ C0
6: end for
7: For x ∈ C, set wx to be the number of points in X closer to x than any other point inC
8: Recluster the weighted points in C into k clusters

instructions:

get points of k = 2
points [(1,1),(1,2),(2,2)]

centroid is placed in middle for this example and

c1 = [(1,1),(1,2)]

This was achieved with the over sample of Euclid foreach() point that satisfies the over sampling requirement.

see example here which uses for each:

# Cluster the data into two classes using PowerIterationClustering
model = PowerIterationClustering.train(similarities, 2, 10)

model.assignments().foreach(lambda x: print(str(x.id) + " -> " + str(x.cluster)))

So you will need to write the distance into the lambda.(if you provide code easier to help you).

K distance = k distance on perimeter which is the bottom red line in diagram

 .foreach(lambda x: kdistance[get average] + then check prob(k prime) of k)

share some code and sample data please


kMean clustering for recommendation I have a file with 50000 rows from a library platform. Each individual row saves a user, and shows the order in which the user, has selected. The books could be from various categories (e.g. roman, history, etc..). There are a total of 10 categories. The categories that user has selected could be for example: 334664.
This means this user has selected a book from categories 3, 4 and 6. How can I use this data to build a recommendation system using the k-means cluster algorithm. If anybody can help me how I can go through the whole process step by step.

Since your data is sequential, you could try with sequential models (LSTM, RNN, GRU) etc.
With which you can you predict what the user will select after the set of books as recommendation. In this way the input sequence length can be anything. (like, 3345, 33456, 334567 etc).
But to answer your question with KMeans.
I assume, all the rows are in same length.

Cluster all data with KMeans
Optimize number of K with Elbow method.
For each cluster take UNIQUE books.
Foe each User,
a. Predict Cluster for the User
b. From the Cluster's Unique books, Find books which are not the user already read. (Cluster books - User books).
c. You can suggest these books to the User.

This is just one the starting method to do it. Depends upon the business you can find many alternatives.
One such I can suggest is create similarity embeddings for each books, and suggest most similar (with distance measures) book to the user.
Hope this helps.


Finding a kernel with feature transformation Suppose we have feature transformation $\Phi(x) = [1, x_1, x_2, x_1x_2]$. Now we want to find the kernel corresponding to $\Phi$.

What I have done is using kernel decomposition, we have:
$$
K(x, y) = \Phi(x) .\Phi(y)\\
K(x, y) = 1.1 + x_1y_1 + x_2y_2 + x_1x_2y_1y_2 \\
K(x, y) = 1 + \sum_{i=1}^{N}x_iy_i + \prod_{i=1}^{N}x_iy_i
$$

$x, y$ are in $\Bbb{R}^2$

Is that kernel valid?

Yes, it is valid.

Kernel is just an inner product in the feature space. In fact, here you have explicitly started from the feature transformation and we understand the feature transformation.


Logistic Regression : Solving the cross-entropy cost function analytically Logistic regression cost function is cross-entropy. It is defined as below:

This is a convex function. To reach the minimum, scikit-learn provides multiple types of solvers such  as : ‘liblinear’ library, ‘newton-cg’, ‘sag’ and ‘lbfgs’.

Is it possible to analytically find the minimum? if yes, what can we say of the computing complexity?

If you use just one neuron (linear->sigmoid), you can find the minimum with too much small error or maybe reach to the minimum using approaches like gradient descent or other optimization algorithms. The reason for finding the minimum is that cross entropy is a non-convex shape, but if you use sigmoid function as the activation function of logistic regression, the cross entropy cost function becomes convex and it is easy to find the only global minimum.