Pros and Cons of Python and R for Data Science So, let's start out by saying I am NOT asking which is better.

I like both of these languages for data science and I think this is a matter of and rather than or since there is no need to choose one of them. My general opinion is that R has more data science depth while Python has more breadth in terms of diversity of application.

I will start this out with a few of my own scattered opinions.

Python

I tend to use Python as the overarching language (sometimes using it to execute scripts in R) since:

Installed standard on Macs and many *nix setups
Easy to install/set up in environments with security restrictions
Has a great suite of tools whether I want to:
Set up a website
Interact with the Apache suite of tools
Build out pipelines
Build and score predictive models
Develop an application / front-end
Has a larger audience if I get stuck and need to ask questions on here
Debug / Unit test
Auto Document A good example of this was in building a Robot some facial recognition stuff over the summer. Raspbery Pi comes with Python by default, the sensors had Python API's, and a few of the web API's I bumped up against had Python API's available. This would have been a difficult application to take on using R.

R

I like to use R for early data explanation since data table tends to load larger sets into memory more quickly than Pandas and the general environment seems more mature for exploratory stuff. If I need mixed models or something similar that isn't Machine Learning "bread and butter" R is more likely to have what I need.  sklearn is pretty good but doesn't feel quite as polished as what I have in R.

On the data viz side, ggplot2 is obviously better than matplotlib (the html based stuff is tough to share in an environment with heavy IP and security restrictions so I tend to lean towards the simple stuff.)

How do you use them?

How do you interact with these languages? Do you mix them together? Specialize in one? What's your rationale? Strengths/weaknesses?

Interaction - Random Facts

Both are good stable languages with interesting complementary qualities. You can get much better packages in one and then stitch them with some data from the other. An example is using time series forecasting and decision trees in R and doing data munging in Python.

Both languages borrow from each other. Even seasoned package developers like Hadley Wickham (Rstudio) borrows from Beautiful Soup (python) to make rvest for web scraping. In addition to that, Yhat borrows from sqldf to make pandasql and many other.

Rather than  reinvent the wheel in the other language developers can focus on innovation because, in the end, the customer does not care which language the code was written, the customer cares for insights.

Mixing Them Up
AM mentioning few approaches to mix them together-

Use a Python package rpy2 to use R within Python . [Demo]

Use Python from within R using the rPython package. [Demo]

Use Jupyter with the IR Kernel. Python and R and makes the interactivity of iPython available to other languages.

Use Beaker Notebook. It allows you to switch from one language in one code block to another language in another code block in a streamlined way to pass shared objects.

Python vs R
Python vs R - This section will answer:

Which will be better?
How to choose one over other?
Specialization

See as I said earlier both are stable and you can choose any or work with both. But when it comes to master one I'll suggest keep these 3-4 guidelines in mind-
Personal Preference
Choose the language to begin with based on your personal preference, on which comes more naturally to you, which is easier to grasp from the get-go. To give you a sense of what to expect, mathematicians and statisticians tend to prefer R, whereas computer scientists and software engineers tend to favor Python.
Project selection
You can also make the Python vs. R call based on a project you know you’ll be working on in your data studies. If you’re working with data that’s been gathered and cleaned for you, and your main focus is the analysis of that data, go with R. If you have to work with dirty or jumbled data, or to scrape data from websites, files, or other data sources, you should start learning, or advancing your studies in, Python.
Collaboration
Once you have the basics of data analysis under your belt, another criterion for evaluating which language to further your skills in is what language your teammates are using. If you’re all literally speaking the same language, it’ll make collaboration—as well as learning from each other—much easier.
Job market
Jobs calling for skill in Python compared to R have increased similarly over the last few years.

Note: Have a look at this infographic by DataCamp. For a better view on it.

My Rationale
In my case am doing both and using them interactively and Customizing them as per my use. You can get something really interesting in one (as I mentioned above) which will be hardly available in other, so it's better to use both together. This is the best way to bridge the gap between these two.
But in the last, it's your call keep the guidelines, your interest, and scenarios in mind and make a clear view on that.
Strength & Weaknesses
R
Strength

R is great for prototyping and for statistical analysis.
It has a huge set of libraries available for different statistical type analysis. Check The Comprehensive R Archive.
RStudio IDE is a definitely a big plus. It eases most of the tedious tasks and fastens your workflow.

Weaknesses

The syntax could be obscure sometimes.
It is harder to integrate to a production workflow.
In my opinion, it is better suited for consultancy-type tasks.
The libraries documentation isn't always user friendly.

Python
Strength

Python is great for scripting and automating your different data mining pipelines. It is the de facto scripting language nowadays.
It also integrates easily in a production workflow. Besides, it can be used across different parts of your software engineering team
(back-end, cloud architecture etc.).
The scikit-learn library is awesome for machine-learning tasks.
Ipython (and its notebook) is also a powerful tool for exploratory analysis and presentations.

Weaknesses

It isn't as thorough for statistical analysis as R, but it has come a long way these recent years
In my opinion, the learning curve is steeper than R, since you can do much more with Python.

The Conclusion
Use R and Python. Learn how they inter-operate together. Start with one and then add the other to your workflow. As I like to remind myself- "choosing the tools should never be the primary problem". When in doubt, use the one that is available and that gets the work done quickly.
Hope it helps!
Ref- Udacity, Quora, Letustweak, kD, DataCamp


Clustering Multiple Networks I'm looking for methods of Community Detection in networks. For example if I have a network of 100 people (each node is a person), how can I cluster nodes? What would be the best approach for grouping these People? I know this question is rather open, but I'm just looking for a nudge in the right direction.

Your question is not clear in a way there are two different Graph Clustering problems. One is having a dataset of different graphs and you would like to cluster similar graphs (in this case each object is a graph), and the other when you have a graph (e.g. a social network) and you would like to group similar nodes inside that graph (here each object is a node).

The first problem needs Graph Embedding which is to transfer graphs into a n-dimensional manifold and from there on, you have a classical clustering problem. For this case have a look at what Horst Bunke has done.

A more simplified approach would be feature extraction from networks and apply classical clustering methods. These features are statistical and topological network measures such as Clustering Coefficient, Assortativity Index, Average Shortest Path Length, Density, Diameter, Average Centrality, etc. This approach is more about the characteristics of real-world networks e.g. biological networks or social networks as it takes into account those characteristics which determine the real-world phenomena.

The second problem is usually called Community Detection in literature. There are several method for doing that based on Modularity Score, Information Theory, Topological Structure or Spectral Graph Theory.


How to use TimeDistributed fo CNN+LSTM? I am trying to classify 6 classes time-frequency domain signal (STFT spectrogram) with a size of 3601x217 pixels. Assume that for each classes have 70 training samples, 20 validation samples, and 10 test samples. Each sample is a PNG image of size 3601x217x1 (grayscale), while in the future I am thinking to expand this into 3601x217x4 (3 color channels + alpha). But right now, I am focusing on grayscale first.

After trying pure LSTM, I found out that the model overfits quickly (training accuracy > 90%, but val_acc stucks at 20%). ow I'd like to try out Time Distributed CNN+LSTM.

Following is the code:

img_width, img_height = 3601,217
train_data_dir = 'sensor1/training'
validation_data_dir = 'sensor1/validation'

num_classes = 6
nb_train_samples = 70
nb_validation_samples = 20
epochs = 50
batch_size = 5

if K.image_data_format() == 'channels_first':
    input_shape = (3, img_width, img_height)
else:
    input_shape = (img_width, img_height, 3)

model = Sequential()
model.add(TimeDistributed(Conv2D(16, (3,3), padding='same', strides=(2,2), activation='relu', input_shape = input_shape)))
model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2))))
model.add(Dropout(0.5))
model.add(TimeDistributed(Conv2D(32, (3,3), padding='same', strides=(2,2), activation='relu')))
model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2))))
model.add(Dropout(0.5))
model.add(TimeDistributed(Conv2D(64, (3,3), padding='same', strides=(2,2), activation='relu')))
model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2))))
model.add(Dropout(0.5))
model.add(TimeDistributed(Flatten()))
model.add(LSTM(units=128, return_sequences=False))
model.add(LSTM(units=64, return_sequences=False))
model.add(Dense(32))
model.add(Dense(num_classes, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

train_datagen = ImageDataGenerator(rescale = 1. / 255)
test_datagen = ImageDataGenerator(rescale = 1. / 255)

train_generator = train_datagen.flow_from_directory(
    train_data_dir, target_size=(img_width, img_height),
    batch_size=batch_size, color_mode='grayscale')

validation_generator = test_datagen.flow_from_directory(
    validation_data_dir, target_size=(img_width, img_height),
    batch_size=batch_size, color_mode='grayscale')

model.fit_generator(
    train_generator, steps_per_epoch=nb_train_samples // batch_size,
    epochs=epochs, callbacks=[plot_losses],
    validation_data=validation_generator,
    validation_steps=nb_validation_samples // batch_size)

The code gives following error:

  File
  "C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\training.py",
  line 496, in _make_train_function
      raise RuntimeError('You must compile your model before using it.')

  RuntimeError: You must compile your model before using it.

What's wrong with the code? I actually compiled the model before calling model.fit_generator:

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

You are specifying the input_shape inside the Conv2D layer, meaning:
model.add(TimeDistributed(Conv2D(16, (3,3), padding='same', strides=(2,2), activation='relu', input_shape = input_shape)))

Instead you should write:
model.add(TimeDistributed(Conv2D(16, (3,3), padding='same', strides=(2,2), activation='relu'), input_shape = input_shape))

or more clearly:
model.add(TimeDistributed(Conv2D(...), input_shape=input_shape))

Furthermore I should note that you are using a TimeDistributed with a Conv2D. This implies that your input_shape should be like this (timesteps, dim1_size, dim2_size, n_channels).

You are using: input_shape=(img_width, img_height, 3)

If you want to take the img_width as timesteps you should use TimeDistributed with Conv1D.

To summarize, always consider that a TimeDistibuted layer adds an extra dimension to the input_shape of its argument-layer.

Lastly, your first LSTM layer with return_sequences=False will raise an error. You must give it a True value.


Debugging Neural Networks I've built an artificial neural network in python using the scipy.optimize.minimize (Conjugate gradient) optimization function.

I've implemented gradient checking, double checked everything etc and I'm pretty certain it's working correctly.

I've run it a few times and it reaches 'Optimization terminated successfully' however when I increase the number of hidden layers, the cost of the hypothesis increases (everything else is kept the same) after it has successfully terminated.

Intuitively it feels as if the cost should decrease when the number of hidden layers is increased, as it is able to generate a more complex hypothesis which can fit the data better, however this appears not to be the case.

I'd be interested to understand what's going on here, or if I've implemented neural net incorrectly?

There are so many ways to go wrong with a neural net that it's going to be difficult to debug. Also, to address your intuition, each additional hidden layer makes learning much harder. With that said, here are some possibilities:

You have added weight decay. Adding more layers adds more weights which increases your regularization cost.
The problem is simple enough that a model with a single hidden layer is sufficient. Adding more hidden layers makes it harder for the network to learn (harder optimization problem).
The optimization method is not doing a great job (I prefer climin to scipy.optimize).
You are using the sigmoid/tanh activation function. The sigmoid function causes the vanishing gradient problem which makes learning hard with more layers. Try using the ReLu function.

Training neural nets takes a lot of practice, luck, and patience. Good luck.


Data Science Groups on Linked In I am interested in joining some Data Science Groups on Linked In.  The relevant list is long.  I don't think it is to my advantage to join all, but I am not sure how to evaluate the individual group's potential to elevate my career.

Is there some clear criteria for which I can evaluate potential groups?

You are limited to
50 groups.

KDNuggets used number of members, growth rate, number of conversations and member engagment in their
analysis of data science groups.


Calculating the Standard Deviation by category using Python I have a datset with Scores and Categories and I would like to calculate the Standard Deviation of these scores, per category. The data look something like this:

Category    Score
AAAA        1
AAAA        3
AAAA        1
BBBB        1
BBBB        100
BBBB        159
CCCC        -10
CCCC        9

What I would then like is the Standard Deviation of each Category. I know that with numpy I can use the following:

numpy.std(a)

But the example I can find only have this relating to a list and not a range of different categories in a DataFame.

You can easily do this using pandas:

import pandas as pd
import numpy as np

df = pd.DataFrame([["AA", 1], ["AA", 3], ["BB", 3], ["CC", 5], ["BB", 2], ["AA", -1]])
df.columns = ["Category", "Score"]
print df.groupby("Category").apply(np.std)


which NN should I use for Time-series dataset, whose pattern change as time goes I am analyzing a time-series dataset using (supervised) tensorflow deep learning. The tensorflow code is given a series of inputs, and based on each input, the NN has to predict output value in near future.

For training there are lots of pre-labeled input/output of past. Each input is an array of state per time step, and each input is labeled with a result value. So far this kind of task is common at deep learning, and I used multiple GRU layers to solve this.

But problem is, I found that the pattern for finding appropriate output actually changes as time goes. For every 10 hours (approximately), the pattern changes so at every start of each pattern period (like first 1-2hours of total 10 hours) the NN is bad at predicting, but after that, prediction rate enhances. Plus, I assume the data also has some noise too.

So far my implementation with GRUs do its job, but I want to find better way to build NN if possible. I currently know some basic supervised learning techniques and some advanced like LSTM, and for unsupervised learning I know DQN and policy-gradient.

Do you know when does the new pattern start? You could reset the hidden state of the RNN each time this happens during training. A more detailed explanation in this paper: https://arxiv.org/pdf/1706.04148.pdf

In that paper, they try to predict the next item in a user's session. They use a GRU to create a user-level representation (which in your case could be the pattern representation) and another GRU to predict the next item based on the current session information.

To do that, they reset the hidden state of the second GRU every time the session changes and reset the hidden state of the first GRU every time the user changes, and I think you could reset the hidden state every time the pattern changes. It is difficult to explain it all right here, I recommend you to read the paper I mentioned before.

EDIT:

To know where are the pattern change points you could train another network, maybe using as labels points where your GRU's accuracy decreases substantially, and then use this network's predictions to reset the hidden state on these points at training time as I mentioned before.


Neural Network returning training data ratio instead of probability I am trying to create a neural network from scratch using numpy.
I have created a network that can classify iris data base to a high degree of accuracy.

I am facing the problem that sometimes instead of predicting a class given input, the network just trains to return the probability distribution of the training sample irrespective of the input.

So, after training the network on 150 samples (50 sample from each class), then on testing on a sample I would get (0.333 0.333 0.333) for each of the three classes, where the expected output was supposed to be, say, (1,0,0).

I had solved that problem by tweaking the Hyperparameters, but I am facing a similar problem with wine data set, for which I cannot do the same.

Has anybody faced this problem before? How did you solve it?

In case of any neural network if you are trying to solve the classification problem then it should generate probabilities.

If you are trying to solve 2 class classification problem then one can opt for sigmoid or softmax. In the case of sigmoid, your neural network only contains 1 output neuron which generates value probability and if probability greater than equal to 50 then belongs to one class and less than 50 then belongs to another class. e.g.(0.333 0.333 0.333) = (0.80) that means belongs class 1 and (0.666 0.333 0.999) = (0.40) that means belongs class 0
In the case of softmax, the neural network should have an equal number of output neurons to the number of categories. e.g.(0.333 0.333 0.333) = (0.80, 0.12, 0.08) that means belongs class 1 and e.g.(0.333 0.333 0.333) = (0.08, 0.20, 0.72) that means belongs class 3


How do "intent recognisers" work? Amazon's Alexa, Nuance's Mix and Facebook's Wit.ai all use a similar system to specify how to convert a text command into an intent - i.e. something a computer would understand. I'm not sure what the "official" name for this is but I call it "intent recognition". Basically a way to go from "please set my lights to 50% brightness" to lights.setBrightness(0.50).

The way they are specified is by having the developer provide a list of "sample utterances" which are associated with an intent, and optionally tagged with locations of "entities" (basically parameters). Here's an example from Wit.ai:

My question is: how do these systems work? Since they are all very similar I assume there is some seminal work that they all use. Does anyone know what it is?

Interestingly Houndify uses a different system that is more like regexes: ["please"] . ("activate" | "enable" | "switch on" | "turn on") . [("the" | "my")] . ("lights" | "lighting") . ["please"]. I assume that is integrated into the beam search of their voice recognition system, whereas Alexa, Wit.ai and Mix seem to have separate Speech->Text and Text->Intent systems.

Edit: I found a starting point - A Mechanism for Human - Robot Interaction through Informal Voice Commands. It uses something called Latent Semantic Analysis to compare utterances. I'm going to read up on that. At least it has given me a starting point in the citation network.

Edit 2: LSA is essentially comparing the words used (Bag of Words) in each paragraph of text. I don't see how it can work very well for this case as it totally loses the word order. Although maybe word order doesn't matter much for these kinds of commands.

Edit 3: Hidden Topic Markov Models look like they might be interesting.

Although not directly answering your question, you may be interested in the field of automated question answering. To answer natural language text questions they must first be understood, which overlaps with your problem.

A good resource is the course by Jurafsky and Manning. Particularly the sections on semantics and question answering may help with what you are looking for. There are accompanying lecture videos available on youtube here.


My small script on value alteration in columns of a data not working I have a data set which has "Speed" as one of the columns (features). The column contains both zero and non-zero values. I want to randomly set 10% of the non-zero values to zeros. This will change the corresponding "class" label to zeros. I mean any value  set to zero, its corresponding class value will be zero as well. I have done this but it is give me errors. Though due to error, I cannot tell it will give me the update/result I want.

file_path = 'Processed_data/data1.csv'
df = pd.read_csv(file_path)
per_change = 0.1
attr = 'Speed'
target = 'Class'
df_spd = df[df['Speed'] > 0.]

num_rows_to_change = int(df.shape[0] * per_change)
num_with_zero_initial = df[df[attr] == 0].shape[0]
assert df_spd.shape[0] > num_rows_to_change, \
    'Number of rows with non-zero speed is less than 10% of the original   dataset.'
df_update = df_spd.sample(num_rows_to_change)
df_update[attr] = 0.
df_update[target] = 0.
df.update(df_update)
update_list = df_update.index.tolist()
num_with_zero_final = df[df['Speed'] == 0].shape[0]
assert num_with_zero_final == num_with_zero_initial + num_rows_to_change, \
    'Number of rows needed to change not equal to number of rows changed.'
df.to_csv('changed.csv')


Help regarding NER in NLTK I have been working in NLTK for a while using Python. The problem I am facing is that their is no help available on training NER in NLTK with my custom data. They have used MaxEnt and trained it on ACE corpus. I have searched on the web a lot but I could not find any way that can be used to train NLTK's NER.

If anyone can provide me with any link/article/blog etc which can direct me to Training Datasets Format used in training NLTK's NER so I can prepare my Datasets on that particular format. And if I am directed to any link/article/blog etc which can help me TRAIN NLTK's NER for my own data.

This is a question widely searched and least answered. Might be helpful for someone in the future whose working with NER.

Training a model, related to information extraction, in general, and named entity recognition/resolution (NER), in particular, is described in detail in Chapter 7 of the NLTK Book, available online at this URL: http://www.nltk.org/book/ch07.html.

Additionally, I think that you might find useful my related answer on Cross Validated site. It has a lot of references to relevant sources on NER and related topics as well as to various related software tools.


Is there an appropriate use of adjusting class weights for a balanced dataset? I ask this because I am currently working with a CNN model built for diagnosis of pneumonia. Originally, I followed a notebook on kaggle to build the model and thereby learn what each bit of code is for, etc.
The dataset used was rather imbalanced, with a far greater number of pneumonia cases than normal (healthy) ones. Hence the model.fit class_weight parameter was set to {0:6.0, 1:0.5}.
(0 being normal, 1 being pneumonia)
Since then, whilst working on the model and making adjustments, I acquired a number of new data to add to the model such that now the dataset is fairly balanced. In fact, I ensure that the data is loaded into the model so that it is exactly balanced, the dataframes used are coded to ensure an equal number of pneumonia and normal cases in the training testing and validation dataframes.
So, accordingly, I am now trying to remove the use of the class_weights parameter as (as far as I understand it) it is not necessary and may impart some bias in the results. However, in doing so, the model no longer seems to improve in accuracy. It essentially stalls on 0.5 indefinitely. Whereas, with the weights applied, I achieve 0.90+ accuracy.
Simply put, is there some reason for this? The code is quite long, but I'm happy to post it if it is deemed required, but I feel like this may be due to my lack of understanding than error in code (as it has otherwise been working fine and as expected). Thanks in advance.
EDIT: For the sake of clarity and understanding, I performed a grid search over possible values for applied weight values. It confirmed an appropriate choice as being 0:~4.0, 1:0.4, but also suggests 0:1, 1:5.0.
EDIT 2: For further clarity, a link to a github containing the model code and output files etc. https://github.com/GeeKandaa/ML-Code

Class Weight can be important even for balanced data if, for example, some class is more significant than others, so loss wrt this class should count more.
One can even think of class weights as unique extra hyper-parameters with their own effect on the outcome (either positive or negative) and treat them as such without interpretation.
Related: How does class_weight work in Decision Tree?


Imputing missing value based on filtering result of another column C1
C2

A
x

A
y

A
z

A
x

A
NaN

A
x

A
x

A
x

B
y

B
y

B
z

B
y

B
NaN

B
y

B
x

B
x

I have to impute missing values in C2 , the imputation should be such that if the missing values corresponding C1 column is A, then filter the datset by A and find the mode of C2 and replace the missing value, similarly for B.
In the above example it should be x for first missing value and y for second missing value

C1
C2

A
x

A
y

A
z

A
x

A
x

A
x

A
x

A
x

B
y

B
y

B
z

B
y

B
y

B
y

B
x

B
x

Tried this, not sure if this is the best way
for i in df['c1'].unique():
  df[(df['c1']==i) & (df['c2'].isnull())]['c2'] = df[df['c1']==i]['c2'].mode()[0]

You can use pandas groupby and transform methods. This maps the groupwise mode to the index of the original dataframe.
df['C2']=df.groupby('C1').transform(lambda x: x.fillna(x.value_counts().index[0]))['C2']
df

The output is:


Can I fine-tune the BERT on a dissimilar/unrelated task? In the original BERT paper, section 3 (arXiv:1810.04805) it is mentioned:
"During pre-training, the model is trained on unlabeled data over different pre-training tasks."
I am not sure if I correctly understood the meaning of the word "different" here. different means a different dataset or a different prediction task?
For example if we pre-train the BERT on a "sentence-classification-task" with a big dataset. Then, should I fine-tune it again on the same "sentence-classification-task" task on a smaller and task-specific data-set or I can use the trained model for some other tasks such as "sentence-tagging"?

The sentence "During pre-training, the model is trained on unlabeled data over different pre-training tasks." means that BERT was pre-trained on normal textual data on two tasks: masked language model (MLM) and next sentence prediction (NSP).  There were no other classification/tagging labels present in the data, as the MLM predicts the text itself and the NSP label is derived from the textual data itself. Both tasks were trained simultaneously from a single textual dataset that was prepared to feed the input text and the expected outputs for both tasks.
Therefore "different" here refers to the two pre-training tasks I mentioned: MLM and NSP.
When fine-tuning, you do not need to train again on the same sentence classification task, you just simply train it on the task you need. It is perfectly fine to fine-tune BERT on a sentence tagging task on your own dataset.


How to predict constant failing of equipment I am trying to predict the failing of an equipment that heat up the liquid in a pipeline using a heat exchanger. The heat exchanger gets build up inside the pipe and thus needs to be flushed every once in a while. There are sensors around the device that are collecting data like temperature, flowrate, pressure, etc every hour. The flushing events happen once or twice a year. The recorded dates for the events are not very specific, only to the month. There is no particular measure that the operators monitor to get an idea of the health of the heat exchanger. The flushing was done neither due to the actual status or scheduled. But the efficiency should be highest after every flushing. I thought about anomaly detection, but that applies to equipment running normal most of the time with anomaly occurs seldom. This failing is constant process and gradual. If there is a pattern the data it should be a gradualy decreasing curve.

Another idea I had was to predict the remaining useful life method. Basically rank the time periods leading to flushing event by how close in time, with closest being the most severe. So it is basically a classification problem. But the thing is the flushing date is not exact which only has the month and year. There is lot of missing values for some of the key measures which might be indicators for on and off time. Plus this is not the same problem as failure prediction because it is not failure. Even if the flushing event doesn't happen, the equipment still works. And before every time the equipment is flushed, the condition of the equipment may vary.

What is the best way to kind of quantify the deteriorate rate of the equipment?

You should try to frame as a prediction problem. Clearly define the target and then map that target to a technique:

Mean time before failure (Survival analysis)
Failure rates per million hours (Estimate Poisson distribution)
Failure within a time window (Binary classification)

Generally, Bayesian methods are used because the sparse available data can be used as a prior.


Should a model be re-trained if new observations are available? So, I have not been able to find any literature on this subject but it seems like something worth giving a thought:

What are the best practices in model training and optimization if new observations are available?
Is there any way to determine the period/frequency of re-training a model before the predictions begin to degrade?
Is it over-fitting if the parameters are re-optimised for the aggregated data?

Note that the learning may not necessarily be online. One may wish to upgrade an existing model after observing significant variance in more recent predictions.

When new observations are available, there are three ways to retrain your model:

Online: each time a new observation is available, you use this single data point to further train your model (e.g. load your current model and further train it by doing backpropagation with that single observation). With this method, your model learns in a sequential manner and sort of adapts locally to your data in that it will be more influenced by the recent observations than by older observations. This might be useful in situations where your model needs to dynamically adapt to new patterns in data. It is also useful when you are dealing with extremely large data sets for which training on all of it at once is impossible.
Offline: you add the new observations to your already existing data set and entirely retrain your model on this new, bigger data set. This generally leads to a better global approximation of the target function and is very popular if you have a fixed data set, or if you don't have new observations to often. However it is unpractical for large data sets.
Batch/mini batch: this is sort of a middle ground approach. With batch, you wait until you have a batch of $n$ new observations and then train your already existing model on this whole batch. It is not offline as you are not adding this batch to your preexisting data set and then retraining your model on it and it is not online as your are training your model on $n$ observations at once and not just a single one. So it's a bit of both :) Mini batch is exactly the same except that the batch size is smaller so it tends towards online learning. Actually online learning is just batch with batch size 1 and offline is batch with batch size the size of the whole data set.

Most models today will use batch/mini batch and the choice for the size of the batch depends on your application and model. Choosing the right size batch is equivalent to choosing the right frequency with which to re-train your model. If your new observation have a low variance with your existing data, I'd suggest larger batches (256-512 maybe) and if on the contrary new observations tend to vary greatly with your existing data, use small batches (8-256). At the end of the day, batch size is kind of like another hyper-parameter which you need to tune and which is specific to your data


Binning long-tailed / pareto data before clustering I want to cluster a set of long-tailed / pareto-like data into several bins (actually the bin number is not determined yet).

Which algorithm or model would anyone recommend?

There are several approaches. You can start from the second one.

 Equal-width (distance) partitioning:

It divides the range into N intervals of equal size: uniform grid
if A and B are the lowest and highest values of the attribute, the width of intervals will be: W = (B-A)/N.
The most straightforward
- Outliers may dominate presentation
- Skewed data is not handled well.

 Equal-depth (frequency) partitioning:

It divides the range into
N intervals, each containing approximately
same number of samples
Good data scaling
Managing categorical attributes can be tricky.

Other Methods

Rank: The rank of a number is its size relative to other values of a numerical variable. First, we sort the list of values, then we assign the position of a value as its rank. Same values receive the same rank but the presence of duplicate values affects the ranks of subsequent values (e.g., 1,2,3,3,5). Rank is a solid binning method with one major drawback, values can have different ranks in different lists.
Quantiles (median, quartiles, percentiles, ...): Quantiles are also very useful binning methods but like Rank, one value can have different quantile if the list of values changes.
Math functions: For example, logarithmic binning is an effective method for the numerical variables with highly skewed distribution (e.g., income).

Entropy-based Binning

Entropy based method uses a split approach. The entropy (or the information content) is calculated based on the class label. Intuitively, it finds the best split so that the bins are as pure as possible that is the majority of the values in a bin correspond to have the same class label. Formally, it is characterized by finding the split with the maximal information gain.


Which type of machine learning to use We are working with a complex application i.e. a physical measurement in a lab, that has approximately 230 different input parameters, many of which are ranges or multiple-value.

The application produces a single output, which is then verified in an external (physical) process. At the end of the process the individual tests are marked as "success" or "fail". That is, despite the many input parameters, the output is assessed in a boolean manner.

When tests fail, the parameters are 'loosened' slightly and re-tested.

We have about 20,000 entries in our database, with both "success" and "fail", and we are considering a machine learning application to help in two areas:

1) Initial selection of optimum parameters
2) Suggestions for how to tune the parameters after a "fail"

Many of the input parameters are strongly related to each other.

I studied computer science in the mid-90s, when the focus was mostly expert systems and neural networks. We also have access to some free CPU hours of Microsoft Azure Machine Learning.

What type of machine learning would fit these use-cases?

With using R, You could look at trees / randomforests. Since you have correlated variables, you could look into Projection pursuit classification trees (R package pptree). And there soon will be a ppforest package. But this is still under  development.  You could also combine randomforest with the package forestFloor to see the curvature of the randomforest and work from there.


multi items forecasting: issue with storing results disclaimer: I am not 100% sure that this is the appropriate place to ask this question.
Here is a little bit of context about the problem.
I have a dataset containing about 1000 products timeseries (about two years of data)
From this data I am forecasting 12 months ahead, with several prediction intervals.
To predict, I made a basic model with statsmodels.tsa.statespace.exponential_smoothing.ExponentialSmoothing model.
I am facing an issue regarding how to store the data once the model produces results at the item level.
Here is a basic reproduction of what my script look like:

seasonal_profile_df.set_index('Id')
forecast_df = pd.DataFrame(seasonal_profile_df.index)

def winter_holts(i):
    fit1 = ExponentialSmoothing(new_df.iloc[:,i], trend=True, seasonal=12).fit()
    prediction_interval = fit1.get_forecast(steps=12).summary_frame(alpha=[0.10, 0.05, 0.01])
    forecast = pd.DataFrame(prediction_interval)
    return forecast

def holts(i):
    fit1 = ExponentialSmoothing(new_df.iloc[:,i], trend=True).fit()
    prediction_interval = fit1.get_forecast(steps=12).summary_frame(alpha=0.10)

    forecast = pd.DataFrame(prediction_interval)
    print("FORECAST",forecast)

    return forecast

for i in seasonal_profile_df.index:
    if seasonal_profile_df['trend'].loc[i] == "trending":
        holts(i)
    else:
        if seasonal_profile_df['seasonality'].loc[i] == "seasonal":
            winter_holts(i)

for each item, the forecasting function returns a dataframe that looks like that:
FORECAST 100221           mean    mean_se  mean_ci_lower  mean_ci_upper
2020-07-31  -4.412599  24.526896     -44.755753      35.930555
2020-08-31  -5.848380  24.526896     -46.191534      34.494775
2020-09-30  -7.284160  24.526898     -47.627317      33.058996
2020-10-31  -8.719941  24.526900     -49.063101      31.623220
2020-11-30 -10.155721  24.526903     -50.498887      30.187445
2020-12-31 -11.591502  24.526908     -51.934676      28.751672
2021-01-31 -13.027282  24.526915     -53.370468      27.315903
2021-02-28 -14.463063  24.526924     -54.806263      25.880137
2021-03-31 -15.898844  24.526935     -56.242062      24.444375
2021-04-30 -17.334624  24.526949     -57.677865      23.008617
2021-05-31 -18.770405  24.526966     -59.113674      21.572864
2021-06-30 -20.206185  24.526986     -60.549487      20.137117

the computations results are truly an issue right now because they are the basis for deeper analysis and they will end up in a posgresql DB.
I am inexperience to this and I am wondering how to deal with the output to be the most efficient possible as I will need to manipulate them later on in the script.

You can store your data using pickle and then load it whenever you want.
import pickle

forecast1 = [1,2,3]
forecast2 = [4,5,6,6]
pickle.dump([forecast1, forecast2], open("forecasts.p", "wb"))

forecast1, forecast2 = pickle.load(open("forecasts.p","rb"))

Or you can directly store your pandas frames as .csv:
df.to_csv("forecasts.csv, header=True, index=False)


How to print feature names in conjunction with feature Importance using Imbalanced-learn library? I used BalancedBaggingClassifier from imblearn library to do an unbalanced classification task. How can I get feature improtance of the estimator in conjunction with feature names especially when the max_features are less than the total number of features? For example, in the following code total number of features equal to 20 but max_features are 8.

from collections import Counter
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from imblearn.ensemble import BalancedBaggingClassifier
from xgboost.sklearn import XGBClassifier

X, y = make_classification(n_classes=2, class_sep=2,weights=[0.1, 0.9], n_informative=3, n_redundant=1, flip_y=0, n_features=20, n_clusters_per_class=1, n_samples=1000, random_state=10)
print('Original dataset shape {}'.format(Counter(y)))

ln = X.shape
names = ["x%s" % i for i in range(1, ln[1] + 1)]

X_train, X_test, y_train, y_test = train_test_split(X, y,random_state=0)
bbc = BalancedBaggingClassifier(base_estimator=XGBClassifier(n_estimators=10,min_child_weight=5,max_depth=3,learning_rate=0.02,colsample_bytree=1,subsample=1,scale_pos_weight=.26), ratio='all',random_state=0, max_features=8)
bbc.fit(X_train,y_train)
for estimator in bbc.estimators_:
    print(sorted(zip(map(lambda x: round(x, 4), estimator.steps[1][1].feature_importances_),names), reverse=True))

I think there is a problem with the above code because always printed features are named x1 to x8 while for example, feature x19 may be among the most important features.

Thanks.

The issue is that you are not introspecting properly the feature importances.

for x in bbc.estimators_:
    print(x.named_steps['classifier'].feature_importances_)

[ 0.  0.  0.  0.  1.  0.  0.  0.]
[ 0.  0.  0.  0.  0.  1.  0.  0.]
[ 0.  1.  0.  0.  0.  0.  0.  0.]
[ 0.  0.  1.  0.  0.  0.  0.  0.]
[ 1.  0.  0.  0.  0.  0.  0.  0.]
[ 0.   0.   0.   0.5  0.   0.   0.   0.5]
[ 0.  0.  1.  0.  0.  0.  0.  0.]
[ 0.  0.  0.  1.  0.  0.  0.  0.]
[ 0.  0.  0.  0.  1.  0.  0.  0.]
[ 0.  0.  1.  0.  0.  0.  0.  0.]

As you can see, the XGBoostClassifier reports the feature importances only on 8 features as defined in the parameters list of the BalancedBaggingClassifier. Those 8 features presented to each XGBoostClassifer are in fact randomly selected for each estimator of the ensemble. So you have to first find which features were subsampled and given to each XGBoostClassifier and then indirectly find the real feature importances.

names = np.array(names)
for x, feat_sel in zip(bbc.estimators_, bbc.estimators_features_):
    feat_imp =  np.nonzero(x.named_steps['classifier'].feature_importances_)
    print(names[feat_sel[feat_imp]])

['x11']
['x11']
['x7']
['x11']
['x11']
['x16' 'x3']
['x7']
['x6']
['x7']
['x7']


Tell me some things a data scientist/statistician needs to do that can't be done in R/Stata/SAS/SPSS Why do I need to learn to program in python or Java and learn data structures? I can't think of a single thing statisticians/data scientists need to do that isn't built in as a package in R/stata.

Why do I need to learn to program in python or Java and learn data structures?

The simple answer - you dont need to lear python or Java or Scala or Julia or anything else if you dont wont to. If you dont know python it doesnt mean that you cant be a data scientist or use ML/DL.
The point is that some tools are better for some tasks, it might be benefitial to lear C++ or Java if you need code that runs faster in production, etc. But you might be totaly okay with well optimized R code. You should do what is best for you and your tasks.
About the things you cant do - I agree with you, you can do basically anything in R, it might be not efficient, but its possible.


Python sample function TypeError: '<' not supported between instances of 'str' and 'int' i GOT this ERROR when doing sampling stratify of a sample dataframe
code is like this :

df = pd.DataFrame(dict(
    A=['a', 'a', 'a', 'b', 'c', 'c', 'c', 'a', 'b', 'b', 'b', 'c', 'a', 'a', 'b', 'b', 'b', 'c', 'c', 'c'],
    B=range(20)

 ))

df.sample(n=100, weights='A',replace=True)

it gives me this error:

  TypeError: '<' not supported between instances of 'str' and 'int'

anyone can help?

Welcome to the site!

I've checked the issue, it's because you are trying to create a sample using a str variable.(df['A']). df.sample cannot create a sample by assigning weights to categorical variable.

I made a small work around i.e.,

df = pd.DataFrame(dict( A=['a', 'a', 'a', 'b', 'c', 'c', 'c', 'a', 'b', 'b', 'b', 'c', 'a', 'a', 'b', 'b', 'b', 'c', 'c', 'c'], B=range(20) ))

#converting the text into categorical codes which are in numeric
df['cat'] = df.A.astype('category').cat.codes

#creating a sample using newly created column
Output = df.sample(n=100,weights = 'cat',replace=True)

Output would have 100 records and created using 'cat' for weights.


1 - Nearest Neighbor , dealing with same distance I know that as a question it may seem stupid, but in case it is applying K NN with k = 1 and I have two neighbors at the same distance , what is the best approach to carry out the classification ?

The best approach would really depend on your application and what is important too you.

However, things you can try include:

Increase K until there is no tie anymore - if you increase to 2, you will likely have another tie, since they are at the same distance already. So, 3 or higher should do the trick.
Include another feature to your classifier - adding another dimension to your space could solve the problem if the values for both these data points are different.
Choose another distance metric - you could have a preferred way of measuring distance, but for ties, chose another metric that will break it.
Establish a rule for breaking ties, e.g.

Pick the class with the most observed data points
Randomly assign a class

More information on your application would help to choose a best approach.


Tensorflow, how many layers does this model have? I'm very new to machine learning and I'm not sure about Tensorflow implementation of neural network model. How many layers does the model below have?
model = Sequential()
    model.add(Dense(200, activation="tanh"))
    model.add(Dropout(0.3))
    model.add(Dense(1, activation='sigmoid'))

I think the answer is 2, because dropout applies to the input layer, but the input shape is (None, 20) and not 200, so does that Dense layer actually follow a implicit input layer?

When talking about feed forward neural networks one distinguishes between input layer, hidden layers and output layer. Usually you have a single input and output layer and one or more hidden layers.
In your case, the input layer with 20 input neurons is not explicitly mentioned in the code but its still there. Further, there is one hidden layer with 200 neurons and one output layer with a single neuron.
So I would say you have a three layer network. But maybe there are other conventions on how to count the number of layers in a feed forward neural network. Some might exclude the input layer.


SQL and JSON database - R I have a database (PostgresSQL) which looks like this:

""results"":
    {""WEBS"":
        {""WEBS_4"": [{""ip"": ""XXXXX"", ""value"": ""YYYY""}]}}

I would like to access the information of ip.

Then I write the following query in R but doesn't give me any info

dbGetQuery(conexion,"SELECT
              data->'results'->'WEBS'->'WEBS_4' ->'ip' as test
                          FROM results;")

And when I write this query

dbGetQuery(conexion,"SELECT                   data->'results'->'WEBS'->'WEBS_4' as test
                          FROM results;")

I get a result like this [{""ip"": ""XXXXX"", ""value"": ""YYYY""}], that is not very useful in R.

Some other parts of my database are highly nested, so I would like to access those parts which are with the [] signs.

Thanks a lot!

Your WEBS_4 is an array with two elements, so there's no ip element in it, which explains why this fails:

dbGetQuery(conexion,"SELECT
              data->'results'->'WEBS'->'WEBS_4' ->'ip' as test
                          FROM results;")

You need to get the zeroth element of WEBS_4, and then you can get the ip element of that. This works in a simple PostgreSQL test I set up using psql:

SELECT  data->'results'->'WEBS'->'WEBS_4'->0->'ip' as test
                          FROM results;
  test
---------
 "XXXXX"
(1 row)

So I suspect the relevant R syntax will be (untested):

dbGetQuery(conexion,"SELECT
              data->'results'->'WEBS'->'WEBS_4'->0->'ip' as test
                          FROM results;")


I want to add demographic data to a data set. Any suggestions on where to find zip code level data? Ideally, I would like to add demographic features to my data.  I have zip codes for each observation, so I was hoping to add in the demographic data by zip code.  Unfortunately, I cannot find any demographic data by zip code.

Does anyone have any suggestions on using the zip code tabulation area data for zip codes?

Thanks!

The US Census American Community Survey can provide a lot of demographics by different geographies, including zip. Check out:

https://factfinder.census.gov/


How each layer of a neural net is responsible for one feature Through my study of neural networks, I came across the idea that each layer of a neural network is responsible for recognizing one feature of the input data. For example, if we build a neural network that classifies cars, buses, vans and bicycles, a layer will be responsible to recognize the tires, another one will be responsible to recognize the size of the vehicle. The question is, why is this true? i.e. each layer appears to perform similar to the others and there is no special design for each one. Is there any way to assign each layer a specific feature or it is done implicitly?

... each layer of a neural network is responsible for recognizing one feature of the input data. For example, if we build a neural network that classifies cars, buses, vans and bicycles, a layer will be responsible to recognize the tires, another one will responsible for recognizing the size of the vehicle.

There are numerous kinds of neural networks and there are different differentiable components that are used inside them to achieve end-to-end learning. Convolutional layers are responsible for finding the features that are essential for reducing the error. These features are shared among different nodes and are not necessarily meaningful to us. They have to be used simultaneously due to being shared among different weights. What you are trying to say is that in convolutional networks, the first layers attempt to find low-level features like vertical and horizontal edges whilst deeper convolutional layers try to find high-level features which are more abstract. But the point here is that they attempt to recognize and the way they do that is by using weights and activations which are shared in different activation maps. Consequently, one neuron is not necessarily responsible for faces or for cars. There may be different neurons and filters which are responsible for them. Take a look at the link which has illustrated that.

The other point is about fully connected layers which are responsible for classification. What they do is finding decision boundaries to classify inputs or estimating a function for regression tasks. They are not feature extractors in a way convolutional layers do. As it is illustrated in the cited link, what they try to do is to separate the input space to make it possible for generalization in the current feature space.

  The question is, why is this true? i.e. each layer appears to perform similar to the others and there is no special design for each one. Is there any way to assign each layer a specific feature or it is done implicitly?

It was not true as mentioned. Although each layer seems similar to other layers, because of the cost function, their corresponding weights are set to decrease the cost function. Consequently, for fully connected layers all the featrues are feed to all the neurons in each layer and the training process decides how to use them. Moreover, the convolutional layers try to find similar patterns, kernels, in the input pattern.


Spacy v2.0.1 custom NER: How to improve training of existing model I implemented custom NER with bellow trained data first time and it gives me good prediction with Name and PrdName. I mentioned code bellow.

if __name__ == '__main__':
TRAIN_DATA = [
            ('My Name is Rajesh', {'entities': [(11, 17, 'Name')]}),
            ('My Name is Bakul', {'entities': [(11, 16, 'Name')]}),
            ('My Name is Pritam', {'entities': [(11, 17, 'Name')]}),
            ('My Name is Rakesh', {'entities': [(11, 17, 'Name')]}),
            ('My Name is Jayeeta', {'entities': [(11, 18, 'Name')]}),
            ('this is the price of bag', {'entities': [(21, 24, 'PrdName')]}),
            ('what is the price of ball?', {'entities': [(21, 25, 'PrdName')]}),
            ('what is the price of jegging?', {'entities': [(21, 28, 'PrdName')]}),
            ('what is the price of t-shirt?', {'entities': [(21, 28, 'PrdName')]}),
              ]

iterations = 20
try:
    model = 'live_ner_model'
    nlp = spacy.load(model)  # load existing spacy model
except:
    model = None
    print("Exception")
    nlp = spacy.blank('en')  # create blank Language class
    print("Created blank 'en' model")

if 'ner' not in nlp.pipe_names:
    ner = nlp.create_pipe('ner')
    nlp.add_pipe(ner)
    print("Create NER")
else:
    ner = nlp.get_pipe('ner')
    print("Exhisting NER")

# Add new entity labels to entity recognizer
for _, annotations in TRAIN_DATA:
    for ent in annotations.get('entities'):
        ner.add_label(ent[2])

# get names of other pipes to disable them during training
other_pipes = [pipe for pipe in nlp.pipe_names if pipe != 'ner']
with nlp.disable_pipes(*other_pipes):  # only train NER
    optimizer = nlp.begin_training()
    for itn in range(iterations):
        print("Statring iteration " + str(itn))
        random.shuffle(TRAIN_DATA)
        losses = {}
        for text, annotations in TRAIN_DATA:
            nlp.update(
                [text],  # batch of texts
                [annotations],  # batch of annotations
                drop=0.2,  # dropout - make it harder to memorise data
                sgd=optimizer,  # callable to update weights
                losses=losses)
        print(losses)

# Save model
output_dir = 'live_ner_model'
if output_dir is not None:
    output_dir = Path(output_dir)
    if not output_dir.exists():
        output_dir.mkdir()
    nlp.meta['name'] = model  # rename model
    nlp.to_disk(output_dir)
    print("Saved model to", output_dir)

# Test the saved model
output_dir = 'live_ner_model'
print("Loading from", output_dir)

nlp2 = spacy.load('live_ner_model')
test_text = """
   what is the price of cup. My Name is Rahim
"""
doc2 = nlp2(test_text)
for ent in doc2.ents:
    print(ent.label_, ent.text)

But when I am trying to trained with some new data which has entity with only PrdName or any other new entity excluding Name in existing model.
Then Name entity prediction goes wrong. I think this issue arises as I updated trained data excluding Name entity.

So is there any way we can improve training by not affecting existing training. Can someone share the idea? If possible please share a sample code.

Environment: Anaconda, spacy=v2.0.1, python=3.7

The model depends entirely on the training data: if you train with some data which has only PrdName as label, the model knows only this label and can predict only this label. You need to provide as much training data as possible, containing all the possible labels.

For the record, NER are usually trained with thousands of sentences in order to account for the diversity of the cases where a NE can appear.


What are the best practices to anonymize user names in data? I'm working on a project which asks fellow students to share their original text data for further analysis using data mining techniques, and, I think it would be appropriate to anonymize student names with their submissions.

Setting aside the better solutions of a url where students submit their work and a backend script inserts the anonymized ID, What sort of solutions could I direct students to implement on their own to anonymized their own names?

I'm still a noob in this area. I don't know what are the norms. I was thinking the solution could be a hashing algorithm. That sounds like a better solution than making up a fake name as two people could pick the same fake name.possible people could pick the same fake name. What are some of the concerns I should be aware of?

I suspected you were using the names as identifiers. You shouldn't; they're not unique and they raise this privacy issue. Use instead their student numbers, which you can verify from their IDs, stored in hashed form. Use the student's last name as a salt, for good measure (form the string to be hashed by concatenating the ID number and the last name).


How to supress previous results in a generative network? Think of a generative network generating animal names.

It's been trained on a set of animal names so this shouldn't be too hard.

But say I want to generate 10 animal names. First I run the network and it produces ZEBRA. Next it produces HORSE. Next it produces HORSE again. This is no good.

I need a way to supress previous answers.

Since I know the patterns of weights that produced the words ZEBRA and HORSE, can I used this knowledge to tweek the network in able to give me a new unique result.

(I don't mean I want all 10 answers at once! I mean I want to run the network again and again but get different results each time.)

In otherwords I want to supress the maxima that gave me the previous answers.

(This task seems quite easy for a human who can list 20-30 animal names without repetition. I'm not sure if this is because the previous animal names are stored in the short term memory or the synapses for the previous animals are supressed somehow).

I need a way to supress previous answers.

The answer to the question depends on the particular network that you are using. What kind of generative model is it? GAN? Autoencoder? Seq2Seq RNN?

Having said that, if your network keeps outputting the same result, this is usually referred to as "mode collapse", which is typical for GANs. If your network is a GAN, it may be worth exploring the Wasserstein Loss that effectively helps you deal with mode collapse.

Generally, a standard path to achieve diversity (and non-repetitive results) in the output of a generative network is to augment your training dataset. Augmentation refers to representing the same data in different ways. In the context of animal pictures, imagine it as providing different pictures of zebras or horses, from different angles or under different light conditions etc.

Also make sure that your training dataset is balanced, i.e. all classes are represented equally in the training set, otherwise the result will be biased.

If you are using a seq2seq network which generates string names of animals, you usually sample those names character by character. There are different sampling strategies in this case, such as greedy sampling (always the most probable character), multinomial sampling (beam search) which can also have temperature scaling. These strategies yield different results and are worth exploring. A very important metric that enhances the diversity of the output of such a network, is the likelihood of a sampled sequence (or its negative log-likelihood). You want to include this likelihood in your loss function to make sure you "push" the network to sample everything in its reach with equal probability. This is explained nicely in this publication and this publication, which use string representations of molecules that are analogous to your animal names task.

If you are using an autoencoder as a generative network, you can combine it with an optimization algorithm, such as particle swarm optimizer, to navigate within its latent space and automatically sample diverse solutions. The key here is the cost function of the optimization algorithm, because inside it you need to penalize repetitive answers. Similar performance can be achieved using reinforcement learning and a likewise penalizing scoring function.

  I'm not sure if this is because the previous animal names are stored in the short term memory or the synapses for the previous animals are supressed somehow

If you are using stateless LSTM units, the memory is reset between independent predictions. If you use stateful LSTM units, previous predictions affect next predictions.

I realize that my answer covered many different approaches, however it may provide you with different directions to explore.


How I can compute Matthews Correlation Coefficient? I want to calculate MCC, I have two CSV files: result.csv and true.csv

result.csv format:

                   id | prob_class1 | prob_class2 | prob_class3 | prob_class4
                    1 |0.34         | 0.4         | 0.32        |0.77
                    2 |0.84         | 0.4         | 0.2         |0.7

true.csv  format:

                     id | prob_class1 | prob_class2 | prob_class3 | prob_class4
                     1 |0            | 0           | 1           |0
                     2 |0            | 1           | 0           |0
                     3 |0            | 0           | 0           |1

result.csv is the output of the multi-classification problem, true.csv is the true result. How I can calculate the Matthews Correlation Coefficient?

Here are the high-level steps:

Load the .csv from disk into an array in memory.
Threshold the probabilities to convert them to predicted target values.
Call sklearn.metrics.matthews_corrcoef.


Collaborative filtering with human-adjusted latent factors Having tried some of movie recommendation engines available on the web I have the feeling they are not satisfactory. I just fail to get movies similar to those I like based on traits interested for me personally.

My guess is that the lack of precision can be overcome with extending the number of latent factors in CF SVD algorithm. Another idea is that latent space can be initially hand-crafted.

E.g. we can take 50 well-studied movies and asses them by 200 traits manually (partially fill item-latent factors matrix). Accept this values as fixed. Then having some of the user ratings we can perform matrix factorization and inherit user-factors and utility matrices as usual, while keeping fixed values.

Any present work on this?

What you are looking for in this case is matrix factorization with side information. And yes, there are quite a lot of works in the area. I would recommend you this paper as a start: Relational learning via collective matrix factorization.


Marginal contribution of a text document I'm trying to build a Shapley value (marginal contribution) of a text document in terms of information content, given that there are several documents on a given topic.
For example, we have 3 reports describing the ocean:
A: {The ocean is blue.}
B: {The ocean is blue and salty.}
C: {The ocean is salty and a home for fish}
If I'm correct, a first step in the algorithm is identifying a set of features (individual pieces of information): {blue, salty, a home for fish}
Moreover, say each feature has an equal weight.
Then I can compute Shapley values of 1/6 for A, 1/3 for B, 1/2 for C.
I wonder if there are well-established algorithms / papers that deal with this problem. I'm coming from economics myself (I'm familiar with Shapley values abstractly), so apologies if the question is too trivial.
Thank you!

From an NLP perspective, the main issue is to identify and extract the terms/relations of interest from the text.
It may be easy if the sentences always appear like these simple examples: one can use pattern matching "the <entity> is <property1> [and <property2>]" (no need for ML).
However with general text it's a quite complex problem which would probably require training an ad-hoc model from annotated data. This would generally involve the tasks of Named Entity Recognition and Relation Extraction.


SGDClassifier fit and partial_fit functions I wanted to know what is the correct way to train the SGDClassier model on new data observations? Should I use the fit function or the partial_fit function? How are they different?

fit(), always initializes the parameters like a new object, and trains the model with the dataset passed in fit() method.

Whereas partial_fit(), works on top of the initialize parameter and tries to  improve the existing weights with the new dataset passed in partial_fit().

It is always good to save the model in persistent storage (say pickle file), for later use or for further training.


What is neural structure learning in tensorflow? What is neural structure learning?
what is the difference between neural network vs neural structure
learning?

It seems that you actually mean "neural structured learning".

From the Tensorflow webpage on their neural structured learning framework, it seems it is just an umbrella term to two types of regularizations: Neural Graph Learning and Adversarial Learning.

Basically you add an extra term to your training loss where you force the internal representations for certain input to be close to the representation of its "neighbors".

The neighborhood is expressed either as a graph (hence leading to neural graph learning) or as a neighbor/no neighbor relation (hence leading to adversarial training).

Therefore, neural structured learning refers to normal neural networks that are trained with extra knowledge on what inputs are "close" to each other.


Dealing with Overfitting Hello and thank you in advance for any answer. I am building a NN for a multi class classification problem. I have pretrained my data through a Word2vec and generated a 300 dimension vector with values. I have 6 classes .. My data consist of 15000 rows(x300 dimensions). My first question is :

What is the number of units? Is it something that we can extract from theory?
Also i have managed to gain 0.85-0.90 training set accuracy but the validation set accuracy is always low: 0.22-0.25. I do not know what way to follow as I am newbie to these kind of stuff. My code is:

import numpy as np
import pandas as pd
import keras
from sklearn.preprocessing import Imputer
from sklearn.model_selection import train_test_split
from keras.models import Sequential
from keras.layers import Dense,Dropout
from sklearn.metrics import confusion_matrix
import matplotlib.pyplot as plt
from keras import regularizers

dataset = pd.read_csv('word2vec.csv')

X = dataset.iloc[:, 0:300].values
y = dataset.iloc[:, 300].values

imputer = Imputer(missing_values = 'NaN', strategy = 'mean', axis = 0)
imputer = imputer.fit(X[:, 0:300])
X[:, 0:300] = imputer.transform(X[:, 0:300])

counter = 0
for iterations in range(1):

    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

    classifier = Sequential()

    classifier.add(Dense(units=70, kernel_initializer='uniform',kernel_regularizer=regularizers.l2(0.0001),
                activity_regularizer=regularizers.l1(0.001), activation='relu', input_dim=300))

    classifier.add(Dropout(0.3))

    # classifier.add(Dense(units=70, kernel_initializer='uniform', activation='relu'))

    classifier.add(Dense(units=6, kernel_initializer='uniform', activation='softmax'))

    classifier.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

    history = classifier.fit(X_train, y_train,validation_split=0.1, batch_size=10, epochs=100,shuffle=True)

    y_pred = classifier.predict(X_test)
    # y_pred = (y_pred > 0.1666)

    classifier.summary()
    counter+=1
    print(classifier.layers)
    print("Running RNN with Dropout Layer")
    print("Number of layers used: "+str(len(classifier.layers)))
    # list all data in history
    print(history.history.keys())
    # summarize history for accuracy
    plt.plot(history.history['acc'])
    plt.plot(history.history['val_acc'])
    plt.title('model accuracy')
    plt.ylabel('accuracy')
    plt.xlabel('epoch')
    plt.legend(['train', 'test'], loc='upper left')
    plt.show()
    # summarize history for loss
    plt.plot(history.history['loss'])
    plt.plot(history.history['val_loss'])
    plt.title('model loss')
    plt.ylabel('loss')
    plt.xlabel('epoch')
    plt.legend(['train', 'test'], loc='upper left')
    plt.show()

Yep - you've overfitted !

Can you see what your accuracy is as a function of the number of training cycles? It would be interesting to see how that looks.

Is there any way you could increase your data? 15000x300 isn't a huge amount of depth for a neural network, considering the width of the data set.

You could look at the impact of increasing your regularisation parameters, or decreasing the depth of your network


How to handle like meaning sentences when working on text summarization Suppose we have a text like Today is a very bad day. Very bad day is today. I wont come to play.
What kind of technique should I use to summarize similar texts like above? From articles, I found over the web till now I think that extractive summarization will give importance to the first two sentences because major key points of the texts(according to the frequency of words) are present in the first two sentences. Similarly, the abstractive summarization technique will also make a summary certainly considering the first two sentences. But, in the ideal case, the third sentence is a must.
What should be my approach considering my text may have many sentences with similar meanings??
I am very new to this any kind of suggestion or help will be great.

In case by 'summarize' you mean in fact 'classify', e.g. 'bad', 'good', etc, then an LSTM/GRU will do the job, just input the whole (truncated probably) sequence, and take the last state:
out, _ = lstm(embedding)
out_state = out[:,-1,:]

Here embedding is whatever word embedding you are using, out is the whole output of LSTM, and batch_first=True (in pytorch), so you take the last state as the input in the fully connected layer.
If you need to make predictions about parts of the text, just add a loop and truncate sentences rather than the whole text.


White circles extraction from a little darker background I'm trying to extract three white circles on the top left corner from the image below:

I've tried to use:

clone = img.copy()

# Threshold grayscaled image to get binary image
ret,gray_threshed = cv2.threshold(img,90,100,cv2.THRESH_TRUNC)
cv2.imshow('gray', gray_threshed)

# Find edges
edge_detected_image = cv2.Canny(gray_threshed, 350, 100)

# Find contours
contours, _= cv2.findContours(edge_detected_image, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)

contour_list = []
for contour in contours:
    contour_list.append(contour)

# Draw contours
cv2.drawContours(clone, contour_list, -1, (255,0,0), 2)

#Resize image
clone = cv2.resize(clone, (800, 800))

cv2.imshow('Objects Detected',clone)

The result was:

I've also tried sth like:

img = cv2.blur(img,(5, 5))

img = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
cimg = cv2.cvtColor(img,cv2.COLOR_GRAY2BGR)

circles = cv2.HoughCircles(img,method=cv2.HOUGH_GRADIENT,dp=1,minDist=2,param1=50,param2=60,minRadius=0,maxRadius=300)

circles = np.uint64(np.around(circles))
for i in circles[0,:]:
    cv2.circle(cimg,(i[0],i[1]),i[2],(0,255,0),2)
    cv2.circle(cimg,(i[0],i[1]),2,(0,0,255),3)

But it showed the error due to the fact that it hadn't found any circles.

Any hints how to solve my kind of problem are welcome. Thanks in advance.

Follow the following steps:

Step 1: use an averaging filter to minimise the noise present in the image

Step 2: threshold the image such that pixels with RGB values within [0, 0, 0] and [1, 1, 1] are set to 0 and rest as 1. This step will lead you to generate a binary image.

Step 3: use Hough transform in OpenCV to detect circles. Tune the parameters of the Hough transform to get your desired output.

Check this out for more details of Hough transform for circles: https://docs.opencv.org/2.4/doc/tutorials/imgproc/imgtrans/hough_circle/hough_circle.html


Export pandas to dictionary by combining multiple row values I have a pandas dataframe df that looks like this

name    value1     value2
A       123         1
B       345         5
C       712         4
B       768         2
A       318         9
C       178         6
A       321         3

I want to convert this into a dictionary with name as a key and list of dictionaries (value1 key and value2 value) for all values that are in name

So, the output would look like this

{
 'A': [{'123':1}, {'318':9}, {'321':3}],
 'B': [{'345':5}, {'768':2}],
 'C': [{'712':4}, {'178':6}]
}

So, far I have managed to get a dictionary with name as key and list of only one of the values as a list by doing

df.set_index('name').transpose().to_dict(orient='list')

How do I get my desired output? Is there a way to aggregate all the values for the same name column and get them in the form I want?


Research in high-dimensional statistics vs. machine learning? (I've posted this question on CV, but I feel it would also be great to hear from experts in DS community.)

As a PhD student starting to think about dissertation topics, I am particularly interested in high-dimensional statistical learning. I wish to find some research review/survey/papers (or webpages, blogs, whatever...) about the state-of-the-art research in this research area, but there seems limited resources I can obtain. My first question is then,

Could you describe some current interesting research topics in
high-dimensional statistics? If you can list any relevant resources
(papers, webpages, etc.), that would be really helpful.

In addition, I've noticed that high-dimensional statistical learning is closed related with machine learning research. For example, the idea of penalized regularization in high-dimensional statistics was used in machine learning domain, like support vector machine, boosting tree, (sparse) additive models, etc. My question is,

what are good research papers about the interplay of
high-dimensional statistics and machine learning?

Last, since high-dimensional statistics was really motivated by genetic research (like gene-expression analysis, or genome-wide association studies), most of applications in high-dimensional research are devoted in that area.

Are there any successful applications of high-dimensional statistics
in areas other than genetics, particularly say, image/text mining,
recommendation, etc, areas where machine learning techniques have
long been used?
A new question to machine learning researchers/practitioners: I might be wrong, but as I understand, most machine learning algorithms are designed for low-dimensional problems (or at least the number of features is smaller that number of observations). Are there any successful applications of machine learning techniques for modeling high-dimensional data?

Any resources/comments are highly appreciated. Thanks.

This makes me think right off the bat of Gaussian processes.

The GP was initially developed by geostatisticians in the 1970s as a model for the distribution of resources over a geographic area; this process is known as kriging (or also Gaussian process regression). But more recently GPs have been used pretty widely in the movement/control/robotics area to model movements and trajectories of robotic limbs etc. over time.

Statistically, GPs are treated as priors over vectors of infinite dimension, such that the distribution of any subset of the values of a vector is Gaussian distributed. There's an online book if you want to read more!


Replace Missing Values with Most Frequent number under Condition I'm trying to replace missing values of column "Age" but under condition of other columns on this data Titanic - Machine Learning from Disaster
df.Age[(df['Sex'] == 0) & (df['Pclass'] == 1)]

I tried to do that using SimpleImputer:
from sklearn.impute import SimpleImputer
Imputer = SimpleImputer(missing_values=np.nan, strategy='most_frequent')

Imputer.fit_transform( pd.DataFrame(df.Age[(df['Sex'] == 0) & (df['Pclass'] == 1)]) )

but it doesn't work and tried to save values to the column:
df.loc[(df.Age.isnull()) & (df.Age[(df['Sex'] == 0) & (df['Pclass'] == 1)]), 'Age'] = Imputer.fit_transform( pd.DataFrame(df.Age[(df['Sex'] == 0) & (df['Pclass'] == 1)]) )

but doesn't work also.
I tried to do it manually using fillna()
df.loc[(df['Sex'] == 0) & (df['Pclass'] == 1), 'Age'].fillna(int(df.Age[(df['Sex'] == 0) & (df['Pclass'] == 1)].mode()), inplace=True)

I tried to use indexes to access rows and update their values:
mod = int(df.Age[(df['Sex'] == 0) & (df['Pclass'] == 1)].mode())
indices = df.loc[(df.Age.isnull()) & (df.Sex == 0) & (df.Pclass == 1), 'Age'].isnull().index
df.loc[ind, 'Age'] = mod
df[(df['Sex'] == 0) & (df['Pclass'] == 1)]['Age'].isnull().sum()

it worked and the output was: 0, but when I'm trying to apply it in for loop it gives me an error
for i in range(1,3):
    for j in range(1,4):
        indices = df.loc[(df.Sex == i) & (df.Pclass == j), 'Age'].isnull().index
        mod = int(df.Age[(df['Sex'] == i) & (df['Pclass'] == j)].mode())
        df.loc[ind, 'Age'] = mod

I want to know what is the wrong of first 2 ways an why the 3rd doesn't work in loop?

This solution works well but I don't know why above don't work!
Imputer = SimpleImputer(missing_values=np.nan, strategy='most_frequent')
for i in range(2):
    for j in range(1,4):
        ls = np.array(df.Age[((df.Sex==i) & (df.Pclass==j))]).reshape(-1,1)
        df.Age[((df.Sex==i) & (df.Pclass==j))] = Imputer.fit_transform(ls)[:,0]
df.Age.isnull().sum()


Linear regression : ValueError: operands could not be broadcast together with shapes (3,) (1338,) I try to use linear regression for insurance data . But had error on the when try to call a function with features parameter.
Here is my code:

def h(x):
    global w
    return np.sum(np.transpose(w)*x)
    raise NotImplementedError()

when try with a simple data it works fine,

w, x = [1,2,3], [2,3,4]
h(x)

the output is : 20

but when try to use the dataset, it errors:

features = dataset.drop(["charges"], axis=1).values
h(features )

it returns error:

ValueError: operands could not be broadcast together with shapes (3,) (1338,)

so the features looks like this:

array([[0.1173913 , 0.        , 0.35698144, 0.        , 1.        ],
       [0.1       , 1.        , 0.48331988, 1.        , 0.        ],
       [0.27391304, 1.        , 0.46674738, 3.        , 0.        ],
       ...,
       [0.1       , 0.        , 0.5496099 , 0.        , 0.        ],
       [0.15217391, 0.        , 0.3117837 , 0.        , 0.        ],
       [0.84782609, 0.        , 0.38216303, 0.        , 1.        ]])

The data i used is insurance.csv from kaggle.com

It looks like you're trying to multiply a matrix and a vector point-wise. Such an operation is not defined. I think you should use X.dot(w), where X is a feature $\bf matrix$ and w is the wights $\bf vector$. np.dot can operate with objects of different nature (like matrices and vectors). So in h I'd write return X.dot(w). Also, I'd call it only with a matrix X (for consistency), even if there's an object it would be $X \in \mathbb{R}^{1 \times d}$ in that case. Watch carefully after shapes of the objects which enter functions.


Plotting multiple datasets on a 3D plot I want to make a 3D scatter plot of multiple data selections on a single plot (i.e same axes).
I know that in 2D this is possible by using par() function like so:

plot(6:25,rnorm(20),type="b",xlim=c(1,30),ylim=c(-2.5,2.5),col=2)
par(new=T)
plot(rnorm(30),type="b",axes=F,col=3)
par(new=F)

(source: http://cran.r-project.org/doc/contrib/Lemon-kickstart/kr_addat.html)
Can I do something like that on a 3D plot, preferably an interactive 3D plot, like the ones created using plot3D from 'rgl' package?

You can use the plot3Drgl library.  E.g.

library("plot3Drgl")
x <- c(1,2,3,4,5)
y <- c(6,7,8,9,10)
z <- c(11,12,13,14,15)
colors <- c(rep("red",3), rep("blue",2))
plot3d(x=x,y=y,z=z,col=colors)

Or you can use plotly:

library("plotly")
x <- c(1,2,3,4,5)
y <- c(6,7,8,9,10)
z <- c(11,12,13,14,15)
colors <- c(rep("red",3), rep("blue",2))
df<-as.data.frame(cbind(x,y,z,colors))
colnames(df)<-c("eat", "pork", "chops", "colors")

p <- plot_ly(data=df, x=~eat, y=~pork, z=~chops, color =~colors, colors=c('#BF382A', '#0C4B8E')) %>% ## Like a pipe?
  add_markers() %>%
  layout(scene = list(xaxis = list(title = 'eat'),
                     yaxis = list(title = 'pork'),
                     zaxis = list(title = 'chops')))
htmlwidgets::saveWidget(as_widget(p), "index.html")

This saves the output as an html file.  In order to get this to work, you'll also need to install pandoc.  On CentOS/RedHat do yum install pandoc pandoc-citeproc.  On Mac OSX, using homebrew, do brew install pandoc.


How to use weights from our data set properly? Should we use them at all? The census data set I'm using: https://archive.ics.uci.edu/ml/datasets/Adult

So, I'm currently using this census data to make observations and predict whether someone is married or not. However, when I plot a categorical graph for the race, it's overwhelmingly white and I was going to just not take race into account at all. Before I did that, though, I realized that I didn't take into account the "final weights" in the data set, which shows the number of people the census believes that the entry represents.

So, I have a few questions:

Should I use these weights when making observations? Am I interpreting this correctly? Or should I just ignore them?

If I were to use these weights, how exactly would I go about using these weights? I want to be able to predict whether someone is married or not based on a few of these factors given, but I'm having trouble figuring out how I should incorporate these weights.

Also, would it be dumb to divide all the final weights by 100 or something similar? All of these weights are in the hundreds of thousands, and I was thinking that if we divide the weights by some constant, then we can just add that number of rows (for each row based on its weight) to our data set.

I'm new to ML in general so forgive me if this is a newbie question.

Firstly, if a variable is imbalanced (Race in your case), you shouldn't worry about. As its a categorical variable, it will be converted to two or more dummy variables.

Secondly, if the weight represents the number of people for each entry, as the following example, you should include the weights.

hypothetical data

Race       Education    Marital    weight
------------------------------------------
White       tertiary    Married      10
White       tertiary    Single        6
Black       tertiary    Married       8
White       secondary   Single        4
...
...

For this example, I would oversample each entry by its weight. Because your real database is:

          X               y
---------------------  --------
Race        Education   Marital
-----------------------------------
White       tertiary    Married
White       tertiary    Married
White       tertiary    Married   10 records
...
White       tertiary    Married

White       tertiary    Single
White       tertiary    Single     6 records
...
White       tertiary    Single
Black       tertiary    Married
Black       tertiary    Married    8 records
...
Black       tertiary    Married
White       secondary   Single
White       secondary   Single
White       secondary   Single
White       secondary   Single

However, I should see few records of your dataset to help you better.


String labels for classification I've recently started playing SciKit ML. I just got my hands on classification algorithms (SGDClassifier, LinearSVC) and I'm not sure how to properly represent feature labels.

Suppose I'm trying to predict a football match outcome. Given N features the result is either 1=WIN or X=DRAW or 2=LOSS. Can/should I label my features using string values?

Eg.:

X_train = [[n0, n1, …, nn], …]
y_train = ['1', 'X', '2', …]

Or maybe I should use boolean matrices instead, like [win,draw,loss], eg:

y_train = [[1,0,0], [0,1,0], [0,0,1], …]

Or maybe I should use a single digit to map each label 1=win and 0=draw, -1=loss, eg:

y_train = [1, 0, -1]

Which one would be best to pick and why? I'm a newbie, thank you for your patience.

Mutually exclusive classes are usually integers; in your case, win=0, draw=1, loss=2 (for instance).

It is mostly a convention and is convenient for implementation.


Probability for label correctness in semi-supervised learning I am aware of the existence of semi-supervised learning approaches, such as the Ladder Network, where only a subset of the data is labeled. Are there any methods or papers which consider correctness probabilities for the labels of that training data subset? That is, some labels may be correct with 100% probability, while others may have only 70% or 45% probability of being correct. Any links to papers or work in this direction are highly appreciated.

I do not know any papers, I would be greatly appreciated if someone would link some.

In my case, I always test by un-labelling my already known labelled Data by using a "traditional" 67% - 33% (train - test) split and checking how the labelling performs in various metrics (Accuracy, Logloss, etc.).

Moreover, there are various categories of Semi-Supervised Learning. For instance, if you would use Active Learning (pool based approach, includes incrementally sample/s each time), you could view how the performance ranges.

Finally, you may use as well Cross-Validation in the same sense for hyper parameter tuning for your Semi-Supervised Learning algorithm. All in all, in my perspective Semi-Supervised Learning, has a lot yet to offer and your question may be rather more domain, data specific and visualsing the Data and Clustering may give you new insights.


Data Augmentation for Regression I want to use deep learning for regression. However, the number of training samples is not large. In image processing, some new samples are generated on the basis of initial data through tasks like rotation to increase the number of data sets. Can I know if there is any algorithm to generate more data based on current samples (data augmentation for regression)?

Yes, you can perturb your data (and targets) in ways that you wish your model to be robust against, for example by adding small amounts of noise (possibly Gaussian) or synthetic anomalies, or by creating meaningful aggregates. Some models also improve just by running the same training data through them more than once, randomly shuffled (“epochs”).


How to understand conv layer to another same conv layer in VGG16? VGG16 struct is:

Img->
Conv1(3)->Conv1(3)->Pool1(2) ==>
Conv2(3)->Conv2(3)->Pool2(2) ==>
Conv3(3)->Conv3(3)->Conv3(3)->Pool3(2) ==>
Conv4(3)->Conv4(3)->Conv4(3)->Pool4(2) ==>
Conv5(3)->Conv5(3)->Conv5(3) ====> FC

The flow :
http://ethereon.github.io/netscope/#/gist/dc5003de6943ea5a6b8b

In keras code https://github.com/keras-team/keras/blob/master/keras/applications/vgg16.py

# Block 1
x = Conv2D(64, (3, 3), activation='relu', padding='same', name='block1_conv1')(img_input)
x = Conv2D(64, (3, 3), activation='relu', padding='same', name='block1_conv2')(x)
x = MaxPooling2D((2, 2), strides=(2, 2), name='block1_pool')(x)

....

It looks like just pass Conv1_1 result to Conv1_2, and the rest too.

I can understand this:

Img->
Conv1(3)->Pool1(2) ==>
Conv2(3)->Pool2(2) ==>
Conv3(3)->Pool3(2) ==>
Conv4(3)->Pool4(2) ==>
Conv5(3) ====> FC

But don't understand why Conv1_1 can connect to a same layer - Conv1_2 :

For example:

Img: 224x224x3
kernel:3x3
depth:3 (channel)
Conv1_1: use 64 kenels. So there are 64 3x3x3 kernel used to scan 224x224x3(with padding 1) ,  then get Conv1_1_feature_map 224x224x64 .

Here, does it mean:

Conv1_2: use 64 3x3x64 kernel to scan Conv1_1_feature_map 224x224x64 , then get Conv1_2_feature_map 224x224x64 ??

If it is, could you explain what is the benifit ? I can't understand the meaning of this flow clearly. It is not clear that why Conv1(3)->Conv1(3)->Pool1(2)  is better than Conv1(3)->Pool1 . On my feeling, another same size Conv looks like just make previous layer output more blurred/ambiguous, doesn't like Pooling layer concentrate previous Layer's output feature.

If I've got the meaning of question, first convolution accepts inputs of size 224*224*3 means that height and width are both 224 and the size of depth or number of activation maps, channels here, is equal to three. The output of this layer will be activation maps with equal height and width as the input, 224, because it is same convolution. The number of activation maps for this layer which is going to be passed to the next layer is equal to 64 because you have 64 filters in this layer. The point is that each filter is of size 3*3*3 to fit to the input. The output of each filter is an activation map of size 224*224*1. The output of filters come together and construct output of size 224*224*64 which means the input of the next layer will have 64 channels, actually depth here. Consequently, the filters of the second convolution layer is of size 3*3*224 to match the entire input of the previous layer. In other words, you will have 64 filters of size 3*3*64 and each will have an output of size 224*224*1. Take a look at here which definitely can help you.

The purpose of writers of VGG net was to make a network which was just deep enough to perform well on ImageNet data-set. CNNs have lots of hyper parameters which setting them is not based on well behaved math stuff. I mean there is no proof to show which hyper parameter is better than the others. They are found based on experience. They are gained practically. Writers of this paper tried to show that if you use same hyper parameters for convolution layers and just make the network deep and deeper, it was much deeper than AlexNet, you will gain good performance without caring about different set of hyper parameters. For understanding what ConvNets do, there is already an answer here which may help you, it contains the interpretation of different layers in CNNs. They have reached to this setting of convolution layers and architecture by experience and there idea was too use just a deep net without complicated hyper parameters.


Prediction Outputs from LSTM NN I'm working with an LSTM network to predict the surface roughness due to biogrowth on ships. I've got a Network that fits the input data I have relatively well, the problem is when I'm using it to predict future roughness values.
I've 3 case scenarios - one where the hull is not cleaned, second where the hull is cleaned and the third where the hull is repainted. Based on the inputs - the idea is the trend drops slightly when the hull is cleaned and considerably when it is repainted.
However when I use the same inputs to predict the roughness for the different cases, the outputs differ in the region where it is supposed to be the same, i.e it is predicting the trend to go upward in the case where the hull is cleaned, but when it is not cleaned it doesn't show this upward trend.
I assume it is seeing the pattern and seeing that as there is a future cleaning - the trend must go up. But this isn't reality. I've tried swapping the LSTM for a basic NN but this still exists.
Any help would be useful.

The problem was that I was using a MinMaxScaler which scaled the inputs differently for the different cases. Once I set them to the same scale, this issue was resolved.


How to validate recommender model in healthcare? In order to validate a recommender model, a usual approach is create a hold-out set that will provide random suggestions (similar to an A/B testing setup).
However, in healthcare applications, this cannot be possible as a random suggestion can put at risk a patient's life.
Hence, what is a reasonable approach to validate the model?

You should still be able to use a validation set to evaluate the model, whether or not you pursue an experimental approach. (Specific features of your model and investigations may tweak these, but this is based on what's already been posted alone).

There is nothing wrong with A/B group assignment and testing in a medical context, with a few caveats (this list is not exhaustive):

The relevant clinical/medical knowledge must be in a state of
equipoise (it's not already clear that one approach is better than
another, or which is better is genuinely not known).
Individuals should be aware that they are participating in a study, and that they are being routed to
group A or B, and have the option to decline their assignment (or,
conversely, they have been made aware of the experimental assignment
and have consented to participate in advance).
An institutional review board should evaluate your proposed
experiment and signed off on it. This, of course, presupposes that
you have access to such a board composed of members able to make
those assessments.

Those can be a tall order, but you don't necessarily have to perform a prospective, double-blind experimental study in order to glean some information. A retrospective study could provide some insight as well, and your process for the validation set would be something like:

Prepare your recommender model
Feed your data through the model, without looking at actual outcomes
Match your model output to actual outcomes to see whether or not
people followed the recommendation (whether they ever saw that
recommendation or not)
Compare the results of people that ended up going with each
recommended approach (A vs. B), as well as those who "followed" the
recommendations or not (Recommended-A-did-A vs. Recommended-A-did-B,
etc.)

Retrospective studies are generally not as good as well-designed, well-executed prospective experimental studies, but they can still provide a lot of information. In situations where prospective experimentation is impossible or undesirable, the information a retrospective study provides may be the very best you can actually get.


Adding extra (meaningful) features does not improve model performance I am struggling with confusion matrices and their outputs. I thought to follow all the steps right, but unfortunately it seems that something is not going well.
I had a dataset built and labelled on my own. It shows a class imbalance so I decided to apply undersampling and oversampling, looking at F1-score and Recall as in many papers and on the web.
The steps were:

split data in train and test (80/20)
apply resampling only on train set
apply pre-processing algorithm (BoW, TF-IDF, ...)
use different classifiers to get results
look at performance using confusion matrices (or alternatively ROC)

I tried with different features: in one dataset with less features engineering, i.e., using only features from Text, I got a maximum value of F1-score equal to 68%. With more features, that I thought to be significant for improving the model, I am getting max 64%, that is weird considering the problem (email classification for spam detection).
In theory, if I extract features only from text, I get a better score rather than extracting also features from email addresses.
I would like to ask you for some tips and suggestions, if you have any, as I think that this cannot be possible, as the expected results should be higher in the second case, when I consider also information from email address (number of dots, suffix, registration date,...).
I am thinking at a problem of overfitting or some other issues with model building.
I would appreciated if you could tell me your thoughts on this.
Thank you for all your help.

I tried with different features: in one dataset with less features engineering, i.e., using only features from Text, I got a maximum value of F1-score equal to 68%. With more features, that I thought to be significant for improving the model, I am getting max 64%, that is weird considering the problem (email classification for spam detection).

Typically this would happen if the model is overfit: not enough data and/or too many features make the model pick patterns which happen by chance in the training data.
Usually with text one has to remove the least frequent words in order to avoid overfitting. You might also want to check the additional features, remove anything which happens too rarely.

Also, the confusion matrix, has given me weird outputs

       0      1
0    [[2036  161]
1    [   1 2196]]

Observations:

True class 0 has 2036+161 = 2197 instances, true class 1 has 1+2196=2197 instances. These results are obtained with the resampled data.
Assuming class 1 is positive: 2196 True Positives (TP), 2036 TN, 161 FP (true positive predicted as negative) and 1 FN (true negative predicted as positive).

recall = 0.999, precision = 0.932. That's an f1-score somewhere higher than 0.95 (probably due to the resampled data).

The second confusion matrix is also clearly obtained with the resampled data, and it shows perfect performance (F1-score is 1).

These matrices show the performance obtained on the resampled data, so it's similar to the performance on the training data. Since the performance on a real test set is much lower, this confirms strong overfitting.