How to save API data into CSV format? Is there an easy way to convert API data with python commands and output the results in CSV format?

Here's what I'm doing now:

import requests

params = {
  "api_key": "fec742c1-c846-4343-a9f1-91c729acd097",
  "format": "jsonp"
  }
r = requests.get('https://data.sa.gov.au/data/api/3/action/datastore_search')
print(r.text)

# This bit of code will write the result of the query to output.csv

with open('output.csv', 'w+') as f:
    f.write(r.text)

This code doesn't work. The API data is available here

On the page you linked there is actually a Python example on how to get the data. It is in Python 2, but I will show you how to make it work in Python 3.

import urllib
import json                 # Used to load data into JSON format
from pprint import pprint   # pretty-print

url = "https://data.sa.gov.au/data/api/3/action/datastore_search?resource_id=fec742c1-c846-4343-a9f1-91c729acd097&limit=5&q=title:jones"
response = urllib.request.urlopen(url)
print(response)
# Just an object:   <http.client.HTTPResponse at 0x7f2618123e10>

We get the text data out by using the read() method:

text = response.read()

The response in this case is a raw string. We can use the json module's function loads to load a string):

json_data = json.loads(text)

pprint(json_data)

returns the following JSON data:

{'help': 'https://data.sa.gov.au/data/api/3/action/help_show?name=datastore_search',
 'result': {'_links': {'next': '/api/3/action/datastore_search?q=title%3Ajones&offset=5&limit=5&resource_id=fec742c1-c846-4343-a9f1-91c729acd097',
                       'start': '/api/3/action/datastore_search?q=title%3Ajones&limit=5&resource_id=fec742c1-c846-4343-a9f1-91c729acd097'},
            'fields': [{'id': '_id', 'type': 'int4'},
                       {'id': 'LGA Name', 'type': 'text'},
                       {'id': 'Tenure type', 'type': 'text'},
                       {'id': 'Very low income <$603 per wk', 'type': 'numeric'},
                       {'id': 'Low income $603-$964 per wk', 'type': 'numeric'},
                       {'id': 'Moderate income $965-$1446 per wk',                            'type': 'numeric'},
                       {'id': 'Total', 'type': 'numeric'},
                       {'id': '_full_count', 'type': 'int8'},
                       {'id': 'rank', 'type': 'float4'}],
            'limit': 5,
            'q': 'title:jones',
            'records': [],
            'resource_id': 'fec742c1-c846-4343-a9f1-91c729acd097'},
 'success': True}

I would suggest using Pandas, which can do a lot of the tedious work for you very easily. It can read straight from a JSON string (our text above). The issue is that is will parse it a little strangely.

There is no simple way to write this directly to a CSV file, because there are nested structures: e.g. under "result" there "fields" and then more values, and CSV files can't display that directly. You need to essentially flatten the structure yourself, perhaps decide what is important or what you want to can leave out.

You could take the JSON formatted json_data above and unpack it manually, removing nested parts, which means looking through the response and making your own Python dictionary with only single level i.e. no nesting. Let's say you do that and have a new flattened dictionary named r. Once you have done that, you can do the following using Pandas to write the CSV file:

import pandas as pd

df = pd.read_json(r)
df.to_csv("output.csv")


Including the dependent variable in your data to perform principal component analysis? Let's say you have a data set with GPA (dependent variable) and Amount of alcohol, Amount of study, IQ, and SAT score as the independent variables. And you want to perform the principal component analysis in R for dimension reduction.

In the csv file you read in R, do you have to have GPA or do you have to remove it?

You need to remove the dependent variable before performing PCA.
Otherwise you are essentially using the dependent variable to explain itself.

Also, think about prediction:
You might want to use your model to predict unknown GPA scores based on information about amount of alcohol, amount of study, IQ and SAT. In that case, it is impossible to include information on GPA scores.


Normalisation results in R^2 score of 0 - Lasso regression I am running a regression analysis on a 7000 row dataset with a train/test split of 70%/30%. I am using one variable X to predict a variable Y.

X ranges between 300 and 810 (mean 712).
Y is an integer (number of occurrences) ranging between 0 and 20 (mean 0.2).

Without standardisation or normalising X, I receive:

Train score:  0.082
Test score:  0.077

However upon normalising (X = (X-X.min())/(X.max()-X.min())), I receive:

Train score:  0.0000
Test score:  -0.0001

Is there something incorrect about normalising for a Lasso regression? The same applies to standardising the data. Would anyone be able to advise me on the best course of action?

Standardizing/normalizing is generally the right thing to do, but it will make little/no difference with just one independent variable if you also adjust the regularization strength.

With more than one independent variable, standardizing assures that the lasso's penalty applies more equally to all the variables.  Without standardizing, a large-scale variable will be penalized less than small-scale ones (because the coefficients will already tend to be small).

In your case, the original model needs a fairly small coefficient on X because its scale is so large, and so you don't receive much penalty from the lasso.  After you standardize, now the model wants a larger coefficient on X, but that means the lasso penalty makes more of a contribution.  That you get an $R^2$ of zero suggests that the lasso penalty is large enough now to push the coefficient to zero, so that the model is just a horizontal line.  If you reduce the regularization strength a bit, you should be able to recover the old model (exactly maybe, with the right adjustment? If I get some spare time I'll look into this).


Problem understanding probabilistic generative models for classification I am a student and I am studying machine learning. I am focusing on probabilistic generative models for classification and I am having some troubles understanding this topic.
In the slide of my professor it is written the following:

which I don't understand.
So far, I have understood that in the generative probailistic models, we ant to estimate $P(C_i|x)$, which is the probability of having class $i$ given a data $x$, using the likelihood and the Bayes theorem.
So, it starts by writing the Bayes rule, but the the slides says that we can write this as a sigmoid, but why?
If I have to try to give an answer to it, I would say because the sigmoid gives a number from $0$ to $1$, and so a probability, but it is just a guess I am doing.
Moreover, it continues by saying that we can use a gaussian distribution for $P(x|C_i)$, and so $P(x|C_i)=N(\mu ,\sigma )$, and so :

I don't understand what it is doing, can somebody please help me?
I don't know if my question is clear so sorry if it is not but I am really confused. If it is not lcear please tell me I will try to edit it. Thanks in advance.
Note: if it can be useful, this has been taken from the Bishop book at page 197

My understanding is:

The first line is OK - it derives from Bayes Rule

Assume that this probability follows a logistic function, that is that

$$P(C_1|x) = \frac{1}{1+exp(-a)}$$

Then if

$$ y = \frac{1}{1+exp(-z)}$$
then
$$ z = ln(\frac{y}{1-y})$$
(some lines below but with $a$ and $\sigma$)

Therefore:

$$ a = ln(\frac{P(C_1|x)}{1-P(C_1|x)})$$

If there are only two classes then $1-P(C_1|x)= P(C_2|x)$.

Then

$$a = ln(\frac{P(C_1|x)}{P(C_2|x)}$$

Using Bayes on $P(C_1|x)$ and $P(C_2|x)$

$$a = ln(\frac{P(x|C_1) P(C_1)}{P(x|C_2)P(C_2)}$$

The whole thing now is to model $P(x|C_i)$. He assumes that it is a normal distribution. And from there you should the second set of equations. The important part (as far as I remember) is that the two covariance matrix (for $C_1$ and for $C_2$) are the same. I do not have access to the book now but I will look it up tomorrow and see if there is any difficult step in the derivations.

tldr: The whole point is he assumes that the probability as a logistic, then he gets the formula for the $a$, then he assumes a normal distribution for $P(x|C_i)$, then he assumes same covariance matrix.


Which algorithm should be used for an accurate job recommendation system I'm building a testing project to get an introduction to DS & ML.
As a person part of the working force, sometimes finding a job is harder than it should be. I thought I could built a testing project to help workers find a job that best match their interests and their skills.

I could use a classifier, but I do not know if regression if the best way to approach this as a first pass. I was thinking of a GA system that could find some way to find way to approach this over time in a X generation. Maybe this isn't the best way for recommending jobs.

What I'm looking for isn't the code for the problem but more of algorithms ideas I should take a look at and implement in any given programming language. I'm looking to implement a system that can take a peak at job descriptions, interests and skills of an individual. I think this doesn't sound too crazy to be able to train an agent and then make it look at every jobs there is on LinkedIn for instance and take a look at 'Software Engineer' and give me 45% matching and here's the three skills that you have to work on or 98% match, please apply.

You are much, much too early in your process to even begin thinking about your models. At this stage, you should be thinking about what the data looks like and how much of it do you need. For starters, what are you measuring? What is the answer that you seek? Is it job satisfaction? Likelihood to land an interview? Likelihood to get a job offer?

Once you have that, how are you quantifying that? Is it a categorical variable? A continuous variable? Then you would have to decide on the likely list of factors that go into that answer and determine how you're going to get that data. Do you need to run a custom survey? Are there other datasets that you can leverage? You need to think about all of these answers before you're ready for any sort of modeling discussion.


Random Forest workflow? I have a data-set comprised of a fairly large number of columns (over 1000) relative to the number of rows (370) that I am currently running a random forest regression on. I am a little confused with respect to the best way to go around various tasks such as feature selection and cross-validation.

At present, I am doing the following:

Splitting the data-set 90/10, and running the RF on all features using default hyper-parameters
Using Grid Search to fine tune hyper-parameters and then running an 'optimized' model with these tunings
Using 5-fold cross validation to evaluate the model
Obtaining feature importance scores to indicate which features are performing best

Given the large number of features I have, and the fairly low performance of my models, I would like to find a smarter workflow.

It seems sensible to me to use the feature importance scores to filter out poor performers, but would I do this before or after cross validation? For instance, would I, after completing the above steps and removing features below a certain thrsehold, then run steps 1-3 again? or would I use feature selection before parameter tuning? Do I only cross validate once? Are there any other obvious steps I should integrate?

I appreciate this isn't an exact science, but if there are some industry standards that I'm either getting wrong, or have missed entirely, then I would much appreciate the feedback.

370 rows is quite a few, RF does bootstrap but it is still few info.
Having too many columns will lead to a more complex model (since the algorithm will work 1 000 dimensions).

Consider doing a pipeline with all the steps and search for hyperparameters and feature selection there. https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html

You can check a simple tutorial here

In summary, try to build a pipeline with all the steps but maybe the problem is in your data 370 rows is a small sample.


How can we detect the existence of outliers using mean and median? How can we detect the existence of outliers using mean and median?

Is it really possible to detect the existence of outliers in a set of data from their feature-wise mean and median?

Suppose, I have a data set with eight features in my hand.

I have juxtaposed their means and medians row by row.

            f1       f2        f3        f4        f5        f6        f7        f8
mean      2.5000    0.1868    0.0148    0.2105    0.2088   79.6583    1.0604    0.0091
median    2.5000    0.1826    0.0001    0.0002    0.0000   -0.0000    0.0000   -0.0000

Can we tell anything about the existence of outliers from these information?

Edit:

  How are the data distributed?

Data set is considered to be normally distributed.

  Do they all have the same mean and variance?

I don't know.

No. Unless you have some other idea of the scale and spread and distribution of the feature values. You can construct data sets that have any given mean and median with no outliers or with one massive outlier.

For example, f2 looks like a well-balanced set of numbers with a very close mean and median, maybe all those values are from a N(0.184,3) distribution. Scale that up by a linear transformation to an N(X,Y) and you'll get a mean and median much like f6. Exercise: find X and Y.


How to implement HashingVectorizer in multinomial naive bayes algorithim I had used TfidfVectorizer and passed it through MultinomialNB for document classification, It was working fine.

But now I need to pass a huge set of documents for ex above 1 Lakh and when I am trying to pass these document content to TfidfVectorizer my local computer hanged. It seems it has a performance issue. So I got a suggestion to use HashingVectorizer.

And I used below code for classification(Just replacing TfidfVectorizer by HashingVectorizer)

stop_words = open("english_stopwords").read().split("\n")
vect = HashingVectorizer(stop_words=stop_words, ngram_range=(1,5))
X_train_dtm = vect.fit_transform(training_content_list)
X_predict_dtm = vect.transform(predict_content_list)
nb = MultinomialNB()
nb.fit(X_train_dtm, training_label_list)
predicted_label_list = nb.predict(X_predict_dtm)

Got error:

  File "/home/rajesh/www/rajesh/docuchief2/project/web/env/lib/python3.6/site-packages/sklearn/naive_bayes.py", line 720, in _count
      raise ValueError("Input X must be non-negative")
  ValueError: Input X must be non-negative

So I got TfidfVectorizer is calculated as per occurrence of words so it works, but HashingVectorizer logic is different which I can not figure out how HashingVectorizer will implement in MultinomialNB.

Can someone please help me with how I can solve this performance issue.
Can I use TfidfVectorizer for a huge training dataset if yes then how?
If not then how can I use HashingVectorizer here?

You need to ensure that the hashing vector doesn't purpose negatives.  The way to do this is via HashingVectorizer(non_negative=True).


Convolutional network for classification, extremely sensitive to lighting I trained a convolutional network to classify images of a mechanical component as good or defective. Though the test accuracy was high, I realized that the model performed poorly on images which had slightly different lighting.

The features that I'm trying to detect are subtle, and lighting seems to trip the model into calling a lot of false negatives. Most, if not all, of the training and test data came from images which were shot under diffused lighting. The new images that I tried the model on were shot with focused lights.

Hoping that histogram equalization(CLAHE) would help, I did equalization on
images before feeding it to the model, did this for training as well as test data. The problem then was that accuracy was high, but the model seemed to have learnt some other correlation between the images and labels. After equalization, everything in the image stands out, the defects become even more subtle and hard to detect even for human eye.

The model prediction changed even when the same component was shot with the same lighting but with the light at different orientations.
So my question is, what can I do, either at the data end or with the model, so that it becomes robust to changes in lighting(or same kind of lighting but changes to orientation of lighting)?

It's called overfitting, your model learned to predict labels based on features which are not actually responsible for particular predictions, so when they are not presented it fails to predict right. Although there are various ways to deal with overfitting (e.g. Dropout) what you seem to need is image augmentation. It is a simple yet very powerful way to train a robust neural network. For your case - just simulate different lighting conditions by e.g. increasing/decreasing pixel values randomly for your inputs for same labels. It's also common practice to add random noise, random rotations, flip etc.

For more info check this stackoverflow answer.

Also, I recently published project where I used various of Augmentation functions which you may find useful. See: https://github.com/Naurislv/P12.1-Semantic-Segmentation/blob/master/augmentation.py


How to support a dynamic shape input for tf.where() Tensorflow (tf.where) function does not support dynamic input shapes.

For example, the following function calculates the svd of a matrix A and tries to identify the singular values greater than a certain threshold to invert them while filling other values with zeros.

def func (A, threshold)
   s,u,v = tf.svd(A)
   condition = tf.greater(s, threshold)
   z= tf.where(condition , tf.reciprocal(s), tf.zeros(s.shape))
   return z

will result in an error:

  Cannot convert a partially known TensorShape to a Tensor: (?,)

Any idea how to overcome this issue?

Just do tf.zeros_like(s) instead of using the shape explicitly.


Tricky stacking models in keras I'm trying to write a model with keras, that is built as shown below:
  |                 +-----+
  +->+------+       |     |
+--->|  NN  |------>|     |
|    +------+       |     |
| |                 |     |
| +->+------+       |     |
+--->|  NN  |------>|  L  |
|    +------+       |  S  |------+-----> Output value
|                   |  T  |      |
|      ...          |  M  |      |
| |                 |     |      |
| +->+------+       |     |      |
+--->|  NN  |------>|     |      |
|    +------+       |     |      |
|                   +-----+      |
|    +-------+                   |
+----| Delay |-------------------+
     +-------+

I have several simple sequential models (marked as NN), that receive two numerical value as input, they calculate some other numeric values (one per each network). These values are passed to LSTM network, which produces a single value as an output and this value additionaly is passed to initial networks (possibly with some delay) as one of two inputs. I work with time series, so calculated final value is passed to network alongside with the next time series value.
I use LSTM to store some "state". It is not quite difficult to build separate "sub-models", but I don't realize, how to join them together as I need, i.e. how to make a final output to be passed to initial networks and how to stack them in the described way (not as a chain).
What I've found: I found keras.layers.Concatenate, but it seems not to be what I'm looking for... But maybe (I hope) I'm mistaken.

You can use the below type code.
import tensorflow as tf
from tensorflow.keras.layers import Input, Dense, LSTM, Concatenate
from tensorflow.keras.models import Model

# input of first NN
input_l1 = Input(shape=(2,))
out_l1 = Dense(1)(input_l1)

# input is 2nd NN
input_l2 = Input(shape=(2,))
out_l2 = Dense(1)(input_l2)

# concat layer output shape will be (None, 2) becuase we concatinated 2 dense layer outputs/
concat_vec = Concatenate()([out_l1, out_l2])

# we need 3d input to LSTM i.e. ( Batch_Size, no of time steps, feature space)
# We have 2 inputs so expanded dim to (None, 2, 1)
expanded_concat = tf.expand_dims(concat_vec, axis=2)

# LSTM
lstm_out = LSTM(15)(expanded_concat)

model = Model(inputs=[input_l1, input_l2], outputs=lstm_out)


Transfer learning VGGish (AudioSet). Impact of zero padding to fit the input size I am trying to train a network on top of the VGGish architecture (https://github.com/tensorflow/models/tree/master/research/audioset/vggish), using (transfer learning) finetuning.

I initially started out with just finetuning the embedding layers, and training from scratch a simple MLP with some regularization techniques. (2 hidden layers FC with ReLU, BatchNorm and dropout, final output currently is FC to 4 classes). It is able to get an accuracy of 1 on the training data.

The input of the VGGish network are spectograms. My collected audiofiles are pretty low quality OGG/vorbis with a sample rate of 8000Hz. This is lower than the audio used in VGGish, therefore, if I create the log-mel spectogram using their code, every spectogram is missing some bins. (see bands 48 and up in the image below, y-axis) note that this happens in every image, so my hypothesis is that this shouldn't really matter. It is essentially just padding to fit the input size of the VGGish network.

Will this have any negative consequences on the ability of the network to learn using my dataset? Do I also need to finetune the ConvLayers earlier on in the network?

Any other thoughts on how I can work around this (potential) issue?It is not feasible to increase the frequency of the recordings, as they are being streamed from low power devices.

I do not have enough (annotated) data to fully retrain, so training a model from scratch is not feasible.

Suggestions on other pretrained networks are also welcome!

Thanks :)

Will this have any negative consequences on the ability of the network to learn using my dataset?

That's really hard to say, but I suspect it does not make a difference for any sounds that aren't exclusively defined by higher frequencies.

As a rule of thumb: If a human can still correctly identify a sound sampled at 8kHz, so should a machine. If the machine cannot do it, because of the missing higher frequencies, it may not have learned the characteristics of the sound to identify in the first place, but something else.

  Do I also need to finetune the ConvLayers earlier on in the network?

I'd start with not re-training those. Having been trained on AudioSet, they should be fairly universal.

  Any other thoughts on how I can work around this (potential) issue?It is not feasible to increase the frequency of the recordings, as they are being streamed from low power devices.

Not really. You could upsample your audio to 22.05kHz and go from there—but you cannot create new information for those higher bands out of thin air. So, I don't recommend upsampling.

  Suggestions on other pretrained networks are also welcome!

Read Look, Listen, and Learn More: Design Choices for Deep Audio Embeddings by Cramer, Wu, Salamon, and Bello. They offer an open, pre-trained embedding model for environmental sounds at https://github.com/marl/openl3 (which, IIRC, is better than VGGish).

You might be able use its output as input for a shallow dense network to train a classifier for your audio (unless of course you way too few samples).


How can we handle cases were the quality of the prediction undermines the quality of the data? Consider the following example:

You have a rare disease whose occurrence seems to depend on a certain number of variables. You build a model which tries to predict patients most likely to be effected by the disease which is partially successful, that is, it predicts likely onset of the disease with some accuracy, but less than desired. Ideally, you update your model as more historical data on patients with this disease comes in, and the accuracy starts to improve.

Eventually you start notifying high risk patients and provide them with steps to counter this disease. Because of this, more and more patients who would have been classified as high risk are actually not catching the disease and therefore decreasing the accuracy of your model. In a sense the model was a 'victim' of its own success.

Are there any strategies for dealing with such prediction scenarios: Where a model designed to predict an undesirable outcome looses accuracy due to it successfully averting the outcome in real world cases?

Because of this, more and more patients who would have been classified
  as high risk are actually not catching the disease and therefore
  decreasing the accuracy of your model.

I'd rather challenge the assumption and say the algorithm didn't miss in this case: it correctly identified a high risk case, which was its original purpose. You're not building a model to identify people who will catch the disease, but to prevent the disease of actually happening.

Always think hard about which metrics make sense and what they represent. If you build a confusion matrix:

Predicted \ Actual | Caught the disease | Did not catch the disease
High risk          | A                  | B
Low risk           | C                  | D

Now, when you use accuracy, you're saying A and D are good, B and C are bad, all with the same weights. But is B really bad? Isn't C way worse than anything else? Or depending on the cost to prevent the disease, maybe B is worse? I don't know, it requires domain knowledge. But point being, using accuracy mindlessly is not a good way of dealing with this situation: adjust your metrics to your problem.


How to train neural network using maximum likelihood principle I often stumble upon papers saying that the NN was trained using maximum likelihood principle. I think I generally understand what maximum likelihood is. But I have trouble understanding how to apply it to neural network training.

How are neural network trained using maximum likelihood principle?

This is pretty common.

The last step for a simple deep neural network would be to determine a way to choose values of $\theta$ that maximize the likelihood of your training data. So, what do you do? Typically, you write a log-likelihood function, and then find the values that maximize it.

In fact, it's relatively trivial to prove that minimizing the loss is equivalent to maximizing the likelihood of the input. Typically, in academic papers, you may see people prefer maximizing likelihood simply for the mere fact that some proofs are easier that way.

Example

Consider the Binary Cross Entropy formula $$CE(y,x,\theta) = - \sum_{i=1}^{n} y_{i} log{p_{\theta}}(y \mid x_{i})+ (1-y_{i})log(1-p_{\theta})(y \mid x_{i})).$$  Naturally, you would optimize this with respect to $\theta$.

We can instead choose to take a probabilistic approach and maximize the likelihood of the data under some probabilistic model.  A natural choice would be to use a Bernoulli distribution, such that $$p(y \mid \pi) = \prod_{i=1}^{y_{i}} \pi_{i}^{y_i}(1-\pi_{i})^{1-y_{i}}.$$

Now, the task here is to train a neural network to estimate $\pi$. The likelihood function is simply $$p(y \mid x, \theta) = \prod_{i=1}^{y_{i}} p_{\theta}(y \mid x_{i})^{y_i} (1-p_{\theta}(y \mid x_{i}))^{1-y_{i}}.$$  We want to maximize the function w.r.t $\theta$.

Now, if you take the log, and we find $$\sum_{i=1}^{n} y_{i} log{p_{\theta}}(y \mid x_{i})+ (1-y_{i})log(1-p_{\theta})(y \mid x_{i})).$$  Thus there is literally no difference in minimizing a cross entropy or maximizing the log likelihood.

Therefore, the only difference is the approach. At the end of the day, you're doing the same thing regardless what you choose to call it.


Q-Learning experience replay: how to feed the neural network? I'm trying to replicate the DQN Atari experiment. Actually my DQN isn't performing well; checking another one's codes, I saw something about experience replay which I don't understand. First, when you define your CNN, in the first layer you have to specify the size (I'm using Keras + Tensorflow so in my case it's something like (105, 80, 4), which corresponds to height, width and number of images I feed my CNN.). In the codes I revisited, when they get the minibatch from the memory, I see they usually fed the CNN without "packing" it on 4 batches. How it is possible? I mean for example if you get 32 random sampled experiences, don't you need to make batches of 4 before feeding it?
Here are an example of what I'm saying: https://github.com/yilundu/DQN-DDQN-on-Space-Invaders/blob/master/replay_buffer.py
https://github.com/yilundu/DQN-DDQN-on-Space-Invaders/blob/master/deep_Q.py
In this code, that's how he/she stores the experiences:

def add(self, s, a, r, d, s2):
        """Add an experience to the buffer"""
        # S represents current state, a is action,
        # r is reward, d is whether it is the end,
        # and s2 is next state
        experience = (s, a, r, d, s2)
        if self.count < self.buffer_size:
            self.buffer.append(experience)
            self.count += 1
        else:
            self.buffer.popleft()
            self.buffer.append(experience)

Then when you need to use them:

 def sample(self, batch_size):
        """Samples a total of elements equal to batch_size from buffer
        if buffer contains enough elements. Otherwise return all elements"""

        batch = []

        if self.count < batch_size:
            batch = random.sample(self.buffer, self.count)
        else:
            batch = random.sample(self.buffer, batch_size)

        # Maps each experience in batch in batches of states, actions, rewards
        # and new states
        s_batch, a_batch, r_batch, d_batch, s2_batch = list(map(np.array, list(zip(*batch))))

        return s_batch, a_batch, r_batch, d_batch, s2_batch

Ok, so now you have a batch of 32 states, actions, rewards, done and next states.

This is how you feed the state batch (s_batch) and next state batch (s2_batch) to the CNN:

def train(self, s_batch, a_batch, r_batch, d_batch, s2_batch, observation_num):
        """Trains network to fit given parameters"""
        batch_size = s_batch.shape[0]
        targets = np.zeros((batch_size, NUM_ACTIONS))

        for i in range(batch_size):
            targets[i] = self.model.predict(s_batch[i].reshape(1, 84, 84, NUM_FRAMES), batch_size = 1)
            fut_action = self.target_model.predict(s2_batch[i].reshape(1, 84, 84, NUM_FRAMES), batch_size = 1)
            targets[i, a_batch[i]] = r_batch[i]
            if d_batch[i] == False:
                targets[i, a_batch[i]] += DECAY_RATE * np.max(fut_action)

        loss = self.model.train_on_batch(s_batch, targets)

        # Print the loss every 10 iterations.
        if observation_num % 10 == 0:
            print("We had a loss equal to ", loss)

In my code (https://bitbucket.org/jocapal/dqn_public/src/master/Deimos_v2_13.py) I get a batch of 32 experiences; then make small batches of 4 experiences and feed the CNN. My question is: am I doing it wrong? And if so, how can I feed 32 experiences when my CNN is waiting for 4 experiences?

Another example of what I'm saying: https://yanpanlau.github.io/2016/07/10/FlappyBird-Keras.html

Input is a 4D tensor [batch_size, height, width, channels] . Single state is already 4 frames stacked together so when you sample a state from the experience replay you sample a 3D tensor [height, width, channels]. When you sample 32 states you actually sample 32 of those 3D tensors and feed them directly to the network. For more details on preprocessing refer to the page 6 of the original DQN paper found here.


Neural network or other algorithms? I have a regression problem, with a million rows or so, around 10-15 features.
What should work better on that particular setting? Neural network or regular regressors?

This is more of question how to select the correct machine learning algorithm, I would refer you to the following blog Which machine learning algorithm should I use?

Regression Algorithms models the relationship between variables that is
iteratively refined using a measure of error in the predictions. Most popular examples are:

Ordinary Least Squares Regression (OLSR)
Linear Regression
Logistic Regression
etc  ...

On the other hand, Artificial Neural Networks models are inspired by the structure and/or function of biological neural networks. "Neural networks currently provide the best solutions to many problems in image recognition, speech recognition, and natural language processing." Neural Networks and Deep Learning/.  Neural Networks are hard to train; thus my recommentation not to start with Neutral Network.


Datasets with 50+ attributes and 2+ tables I'm looking for a data set to use in my Master Thesis. I've looked on Kaggle but all of them seem to be smaller data sets. Ideally it would be CSV files with 50+ attributes and 2+ tables, but not much bigger than that as it still needs to be appropriate for a Master Thesis.
Could anyone recommend websites to look for such (free to use) data sets?

You can find the dataset from the following links.

IEEE Dataport (https://ieee-dataport.org/datasets)
https://datasetsearch.research.google.com/


Ways to simulate weather data over several periods (Python or R)? I have a time series dataset that has several variables for a state/province for fixed periods of time. That is for state A, there are samples from April 2017 to July 2019. Of course, I thought adding precipitation and temperature variables would be a great idea. I tried finding some relevant external data but most of it is abstract and spread out. How would one simulate dynamic data in Python with varying means, highs and lows for say six months on a daily basis, taking into account average temperatures/precipitation for each month?

So if I have the mean temperatures (C) as below for state A:

year    Jan   Feb   Mar   Apr   May   Jun
2017    5.5   6.0   12.0  15.0  20.0  25.0

I would like data to be simulated as below without really doing for each month since that would make the whole task very tedious:

Duration     Temp
2017-01-01   5.0
2017-01-02   5.1
2017-01-03   4.9
.
2017-03-01   7.8
2017-03-02   9.0
2017-03-03   9.5
.
2017-06-30   26.7

Are there ways to achieve this in Python (or R)?

What about creating a Pandas DataFrame and adding a new column such as "Temp_simulated" and simulate the temperature?


regression with noisy target vairable How can I approach a regression problem where the input data is not noisy but the target variable is noisy? Are there any regression algorithms that are robust to a noisy target variable?
Also, is it possible to de-noise the target variable somehow? If so, how?

It depends how much noise:

If it's only a little noise, say for instance 2% of the target values are off by a small value, then you can safely ignore it since the regression method will rely on the most frequent patterns anyway.
If it's a lot of noise, like 50% of the target values are totally random, then unless you can detect and remove the noisy instances you can forget it: the dataset is useless.

In general ML algorithms are based on statistical principles, to some extent their job is to avoid the noise and focus on the regular patterns. But there are two things to pay attention to:

Is the noise truly random, or does it introduce some biases in the data? The latter is a much more serious issue.
Noisy data is even more likely to cause overfitting, so extra precaution should be taken against it: depending on the data, it might be necessary to reduce the number of features and/or the complexity of the model.


how to implement a hierarchical clustering technique using parallel execution in R In R, currently to implement wards method for hierarchical clustering, I use the following code - results <- hclust(data, "ward.D2").
However as my data size has increased exponentially, I wanted to implement parallel execution.

Since I found no package, I tried to re-write the hclust method, however most of its code is written in fortan.

Does there exist any way to implement parallel execution here?

The complexity of that algorithm is O(n³), and it needs O(n²) memory.

So if your data grows "exponentially", you better settle for a sampling-based approach!

Seriously: benchmark the run time and memory requirements for 5k, 10k, 20k, 40k, 80k instances. You should be able to observe something between O(n²) (for computing the distance matrix) and O(n³) for the clustering. Then estimate how long this will take (and how much memory you'll need). Even if you are very optimistic, divide this by the number of threads M you can run. A more realistic value will be M/2 if you can run M threads.

When you are really convinced that you'll be okay with the resulting runtime, just implement the algorithm yourself. It is a very simple algorithm. No need to study the Fortran code. But most likely your code will be 2x slower (20x if using pure Python or R) than a well written old Fortran code...


How to get similar visualization to R fviz_cluster function in Python? It looks like R has some cool visualization function for clusters that gives output like this:

The input is 2D Points and labels for them.
How can i get same visualization in Python?

In Wolfram Language you may use FindClusters to identify the clusters, ConvexHullMesh to calculate the convex hull of the clusters, and ListPlot with the Prolog option and Labeled to display.

First we will collect some example data from the USA "AdministrativeDivision" states.  The "Name" property includes the county so this is removed with StringDelete.

data = Select[FreeQ[_Missing]]@EntityValue[
    EntityClass["AdministrativeDivision", {"ParentRegion" -> Entity["Country", "UnitedStates"]}],
    {"HomeOwnershipRate", "HealthInsuranceCoverageRate", "Name"}];
data = MapAt[StringDelete[", United States"], {All, -1}]@data;
First@data

{70.7% , 85.2%, "Alabama"}

For each state we get two "Percent" Quantity objects and the state name.

We can FindClusters on this list by using all but the last item in each list (Most) for the cluster algorithm but returning the entire list (Identity), which includes the name, as the cluster item.

clusters = FindClusters[Rule @@ Through[{Most, Identity}@#] & /@ data, Method -> "MeanShift"];

The hulls of clusters with ConvexHullMesh.

hulls = ConvexHullMesh /@ Map[Most] /@ clusters;

Then ListPlot clusters with each point Labeled while using Prolog to plot the hulls beneath the points.

ListPlot[
 Map[Labeled[Most@#, Last@#] &] /@ clusters,
 PlotStyle -> ColorData[110],
 PlotTheme -> "Detailed",
 FrameLabel -> {"HomeOwnershipRate", "HealthInsuranceCoverageRate"},
 Prolog -> {
   Opacity[.1],
   MapIndexed[
    {ColorData[110] @@ #2,
      EdgeForm[{Thin, ColorData[110] @@ #2}], #1} &,
    hulls
    ]}
 ]

The default behaviour is to automatically use Tooltip in places where labels would overcrowd.

Hope this helps.


How to calculate mean and standard deviation of all features in a class identified by k-nearest neighbors? I have classified my data into several neighborhoods using k nearest neighbors. I need to efficiently calculate the mean and standard deviation for all features of data points belonging to a particular neighborhood.
I am using sklearn.kneighbors.

If you append the predicted neighbourhood onto your data df (let's call this neighbourhood), then using groupby and transform within a loop should do the trick.

As an example:

features = [var_1,var_2,...] # a list of the features to run over
for col in features:
  df[col+'_mean'] = df.groupby('neighbourhood')[col].transform('mean')


Balancing XGboost still skews towards the majority class I have unbalanced dataset for multiclass classification and I tried to use the class weights option in XGboost and the classifier still tends to favor the majority class. I am not sure if I need to tune something else or how I should approach this.
If the algo was predicting all over the place I could still understand but not sure why it still skews towards the majority class. Any pointers?
Update: What I mean is that majority of the predictions are still for the major class which is roughly 30% in both test and train

That is completely normal. You should remember that the model will basically learn a statistical function given by your data (Intuitively), and since your data is skewed, it will learn by the majority class.

To overcome that, you can treat the imbalance characteristic of data set using two types of approaches: sampling and cost-sensitive learning. Basically, in sampling techniques we have undersampling and oversampling, that is, we can reduce our data set to balance data or increase it with artificial methods. Some examples:

Random Oversampling: essentially, this technique increases minority class by randomly sampling it.
SMOTE: random oversampling has a overfitting problem as we are just replicating data considerably. To overcome that, smote works, in a simplified way, interpolating points on the feature space for nearest neighbors in minority classes.

For undersampling techniques:

Random Undersampling: this method simple undersample the majority class. We can loss information, but if points are close, it can give good results
NearMiss: In a simplified way, this model selects the closest points to the minority class.

In cost-sensitive learning techniques, we use cost matrix 'punishing' models that give wrongs score to a specific class. For example, let's say you want to give better results for class C1, you can create a matrix where the misclassification of C1 is higher than other classes.

In python we have two excellents libraries to deal with this problems:

https://imbalanced-learn.readthedocs.io/en/stable/api.html
https://wwwen.uni.lu/snt/research/sigcom/computer_vision_lab/costcla_a_cost_sensitive_classification_library_in_python

They have excellent tutorials on the topic so you can get a deeper overview of methods discussed and others ones. I hope it helps.


Drawing 1D CNN architecture How can I draw CNN Architecture like this one here:

You can find a python script over here: https://github.com/gwding/draw_convnet

I hope it's helpful


Pre-trained CNN for feature extraction I'm trying to classify images. I want to run each image through a pretrained CNN to apply convolution and pooling and end up with a smaller picture/matrix where the value of each pixel is a feature. Then I want to pass that to an SVM for classification.

Can someone provide code to get started with feature extraction with CNN and SVM ?

Here's some starter code:

from keras.applications.vgg19 import VGG19
from keras.preprocessing import image
from keras.applications.vgg19 import preprocess_input
from keras.models import Model
import numpy as np

# define the CNN network
# Here we are using 19 layer CNN -VGG19 and initialising it
# with pretrained imagenet weights
base_model = VGG19(weights='imagenet')

# Extract features from an arbitrary intermediate layer
# like the block4 pooling layer in VGG19
model = Model(inputs=base_model.input, outputs=base_model.get_layer('block4_pool').output)

# load an image and preprocess it
img_path = 'elephant.jpg'
img = image.load_img(img_path, target_size=(224, 224))
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
x = preprocess_input(x)

# get the features
block4_pool_features = model.predict(x)

You can then use these features for passing to a SVM classfier.
Here are some additional links for reference:

Understanding CNNS

Keras pretrained networks

Coding a convolutional neural network in keras


What is normalization for? I am new in python and data science (and not great in math). I am learning machine learning. I got following normalize function. Can you please explain what does this normalize function do?

def normalize(array):
    return (array - array.mean()) / array.std()

Also please explain what this array - array.mean() does?

Also please explain what this array - array.mean() do?

Basically, it is doing memberwise subtraction operation after broadcasting. np.mean function finds the mean in your array and its result will be a scalar, a single number. Your array is a numpy array and the result of the latter term is a single value as mentioned. Consequently, the single value gets extended to the shape of the former term. then a memberwise subtraction will be performed for each entry of the array and the result will have the same shape as the former term.

  Can you please explain what does this normalize function do?

Normalizing data is done for accelerating optimization. If you have features with different scales, it will take too much time for your optimizer function to find optimal points. Suppose you have age feature which can change between 0 to 150 (!) and salary which can be changed from 0 to whatever, like 500,000,000 $. your optimization algorithm used in your ML model will take too much time, if possible, to find appropriate weights for each feature. Moreover, if you don't scale your data, your ML algorithm may take too much care to features with large scales.


Best anomaly detection algorithm based on two conditions Choosing the right anomaly detection algorithm seems quite hard at the moment.
It might be because I am bombarded with so many alternatives likes clustering, K-Means DBSCAN and so many others.

On my side I have a csv files with thousands of lines, the columns have header that either show the name of the file and the features.

I’m my case this is how the file would look like, keep in mind the file is way larger.

Values columns is the one I want to check about anomalies, in this case I would get a lot of anomalies because the numbers belong to different units-

So first I would need to filter Unit to lets say meters and then check values column data for any anomalies

I would appreciate some advice on what would be the best approach to tackle this problem

Since your data is one-dimensional and numeric, I don't think you need any fancy clustering technique.  Clustering is useful when your data points have multiple attributes.  When there is just a single attribute, then all you need is a good definition of "anomaly".

For example, suppose you decide that an anomaly is any point more than two standard deviations from the mean.  It's easy to find such points using pandas:

import numpy as np
import pandas as pd
from scipy import stats

threshold = 2.0  # in standard deviations
input_file = "path/to/my/file.csv"
target_unit = "meters"

# read the file into a pandas DataFrame
df = pandas.read_csv(input_file)
# filter to only include the target unit
df = df[df['Units']==target_unit]
# compute z-scores for the `values` columns
df['values_z'] = np.absolute(stats.zscore(df['values'].values))
# threshold z-score to identify anomalies
is_anomaly = df['values_z'] > threshold
anomalies = df[is_anomaly]

# `anomalies` is now a DataFrame that contains the anomalous points
# you can consume this however seems appropriate
# for example, you can write the anomalies to a separate file:
anomalies.to_csv('path/to/output.csv')

```


Covariate shift detection Is there any standard approach for detecting the covariate shift between the training and test data ? This would be useful to validate the assumption that covariate shift exists in my database which contains a few hundred images.

There are methods like the Kullback-Leibler divergence model, the Wald-Wolfowitz test for detecting non-randomness and covariance shift.

A simple test for quick analysis of covariance test would be to build a machine learning model, where the model is repeatedly tested with inputting training data and the production data.

In case, the model can make out the difference between the training and production datasets then it can be a sign of covariance shift.


Import data from google drive to Kaggle Kernel I want to import a csv file from google drive . I tried using the link in add dataset tab but it is taking some thing else as "Open". Please see the image.

Go to your Google Drive and right click the dataset you want to upload to Kaggle.
Generate the shareable link and the code that comes after https://drive.google.com/open?id=, it should be a long code (like 3tlxSE__5eL3q7Zb31M3CEKHlieGWZYgM)
Insert the long code at the end of https://drive.google.com/uc?export=download&id=
Copy that link. Should look something like https://drive.google.com/uc?export=download&id=3tlxSE__5eL3q7Zb31M3CEKHlieGWZYgM
When uploading a dataset to Kaggle, select to add a file from remote URL.
Insert copied URL.
Done!


What algorithms are good to predict next numbers? Let's consider we have several hundreds of numbers like ( 1, 2, 5, 8, 7, 15, 19, 8, 4, 6, ...) those are closed numbers of a stock on consecutive days for example. I like to know what algorithms are good to predict next numbers?  Is it possible to find a relationship between this sequence of numbers or we need extra input data to make a prediction?

I am completely new on this and seeking some ideas and guidance for a new comer in data science field(although I have some experience in machine learning and image and voice recognition).

Hi this can be approached as a timeseries regression problem. Deep learning (tools like Tensorflow or Keras a higher level API) can be very effective in this. It depends on many factors like the amount and quality of your data, the architecture, the complexity of the problem and hyper parameters whether it will be useful to you. machinelearningmastery.com has many examples of LSTM have a look at it! Maybe start with this. As you can see in the examples in 2 having the numbers you mentioned and the sequence in which they occured could be enough to predict the next number(s).  Usually adding more information to your neural network should make it improve.


"Each agent was evaluated every 250,000 training frames for 135,000 validation frames" What does this sentences stands for? in DQN nature paper? In nature paper of DQN by DeepMind, DQN is compared to linear function but they does not said what is this linear function? They compared with some linear functions?

0- What is the meaning of this sentences "Each agent was evaluated every 250,000 training frames for 135,000 validation frames" in DQN nature paper?

2- What about meaning of this sentences:  "Note that these evaluation episodes were not truncated at 5min leading to higher scores on Enduro"

1- Does this correct:we just put a network with an input and an output?
I do not sense why the performance of it is very how?

TheperformanceoftheDQNagentiscomparedwiththeperformanceofalinearfunctionapproximator on the 5 validation games (that is, where a single linear layer was used instead of the convolutional network, in combination with replay and separate target network). Agents were trained for 10 million frames using standard hyper parameters,and three different learning rates. Each agent was evaluated every 250,000 training frames for 135,000 validation frames and the
 highest average episode score is reported. Note that these evaluation episodes were not truncated at 5 min leading to higher scores on Enduro than the one s reported in Extended Data Table2. Note  so that the number of training frames was shorter (10 million frames) as compared to the main results presented in Extended Data Table 2 (50 million frames).

0: After 250,000 steps in the training loop they start a validation environment to check the current performance without fitting the weights of the current neural networks.

2: One episode of a validation episode just takes at most 135,00 frames (steps) and were not truncated by time.

1: Some games are more difficult to learn than others and the DQN approach is more suitable to some games than to others.


How do you avoid 'analysis paralysis' when choosing a method to implement? When you have multiple methods to accomplish a task, how do you choose which one to implement?

This is a very broad question but in general you can define some criteria that you want the method to meet (eg low error, efficiency, scalability) and then you score the methods, which one solves your problem best.


Make use of relationships on recommendation systems I have a data set of user rating for movie as

user_name, product_name, user_rating

and I am using this data to recommend new movie to user (collaborative filtering).

I also have another data set which show relationships between user like

user_1, CO_WORKER, user_2
user_1, FATHER_OF, user_3
user_3, FRIEND_OF, user_4 etc

Now what are the ways to include 'relationship data' along with 'ratings data' to produce better recommendations ?

example : user_1 might like the movies which user_2 likes since they are co_workers

There are a number of ways to make use of this information, including various graph-based methods of recommendation.
Collaborative Filtering using Weighted BiPartite Graph
Projection
A Recommendation System for Yelp
It's also common to use multiple approaches to build a hybrid recommendation system.


Historical exchange rate data via World Bank API? I can see a massive list of indicators available via the World Bank API here (may take 1-2 minutes to load)
I found exchange rates for various currencies into USD, but cannot see how to get exchange rates between currencies which aren't the USD.
Is this possible via the World Bank API?

I don't think so, since:

USD is usually used as a reference.
Providing exchange rates for all possible pairs would be to much data. If there are 100 countries, there would be 100! (factorial) different pairs.
Exchange rates for all different pairs can be calculated manually by using the USD rates of two pairs.


Cannot understand feature extraction I'm following an AI course and we've just entered the deep learning chapter. Speaking about the difference between classic machine learning models and deep learning, it turns out one of the most relevant points in favor of a neural network is that it doesn't need the feature extraction phase. That's because it can implicit learn what are the most important features directly on data we give it as input. Conversely, a machine learning system heavily relies on feature extraction before it can start training.
I've found some useful questions about this topic on this site, but still I'm a bit confused in understanding what feature extraction actually is. So I've asked my teacher to provide me an example.
He gave me this example: suppose we have a huge dataset of images and we want to classify them based on the presence or absence of at least one tree in the picture. So, an image will be classified as 1 if it contains a tree, -1 otherwise.

in the case of a neural network, we simply feed the images to the net with their labels. That's all. The network will learn what a tree is and will optimize its weights in order to detect them
on the contrary, if we want to use a machine learning algorithm, we also have to tell the model what a tree actually is, in addition to the labeled images. This part is the feature extraction phase

I don't understand how can I tell to a model what a tree is inside an image. I thought discover it is part of the task of the machine learning algorithm.

Preliminary note
as far as I know, people may use the term feature extraction in slightly different ways:

referring to automatic methods used for dimensionality reduction which involve transforming the features (and not only selecting a subset of features).
referring to the general process of designing and engineering the features before training/testing a model on these features.

Please be aware that I'm not 100% sure that the terminology I use myself is standard. (but I don't think it matters for the question you're asking).
Answer
Used in the context you mention, the term certainly refers to the general process of feature engineering (2nd point above).
In traditional ML models the model directly uses the features exactly as they are provided to the training algorithm. For example, a decision tree classification model tries to find the best conditions in the features to discriminate the data. This means that the algorithm selects a single feature and directly uses its range of values to build a condition, something like this:
 if color == 'green' then go to subtree 1 else go to subtree 2

Thus the type of information that can be used by a model is strictly limited by the algorithm and the features which are provided to it: there's no way to consider any information which is not directly available as a feature. As a consequence the stage of feature engineering is crucial. For example in text classification there are a lot of choices to make: provide the words frequency or TFIDF as features? Provide all the word or remove stop words? Lemmatize or not? Add POS tags? And these are just some very standard questions, specific tasks may require a very detailed analysis. Importantly, it's not possible to just provide all the features one can think of because this would almost certainly cause the model to overfit due to the high number of features. Generally in this stage the goal is to give the model the best and most precise clues that will help it make correct predictions, and also avoid giving it too many irrelevant features to prevent overfitting.
That's the stage that DL doesn't need: thanks to a much more complex architecture and algorithmic process, a DL model can itself select and combine the features it needs to perform well.

I thought discover it is part of the task of the machine learning algorithm.

Sure, but in traditional ML in order for the model to discover the most important patterns for a specific task, the features have to be prepared in a way which maximizes the chances of the model to find these. Simile: if a teacher tells their students to revise page 63 of the textbook for the test, it's likely that the students will perform better than if they have no clue about the topic and need to revise the whole textbook.


How can I create a negation of the sentence? What are some ways that I can generate the negation of a sentence such that the output sentence reflects the negation of original sentence?

If the sentence is: That is an apple.

Then, the expected negation could be: That is not an apple.

Another example:

Sentence: Rita is so hot.
Negation: Rita is boring.

You could look in to Style Transfer. This is an active field of research.


Publicly available news APIs/datasets? In addition to our list of publicly available datasets, I'd like to know if there is any list of publicly available news datasets/crawling APIs. It would be very nice if alongside with a link to the dataset/API, characteristics of the data available were added. Such information should be, and is not limited to:

the name of the news network / news aggregator;
what kind of news information it provides (title, snippet, full-article, date, author, url, ...);
whether it allows for crawling its contents via an API (and rate: 10/min, 1k/month, ...);
whether it simply provides a snapshot of the whole dataset.
news sources covered
date range of news content coverage

Any suggestions and further characteristics to be added are very welcome.

There is GoogleNews package for Python. It allows queries by data range, keyword, and language.


What is the difference between Hadoop and noSQL I heard about many tools / frameworks for helping people to process their data (big data environment).

One is called Hadoop and the other is the noSQL concept. What is the difference in point of processing?

Are they complementary?

Hadoop is not a database, hadoop is an entire ecosystem.

Most people will refer to mapreduce jobs while talking about hadoop. A mapreduce job splits big datasets in some little chunks of data and spread them over a cluster of nodes to get proceed. In the end the result from each node will be put together again as one dataset.

Let's assume you load into hadoop a set of <String, Integer> with the population of some neighborhoods within a city and you want to get the average population over the whole neighborhoods of each city(figure 1).

figure 1

    [new york, 40394]
    [new york, 134]
    [la, 44]
    [la, 647]
    ...

Now hadoop will first map each value by using the keys (figure 2)

figure 2

[new york, [40394,134]]
[la, [44,647]]
...

After the mapping it will reduce the values of each key to a new value (in this example the average over the value set of each key)(figure 3)

figure 3

[new york, [20264]]
[la, [346]]
...

now hadoop would be done with everything. You can now load the result into the HDFS (hadoop distributed file system) or into any DBMS or file.

Thats just one very basic and simple example of what hadoop can do. You can run much more complicated tasks in hadoop.

As you already mentioned in your question, hadoop and noSQL are complementary. I know a few setups where i.e. billions of datasets from sensors are stored in HBase and get then through hadoop to finally be stored in a DBMS.


How to feed two (or more) time-series data sets to a neural network? I want to train the network based on two sets of data. For example, I want the network to predict the humidity based on past humidity trends AND past temperature trends. In this case, how should I organize the input layer? I would think that for just one time-series, I would use a regular sliding window on the series. With two series, do I just present both windows (from the humidity and temperature series) to the input layer that is twice the size of the window? if not, how else can I configure the input layer so that it doesn't confuse between the two sets of training data?

Do I just let the network sort this out by itself or is there a preferred method of presenting two (or more) sets of training data to the network? Thanks.

The whole idea of NN is that it learns by itself to give the appropriate weights and biases to the input neurons, so yes - just give your network the 2 series and it should return the appropriate weight and bias for each neuron in the input.


How many coefficients does the Logistic regression model has as a function of the number of features? I have built a logistic regression model using Python anaconda and was surprised to see that the number of model coefficients turned out to be proportional to the training sample size i.e.

My training data is:

print('Training data is type %s and shape %s' % (type(os_X_train), os_X_train.shape))

and outputs:

Training data is type <class 'pandas.core.frame.DataFrame'> and shape (174146, 11)

Then the model is:

logreg = LogisticRegression(penalty='l2',solver='lbfgs',max_iter=1000)
model = make_pipeline(preprocess, logreg)
model.fit(os_X_train, os_y_train)
print(logreg.coef_.shape)
print("Model coefficients: ", logreg.intercept_, logreg.coef_)

This outputs:

(1, 153024)
Model coefficients:  [12.02830778] [[ 0.42926969  0.14192505 -1.89354062 ...  0.008847    0.00884372 -8.15123962]]

To my understanding the number of model coefficients should be the number of columns for the predictor variables or features plus one the intercept, or?

You are correct that the number of parameters depends on the number of features and it of the number of observations. You are seeing this error because you and the computer do not agree on what your input data mean.

Sklearn assumes your data to be $n\times p$, where each row represents an observation, and each column represents a variable. I think you’re doing the transpose if that, $p \times n$, where each column represents an observation. As you have more and more observations, you have more and more columns, telling sklearn that there are additional features.

If you transpose your predictor data, the functions should behave as expected.

This isn’t unique to logistic regression. You would have this happen with any model. Say you trained a k-NN on 80 observations of 6 features. Then you test on 20 observations of those 6 features. Python should come back like, “You gave me 80 features for training and now only 20 for testing. What gives?”


Orange3 does not show Geo Map tool on OS X I am using Orange 3.3.11 and I cannot find "Geo Map" widget on OS X.

But the documentation says there is such a widget. How can I find this widget?

Take a look if you have the widget "Map" in your Visualization group. It might be the same widget, just with the different name.

It looks "Geo Map" widget appears for Orange 3.4.2 for Linux, in Visualize group:

However, from this commit in Orange source code, it looks the previous name for this widget was just "Map".


Data Science education curriculum design and guidelines in Computer Science and other Disciplines I am studying data science education curriculum design and guidelines. All resources I have up to now is about data science in statistics education and library and information science education.

Is there any work on data science in computer science/software engineering education (adding data science courses to a traditional software engineering curriculum) ,management and other disciplines?

An example of including the curriculum for statistics is this paper

Yes, data science work do exist in domains of management and computer science.

I think data science is a huge domain in itself to include in any of the curricula. So, I would explain how data science is used in the domains:

Management:
Data Science used in a lot of ways. I use data science and analytics in every phase of the customer acquisition and retention funnel; which includes designing models for using the customer event tracking data and identify patterns and predict churn. Here is an example article from the Moz blog, which explains about the use of deep for churn prediction and customer retention.
Computer Science:
Most of the data science has roots in computer science basics. However, including the concepts of distributed computing would help the students understand and appreciate the concept of Big Data, when they later enter the industry.

So, I think data science is a huge domain in itself, and has it's share of unique applications in a wide range of domains. However, bulk of data science concepts and techniques have taken roots in math, stats and computer science basics and concepts. So, I think the knowledge of those domains would be helpful and suffice.

So, these are some links which can help in curriculum re-design for accommodating data science concepts in the management and CS curricula:

MBA speciality.
Curriculum of the Data Science and Business Analytics program at the Santa Clara University.
Curriculum of the Masters of Data Science in Virginia University.

The third bullet is something very close to how data science can be included or blended with the Computer Science degree.


Suspiciously low False Positive rate with Naive Bayes Classifier? I am performing phishing URL classification, and I am comparing several ML classifiers on a balanced 2-class data-set (legitimate URL, phishy URL).

The ensemble and boosting classifiers such as Random Forest, Ada Boost, Extra Trees etc AND K-NN  achieve an accuracy about 90%, and a False Positive Rate about 11-12%. (fig.)

On the other hand, classifiers such as SVM, Logistic Regression, Multinomial NB and Bernoulli NB seem to perform poorly with accuracies fluctuating between 70% - 80% and higher false positives.

Here is the thing.
I also tried Gaussian NB and although it yields by far the worst accuracy 58.84% it has an incredibly low False Positive Rate 2.14% (and thus a high FNR)

I have no idea why this is happening. Any ideas?
Why some classifiers perform so poorly and others not?

I parametrized them all with Grid Search, they are used on the same dataset (about 30k records of each class) and I perform a 3-fold cross validation. It doesn't make any sense to me, especially for SVM. At last I use about 20 features.

P.S: I use python's sk-learn library

I find the easiest way for people to understand this is to think of the confusion matrix. Accuracy score is just one measure of a confusion matrix, namely all the correct classifications over all the prediction data at large:
$$\frac{True Positives + True Negatives}{True Positives + True Negatives + False Positives + False Negatives}$$
Your False Negative Rate is calculated by:
$$\frac{False Negatives}{False Negatives + True Positives}$$
One model may turn out to have a worse accuracy, but a better False Negative Rate. For example, your model with worse accuracy may in fact have many False Positives but few False Negatives, leading to a lower False Negative Rate. You need to choose the model which produces the most value for your specific use case.
Why do some classifier perform poorly? While an experience practitioner might surmise what could be a good modeling approach for a dataset, the truth is that for all datasets, there is no free lunch... also known as "The Lack of A Priori Distinctions Between Learning Algorithms" You don't know ahead of time if the best approach will be deep learning, gradient boosting, linear approaches, or any other number of models you could build.


Methods for learning with noisy labels I am looking for a specific deep learning method that can train a neural network model with both clean and noisy labels.

More precisely, I would like this method to be able to leverage noisy data as well, for instance by not fully "trusting" noisy data, or weighting samples, or deciding whether to use a specific sample at all for learning. But primarily, I am looking for inspiration.

Details:

My task is sequence-to-sequence NLP,
I have both clean pairs of sequences of (clean input, clean output) and noisy ones (noisy_input, noisy_output),
I know for certain which samples in my data are noisy, and if possible, I would like the desired method to make use of this information

I am very glad to give more information about my use case if needed.

Edit: Noisy vs. negative examples

  First, I wouldn't use the word "noisy" here because if you know which instances are "wrong" then these are not noise, they are negative examples.

My view is that the data I have are noisy examples, but not "negative". Using an example from machine translation from German to English:

clean (equivalent meaning)

DE Wenn es um die Medien geht, lebt Amerika in einem Paralleluniversum.
EN Regarding media, the US are living in a parallel universe.

noisy (meaning overlap)

DE Wenn es um die Medien geht, lebt Amerika in einem Paralleluniversum.
EN Regarding media, the US are weird.

negative (unrelated)

DE Wenn es um die Medien geht, lebt Amerika in einem Paralleluniversum.
EN Is Math related to science?

First, I wouldn't use the word "noisy" here because if you know which instances are "wrong" then these are not noise, they are negative examples. In my opinion "noisy" is when positive and negative cases are mixed together in a way that makes it difficult (or impossible) to distinguish between them. I think this matters because you're more likely to find similar use cases and relevant methods using this terminology.

I don't have a precise method to suggest but I would check the state of the art in machine translation: it's also a sequence-to-sequence task in which there are potential positive/negative cases. In particular there has been some work done in MT quality estimation, where the goal is to predict the quality of a translation for a sentence. This might be related because it's about labeling or quantifying how good a translation is, and I would assume that there are works which re-use labelled/scored translations (including potentially wrong ones) in order to obtain a better model. Unfortunately I don't have any pointers since I haven't followed the field recently.


How do you generalize from stochastic to batch learning using gradient descent? I understand pretty OK how to derive the formulas and implement stochastic gradient descent for a deep neural network (even though the total derivative magic for hidden layers is a bit pushing my limits).

I'm struggling to grasp the fundamentals of generalizing the method to batch or mini batch training.

From what I understand, in batch training you feed forward all of your batch's examples and then backpropagate the error.

In stochastic backpropagation most computations involve taking the derivative of the error with respect to neurons' activations. But in the case of batch training there is one activation per training example. Do you average them or something ? Or is the math radically different?

Is there a simple example somewhere that doesn't involve tensors?

Do you average them or something ?

Yes! The gradient of the loss for the minibatch is just a simple average of the gradients of the individual examples in the minibatch. See Section 8.1.3 of the Deep Learning Book:

  Hence, we can obtain an unbiased estimator of the exact gradient of the generalization error by sampling a minibatch of examples $\{x(1),...x(m)\}$
  with corresponding targets $y(i)$ from the data-generating
  distribution $p_{data}$, then computing the gradient of the loss with respect
  to the parameters for that minibatch:
  $$\hat{g} = \frac{1}{m} ∇_\theta \sum_{i} L(f(x^{(i)}; \theta), y^{(i)}).$$


How do I get my Neural network to ignore certain values? I was wondering if there was a way that I can get my CNN encoder-decoder neural network to completely ignore certain values in my data (2d images).
There are some pixel values of 0 that never change from the input to the target. They are meant to be ignored. I've tried some self attention layers and I even multiply the 'mask' image at the end of the encoder-decoder network, but it still does not ignore these values, causing a lot of unnecessary noise and inaccuracy in my results.
How can I ensure these values are being ignored at every convolutional layer?

CNNs work because of those zeros (the zeros create the boundary on which the change is values is the highest i.e. what networks learn). They are not the problem. Look at your regularizing your network (use dropout, reduce filters if overfitting or increase if underfitting).