Is it a good idea to use tensorflow instead of numpy for numerical approximations? I intend on perfroming some numerical approximations for a problem in physics.

The main gist of the program will be to perform a svd on large sparse matrices and also calculating the trace of a large matrix.

I used numpy/scipy and multiprocessing modules in python to do this, but it is not fast enough.

I also implemented the same code using tensorflow on some gpus. Tensorflow does the calulations much faster, but it takes a long time to send the data into the gpus and bring it out. Each instance of data is sent individually as float values.  Is there any way to send all the data at once into the gpu?

Is it a good or bad idea to use tensorflow for numerical calculations?

Below is a snippet of my code....

def Expectation_Value(density,N):
   expec=0
   for i in np.arange(1,N+1):
       a=tf.linalg.trace(tf.matmul(Sigma(i,N,Sz),density))
       expec+=a
return expec/N

def main():
   df=pd.read_csv('.....')
   sess=tf.Session()

   for l in range(len(df)):
   Delta=df['Delta'].iloc[l]
   Omega=df['Omega'].iloc[l]
   Gamma=df['Gamma'].iloc[l]
   J=df['J'].iloc[l]

    Sz_tens=tf.zeros(len(df))
    Expec=Expectation_Value(DMT,N)
    Sz_tens[l]=Expec
Sz_arr=Sz_tens.eval(session=sess)
df['Sz']=Sz_arr
sess.close()
df.head()

TensorFlow is a library that supports dataflow paradigm which makes it a good candidate for numerical calculations.

In general, TensorFlow data loading performance can be optimized using tf.data module. In particular, there is a concept called GPUs starvation, where GPUs are waiting for data to be loaded. GPU starvation can be minimized in TensorFlow with Pipelining, overlapping the preprocessing and model execution of a training step.

In particular, your code can be optimized by refactoring to Expectation_Value operate on vectors instead of scalars. Right now N and i are scalars and there is a for-loop that processes elements one at a time. tf.linalg.trace can take a tensor as an input so that N and i could be vectors. The function will be much faster once it operates on vectors and no longer uses a for-loop.


Can we access HDFS file system and YARN scheduler in Apache Spark? We can access HDFS file system and YARN scheduler In the Apache-Hadoop. But Spark has a higher level of coding. Is it possible to access HDFS and YARN in Apache-Spark too?

Thanks

HDFS

Spark was built as an alternative to MapReduce and thus supports most of its functionality. In particular, it means that "Spark can create distributed datasets from any storage source supported by Hadoop, including your local file system, HDFS, Cassandra, HBase, Amazon S3, etc."1. For most common data sources (like HDFS or S3) Spark automatically recognizes schema, e.g.:

val sc = SparkContext(...)
val localRDD = sc.textFile("file://...")
val hdfsRDD  = sc.textFile("hdfs://...")
val s3RDD    = sc.textFile("s3://...")

For more complicated cases you may need to work with lower-level functions like newAPIHadoopFile:

val hBaseRDD = sc.newAPIHadoopRDD(conf, classOf[TableInputFormat],
      classOf[org.apache.hadoop.hbase.io.ImmutableBytesWritable],
      classOf[org.apache.hadoop.hbase.client.Result])
val customRDD = sc.newAPIHadoopRDD(conf, classOf[MyCustomInputFormat],
      classOf[MyCustomKeyClass],
      classOf[MyCustomValueClass])

But general rule is that if some data source is available for MapReduce, it can be easily reused in Spark.

YARN

Currently Spark supports 3 cluster managers / modes:

Standalone
Mesos
YARN

Standalone mode uses Spark's own master server and works for Spark only, while YARN and Mesos modes aim to share same set of system resources between several frameworks (e.g. Spark, MapReduce, Impala, etc.). Comparison of YARN and Mesos may be found here, and detailed description of Spark on YARN here.

And, in best traditions of Spark, you can switch between different modes simply by changing master URL.


difference btw data distribution and frquency distribution I have a dataset with 'n' features and corresponding labels(binary in nature). How can I calculate the data distribution and frequency distribution of the same? What is the difference btw the two?

If you don't know the definition, how could you calculate anyway?

I looked at a few reliable definitions,

https://www.spss-tutorials.com/frequency-distribution-what-is-it/
https://www.statisticshowto.datasciencecentral.com/data-distribution/
http://makemeanalyst.com/observational-studies-and-experiments/population-distribution-sample-distribution-and-sampling-distribution/

The differences are subtle, and sometimes depend on who you ask.

What I conclude is that the frequency (or sample) distribution is statistics on an actual sample, counted per bin, maybe percentages added to the statistics.

The (population) data distribution is the distribution that you'd expect from the whole population.

For fair coin tosses, the data distribution would be 50/50, though a sample distribution of 10 could give 6/4.

My advice, either use the textbook definitions, or present the statistics that you see fit
for your analysis. Repeat the definition if necessary.

If you have a large enough random sample the frequency distribution becomes an estimate for the data distribution anyway (but sometimes you have to prove this to show your sample is random).

When you have $n$ features, you repeat this for all features.
E.g. when you have people's 'gender', 'married', 'smokes', 'employed', features, you have to repeat for all these features.


Unsupervised Clustering for n-length word arrays I have a series of arrays

[Apple,Banana,Cherry,Date]
[Apple,Fig,Grape]
[Banana,Cherry,Date,Elderberry]
[Fig,Grape]

and I would like to build some clusters that associate the arrays into groups based on overlap

Group1: Array1 and Array3 as they have 3 common words
Group2: Array2 and Array4 as they have 2 common words
etc..

I was thinking kmeans but there is really not a distance calculation - more like an overlap one.

Does anyone have a suggestions?

Thanks!

Assuming the dimensionality is reasonable, I would not use K-means or any generic algorithm, instead I would write a code which directly gives me the exact result by building a map of the groups:

// Assuming data is an array of size N containing all the arrays
// clusters is a map associating each group with a set of arrays
for i=0 to N-1
  for j=i+1 to N-1
    group = overlap(data[i], data[j])
    add data[i] to the set clusters[group]
    add data[j] to the set clusters[group]

An alternative version if the number of different values and size of the sets allow it and/or if it's possible to precompute the groups of interest:

for i=0 to N-1
  for every subset S of data[i]
    add data[i] to the set clusters[S]


How large of a value should a weight have in a neural network? If you're assigning random values to the weights in a neural network before back-propagation, is there a certain maximum or minimum value for each weight ( for example, 0 < w < 1000 ) or can weights take on any value? Could a network potentially have weights of 0.1, 0.0009, and 100000?

One of the problems that can occur when training a neural network is known as the exploding gradient problem. A poorly initialised network could lead to a large increase in the norm of the gradient during training. These larger values will basically run the weights out of the number precision of the computer, resulting in NaN values.

This post gives more information on the exploding gradient problem and how to solve it. A related post discusses different initialization strategies.


How to do feature selection after using pre-trained word embeddings? I am working on a multiclass text classification problem. I want to use the top k features based on mutual information (mutual_info_classif) for training my model.

I started this project on ML models:

I used tfidf for feature extraction and then used mutual_info_classif for feature selection.

    svc = Pipeline([('tfidf', TfidfVectorizer()),
                   ('ig', SelectKBest(mutual_info_classif,k=1000)),
                   ('clf',LinearSVC(multi_class='ovr')),
              ])
    svc.fit(X_train, y_train)
    y_pred = svc.predict(X_test)

This was pretty straight-forward.

Next, I started to work on RNN (more specifically a simple LSTM model)and that is where I am having a problem.

I have used pre-trained word embeddings from GloVe (300d) to get features from my data.
The embedding matrix I feed into the embedding layer of my RNN has the shape (4293,300), 4293 is the number of unique words found in my data and 300 is the dimension.

My questions are: Is there any way that I can use the top 1000 words (features) out of these 4293 based on mutual_info_classif? Is it even possible to do so? If yes, then should it be done before making the embedding_matrix or after?

When using a RNN, you don't feed all the data at once, you usually have a seq2seq model.
The models are created with an encoder-decoder architecture. The LSTM is used in the encoding phase.

So, let's say you have a text of 78 words. You will feed the embedding vector (size 300) of those 78 words, 1-by-1 into your LSTM and in the end you will get a hidden vector which represents your sentence.

Then, you can take this hidden vector and use it for classification (with a feedforward neural network, for example).

So, it doesn't matter that you have 4293 unique words in your data. You need to feed your LSTM a sequence of size [<number of words in sentence> × 300].


How many possible hypotheses The training set contains $p$ papers. Each paper is annotated as research or non-research. To develop the research paper filter, we consider the $W$ most frequent phrases in a paper. The research paper filter will use the presence/absence of these $W$ phrases to decide if the paper is indeed a research paper or not.

How many possible hypothesis are there?
I have $2^W$ possible hypothesis, but if we are including conjunctive hypothesis symbols, then I would say $4^W$. Is my logic correct?

How many of them are consistent?
For this question, I have no idea where to begin. Any insight?

I'm not seeing where the relationship between hypotheses and research paper is. I know what the relationship is in reality, but your question is a bit unorganized. Perhaps we could think about it in a different way...

I would first create a definition of possible hypotheses (i.e., $H_{0}$ and $H_{1}$ are my two possible hypotheses, and every hypothesis will have both a null and an alternative), and then associate a $H_{0}$ with a $H_{1}$ such that a combination of both null and alternative hypotheses equate to one hypothesis.

With this, it's a matter of iteratively counting.

With $p \in P$, where $p$ is a paper in the set of papers $P$,
With $w \in W$, where $w$ is a word in a set of words $W$ that indicate whether $p$ is a research paper or not,
With $i$ as the number of times a $w$ occurred in a $p$,
count the number of times a $w$ occurs in each $p$ and iterate $i$ accordingly.


Poor performance shown on Rare event modeling I am working on a Rare event classification problem. I Have 95% of the data as a majority class and 5% of the data as the minority class. I use classification trees algorithm. I am measuring the goodness of the model using confusion matrix.

As the i have the minority class just 5% of the total data, even though my prediction performance of minority class is close to 70%, the total number of errors are high.

For example, here is my confusion matrix.
             0           1
     0     213812      7008
     1     29083       16877

Though the Minority class(class 1) has predicted 16877 times correctly(70% and the misclassifcation is just 30%, but the absolute value of the misclassifcation is very high(29083) comparing to the correctly predicted minotriy class (16877). Which makes the solution less usable for the business.

Is there any idea on handling these kind of  issues in such rare event modelling.

Kind note: I have  balanced the target variable using the SMOTE algorithm before applying Classification tree.

If you are willing to use the caret package in R and use random forests, you can use the method in the following blog post for downsampling with unbalanced datasets: http://appliedpredictivemodeling.com/blog/2013/12/8/28rmc2lv96h8fw8700zm4nl50busep

Basically, you just add a single line to your train call. Here is the relevant part:

> rfDownsampled <- train(Class ~ ., data = training,
+                        method = "rf",
+                        ntree = 1500,
+                        tuneLength = 5,
+                        metric = "ROC",
+                        trControl = ctrl,
+                        ## Tell randomForest to sample by strata. Here,
+                        ## that means within each class
+                        strata = training$Class,
+                        ## Now specify that the number of samples selected
+                        ## within each class should be the same
+                        sampsize = rep(nmin, 2))

I have had some success with this approach in your type of situation.

For some more context, here is an in-depth post about experiments with unbalanced datasets: http://www.win-vector.com/blog/2015/02/does-balancing-classes-improve-classifier-performance/


Object detection model using images which have single instance of class per image-can it detect multiple instances in a single image? I tried to train Faster R-CNN to detect multiple instances of a single class (eg. pomegranate on a pomegranate tree) but the training data consisted of 200 images where each image had only one object labeled. The model is detecting an object but with very low probability. Is the model trying to find a single instance of the class but getting confused?

model name: Faster RCNN inception v2

Why does the model detect objects more efficiently when zoomed in closer?

Is there something wrong in the training process or something else?

I believe that the resulting accuracy of your model is due to a mix of reasons. Firstly, the data-set is really small, so you cannot expect to have an efficient model just by using your 200 images.

Proposed Solution #1: Data Augmentation. Enlarge your data-set by processing your images, crop, rotate, etc Examples with code. As for labeling the bounding boxes, you could do it manually or by creating functions to transform the co-ordinates based on the image processing method.

Proposed Solution #2: Also, try a different approach. I think that a version of the Yolo algorithm would be more appropriate for multiple instance detection YOLO-read_the_docs_link because R-CNN's search algorithm is fixed, no learning happens during selection, which might lead to bad region proposals resulting to fewer detection and low probability outputs.

  Why does the model detect objects more efficiently when zoomed in closer?

The obvious reason is that it is easier for your model to detect them, given the fact that the data-set is small. Also, it depends on the labeling, if your ground truth contains only large bounding boxes, then it would detect bigger pomegranates more efficiently. Adding cropped instances of your data-set to the train set might help with this issue too.


What is the difference between GPT blocks and BERT blocks Nowadays many applications only use the Encoder and Decoder part of the Transformer respectively. I am having trouble understanding the difference though.
If GPT uses Decoder only and BERT uses Encoder only does this mean that the only difference between the two is basically in the masking part?
The cross attention layer in the Decoder is omitted since there is no Encoder within GPT right?

BERT is a Transformer encoder, while GPT is a Transformer decoder:

You are right in that, given that GPT is decoder-only, there are no encoder attention blocks, so the decoder is equivalent to the encoder, except for the masking in the multi-head attention block.
There is, however, an extra difference in how BERT and GPT are trained:

BERT is a Transformer encoder, which means that, for each position in the input, the output at the same position is the same token (or the [MASK] token for masked tokens), that is the inputs and output positions of each token are the same.

GPT is a Transformer decoder, which means that it is meant for autoregressive inference. This means that the tokens in the input are shifted one position to the right with respect to the output, that is, if the output is [the, dog, is, brown, </s>], the input is [<s>, the, dog, is, brown, </s>].

Apart from that, at inference time BERT generates all its output at once, while GPT is autoregressive, so you need to iteratively generate one token at a time.


Q learning Neural network Tic tac toe - When to train net This is another question I have on a q learning neural network being used to win tic tac toe, which is that im not sure i understand when to actually back propogate through the network.

What i am currently doing is when the program plays through the game, if the number of game sets recorded has reached the max amount, every time the program makes a move, it will pick a random game state from its memory and back propagate using that game state and reward. this will then continue every time the program makes a move as the replay memory will always be full from then on.

The association between rewards and game state and action from history, is that when a game has been completed, and the rewards have been calculated for each step, meaning that the total reward per step has been calculated, the method i use to calculate the reward is:

Q(s,a) += reward * gamma^(inverse position in game state)

in this case, gamma is a value predetermined to reduce the amount that the reward is taken into account the further you go back, and the inverse position in game state means that if there have been 5 total moves in a game, then the inverse position in game state when changing the reward for the first move would be 5, then for the second, 4, third 3 and so on. this just allows the reward to be taken less into account the earlier the move is.

Should this allow the program to learn correctly?

This update scheme:

Q(s,a) += reward * gamma^(inverse position in game state)

has a couple of problems:

You are - apparently - incrementing Q values rather than training them to a reference target. As a result, the estimate for Q will likely diverge, predicting total rewards that are impossibly high or low. Although in your case with a zero sum game and initial random moves, it may just random walk around zero for a long time first.
Ignoring the increment, the formula you are using is not from Q-learning, but effectively on policy Monte Carlo control, because you use the end-of-game sum of rewards as the Q value estimate. In theory, with a few tweaks this can be made to work, but it is a different algorithm than you say you want to learn.

It is worth clarifying a few related terms (you clearly know these  already, but I want to make sure you have them separated in your  understanding of the rest of the answer):

Reward. In RL, a reward (a real number) can be returned on every increment, after taking an action. The set of rewards is part of the problem definition. Often noted as $R$ or $r$.
Return (aka Utility). The sum of all - maybe discounted - rewards from a specific point. Often noted as $G$ or $U$.
Value, as in state value or action value. This is usually the expected return from a specific state or state, action pair. $Q(S_t, A_t)$ is the expected return when in state $S_t$ and taking action $A_t$. Note that using $Q$ does not make your algorithm Q-learning. The $Q$ action value is the basis for several RL algorithms.

Your formula reward * gamma^(inverse position in game state) gives you the Return, $G$ seen in a sampled training game, $G_t = \gamma^{T-t} R_T$ where $T$ is the last time step in the game. That's provided the game only has a single non-zero reward at the end - in your case that is true. So you could use it as a training example, and train your network with input $S_t, A_t$ and desired output of $G_t$ calculated in this way. That should work. However, this will only find the optimal policy if you decay the exploration parameter $\epsilon$ and also remove older history from your experience table (because the older history will estimate returns based on imperfect play).

Here is the usual way to use experience replay with Q learning:

When saving experience, store $S_t, A_t, R_{t+1}, S_{t+1}$ - note that means storing immediate Reward, not the Return (yes you will store a lot of zeroes). Also note you need to store the next state.
When you have enough experience to sample from, typically you do not learn from just one sample, but pick a minibatch size (e.g. 32) and train with that many each time. This helps with convergence.
For Q-learning, your TD target is $R_{t+1} + \gamma \text{max}_{a'} Q(S_{t+1}, a')$, and you bootstrap from your current predictions for Q, which means:

For each sample in the minibatch, you need to calculate the predicted Q value of all allowed actions from the next state $S_{t+1}$ - using the neutral network. Then use the maximum value from each state to calculate $\text{max}_{a'} Q(S_{t+1}, a')$.
Train your network on the minibatch for a single step of gradient descent, with NN inputs $[S_t, A_t]$ and training label of the TD target from each example.
Yes that means you use the same network to first predict and then learn from a formula based on those predictions. This can be a source of problems, so you may need to maintain two networks, one to predict and one that learns. Every few hundred updates, refresh the prediction network as a copy of the current learning network. This is quite common addition to experience replay (it is something that Deep Mind did for DQN), although may not be necessary in your case for a game as simple as Tic Tac Toe.
The TD target is a bootstrapped and biased estimate of expected $G$. The bias is a potential source of problems (you may read that using NNs with Q-learning is not stable, this is one of the reasons why). However, with the right precautions, such as experience replay, the bias will reduce as the system learns.

In case you are wondering, it is the use of both $S_t$ (as NN input) and $S_{t+1}$ (to calculate TD target) in the Q-learning algorithm, which effectively distributes the end-of-game reward back to the start of the game's Q value.
In your case (and in many episodic games), it should be fine to use no discount, i.e. $\gamma = 1$

From your previous question, you noted that you were training two competing agents. That does in fact cause a problem for experience replay. The trouble is that the next state you need to train against will be the state after the opponent has made a move. So the opponent is technically viewed as being part of the environment for each agent. The agent learns to beat the current opponent. However, if the opponent is also learning an improved strategy, then its behaviour will change, meaning your stored experiences are no longer valid (in technical terms, the environment is non-stationary, meaning a policy that is optimal at one time may become suboptimal later). Therefore, you will want to discard older experience relatively frequently, even using Q-learning, if you have two self-modifying agents.


What should I use as training data for base (level 1) classifiers in ensembling? Can I just take all training data that I have, train the base models on them and then take their results and use them for training level 2 model? Is this a good practice, or should it be done differently?

You can do that, but your model will not generalize well. You should not use base-model predictions from data, which were used to fit the base model.
Thus, you have to get the base model predictions for the training data using cross-validation. This is called "model stacking".

This page has a good explanation:

Split your training data into subsets, predict the target for each subset using all other subsets.

Fit the base model on the whole training data and predict the target for the test set.

Do this for multiple base models. Now you have train and test set predictions for each base model. In this example we have two base models:

Fit an ensemble model on the base training predictions and evaluate the performance on the base test predictions.


UniLM - Unified Language Model for summarization The UniLM claims to be the best approach for summarization task. But there doesn't seem to be any tutorial or how-to section in the README.md or any other blog. How exactly can I use this state-of-the-art library for abstractive summary generation?
Github link
Paper
P.S. A newbie in NLP. Sorry if this is a dumb question.

Here's what you should do

Prepare your dataset: Follow similar instructions as described in the paper and preprocess your dataset. This will be your major task as after this you will only have to fine-tune the model. If you don't have a dataset, you can use the dataset used in this research paper, which can be downloaded from here.
Download the pre-trained model. Or you can choose to start from the provided fine-tuned model checkpoint (from the link). You will have to check which version of the model works best for your dataset. If you select model fine-tuned for summarization task and your dataset is similar to CNN/DailyMail dataset [37] and Gigaword [36] you can skip fine-tuning.
Fine-tune model: In this step, you will be using the command mentioned in the readme of the Github repository. Note that, there are some parameters that should be according to the language model you have downloaded in the previous step. Based on the size of your dataset you can change the number of epochs in the following command. You should also note that this will require GPU. The repository readme recommends 2 or 4 v100-32G GPU cards for finetuning the model.

    OUTPUT_DIR=/{path_of_fine-tuned_model}/
    MODEL_RECOVER_PATH=/{path_of_pre-trained_model}/unilmv1-large-cased.bin
    export PYTORCH_PRETRAINED_BERT_CACHE=/{tmp_folder}/bert-cased-pretrained-cache
    export CUDA_VISIBLE_DEVICES=0,1,2,3
    python biunilm/run_seq2seq.py --do_train --fp16 --amp --num_workers 0 \
      --bert_model bert-large-cased --new_segment_ids --tokenized_input \
      --data_dir ${DATA_DIR} \
      --output_dir ${OUTPUT_DIR}/bert_save \
      --log_dir ${OUTPUT_DIR}/bert_log \
      --model_recover_path ${MODEL_RECOVER_PATH} \
      --max_seq_length 192 --max_position_embeddings 192 \
      --trunc_seg a --always_truncate_tail --max_len_a 0 --max_len_b 64 \
      --mask_prob 0.7 --max_pred 48 \
      --train_batch_size 128 --gradient_accumulation_steps 1 \
      --learning_rate 0.00003 --warmup_proportion 0.1 --label_smoothing 0.1 \
      --num_train_epochs 30

Evaluate your model: Use biunilm/decode_seq2seq.py to decode (predict the output of evaluation dataset) and use the provided evaluation script to evaluate the trained model.
Use the trained model: In order to use this model to make a prediction, you can simply write your own python code to:

load Pytorch pre-trained model using pytorch_pretrained_bert library as used in decode_seq2seq.py file
Tokenize your input
predict the output and detokenize the output.

Here is the logic which you can use:

model = BertForSeq2SeqDecoder.from_pretrained(long_list_of_arguments)
batch = seq2seq_loader.batch_list_to_batch_tensors(input_batch)
input_ids, token_type_ids, position_ids, input_mask, mask_qkv, task_idx = batch
traces = model(input_ids, token_type_ids,position_ids, input_mask, task_idx=task_idx, mask_qkv=mask_qkv)

Note that this is not the complete logic. This code just show how github repository code has handled the saved model and used it to make predictions. Use traces to convert ids to tokens and detokenize the output tokens (as used in the code here). The detokenization step is necessary as the input sequence is tokenized to subword units by WordPiece.

For reference here is the code which loads the pre-trained model. You can go through the loop to understand the logic and implement it in your case. I hope this helps.


How to deal with skewed imbalanced image dataset to work with CNN? I am working on multi-class classification problem on an image dataset. There is one class with 80% of the images and rest 20% is divided into rest 6 remaining classes. If I have to apply the image-augmentation technique, what ratio of count of images between each class I have to maintain?

I think that you first need to consider what are the class proportions in the data that you are going to eventually use the model for. It is important to have a test set that comes from the same data generating distribution as the real data. Now, in case the proportions in this data are the same as you wrote than it is actually a good thing to have more examples for the classes that are more common. It allows your model to give those classes a bigger prior and focus more on those classes, extract more features relevant to those classes particularly.

In case you'd like to reduce this skew (for example because the proportions in the real data are different or the imbalance is so big that your model learns to assign always just the most common class and ignores others) you can use resampling: define what class proportions would suit you and randomly sample examples using those ratios.


confusing regarding to kmeans clulstering for data correlation I am trying to think through my process before doing any real coding. However, got really confused easily.

Say I have 100 instruments and I know their price movements every day for a year. So I can create a movement matrix

A =[[I1-1, I2-1, .... I100-1],  (I1-1 is price for instrument 1 on day 1)
    [I1-2, I2-2, .... I100-2],
    ....
    [I1-365, I-2365, .... I100-365]
    ]

Then for each instrument, I can calculate a price movement correlation between other instruments for the whole year.

   C =[C1-2, C1-3,...C1-100,C2-3,....C99-100] (C1-2 is the price movement correlation between instrument 1 and 2 for the whole year)

Then I would like to apply a K-Means clustering algorithm to classify the correlation into say 10 categories. So in theory, I created 10 categories that the prices turned to move together.

However, the more I think about it, the more it is not correct. For example, if this is my Correlation result:

 C =[0.35, 0.59,...0.88(C1-100),0.48,....0.99(C99-100)]

isn't it K-Means clustering may classify C1-100, C99-100 in one cluster, and C1-2, C1-3, C2-3 in another cluster.

When I read that, it means instrument 1,100, 99 in one category, and instrument 1,2,3 in another category. But I would like each instrument only available in one category, so looks like there is a hole in my idea or maybe my idea is totally wrong?

You will not get what you seek this way, but you are on the right path. Use the correlation between two instruments as a measure of similarity, and then perform spectral clustering with this measure as the kernel.

Basically, you will start with your correlation matrix $R$, and build the corresponding Laplacian matrix $L$. The eigen vectors corresponding to the smallest eigen values of $L$ will give you a projection space in which you can perform k-means clustering.

This technique is efficient if you have a good similarity measure, but only works for reasonably sized datasets (because you need the eigen decomposition of the $n \times n$ matrix).


Imputing missing values by mean by id column in R This is fairly straight forward but i am unable to do it. My data frame has a id variable which is repeating. For same id I want to replace the NAs in other continuous variable(rating and sur) with their corresponding mean. can anyone pls suggest

ID rating Sur
101 60 0.7687
101 78 NA
101 NA 0.765
102 60 NA
102 NA 0.654
102 75 0.435
103 NA 0.576
103 68 0.875
103 70 NA

If you want the mean, you could use dplyr syntax:

df = structure(list(ID = c(101L, 101L, 101L, 102L, 102L, 102L, 103L,103L, 103L),
                    rating = c(60L, 78L, NA, 60L, NA, 75L, NA, 68L, 70L),
                    Sur = c(0.7687, NA, 0.765, NA, 0.654, 0.435, 0.576, 0.875, NA)),
               .Names = c("ID", "rating", "Sur"), class = "data.frame", row.names = c(NA,-9L))

library(dplyr)
df %>%
  group_by(ID) %>%
  mutate(rating = ifelse(is.na(rating), mean(rating, na.rm = TRUE), rating),
         Sur = ifelse(is.na(Sur), mean(Sur, na.rm = TRUE), Sur))

     ID rating     Sur
  <int>  <dbl>   <dbl>
1   101   60.0 0.76870
2   101   78.0 0.76685
3   101   69.0 0.76500
4   102   60.0 0.54450
5   102   67.5 0.65400
6   102   75.0 0.43500
7   103   69.0 0.57600
8   103   68.0 0.87500
9   103   70.0 0.72550

for other imputations you can look at packages like mi, mice, caret, amelia or simputation. And I'm probably forgetting a lot of others. Personally I like the simplicity of simputation.

For example median imputation is just one line:
impute_median(df, rating + Sur ~ ID)


Is a data set considered to be linearly separable if it can only be separated by multiple hyperplanes? For example, on the linear separability Wikipedia article, the following example is given:

They say "The following example would need two straight lines and thus is not linearly separable".

On the other hand, in Bishop's 'Pattern Recognition and Machine Learning' book, he says "Data sets whose classes can be separated exactly by linear decision surfaces are said to be linearly separable".

Under Bishop's definition of linear separability, I think the Wikipedia example would be linearly separable, even though the author of this Wikipedia article says otherwise. This is because Bishop says that we can use multiple linear decision surfaces (hyperplanes) to separate the classes, and it's still considered to be linearly separable data. Bishop implies this by referring to linear decision surfaces, plural, not singular.

Logically, I agree with Bishop. After all, the classes in the Wikipedia example are being separated by linear decision surfaces. So how can one then turn around and say the data set isn't linearly separable? Well, perhaps you could enforce the rule that a data set is only linearly separable if we can separate $N$ classes with $N-1$ decision surfaces. But why would you define linear separability in this way?

So, is the Wikipedia example linearly separable or not?

In simple terms: Linearly separable = a linear classifier could do the job. You could fit one straight line to correctly classify your data.

Technically, any problem can be broken down to a multitude of small linear decision surfaces; i.e. you approximate a non-linear function with a high number of small linear boundaries. That's what Neural Networks are doing with ReLU activations btw, but nobody would say that Neural Networks are linear models.


What should I care about while stacking as an ensemble method? I'm using SMO, Logistic Regression, Bayesian Network and Simple CART algorithms for classification. Results form WEKA:

Algorithm               Sensitivity (%)       Specificity (%)         Overall accuracy (%)
Bayesian Network            57.49                 76.09                    65.24
Logistic Regression         64.73                 69.86                    66.87
SMO                         54.32                 79.20                    64.69
Simple CART                 71.88                 61.51                    67.56

SMO gives the best result for my classification problem, since it correctly classify the 79.20% of the class which is important for me. I want to increase this accuracy by stacking. I tried to combine some of them. In most of the cases I couldn't increase the accuracy but stacking SMO with Logistic Regression made a little increment in accuracy.

How can I explain why stacking SMO with Logistic Regression is better than others?

Is there any generalization such as combining tree classifiers gives good result in stacking? What should I care about while stacking?

EDIT:

                                Bayesian Network    Logistic Reg.   SMO         CART
Kappa statistic                   0.3196             0.3367         0.3158      0.3335
Mean absolute error               0.3517             0.4164         0.3531      0.4107
Root mean squared error           0.5488             0.4548         0.5942      0.4547
Relative absolute error (%)      72.3389              85.65        72.6299      84.477
Root relative squared error (%) 111.3076            92.2452       120.5239     92.2318
Weighted Avg. of F-Measure        0.653               0.671          0.676     92.2318
ROC Area                          0.725               0.727          0.668       0.721

Total number of instance is 25106. 14641 of them is class a, and 10465 of them belong to class b.

=== Confusion Matrix of Simple CART ===
     a     b   <-- classified as
 10524  4117 |     a = 0
  4028  6437 |     b = 1

=== Confusion Matrix of SMO ===

    a    b   <-- classified as
 7953 6688 |    a = 0
 2177 8288 |    b = 1

=== Confusion Matrix of Logistic Regression ===

    a    b   <-- classified as
 9477 5164 |    a = 0
 3154 7311 |    b = 1

Since SMO is successful at class b and CART is successful at class a, I tried to ensemble these two algorithms. But I couldn't increase the accuracy. Then I tried to combine SMO with Logistic Regression, the accuracy is increased a little bit. Why ensembling SMO with Logistic Regression is better than ensebling SMO with CART, is there any explanation?

To directly answer your question about stacking: you should care about minimizing 1) bias, and 2) variance. This is obvious, but in practice this often comes down to simply having models which are "diverse". (I apologize that link is behind a paywall, but there are a few others like it and you may well find it other ways)

You don't want ensembles of like-minded models - they will make the same mistakes and reinforce each other.

In the case of stacking, what is happening? You are letting the outputs of the probabilistic classifiers on the actual feature input become the new features. A diverse set of classifiers which can in any way give signals about edge cases is desirable. If classifier 1 is terrible at classes A, B, and C but fantastic at class D, or a certain edge case, it is still a good contribution to the ensemble.

This is why neural nets are so good at what they do in image recognition - deep nets are in fact recursive logistic regression stacking ensembles! Nowadays people don't always use the sigmoid activation and there are many layer architectures, but it's the same general idea.

What I would recommend is trying to maximize the diversity of your ensemble by using some of the similarity metrics on the classifiers' prediction output vectors (ie, Diettrich's Kappa statistic) in training. Here is another good reference.

Hope that helps.


Why is stochastic gradient descent so much worse than batch GD for MNIST task? Here the code from Tensorflow tutorial:
A Multilayer Perceptron implementation example

With batch size = 100 we quickly got Accuracy: 94.59%.

If I set the batch size to be one, the training takes ten times more time but the accuracy is only nine percent.

I have tested different learning rates with no luck. SGD performance is terrible for small batch sizes. We can expect that SGD performance will be lower, but not ten times less!  What is the reason for this performance loss?

Why is stochastic gradient descent so much worse then batch GD for MNIST task?

It isn't inherently worse. Instead, by changing just one parameter on its own you have adjusted the example outside of where it has been "tuned" to work, because it is a simplified example for learning purposes, and it is missing some features that most users of NNs would consider standard.

The batch size of 1 is performing just fine. In fact, although it takes longer to process the same number of epochs, each epoch actually has more weight updates. You get 100 times as many weight updates, although each one has far more noise in it than a batch size of 100. It is these extra weight updates, plus extra time spent running interpreted Python code for 100 times as many batches, which adds the large amount of time.

The problem with accuracy is that the example network has no protection from overfitting. By running so many more weight updates, the training starts to learn the precise image data to match each digit, in order to get the best score. And by doing that exclusively it learns rules that work really well on the training data but generalise really badly to new data in the test set.

Try batch size of 1 and number of epochs = 3 (I tried this and got accuracy of 94.32%). Basically that is using early stopping as a form of regularisation. It's not the best form of regularisation, but it is quick to try and often effective - the problem is how to tell when to stop, so you need to measure a test set (often separate to final test set, called cross-validation set) at any potential stopping point, and save the best model so far. That will obviously involve adjusting the example code.

Probably the 15 epochs in the original example has been chosen carefully so as to not make overfitting a problem with batch size of 100, but as soon as you change batch size, without any other form of regularisation, the network is very likely to over-fit. In general neural networks strongly tend to over-fit, and you have to spend time and effort to understand and defend against this.

Have a look at regularisation in TensorFlow for other options. For this kind of problem, dropout (explained lower down page in the link) is highly recommended, which is not purely regularisation, but works to improve generalisation for many neural network problems.


Hub removal from graphs I have a graph with vertices that represent some entities and the edges are weighted as the correlation between two such entities.

I would like to break this graph into several subgraphs with high inner-correlation.

My problem is that I have a few 'hubs', with high correlation to a lot of different entities.

How can I detect such hubs in order to remove them?

Depends on how you define "hubs". In Network Science, a hub is simply a node with high degree i.e. those node who contribute to the power-law nature of the degree distribution the most. But you can also find other definitions for instance according to the information flow in the network where hubs are defined as those node that are critical in the process information flow (also called central nodes).

My Suggestions

Degree Distribution: The simplest approach would be to choose high degree nodes as hubs. To ensure your results a bit more I recommend to calculate the summation of all correlations correspond to each node and have a look at these numbers as well. In this case you are looking for nodes which have high degrees and contain a larger value of correlation.
Centrality Measure: or Betweenness introduced by Linton Freeman which again somehow measures the influence of a node in the process of information flow over the network. Calculating the centrality for a vertex $v$ in a graph has basically 3 steps:

For each pair of vertices $(s,t)$, compute the shortest paths between them.
For each pair of vertices $(s,t)$, determine the fraction of shortest paths that pass through the vertex in question (here, vertex $v$).
Sum this fraction over all pairs of vertices $(s,t)$.

For more information read this carefully and in case you need any help (specially for implementation) just drop me a line in the comments :)

Good Luck!


How can I explain the cause of different performances for two different LSTM models and improve the performance? I've built two different models for Load Forecasting. Dataset has six features. The performance evaluation metric is the Mean Absolute Percentage Error(MAPE). Both models are based on LSTM. Here is the first model and performance.

model = Sequential()
model.add(LSTM(20, input_shape=(1,6),init= 'uniform', return_sequences=True)) #input shape (1,5) to (1,6)
model.add(LSTM(20,init= 'uniform', activation='relu'))
model.add(Dense(256,init= 'uniform', activation='relu'))
model.add(Dense(256,init= 'uniform', activation='relu'))
model.add(Dense(1,init= 'uniform', activation='relu'))

For 30 epochs:

TRAIN loss = 3.7549, TEST LOSS : 3.5419
Time Requered: 5.9089 minutes

Second model is stacked-LSTM. Here is the code:

model = Sequential()
model.add(LSTM(20, input_shape=(1,6),init= 'uniform', return_sequences=True)) #input shape (1,5) to (1,6)
model.add(LSTM(20,init= 'uniform', activation='relu',return_sequences=True))
model.add(LSTM(20,init= 'uniform', activation='relu',return_sequences=True))
model.add(LSTM(20,init= 'uniform', activation='relu',return_sequences=True))
model.add(LSTM(20,init= 'uniform', activation='relu'))
model.add(Dense(1,init= 'uniform', activation='relu'))

For 30 epochs:

TRAIN loss =  5.0825, TEST LOSS : 5.1821
Time Requered: 5.7295 minutes

In both case I have compiled models in this way:

model.compile(loss='mean_absolute_percentage_error', optimizer='adam', metrics=['acc'])

Definitly, second model is working better. But why second model(stacked-lstm) is not working well? And how can I improve better than that? For the first model after 150 epochs gradient vanishes whene loss is around 3.01. I'm aiming for less than 2.

LSTM especially works for time-series data. I'm assuming these features are not dependent temporally, so it's not ideal to use LSTM or any RNN variant.

Remove the LSTM layers, reduce the number of neurons, use regularizer, dropout.
Also, using ReLU in the last layer doesn't make any sense. If it's a regression problem, just use linear activation instead.

To reduce the error, do a hyperparameter tuning with a randomized search or grid search.


How do you, analytically, show you are not using too many features? One of the managers at my company asked if there is a I could include a metric demonstrates that the my model is not using too many Features/Variables. Is there a metric or best practice that does this? Do you have a good way of communicating this to the business?
Specifically, she asked if I could include AIC about my models. After I asked some questions though, this is not quite what management is looking for because, unless I'm mistaken, AIC is only useful for comparing models with the same features.
In any case, I was able to use train-test validation for the model I was presenting, so AIC would have been overkill. To deal with this problem in practice I create a baseline model which uses all the variables, and then pair down as many variables as I can without sacrificing too much performance (on the train and test sets).

My approach would be quite similar to yours with a baseline, maybe just a bit more general:
Assuming it's practical (i.e. the training process is not too costly), you could train/test multiple models with various number of features, e.g. 10%, 20%...,100% of the features. For every subset you use an appropriate feature selection method, preferably something like genetic algorithm but it might be too costly. If it's not possible a simple individual feature selection method, e.g. using information gain, but it's likely not to be optimal. Once all the models have been trained, tested and evaluated, a plot of the performance as a function of the number of features should (hopefully) show a curve which increases less and less but never reaches the plateau. If it does reach the plateau before your chosen number of features, it means that you could actually cut down the number features.


tensorflow categorical data with vocabulary list - Expected binary or Unicode string, got [0,1,2,...] I'm brand new to machine learning (having just completed the google machine learning crash course) and thought it would be good to try my hand at a Kaggle competition as a good starter to some real problem solving. I'm using tensorflow and Python 3, all up to date (the kaggle online jupyter notebook)

The data is formatted in a dataframe like below

|Identity | Cuisine | Ingredients                |
|---------|---------|----------------------------|
|1        | italian | [beans, milk,..., tomatoes]|
|2        | indian  | [chicken, curry leaf,...]  |

I have made a vocabulary list generator to create a vocabulary set, and replace instances of those words in the ingredients array with the index of the ingredient in the vocabulary set, so my original data looks like below.

|Identity | Cuisine | Ingredients |
|---------|---------|-------------|
|1        | italian |[0, 1,..., 4]|
|2        | indian  |[5, 6,...]   |

I seperate the labels (cuisine) and the features (ingredients) into 2 seperate dataframes for ease, and I am using a tf.feature_column.categorical_column_with_vocabulary_list and subsequent tf.feature_column.indicator_column for the ingredients array.

I now however have an issue with my model not being able to read the ingredients column, and get the error

TypeError: Expected binary or unicode string, got [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

my input function is as follows

def input_fn(features,labels,batch_size,num_epochs=None,shuffle=True):
    ds = Dataset.from_tensor_slices((features,labels))
    ds = ds.batch(batch_size).repeat(num_epochs)

    if shuffle:
        ds = ds.shuffle(10000)

    feature_batch, label_batch = ds.make_one_shot_iterator().get_next()
    return feature_batch, label_batch

which is fed into a simple function as below

training_func = lambda: input_fn(training_example,training_target,batch_size)
validati_func = lambda: input_fn(validation_example,validation_target,batch_size)

optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)
optimizer = tf.contrib.estimator.clip_gradients_by_norm(optimizer, 5.0)

classifier.train(
    input_fn=training_func,
    steps=steps_per_period
)

My urgent question is how do I fix this TypeError

In addition I also want to know if there a best practice for handling this format of data? (and if there is any built-in functionality to handle this)

I'm not completely familiar with TF API, but here's what I think is happening.

The library tells you that it can handle a binary column or a string. But you have all the ingredients listed in a single column. So the integer conversion of ingredient label is not helping.

You can instead create one column per possible list of ingredient and setting it to 1 if that ingredient is present or absent. For example, Italian cuisine will have column for tomatoes or garlic set to 1 for many records.

You can read more about get_dummies function in pandas library. If the original ingredient list comes in form of text, you can read up more about text feature extraction / bag of words APIs in scikit-learn libary.


K-means Clustering algorithm problems I am trying to implement k-means clustering algorithm, but I am confused about calculating the distance and update(move) cluster centroids. For example, let's say that I have 2 features. One of them is weight={2,4,6,8,11,14,21} and the other one is height={4,6,7,8,9,12,14}. So, in the coordinate system my points are x1={2,4},x2={4,6},x3={6,7} and so on. Then, I initialize the cluster centroids randomly, doesn't matter how many there are for now, but they have coordinates too. Let's say μ1={4,2}. At this point, I understand how do I calculate distance with Euclidean distance.

My code for calculating distance:

def get_distance(x1,x2,s1,s2):
      return np.sqrt(np.power(s1-x1,2)+np.power(s2-x2,2))

Now I get a distance.

My first question is how cluster assignment step(first step in loop) will know which centroid assign to c(i).I mean,am I supposed to look at each centroid for understand which sample(x(i)) is close to it and then I should assign centroid to c(i), right?

My second question, let's say I got distances and I have c(1,2..,n) array now. Second step in algorithm which is called move(update) centroid step, we are calculate μ. According to formula, this μ is the average of points assigned to clusters so for example μ1=[x(3) + x(4) + x(6)] / 3. However, here our μ was a point in coordinate system, right? I mean, μ1 was {4,2}. How can this be possible? It's a point not a variable. It has coordinates. Well, if it will become a variable let's say μ1=5, how can I subtract ||x(i)-μ|| then? x is a coordinate.

My last question is very simple. For this example, I have two features weight and height. What is the maximum number of features that we can use in k-mean? Is it possible to use k-mean algorithms for many many features? For instance, my first feature is height and the second is weight and the third is width fourth and so on.

I hope, I explained my problem clearly. If not, sorry for the bad English. I think these three questions are independent questions, so you can answer one of them.

Thanks.

Wikipedia says: "Assign each observation to the cluster whose mean yields the least within-cluster sum of squares (WCSS)"

I think in your case, this is translatable to: $c_i$ is assigned to the closest centroid by euclidean distance.

For your second question, the centroid should $\mu$ should have the same number of dimensions as each training point $x_i$. They are both points in the co-ordinate system.

You can use a high number of features with K-means, for example, text analytics might reduce a corpora of news articles to 10,000+ dimensions. Depending on the package you use these might be represented as a sparse matrix.


What regressors are recommended with text modeling? For the sake of my own exploration, I am working on a sales prediction project. I am using text extracted from a set of books to build a predictive model.

With scikit learn, I have created a Tfidf, and together with the numeric sales numbers, I created a SGDRegressor. But I'd like to practice with other models.

I am looking over the options available to me, and I'm wondering, what other algorithms might be useful in this kind of regression scenario? And/or, what other algorithms in scikit learn will take a tfidf as the dataset?

In sklearn Anything that will take sparse data can take output from TFIDF.

In sklearn basically all data any model can take is either dense (normal array) or sparse (that only stores the location of values != 0). You can convert from sparse to dense, but chances are you'll run out of memory if you try.

I'm quite a big fan of using linear algorithms on text data (sgdregressor for example have a ton of different options you can play with). But other algorithms like randomforest that Sergey mentioned, and naive bayes models can also work with that kind of data. Basically what you are looking for is anything that can take sparse input data.

(One thing I've done in the past when working with text+other data is taking output from an sgd's analysis of the data and feeding that + the other data to another algorithm like randomforest. It's a simple and quite powerful method)


How to setup my Ubuntu PC for these Deep Learning frameworks? I need to use both NVIDIA Digits and tensorflow Object Detection API for different deep learning purposes. I am a bit lost with the compatibility issues, especially for what concerns CUDA. So which version should I choose for DIGITS, Tensorflow-gpu, CUDA and Object Detection API in order to make everything work?

An easy way to run different versions of multiple frameworks alongside each other is to use Docker. With Docker, you can create a new "Container", kind of a light-weight virtual machine, for each project. Docker handles all hardware abstraction and separates the different Frameworks.

Note: To use the GPU within a Docker container, you need to install Nvidia-Docker, as a "normal" Docker installation can not share the GPU with a container.

In your example, you could have a Docker container for Digits, and a separate one for TensorFlow - then the two frameworks will be completely isolated from each other.

This approach comes with multiple advantages:

You can run the same container on a different machine without installing e.g. TensorFlow - all you need is (Nvidia-)Docker.
You can easily use different (TensorFlow) versions for different projects, as each project gets it's own container which is separated from the system.
Containers can use different CUDA versions, e.g. CUDA 8 for Digits and CUDA 9 for TensorFlow.
There are pre-built containers for almost all existing deep learning frameworks. Nvidia even provides some GPU-optimized containers with their free NGC (Nvidia GPU Cloud) service.


Implementing spatio-temporal convolutions in pytorch I am trying to implement a layer to perform the (2+1)D convolutions described in this paper: https://arxiv.org/pdf/1711.11248.pdf

The basic idea is as follows: Let's say I have a 3D convolutional layer that takes in an input with $N_{i-1}$ channels, and performs $N_i$ 3D convolutions on a 3D tensor representing $T$ 2D frames taken over $T$ timesteps.

Instead of doing 3D convolutions over the tensor, the approach is to replace the 3D convolution with a 2D convolution followed by a 1D convolution along the temporal axis. In particular, if you wanted to perform $N_i$ 3D convolutions with kernels of size $N_{i-1} \times t \times d \times d$, you instead perform $M_i$ 2D convolutions with filters of size $N_{i-1} \times 1 \times d \times d$ followed by $N_i$ 1D convolutions along the temporal axis of size $M_i \times t \times 1 \times 1$.

Here is the tensorflow implementation of this type of layer: https://github.com/facebookresearch/R2Plus1D/blob/master/lib/models/video_model.py

I am having trouble implementing this type of layer in pytorch, however. This is what I have right now, and I'm clearly doing something wrong. I would appreciate it if someone could point me in the right direction as to how I would go about performing this type of convolution. I'm setting the intermediate channel dimension $M_i$ to 20 in this snippet:

class hybrid3d(nn.Module):

def __init__(self, n_classes):
    super(hybrid3d, self).__init__()
    t, d, d = 3, 3, 3
    self.conv1_1 = nn.Conv2d(3, 20, kernel_size=(1, d, d), stride=[1,1,1] , padding=[0,0,0])
    self.relu1_1 = nn.PReLU()
    self.conv1_2 = nn.Conv1d(20, 12, kernel_size=(t, 1, 1), stride=[1,1,1], padding=[0,0,0])
    self.relu1_2 = nn.PReLU()

def forward(self, x):
    x = self.relu1_1(self.conv1_1(x))
    x = self.relu1_2(self.conv1_2(x))
    return x

import torch as t

class Net_1(t.nn.Module):
    def __init__(self):
        super(Net_1, self).__init__()
        self.conv3d = t.nn.Conv3d(3, 8, (2, 3, 3))
        self.relu = t.nn.ReLU()

    def forward(self, x):
        x = self.relu(self.conv3d(x))
        return x

net_1 = Net_1()

video_1 = t.empty(1, 3, 6, 72, 108).normal_()

%%time
net_1(video_1).shape

Wall time: 18 ms
torch.Size([1, 8, 5, 70, 106])

class Net_2(t.nn.Module):
    def __init__(self):
        super(Net_2, self).__init__()
        self.conv2d = t.nn.Conv2d(3, 8, (3, 3))
        self.conv1d = t.nn.Conv1d(8, 8, (2))
        self.flatten = t.nn.Flatten(start_dim=2)
        self.relu = t.nn.ReLU()

    def forward(self, x):
        x = self.relu(self.conv2d(x))
        x = self.flatten(x)
        x.transpose_(0, 2)
        x = self.relu(self.conv1d(x))
        x.transpose_(0, 2)
        x.transpose_(0, 1)
        return x.view(5, 8, 70, 106)

net_2 = Net_2()

video_2 = t.empty(6, 3, 72, 108).normal_()

%%time
net_2(video_2).shape

Wall time: 58 ms
torch.Size([8, 5, 70, 106])


Modeling 3 variable equation The data produced from my humidity sensor at different temperature is as follows (forget about purpose of frequency and its source)

on re-arrangement (temp in deg C)

In this case temperature and frequency are independent variables and humidity is dependent on both of them. Humidity seems directly proportional to frequency if analyzed for humidity Vs Frequency and temperature seems inversely proportional to frequency while analyzed Temp Vs Frequency.

I need to be able to make an equation which gives Humidity upon solving with known temperature and frequency. For example if I input F = 8179 and T = 20.5, H should be near to 42%.

I tried polyfit and interp2 from MATLAB for frequency vs humidity and got much errors and tried least square method manually but ended getting nothing near. Specifically I do not have idea on modeling for 3 Variable equation. What may be the way for modeling 3 variable equations upon this small sets of data. Thanks.

You want to fit the model $H = \beta_0 + \beta_1 T + \beta_2 H + \epsilon$

The standard way to enter the data to fit this problem is to enter 3 columns, one for T, one for F, one for H.

T    F     H
20  8179    40
30  8044    40
40  7923    40
20  8343    60
30  8250    60
40  8130    60
20  8528    80
30  8430    80
40  8320    80

In R, the model can be fitted with the command:
lm(H~T+F)

Without checking the appropiatemes of the model, the result is:
Coefficients:
(Intercept)   $\;\;\;\;\;$         T     $\;\;\;\;\;\;\;\;\;\;\;\;\;$      F
  -845.4072 $\;\;\;\;\;$    1.1911   $\;\;\;\;\;$    0.1056
with Rsqr= 0.9958.

When F = 8179 and T = 20.5, the predicted value of H = -845.4072 + 1.1911*20.5 + 0.1056*8179 =  42.39791


AlexNet second layer understanding It has been a long I am confused on understanding some of the AlexNet architecture :
The output of the first conv layer is 55x55x48 (96 considering the division between GPUs but let's stick to 1 GPU so depth 48). Then max pooling is applied and there come my problem.
When applying max pooling, the result is 27x27x48 right ? If so, how is applied the next convolution over this result (with 5x5x48 filters) to output 27x27x128 ? I finally don't see how and when to apply max-pooling in between convolutions. I must miss something here...

Okay, I got it. If anyone interested, they use 5x5 filter but with padding 2 and striding 1 so that with bias it doesn't change the 2D dimension of the output when applied on the result of max-pooling. The info on padding isn't present on the original paper...


Reason for Training and test loss sudden increment after some epochs keras We know that if training and test loss are different from each other, our model is over-fitting. However, if both get high after some epochs, how can we justify it?

One way to solve it is to reduce the learning rate to 0.0001.

But, actually I would like to know the theoretical reason behind this sudden increment.

This might be happening because of high learning rate.

The loss function is convex and the model needs to reach the minima. Learning rate defines the magnitude of step you take towards minima. If you take high learning rate, you might overshoot the minima, hence sudden increase in loss value.


Considering the output of a BLSTM in pytorch, what's the order of the elements? I am currently using pytorch to implement a BLSTM-based neural network. I understand that the output of the BLSTM is two times the hidden size. However, I am currently unable to find out whether this is ordered as [forward_state_0, backward_state_n, forward_state_1, backward_state_n-1,..., forward_state_n, backward_state_0] or as [forward_state_0, forward_state_1,..., forward_state_n, backward_state_n, backward_stat_n-1,...,backward_state_0] or something else. I'd like to feed a pairwise maximum of the outputs to the next layer so the most import thing for me is which are the corresponding forward and backward states.

Your answer is in the documentation of the code you linked in your comment:

  For the unpacked case, the directions can be separated using output.view(seq_len, batch, num_directions, hidden_size), with forward and backward being direction 0 and 1 respectively. Similarly, the directions can be separated in the packed case.

So you just need yo separate the directions as specified above and then take the element-wise maximum with torch.max.


Neural Network Backpropagation problems I'm working on an implementation of a neural network so I can really grasp how these magic boxes work. However the neural network I've written code for doesn't work and I think it's due to my implementation of backpropagation

[pseudo code]

'last layer only
for each node (i) in last layer
error(i) = outputError(i)
end for

'all hidden/inner layers
for each layer (i) in layers (not including layer 0 or last)
 for each neuron (j) in layer i
  for each neuron (k) in layer i+1
   error(i)(j) +=error(i+1)(k) * weight(i)(j,k)
  end for

  errors(i)(j) = errors(i)(j) * activation(i)(j) * (1-activation(i)(j))

  for each neuron in layer i+1
   deltaWeight(i)(j,k) = -error(i)(k)*activation(i)(k)*learningRate/NumExamples
   deltaBias(i)(k) = -error(i)(k) * bias(i)(k)*learningRate/NumExample
  end for
 end for
end for

Then I update all the weights with this:

weight(i)(j,k) = weight(i)(j,k) + deltaWeight(i)(j,k)

Now the problem I have is that the output doesn't seem to get any better. It certainly changes but it doesn't seem to minimise the cost function at all, does anyone know why?

Some reasons the network may not be getting better:

Learning rate is too high (jumps back and forth forever)
Learning rate is too low (when divided by training set size, goes to zero)
You're replacing the error signal with the weight gradient signal.  The reuse of error (based on the indentation) seems to be right on the same line as the inner loop.  I'm not sure if that's correct.  First, backprop your errors, THEN calculate the gradient based on the activation of the hidden neurons and delta sigmoid(error).


What could be the value of having a business analyst in a data science team? Correct me if I am wrong, but while business analysts are often an important role in software delivery teams, but it seems not the case in the data science field.

Commonly available data scientist skill sets state also importance of story-telling skills and understanding customer's business, but is that enough?

I believe the integration of Business Analyst(s) into data science team makes great sense.

For a number of reasons:

there will be a lot of interaction between business proper and the data science team. BAs are used to acting as an interface layer between IT and business functions. This is much harder than it seems, as IT and (for example) Sales & Marketing people are by nature extremely different. It takes a lot of patience and experience to get them both on the same page. Hint: when talking to Sales or Accounting do not start with an icebreak about the virtues of Open Source!
BAs usually have better domain knowledge than DSs. They can be of great help in initial phases of the project as they tend to have a more natural "feeling" for the business (e.g. which outcomes are possible, and which not, are we forecasting a rare event and some under/over sampling is in order, what sort of results are no news and what kind are so surprising that they should be brought to board attention immediately etc.). Data Science types are good at finding the correct answers once a problem is formulated, but asking the right question is arguably more important. And it is a task a BA can help with.
Business Analysis is around for longer, so there is a deep and broad pool of experienced people around. Data Science is young but sexy and looks nice in a resume, so it will be easy to persuade a smart guy/gal with strong understanding of the underlying business and good relationship with the key players in the company to become a project member. It is a win win situation.


One-hot vector for fixed vocabulary given a vocabulary with $|V|=4$ and V = {I, want, this, cat} for example.
How does the bag-of-words representation with this vocabulary and one-hot encoding look like regarding example sentences:

You are the dog here
I am fifty
Cat cat cat

I suppose it would look like this

$V_1 = \begin{pmatrix}
0 \\
0 \\
0 \\
0 \\
\end{pmatrix}$

$V_2 = \begin{pmatrix}
1 \\
0 \\
0 \\
0 \\
\end{pmatrix}$

$V_3=\begin{pmatrix}
0 \\
0 \\
0 \\
1 \\
\end{pmatrix}$

But what exactly is the point of this representation? Does is show the weakness of one-hot encoding with a fixed vocabulary or did I miss something?

library(quanteda)

mytext <- c(oldtext = "I want this cat")
dtm_old <- dfm(mytext)
dtm_old

newtext <- c(newtext = "You are the dog here")
dtm_new <- dfm(newtext)
dtm_new

dtm_matched <- dfm_match(dtm_new, featnames(dtm_old))
dtm_matched

$V_1$
Document-feature matrix of: 1 document, 4 features (100.0% sparse).
         features
docs      i want this cat
  newtext 0    0    0   0

$V_2$
Document-feature matrix of: 1 document, 4 features (75.0% sparse).
         features
docs      i want this cat
  newtext 1    0    0   0

$V_3$
Document-feature matrix of: 1 document, 4 features (75.0% sparse).
         features
docs      i want this cat
  newtext 0    0    0   3

Of course when using a "one hot" vectorizer, "cat" in $V_3$ would be 1 (instead of the count).


Are there Machine Learning Models for Networks? I am working on a regression problem, where the goal is to estimate historic traffic volumes throughout a transportation network. I have traffic counters at 100 locations, so a model can learn the relation between traffic volumes and a number of explanatory variables (e.g., speeds, road characteristics, weather). Afterwards, I can apply the model to estimate historic traffic volumes in places where I don't have traffic counters.

My neural network works reasonably well, but I am wondering if there are machine learning models that could explicitly account for the topology of my road network and the fact that traffic on neighboring road links is highly correlated. I could add "traffic volume at the closest traffic counter" as an input variable to my ANN, but I am wondering if there is a more intelligent approach.

In this regard, I came across Bayesian networks, which can account for the network topology and correlation. However, they seem applicable to cases when we have sensors at 100 locations and we want to predict the traffic state (at these 100 locations) at a future time point. On the other hand, I have measurements at 100 locations and are looking to estimate traffic at a different location for the same time point.

Any suggestion is much appreciated!

Coming from the related field of measuring and predicting network security, I'd strongly suggest trying a time-series forecasting. I assume your data is time-stamp based (network congestion values, sampled at some interval? if not skip to the second idea)

1st idea:
I'd borrow from time-series the concept of flattening the 100 measurements into 1 datum.
So instead of:

[t1+delta1, location1, messurement1]

[t1+delta2, location2, messurement2]

[t1+delta3, location3, messurement3]

Fold into:

[t1-bucketed,  loc1, mess1, loc2, mess2, loc3, mess3.]

This would help the model "grasp" the relation between the different measurements, with emphasis on the time axis

2nd idea
Flatten into each measurement row the closest measurements by topology (or even the entire 100 neighbors) into:

[mess, topol-1-mess, topol-1-dist, topol-1-other, topol-2-mess, topol-2-dist, topol-2-other, ..]

This would help the model "grasp" the relation between a specific measurement and its neighborhood measurements, with emphasis on the topology features from each measurement

Please let us know if that helped :)


Is there any technique, which can decide no. of bags for Bagged Logistic regression? In case of Bagged Logistic regression, people suggest more the bags better will be results.
There should be some threshold for more. Is there any technique available which can suggest no. of bags for the type of data available.

For ex.
Details about data
No. of features - 6
No. of records - 1 M\
Bag size - 20%
No. of Bags - ?

For above mentioned details, Is there any way to devise no. of bags?

From the start I have to state that I am not aware about any paper regarding number of bags for bagged logistic regression. Therefore my assertions apply generally, for any bagged ensemble.

The short answer is no, and I do not think that if possible to be constructed. There are a couple of reasons that I will describe above.

The first reason is that it depends on the complexity of the joint probability you want to estimate. Technically any model you build for prediction purposes aims to estimate a conditional probability space on output variable giving the joint input variables. Doing bagging puts the problem if the estimations from selected samples does cover the that relation in all places. To exemplify, you can have one input categorical variable with 2 levels or with 10 levels. I think more estimations are needed for the later case.

Second reason if the sample itself is representative. Often in practice the sample is not purely a random sample. It contains various interdependencies which can lead to problems. For example if you collect data from multiple countries and you have more samples from some countries simply because of costs, but not related with the phenomena you study.

Third reason is that 20%. A typical bootstrap sample have the same size as original. Reducing the size is interesting because of performance gain or because of induced artificially variance which makes the samples "more independent". However estimating what is the effect of decreasing the sample size is again hard to estimate.

Fourth factor could be also the signal to noise ratio. In other words if you data is noisy enough you need more samples to surface to interesting signal in a stable way.

What you can do?

Since finding the answer to your question is hard, and knowing it is close to similar with knowing the true model, there are things which can be follow in order to find a proper number of bags. What I recommend would be to build repeatedly models with increasing number of bags and see the error estimates. Usually when those estimates have a low variance and adding bags does improve the performance significantly than you can assume you need no more bags and bagging with your chosen model is not able to capture any other structure in your data.


how can autoencoder reduce dimensionality? I can't understand how is dimensionality reduction achieved  in autoencoder since it learns to compress data from the input layer into a short code, and then uncompress that code into the original data I can' t see where is the reduction: the imput and the putput data have the same dimensionality?

Autoencoders are trained using both encoder and decoder section, but after training then only the encoder is used, and the decoder is trashed.

So, if you want to obtain the dimensionality reduction you have to set the layer between encoder and decoder of a dimension lower than the input's one. Then trash the decoder, and use that middle layer as output layer.


Clustering with arrays / vectors as features? I am trying to cluster a large set of documents of which I have a DOC2VEC representation. But I want to cluster them with more features, thus resulting in having both a vector (numpy.asarray(doc2vec_for_document)) and single values as features.

Is this possible? I would like to try clustering with both K-means and DBSCAN of the SKLearn module in Python.

My dataset would look like this:
| Document | DOC2VEC | extra_ft1 | extra_ft2 | ...

Also, what if I had multiple vectors instead of 1 vector and many values?

You need to define a distance function that yields the desirable output.

Usually it will be enough if you can construct it such that d(a,b)for your purpose.

Doc2vec vectors are a bit tricky because they have so many dimensions, and a very strange geometry. It doesn't even seem to be clear whether cosine or Euclidean is to be used on these vectors...

Either way, you have to carefully balance the different features. In the other answer, minmax or stddev scaling was proposed. On one hand, that likely ruins the doc2vec properties, on the other hand this will but much more weight on the word vectors than on the other attributes.

For DBSCAN, you can also follow the "Generalized DBSCAN" approach. Here, the idea is just to define different thresholds for different features. Then neighbors must satisfy all thresholds. I.e. have doc2vec cosine less than A, and other features' distance less than B.
This is likely easier than balancing them as factors inside a single distance function.

But nothing saves you from weighting different feature (sets} carefully.


Is Gini coefficient a good metric for measuring predictive model performance on highly imbalanced data I am evaluating a Credit Risk model that predicts the estimated likelihood of customers defaulting on their mortgage accounts. The model is a Logistic Regression estimator and was built by another team. They use the Gini metric to measure the performance of the model. They achieved 87%. Upon evaluation, I found that the recall was 51% whilst the error rate of the non rare event class (do not default) was 0.9%. Am I correct in thinking that the Gini is actually a misleading metric in this case because it doesn't really show the extremely poor predictive performance of the rare event class? I have questioned them about this and tried to recommend them to use precision/recall metrics as well as confusion matrices and a precision-recall trade-off graph but they quickly dismissed me.

Any advice would be much appreciated.

The Gini Coefficient can also be expressed in terms of the area under the ROC curve (AUC): G = 2*AUC -1 link. The ROC curve, on the other hand, is influenced by class imbalance through the false positive rate FP/(FP+TN). If the number of negatives is a lot larger, this could be a potential issue.

In short, the Gini Coefficient has similar pros and cons as the AUC ROC metric.


What is the point of tensors in CNNs? Why not simply reshape the data into matrices? Take the following tensor:
$$
\left[\begin{array}{cc}
a & b & c\\
d & e & f\\
g & h & i\\
\end{array}\right]
$$
$$
\left[\begin{array}{cc}
j & k & l\\
o & n & m\\
p & q & r\\
\end{array}\right]
$$

Where each matrix represents a channel.

This could be reshaped fairly easily into a vector:
$$
[a,d,g,b,e,h,c,f,i,j,o,p,k,n,q,l,m,r]
$$

And then concatenated row-wise with other vectorized tensors to form a typical flat-file dataset of dimension $N \times P$, where $N$ is the number of training samples and $P$ is the product of the tensor dimensions.

Rather than futzing with a convolutional layer, one could simply constrain ones weights to be zero in the subsequent layer.

If $X$ is a flat $N\times P$ dataset of concatenated vectorized tensors, then the convolutional weights would form a sparse matrix, with the first two columns of a $P \times 4$ convolutional "layer" being $2\times 2\times 2$ filter being
$$
\left[\begin{array}{c}
w 0\\ w0\\ 00\\ ww\\ ww\\ 00\\ 0w\\0w\\00\\w0\\ w0\\00\\ww\\ww\\0w\\0w\\0w\\00
\end{array}\right]
$$

This seems to me more intuitive than the tensor formulation, and could be computed fairly simply using sparse matrix packages.  Perhaps it is partly matter of taste.  But I'm curious: is there anything special about the tensor paradigm -- either mathematically or computationally -- that is superior to the flattened representation?  I understand that computers convert matrix algebra to for-loops "under the hood", but doesn't the advent of the GPU make such explicit looping irrelevant?

Tensors come pretty natural in convolutionals networks.

Local pixel information matters: if $e$ is a pixel in your example above, it's important to know that $a$ through $i$ are its neighbors. This information gets lost when you reshape an image into a vector. Look how a convolutional layer works.

Training usually is done in batches, which is another dimension as far as a neural network is concerned. You don't want to mix different images into a matrix, they are totally independent.
Finally, there is depth channel. Initially it's one of R, G, B channels of the input image, and then each channel corresponds to a filter applied to the previous conv layer. Once again, filters are independent, it doesn't make sense to mix them up until the final layer.

So, in total there are 4-rank tensors going through the conv net. It's not only more intuitive (each dimension has a meaning), but also results in higher accuracy, because it employs all meaningful information from the images.


Random Forest Model Giving Same Accuracy for different feature sets after tuning I am having this weird issue and cannot seem to find a solution. I am trying to tune a different random forest model for every different feature-set. Basically from a given data set, I have created 3 different feature sets. In first case you have all features, in second case you have slightly less number of features, and finally for the 3rd feature set you just have 5 features.
I am trying to tune the model with RandomizedSearchCV.

param_grid = {'n_estimators' : range(1,100),
'max_depth' : np.linspace(1, 50, 5, endpoint=True),
'min_samples_split' : np.linspace(0.1, 1.0, 10, endpoint=True),
'min_samples_leaf' : np.linspace(0.1, 0.5, 5, endpoint=True),
'max_features' : range(1,X.shape[1]),
"criterion": ["gini","entropy"],}

grid = RandomizedSearchCV(clf_rf, param_distributions = param_grid, cv = 5, scoring = 'roc_auc', verbose=True, n_jobs=-1, random_state=1, return_train_score=False)
grid.fit(X,y)

Next, I use the grid.best_estimator_, the best model found by RandomizedSearchCV,   to find  to find the cross validated score.

estimator_cv = grid.best_estimator_
cv_results = cross_validate(estimator_cv, X, y, cv=10)
cv_accuracy = cv_results['test_score'].mean()
cv_accuracy

The issue I am facing is that after following the above steps for the the 3 different feature-sets, I am getting the same cross validated accuracy score of 0.8873372903702229. The roc_auc however is changing.

The fact that their accuracy is the same but their AUC score is different is encouraging. First and foremost, it would be good for you to inspect what is learned by the various trees. A good way to do that is to follow the instructions on that link.

What I think is that all 3 algorithms essentially learned that the same features are useful and that these features are present in each feature set. That is what random forests do, given a lot of features, they inherently "decide" which features are worth using.


Categorical data for sklearns Isolation Forrest I'm trying to do anomaly detection with Isolation Forests (IF) in sklearn.
Except for the fact that it is a great method of anomaly detection, I also want to use it because about half of my features are categorical (font names, etc.)

I've got a bit too much to use one hot encoding (about 1000+ and that would just be one of many features) and I'm anyway looking for a more robust way of data representation.

Also, I want to experiment with other clustering techniques later on, so I don't want to necessarily do label encoding as it will misrepresent the data in euclidean space.

I have thus a two part question:

How will label encoding (ie. ordinal numbers) affect tree based methods such as the Isolation Forest? Seeing as they aren't distance based, they shouldn't make assumptions about ordinal data, right?
What other feature transformations can I consider for distance based models?

I coded isolation forest with dataset containing both categorical and numeric features, and it is working properly. How is it possible.?


Impurity measures in decision trees I have recently stepped into impurity based criteria for decision trees and I was just wondering why do we really need an impurity based criteria model such as the Gini index? What if we could simply label the entity with the majority class? What would be the effect on results?

In general, every ML model needs a function which it reduces towards a minimum value.

DecisionTree uses Gini Index Or Entropy.
These are not used to Decide to which class the Node belongs to, that is definitely decided by Majority.
At every point - Algorithm has N options(based on data and features) to split. Which one to choose.

The model tries to minimize weighted Entropy Or Gini index for the split compared to the parent. Which indirectly imply the cleanliness of split.
 Entropy is not used because it uses log which has a higher computational cost.

Check this sample code

#Gini
a = [0, 0, 0, 0, 1, 1, 1, 1, 1, 1]
p(0) = 0.4, p(1) = 0.6
gini = 0.4*(1-0.4) + 0.6*(1-0.6) #0.40

########Split - I

#Let's split and calculate the weighted dip
a1 = [0, 0, 0, 0, 1] ; a2 = [1, 1, 1, 1, 1]
gini_a1 = 0.8*(1-0.8) + 0.2*(1-0.2) #0.32
gini_a2 = 1*(1-1) # 0

#Weighted
gini = (5/10) * gini_a1 + (5/10)*gini_a2 #0.16

########Split differently
#Let's split and calculate the weighted dip
a1 = [0, 0, 0, 0] ; a2 = [1, 1, 1, 1, 1, 1]
gini_a1 = 1*(1-1) #0 - clean split
gini_a2 = 1*(1-1) # 0

#Weighted
gini = (5/10) * gini_a1 + (5/10)*gini_a2 # 0

Obviously 2nd split is better because Gini reduction is more
Case 1 - 0.40 --> 0.16
Case 2 - 0.40 --> 0

You may think, why not simply use Accuracy to decide the split.
Check this blog


How is "relevance" defined in information retrieval outside the context of systems with user feedback? I've seen information retrieval systems that return some results from a query, and then the user rates these results as either "relevant" or "not relevant".

What can you do if you do not have user feedback?

E.g. suppose your system returns some ranked results from a query. Suppose you have no pre-defined notion of what is relevant, and suppose you cannot receive any kind of user feedback. What can you do?

This is important, because information retrieval evaluation metrics are based on relevance. Maybe it isn't possible to define relevance without user feedback, if so, maybe some information retrieval evaluation metrics without dependency on user feedback can be suggested?

There is no formal definition for the concept of relevance, because relevance depends completely on the context and is therefore highly subjective. This is why the best way (some might say the only way) to evaluate relevance is to actually ask users what is relevant for them.

For any ML-based task, one needs to design a proper evaluation framework in order to  control and measure the quality of the results. Naturally the evaluation method should be chosen so that it reflects as much as possible the level of quality with respect to the goal of the task, i.e. what one would intuitively expect from it. Evaluation metrics are almost always simplified indicators of this "level of quality", so what matters is how well they correlate with what a user would expect from the system: sometimes even a perfectly standard evaluation measure might not to be suited to the goal of the task.

My point is that evaluation is a matter of analysis and design. There are infinitely many options, but the point is to select the most appropriate one for the job. Here are some of these options:

The ideal case is to have annotated data (e.g. user feedback) directly suited to the data and the task: then it's just a matter of counting how often the prediction is correct.
It's common to evaluate a system against another annotated dataset X, assuming that the task is similar enough so if the system works well on X then it will work well on the real dataset.
Another less than ideal way is to evaluate against the predictions of another reference system X: in this case X is considered the gold standard, so there is no way for the tested system to perform better than X.
Indirect evaluation: if there is another task being performed at a later stage with the predictions, and this task can be evaluated more easily than the IR task itself.
Heuristics: that would be the least reliable kind of evaluation, but it's better than nothing. It ranges from simply counting the number of words in common between the query and top N results to developing complex methods using third-party resources.


Remove all columns where the entire column is null I have a very dirty csv where there are several columns with only null values.

I would like to remove them. I am trying to select all columns where the count of null values in the column is not equal to the number of rows.

clean_df = bucketed_df.select([c for c in bucketed_df.columns if count(when(isnull(c), c)) not bucketed_df.count()])

However, I get this error:

SyntaxError: invalid syntax
  File "<command-2213215314329625>", line 1
    clean_df = bucketed_df.select([c for c in bucketed_df.columns if count(when(isnull(c), c)) not bucketed_df.count()])
                                                                                ^
SyntaxError: invalid syntax

If anyone could help me get rid of these dirty columns, that would be great.

[Updated]: Just realized it is about pyspark!
It is still simple! A concrete example (idea heavily borrowed from this answer):
Creating a dummy dataset
import pandas as pd
import numpy as np
import pyspark.sql.functions as sqlf

main= pd.DataFrame(np.random.randint(0,100,size=(100, 4)), columns=list('ABCD'))

main["E"]= np.NAN
main["F"]= np.NAN

df = sqlContext.createDataFrame(main)

Function to drop Null columns
def drop_null_columns(df):

    """
    This function drops columns containing all null values.
    :param df: A PySpark DataFrame
    """

    null_counts = df.select([sqlf.count(sqlf.when(sqlf.col(c).isNull(), c)).alias(c) for c in df.columns]).collect()[0].asDict()
    to_drop = [k for k, v in null_counts.items() if v >= df.count()]
    df = df.drop(*to_drop)

    return df

Outcome
df_dropped = drop_null_columns(df)


Two ways of optimize the same function? I'm actually reading this tutorial about deepLearning and in particular about Logistic Regression.

I don't get why it first says to optimize logistic regression taking the max Probability and after using the Log loss function ?

Sorry you can explain me the point where him will use the Argmax and where Will use the Loss ?

you would not need only 1 of this 2 ?

The argmax is to get the class label prediction y_pred given a sample x.

# symbolic description of how to compute prediction as class whose
 # probability is maximal
 self.y_pred = T.argmax(self.p_y_given_x, axis=1)

The Loss function is used as optimization objective function to solve the coefficients W and b based on training data. On the tutorial, the loss function is the reverse the likelihood.


Appropriate way to store data in R I have data, which looks like this:

These data are only for one subject. I will have a lot more.
These data will be analyzed in R.

Now I'm storing them like this:

subject <- rep(1, times = 24)
measurement <- factor(x = rep(x = 1:3, each = 8),
                      labels = c("Distance", "Frequency", "Energy"))
speed <- factor(x = rep(x = 1:2, each = 4, times = 3),
                labels = c("speed1", "speed2"))
condition <- factor(x = rep(x = 1:2, each = 2, times = 6),
                    labels = c("Control", "Experm"))
Try <- factor(x = rep(x = 1:2, times = 12),
              labels = c("Try1", "Try2"))
result <- c(1:8,
            11:18,
            21:28)

dt <- data.frame(subject, measurement, speed, condition, Try, result)

What is the appropriate way to store these data in R (in a data frame)?

Without more information all I can say is that:

the say you're storing it is fine in general
you can further transform/store your data depending on your use case

To expand on #2, if I want to study Distance vs Energy across all subjects, then I would format my data like this:

> library(reshape2)
> dt2 <- dt[dt$measurement %in% c('Distance','Energy'),]
> dt_cast <- dcast(dt2, subject+Try~measurement+speed+condition, value.var='result')

The transformed data (dt_cast) would then look like:

  subject  Try Distance_speed1_Control Distance_speed1_Experm Distance_speed2_Control
1       1 Try1                       1                      3                       5
2       1 Try2                       2                      4                       6
  Distance_speed2_Experm Energy_speed1_Control Energy_speed1_Experm Energy_speed2_Control
1                      7                    21                   23                    25
2                      8                    22                   24                    26
  Energy_speed2_Experm
1                   27
2                   28

Allowing me to, for example, look at the relationship between the Distance_speed1_Control vs Energy_speed1_Control columns.

Basically subset/aggregate your data and then use the dcast to get the rows and columns the computer needs.


Conjugated gradient method. What is an A-matrix in case of neural networks I am reading about conjugated gradient methods to understand how they exactly work. I understand that a pair of vector $u$ and $v$ are conjugated with respect to $A$ if $u^TAv=0$. I also read that $A$ is symmetric, positive definite matrix.

I am trying to find out how is that related to training of neural network by minimalising mean-square error function using CG method. What will be $A$ matrix in that case? How $A$ matrix is connected to weights of neural network. And is it still symmetric and positive definite? I read What is conjugate gradient descent? this thread and resources linked there, but I still can't figure it out.

I'm sure I'm missing something simple, but could you give me a bit of explanation?

Thank you,

Max

The traditional conjugate gradient descent is an increment on the gradient descent that just takes a direction that is fully orthogonal to the previous descent direction. There is no $A$ matrix in that case.

There are different rules (you can check some in my old optimization toolbox at https://github.com/mbrucher/scikit-optimization/blob/master/scikits/optimization/step/conjugate_gradient_step.py). If I remember properly FR combined with strong Wolfe-Powell line search rule give one of the best answer. The issue is that it requires more computation, which is why line search is never used in neural networks optimization.


Best file format for transfer of EHR data I am working on a clinical trial where we have several sites sending us EHR data.  The sites are currently sending the data in excel files.  I have a feeling someone's opening the files because 3 of the files have 64,999 rows exactly, and excel 2007 cuts off at 65,000.

I am working in python, but I am trying to prevent the people at the local sites from opening the files in excel.

What's the best format for the files to be sent to me such that they can't be opened and cut off in excel?  csv unfortunately can also be opened in excel.

If your are using python, I suggest using Numpy's npy format. An NPY file is a NumPy array file created by the Python software package with the NumPy library installed. It contains an array saved in the NumPy (NPY) file format. NPY files store all the information required to reconstruct an array on any computer, which includes dtype and shape information.

You can also consider using hdf or hdf5 formats both of which are supported in Python. Hierarchical Data Format (HDF) is an open source file format for storing huge amounts of numerical data. It’s typically used in research applications (meteorology, astronomy, genomics etc.) to distribute and access very large datasets without using a database. One can use HDF5 data format for pretty fast serialization to large datasets.


Why does the naive bayes algorithm make the naive assumption that features are independent to each other? Naive Bayes is called naive because it makes the naive assumption that features have zero correlation with each other. They are independent of each other. Why does naive Bayes want to make such an assumption?

By doing so, the joint distribution can be found easily by just multiplying the probability of each feature whilst in the real world they may not be independent and you have to find the correct joint distribution. It is naive due to this simplification.


how to derive the steps of principal component analysis? I'm trying to understand the theoretical reasoning behind the method, but I can't understand a particular step in the middle of this page.

"The constraint on the numbers in $v_1$ is that the sum of the squares of the coefficients equals $1$. Expressed mathematically, we wish to maximize
$$\frac1N\sum_{i=1}^NY_{1i}^2$$

where
$$y_{1i}=v_1'z_i,$$

and $$v_1'v_1=1$$

(this is called "normalizing" $v_1$).
Computation of first principal component from $R$ and $v_1$. Substituting the middle equation in the first yields

$$\frac1N\sum_{i=1}^NY_{1i}^2=v_1'Rv_1."$$

I don't understand how $R$ suddenly appeared in this equation. The right hand side "$v_1' R v_1$" seems to have appeared out of nowhere.

Please help

\begin{align}
\frac1N\sum_{i=1}^N Y_{1i}^2 &= \frac1N\sum_{i=1}^N (v_1'z_i)(z_i'v_1)\\
&=v_1'\left(\frac1N \sum_{i=1}^N z_iz_i' \right) v_1 \\
&= v_1'Rv_1
\end{align}

where $R=\frac1N \sum_{i=1}^N z_iz_i'.$


Calculating a Confusion Matrix Can someone help me understand how to find the values of a confusion matrix?

I know that essentially a confusion matrix looks like this:

True Positive |False Positive
False Negative|True Negative

I've encountered a problem where I have around 1000 cases in my test data.

In the approximate middle of its ROC chart there is a point where the false positive rate is 0.5, the true positive rate is 0.7, and the accuracy is 0.7.

This is my approach to this problem and I just want to verify if I am doing it correctly:

To calculate the count of True Positives:

$$0.7 * 1000\,Cases = 700\,Cases$$

To calculate the count of False Positives:

$$0.5 * 1000\,Cases = 500\,Cases$$

I would assume calculating the True Negatives would be $1000 - 700 = 300$.

Then solving for False Negatives =

$$.7 (Accuracy) = 700 + 300 / 700 + 300 + 500 + FN$$

Can someone confirm if this is the right approach?

Your approach is not correct.

Some hints that should help you to understand your mistakes

the sum over all elements of the confusion matrix should be 1000 in your case.
TPR=TP / Num_positives

Good guide to confusion matrix terminology


Histogram of some values only I have a pandas dataframe df, and I want to show the histogram.

df.hist(bins=100, label="myhist")

Now, for some reason I have lots of zeros in this df, so I only want to show the values between 1 and 100. I tried

df.hist(bins=(1,100), label="myhist")

but that gives a flat line which has nothing to do with the data.

How do I do it?

Ok, after some digging around I found that I can pass a range = (1,100) and that does the trick.


File format where column names are repeated on each row I have received a dataset in text file with the following format

col1=datac1r1,col2=datac2r1,col3=datac3r1
col1=datac1r2,col2=datac2r2,col3=datac3r2
col1=datac1r3,col2=datac2r3,col3=datac3r3
col1=datac1r4,col2=datac2r4,col3=datac3r4

Each row is a unique entry, with columns separated by comma, just that the column name is repeated in each element.

I need to parse this in R and analyze it. I have worked with csv files extensively, but I have never seen this format before.

Is it a std. format I can import it in? Or do I need to write a script to convert it into a csv format?

Using Miller (https://github.com/johnkerl/miller)

mlr --ocsv unsparsify input.txt

you will have this CSV

col1,col2,col3
datac1r1,datac2r1,datac3r1
datac1r2,datac2r2,datac3r2
datac1r3,datac2r3,datac3r3
datac1r4,datac2r4,datac3r4