Best approach to find optimal solution to linear equation by group in R I am currently modeling a pricing and discount system in R.

My data frame looks as follows:

df = structure(
  list(
    Customers = structure(
      c(1L, 1L, 1L, 2L, 2L, 2L),
      .Label = c("A", "B"),
      class = "factor"
    ),
    Products = structure(
      c(1L,
        2L, 3L, 1L, 2L, 3L),
      .Label = c("P1", "P2", "P3"),
      class = "factor"
    ),
    Old_Price = c(5, 2, 10, 7, 4, 8),
    New_Price = c(6, 3, 9,
                  6, 3, 9)
  ),
  class = "data.frame",
  row.names = c(NA,-6L)
)

There are several customers who buy different products with an "Old Price" and a "New Price". I now want to identify one discount parameter (a real from -1.0 to 1.0) for each customer that minimizes the difference of Old Price and New Price.

Because I do not know a lot about optimizations, etc. my current approach would be to do the following, which seems horribly inefficient and might not lead to the optimal solution anyway:

df %>%
    mutate(Individual_Discount = (Old_Price-New_Price)/New_Price) %>% # Identify optimal discount individually
    group_by(Customers) %>%
    mutate(Optimal_Discount = mean(Individual_Discount)) # Average individual discount to get approximate discount for customer

What's the best approach to solve a case like this and how can I implement it in R?

Update:

To clarify the problem more plainly. There is a data frame that looks like this:

Customers | Product | Old Price | New Price | Delta  | Discount | Discounted New Price
CustA     | ProdA   | 10.00     | 12.00     |  2.00  | -0.167   | 10.00
CustA     | ProdB   | 30.00     | 25.00     | -5.00  |  0.2     | 30.00
CustB     | ProdA   | 15.00     | 12.00     | -3.00  |  0.25    | 15.00
CustB     | ProdB   | 20.00     | 25.00     |  5.00  | -0.2     | 20.00

The discount represents the optimal discount to reduce the difference between the old and new price to zero (So a new price 2 would be calculated as New Price + New Price * Discount). However each customer can only get one discount, so which discount should I choose per customer to minimize the remaining deltas (the delta between the discounted new price and the old price)?

Update2: Mathematical relationships

Delta = New_Price - Old_Price

Discount = Delta / -New_Price

Discounted_New_Price = New_Price+New_Price*Discount

Update3:

I have fitted a linear model based on the comments but a "linear discount" based on the gradient of the grouped linear model yields worse results than my "mean hack":

df %>%
  group_by(Customers) %>%
  do({ co <- coef(lm(Old_Price ~ New_Price, .))
       mutate(., linear_discount = co[2])
  }) %>%
  ungroup %>%
  mutate(linear_discount = 1/linear_discount-1) %>%
  mutate(linear_price = New_Price+New_Price*linear_dis

The results are

Customers | Product | Old Price | New Price | Linear Discount  | Linear Price | Discounted New Price
CustA     | Prod1   | 05.00     | 06.00     |  -0.25           | 4.50
CustA     | Prod2   | 02.00     | 03.00     |  -0.25           | 2.25
CustA     | Prod3   | 10.00     | 09.00     |  -0.25           | 6.75
CustB     | Prod1   | 07.00     | 06.00     |   0.50           | 9.00
...

You could use linear regression without intercept in order to accomplish this.

If I understand correctly, your implementation of linear regression in Update 3 had two issues: first, you fit the linear model with intercept, but then drop the intercept when applying the model. Second, there is a typo in the following line

linear_discount = 1/linear_discount-1

In the code snippet below I extended your analyses by two approaches: linear regression without intercept and linear regression with intercept. The former case should correspond exactly to your requirements, i.e. using a single relative discount rate per customer. The latter one corresponds to the case with relaxed assumptions, i.e. one relative discount rate and one absolute discount (independent on the price level).

Note that for the illustrative dataset you provided the optimal (in terms of mean squared error) discount rate would be zero.

df_ext <- df %>%
  # mean hack
  mutate(Discount_indiv = (Old_Price-New_Price)/New_Price) %>% # Identify optimal discount individually
  group_by(Customers) %>%
  mutate(Discount_mean = mean(Discount_indiv),
         Old_Price_mean = New_Price + New_Price*Discount_mean) %>% # Average individual discount to get approximate discount for customer

  # incorrect linear regression (fitted with intercept, then intercept dropped)
  do({ co <- coef(lm(Old_Price ~ New_Price, .))
  mutate(., Discount_wrong = co[2])
  }) %>%
  ungroup %>%
  mutate(Discount_wrong = 1 / Discount_wrong-1) %>%
  mutate(Old_Price_wrong = New_Price+New_Price*Discount_wrong) %>%

  # correct linear regression w/o intercept
  group_by(Customers) %>%
  mutate(Discount_regr = coef(lm(Old_Price ~ New_Price -1))[1] - 1,
         Old_Price_regr1 = New_Price+New_Price*Discount_regr) %>%

  # correct linear regression with intercept (i.e. two discounts per customer)
  mutate(
    Discount_abs = coef(lm(Old_Price ~ New_Price))[1],
    Discount_rel = coef(lm(Old_Price ~ New_Price))[2] - 1,
    Old_Price_regr2 = New_Price + New_Price*Discount_rel + Discount_abs) %>%

  #calculate residuals
  ungroup() %>%
  mutate(Resid_mean = Old_Price_mean - Old_Price,
         Resid_wrong = Old_Price_wrong - Old_Price,
         Resid_regr1 = Old_Price_regr1 - Old_Price,
         Resid_regr2 = Old_Price_regr2 - Old_Price)

#transform data for visualisation
df_gat <- select(df_ext, matches("Customers|Price")) %>%
  gather(key="Approach", value="Old_Price", -Customers, -New_Price)

ggplot(df_gat, aes(x=New_Price, y=Old_Price, group=Approach, colour=Approach, shape=Approach)) +
  geom_line() + geom_point(size=3) + facet_wrap(~Customers, ncol = 1)

select(df_ext, matches("Customers|Disc"))
select(df_ext, matches("Customers|Price"))

#calculate mean square error for all approaches
select(df_ext, matches("Customers|Res")) %>%
  group_by(Customers) %>%
  summarise_all(~mean(.^2))

Results for all approaches: MSE by customer & some visualizations

  Customers Resid_mean Resid_wrong Resid_regr1 Resid_regr2
  <fct>          <dbl>       <dbl>       <dbl>       <dbl>
1 A               1.71        3.62       1.          0.222
2 B               1.71       11.5        1.000       0.222


KL-divergence returns infinity Given an original probability distribution P, I want to measure how much an approximation Q differs from the initial distribution. For that I calculate the KL-divergence via scipy.stats.entropy, which returns infinity due to the large difference. However, as with time the approximation becomes better, I still want to quantify the divergence between the two sets.

The question is, is there any hack to avoid inf values or should I circumvent the behaviour by using some other distance measure?

The infinity is due to divide by zero.
Just replace any zero value with a very small value.
This question is very common , if you search for it you will find many questions similar to it .
Remember that kl is not a metric


MLP incremental learning In my project I use 2 MLP ANNs: for classification and for prediction. Basic training dataset is stored in database (MS SQL Server). But we need to make future incremental learning possible. It means that ANN is used in industrial realtime system, operator marks observations in database as objects of a particular class and ANN must take it into account. How can I do it?

If your old data is representatively of the underlying population and there is no radical shift of the underlying true model over time. Than you don't need to update you model as new data comes in. You might want to retrain you model with the new dataset periodically, but you don't need to do it in real time on your production machine.

If the underlying true does change radically overtime and you don't have any existing data to captures those changes then doesn't matter what algorithm you use, it's not going to work. The information is simply not there.


Document embedding vs locality sensitive hashing for document clustering I would like to compare two methods: locality sensitivity hashing  and document embedding to get the similarity between two documents. Both of those methods encode information of a document in a vector which I would like to use to find similar documents in a very large corpus (potentially more than 100 000 documents). Have anybody ever compared those two methods and what are the advantages of each of them.

Cheers in advance

Locality sensitive hashing (LSH) is a search technique. With it, similar documents get the same hash with higher probability than dissimilar documents do. LSH is designed to allow you to build lookup tables to efficiently search large data sets for items similar to a given item. It is also a probabilistic method in that it can generate false positives and false negatives. While there are ways to train LSH, most LSH is untrained. That's because LSH has been studied more in the search setting than the machine learning setting.

Embeddings are a machine learning technique to capture semantic information for use in some downstream task, such as clustering or classification. Typically semantically similar items get similar (but not the same) embeddings. Embeddings are trained from data. There are many unsupervised algorithms (word2vec, glove) and there are supervised methods too (auto-encoders, hidden layer output from deep models).

Embeddings can be used to map items into a space where near neighbor search would find semantically similar items. However, on large data sets you would still need to index the data to search efficiently, which raises the possibility of doing LSH on embeddings. That way you get the benefit of a trained model that learns the distribution of your data set and the benefit of a fast lookup table.


What model should I choose for Jobs recommender system? Now, I have enormous number of resumes of someone who search for next jobs.
There are infomation about how old they are, what schools they graduated,and
what jobs they experienced.etc...

I want to make a recommender system using those data.
In the system, if I get new resume, I want to recommend certain jobs for him.

I have some idea what models will I use, for example,
collaborative filtering, recurrent neural network, word2vec etc...

But I couldn't find best method for this problem.
If someone Knows good idea, please tell me that.

This is an example of one data,

[age,education,skills,past_job1,job1's occupational category, past_job2,job2's occupational category,
...,]

All data is stored in string.

One method is using locality-sensitive hashing (LSH), an approximate nearest neighbor search method.

Each document, both resumes and job descriptions, is hashed into the same space. There are several ways to perform the hashing. An older method is shingling. The entire process is outlined in Chapter 3: Finding Similar Items in Mining of Massive Datasets. A newer method is doc2vec.

Once all the documents are in the same space, LSH maps similar items to the same “buckets” with high probability. It is a type of hashing where collisions are features, not bugs. The collisions are a variation of clustering. Given a new document, all similar documents are retrieved also. Given a resume, retrieve similar job descriptions. Given a job description, retrieve similar resumes. (After retrieval, filter out documents in the same document class)

Since LSH is an approximate algorithm, it scales well to millions of documents and handles noisy data. Resumes and job descriptions are noisy descriptions so it is appropriate to have an algorithm that can handle noise.


Help getting corresponding dataframe values I have two dataframes:

self.thisSession_df:
    id   exerciseId  sets
    1       1         12
    2       1         14
    2       2         15
    2       2         15

self.exercises_df:
    id   exerciseName
    1      Squat
    2      Pullup

I would like to find a way to replace the exerciseId in self.thisSession_df with the corresponding name from self.exercises_df

Hopeful result:

self.thisSession_df:
    id   exerciseId    sets
    1       Squat       12
    2       Squat       14
    2       Pullup      15
    2       Pullup      15

I tried a solution that I found on here and modified it to come up with:
self.thisSession_df['exerciseId'] = self.thisSession_df['exerciseId'].map(df1.set_index('id')['exerciseName'])

This gives me error: string indices must be integers

I would appreciate a nudge in right direction!

I would recommend merging your two dataframes to get the exercise names:

exercise_name_df = thisSession_df.merge(exercises_df, left_on='exerciseId', right_on='id')

This will give you a dataframe like

exercise_name_df:
    id   exerciseId  sets  exerciseName
    1       1         12      Squat
    2       1         14      Squat
    2       2         15     Pullup
    2       2         15     Pullup

Then you can replace exerciseId with the name if you really want to:

# reassign id to name
exercise_name_df['exerciseId'] = exercise_name_df['exerciseName']
# drop redundant column
exercise_name_df.drop(columns=['exerciseName'], inplace=True)

I would argue that the above method is more readable than the map() solution. But I think the map() solution should also work with a small tweak:

thisSession_df['exerciseId'] = thisSession_df['exerciseId'].map(exercises_df.set_index('id')['exerciseName'])


What are some data sources with strong positive and negative correlations in feature and sample space? For example, financial data has in some cases strong positive and negative correlations between different stocks (feature space) and also strong positive and negative correlations between specific dates (sample space), if surprising positive or negative news was released on those dates.

What are other sources of continuous data, such as audio, visual, or data obtained by some measurement instrument or process, that display the above characteristics? See also "transposable data".

Importantly, it seems that in the sample space often negative correlations are non-sensical or non-causal, which limits the types of data sources with positive and negative correlations in both sample and feature space.

A couple of spatio-temporal datasets are likely to exhibit this characteristic:

Transportation demand across different locations of a city over different hours of a day. While residential locations have high demand originating there in the morning hours, and low demand in the evening, the reverse is true of work locations. The demand pattern of these two location types will be negatively correlated, whereas that of two different residential locations (or two locations of the same type) will be positively correlated. From the point of view of hours, correlation between two morning hours will be positive, whereas between a morning and an evening hour will be negative.
Temperature pattern across days, for different cities across the world. Here, data for cities in the northern and southern hemisphere will be negatively correlated.


Multiple-output vs single-output NNs I'm trying to build a 5 input-5 output model using LSTM, where all the outputs are the same features as the inputs, predicted in the future.

My question is: is it better to build 5 models, each with the same 5 inputs, but predicting just 1 of the 5 sequences at a time, or is it the same as building 1 model predicting all 5 sequences?
In other words, is the accuracy per predicted sequence going to be higher with 5 separate models or will it be the same as 1 model with 5 outputs.

The reason for my confusion is that, in the case of the multiple output model, the hidden layer will be the same; so how would the algorithm go about optimizing the weights so as to minimize error for all output sequences?

In general, you cannot know which is better until you try both with lots of data and evaluate them with a strong statistical test.
I know you won't like this answer but this is how neural networks work.
Hope this helped


Is smoothing in NLP ngrams done on test data or train data? Is smoothing in NLP ngram done on test data or train data?
Since smoothing is to avoid the language model predicting 0 probability of unseen corpus (test). So I wonder is smoothing done on test data only? Or on train data only? Or both? I don't seem to find an answer to this yet.

Is smoothing in NLP ngram done on test data or train data?

In short: both.
Smoothing consists in slightly modifying the estimated probability of an n-gram, so the calculation (for instance add-one smoothing) must be done at the training stage since that's when the probabilities of the model are estimated.
But smoothing usually also involves differences at the testing stage, in particular for assigning a probability to unknown n-grams instead of 0.


How to define discrete action space with continuous values in OpenAI Gym? I am trying to use a reinforcement learning solution in an OpenAI Gym environment that has 6 discrete actions with continuous values, e.g. increase parameter 1 with 2.2, decrease parameter 1 with 1.6, decrease parameter 3 with 1 etc.

I have seen in this code that such an action space was implemented as a continuous space where the first value is approximated to discrete values (e.g. 0 if it is < 1 or 2 if it is < 2 and > 1).

Does anybody know if the above solution is the correct way to implement such an action space? Or does Gym offer another way?

Here is a sample environment which demonstrates this. It relies on the environment to successfully filter out the correct continuous control element

import gym
from gym.spaces import Dict, Discrete, Box, Tuple
import numpy as np

class SampleGym(gym.Env):
    def __init__(self, config={}):
        self.config = config
        self.action_space = Tuple((Discrete(2), Box(-10, 10, (2,))))
        self.observation_space = Box(-10, 10, (2, 2))
        self.p_done = config.get("p_done", 0.1)

    def reset(self):
        return self.observation_space.sample()

    def step(self, action):
        chosen_action = action[0]
        cnt_control = action[1][chosen_action]

        if chosen_action == 0:
            reward = cnt_control
        else:
            reward = -cnt_control - 1

        print(f"Action, {chosen_action} continuous ctrl {cnt_control}")
        return (
            self.observation_space.sample(),
            reward,
            bool(np.random.choice([True, False], p=[self.p_done, 1.0 - self.p_done])),
            {},
        )

if __name__ == "__main__":
    env = SampleGym()
    env.reset()
    env.step((1, [-1, 2.1]))  # should say use action 1 with 2.1
    env.step((0, [-1.1, 2.1]))  # should say use action 0 with -1.1
```


In CycleGAN are there two different generators and two different discriminators? I am trying to assimilate the contents of this paper. I have a confusion about how many different networks are there in the architecture of CycleGAN. To my understanding, the concept of cycle means that there is only one generator and one discriminator which changes roles (Discriminator as generator; and Generator as Discriminator) depending on what loss we want to calculate (whether Y -> X or X -> Y). However, my friend I have a confusion and we have been debating about this without a consensus. He says that there are totally four different networks. Two generators G1 and G2; and two discriminators D1 and D2. What is the truth ?

There are two generators and two discriminators.

Check out the code here


Can the learning rate be considered both a parameter AND a hyper-parameter? Here is my understanding of those 2 terms:

Hyper-parameter: A variable that is set by a human before the training process starts. Examples are the number of hidden-layers in a Neural Network, the number of neurons in each layer, etc. Some models don't have any hyper-parameters, like the linear model.

Parameter: A variable in which the training process will update. For instance, the weights of a Neural Network are parameters as they are updated as we train the network and there is no human intervention on the process. Another example would be the slope and the y-intercept in a simple linear model.

Having said that, what would the learning rate parameter ($\eta$) be?

$$
\Theta_{i+1} = \Theta_{i} + \eta \nabla J(\Theta_{i} )
$$

My understanding is that $\eta$ is set before the training starts to a large value but then, as the training progresses and the function gets closer and closer to a local minimum, the learning rate is decreased. In that case, doesn't the learning parameter satisfy both the definitions of a parameter and of a hyper-parameter?

My understanding is that $\eta$ is set before the training starts to a large value but then, as the training progresses and the function gets closer and closer to a local minimum, the learning rate is decreased. In that case, doesn't the learning parameter satisfy both the definitions of a parameter and of a hyper-parameter?

No it does not, because you are at all times controlling the learning rate without reference to the data. Typically both the learning rate and the learning rate schedule are considered hyper-parameters, so by adding a learning rate schedule you have not transformed learning rate into a parameter, but instead added a new hyper-parameter that may need tuning!

The learning rate is also is not part of the model as it would be used in production, so it is a training hyper-parameter. Although that is not a strict separation, this also applies to things that are neither hyper-parameters nor parameters, such as the current momentum values, which are used to improve training rate, and are usually stored in a some data structures alongside the model being trained. Likewise most adaptive learning rate structures (used in e.g. RMSProp, Adagrad etc) are in this category - neither model parameter nor hyper-parameter.

It might be possible to argue that for a continuous online model, the learning rate could be a model parameter, as it is part of the production model being used to make predictions - provided it was still being used, but adaptive, depending on data seen to date, and/or current learning progress. Then if you had to feed in an initial learning rate you could maybe say that the learning rate was both a parameter and hyper-parameter of the ANN model. Although you would likely have to explain the scenario in full to be understood, it would be more usual to just say you were using an adaptive learning rate.


Model trained on CIFAR 10 performing badly when fed with internet downloaded images My model is based on Shallow Net.
When I am training my model, the results are:
loss: 1.1398 - accuracy: 0.6093 - val_loss: 1.2309 - val_accuracy: 0.5657

Then I downloaded 20 images (2 for each class) from the net to check the performance.

Labels corresponding to this dataset should be:0,0,1,1,2,2,3,3,4,4,5,5,6,6,7,7,8,8,9,9].
But my model's prediction is: [0,0,0,1,0,1,5,0,0,5,2,2,0,0,5,2,0,0,1,9].
The accuracy is: 0.2 which is quite low as compared to 0.5657.
My code to load these datasets:
for file in os.listdir("C:/Users/....."):
    img_arr=cv2.imread(os.path.join(os.getcwd(),"Dataset",file))
    img_arr=cv2.resize(img_arr,(32,32))/255
    img_arrs.append(img_arr)

img_arrs=np.array(img_arrs)
img_arrs=img_arrs.reshape(20,32,32,3)
model=load_model("weights.hdf5")
pred=model.predict(img_arrs).argmax(axis=1)

What could be the reason behind this? Can someone give me an insight?
Edit:(Added training code)
(x_train,y_train),(x_test,y_test)=cifar10.load_data()
x_train=x_train.astype(float)/255
x_test=x_test.astype(float)/255
lb=LabelBinarizer()
y_train=lb.fit_transform(y_train)
y_test=lb.transform(y_test)

labelNames = ["airplane", "automobile", "bird", "cat", "deer","dog", "frog", "horse", "ship", "truck"]

model=ShallowNet.ShallowNet.build(width=32, height=32, depth=3, classes=10)
sgd=SGD(0.001)
model.compile(optimizer=sgd,loss="categorical_crossentropy",metrics=["accuracy"])
H=model.fit(x_train,y_train,validation_data=(x_test,y_test),batch_size=32,epochs=50,verbose=1)

Here are some possibilities that come to mind:

The ordering of the dimensions. Depending on the network you are training with and the original training data, you may need to transpose some of the dimensions and/or reshape them differently. The original cifar 10 data is a little strange in that the color-dimension precedes the height/width dimensions. It may help to visualize both an image from the training data and the internet data side by side with the same function and ensure they are displaying the same way as a test of this.
Was any pre-processing performed on the training images, for example rescaling of values? If so, the same pre-processing should be performed on these images. Even if pre-processing was not explicitly performed image data can be stored in a variety of different ways. I would check the range of values on your training/validation images and compare against the images you are getting from the net.
Something funky is possibly happening when these images are resized. I would suggest visualizing the above examples after they have been resized and confirming that they are reasonable.
Another possibility is that something unexpected is happening when loading the model - it may seem redundant given the validation results but I would nonetheless take a set of 20 or so images from the original validation set and check the predictions after retrieving the model with the above load_model function.


Classes of neural nets and their applications Would you say, you could design,  tune and/or train any DNN for any application, or do their designs inherently postulate some specialization?
Is there such a review?

For example, are CNNs better for the spatial domain (e.g image analysis), and RNN for temporal (e.g. time series data, or speech), or it does not matter?

Convolutional Neural Networks have consistently outperformed other methods for image recognition and related tasks. In fact, beyond a certain image size it is not practical to train a fully-connected network on raw pixels due to the enormous number of parameters that would be required. The number of parameters in a CNN however grows much more slowly as you increase the input image size, since they usually do not require larger filters, and the number of parameters per filter is independent of input size.
With that said, there is a much more remarkable demonstration of the power of the architecture of a CNN for image processing in the recent paper Deep Image Prior. The abstract reads:

Deep convolutional networks have become a popular tool for image generation and restoration. Generally, their excellent performance is imputed to their ability to learn realistic image priors from a large number of example images. In this paper, we show that, on the contrary, the structure of a generator network is sufficient to capture a great deal of low-level image statistics prior to any learning. In order to do so, we show that a randomly-initialized neural network can be used as a handcrafted prior with excellent results in standard inverse problems such as denoising, superresolution, and inpainting. Furthermore, the same prior can be used to invert deep neural representations to diagnose them, and to restore images based on flash-no flash input pairs.
Apart from its diverse applications, our approach highlights the inductive bias captured by standard generator network architectures. It also bridges the gap between two very popular families of image restoration methods: learning-based methods using deep convolutional networks and learning-free methods based on handcrafted image priors such as self-similarity.

So they were able to demonstrate that the raw network prior to any training was powerful enough for sophisticated tasks, which is the strongest and most direct illustration of the power of the specific architecture itself that I have seen.
The answer is not so clear for time series prediction, since CNNs with various tricks have performed on par with RNNs for some datasets, but in general RNNs seem much more suited to the problem, since they can handle inputs of arbitrary length, and they can store state.
However, either method will significantly outperform previous methods such as passing in a sliding window of fixed size into a fully-connected network. Even when a fully-connected network is combined with some state-preserving model such as ARIMA it still cannot match the performance of CNNs or RNNs. See Deep Learning for Time-Series Analysis.


Data Science Tools Using Scala I know that Spark is fully integrated with Scala. It's use case is specifically for large data sets. Which other tools have good Scala support? Is Scala best suited for larger data sets? Or is it also suited for smaller data sets?

Re: size of data

The short answer

Scala works for both small and large data, but its creation and development is motivated by needing something scalable.  Scala is an acronym for “Scalable Language”.

The long answer

Scala is a functional programming language that runs on the jvm.  The 'functional' part of this is a fundamental difference in the language that makes you think differently about programming.  If you like that way of thinking, it lets you quickly work with small data.  Whether you like it or not, functional languages are fundamentally easier to massively scale.  The jvm piece is also important because the jvm is basically everywhere and, thus, Scala code can run basically everywhere.  (Note there are plenty of other languages written on the jvm and plenty of other functional programming languages, and languages beyond Scala do appear in both lists.)

This talk give a good overview of the motivation behind Scala.

Re: other tools that have good Scala support:

As you mentioned, Spark (distributable batch processing better at iteratative algorithms than its counterpart) is a big one.  With Spark comes its libraries Mllib for machine learning and GraphX for graphs.  As mentioned by  Erik Allik and Tris Nefzger, Akka and Factorie exist.  There is also Play.

Generally,  I can't tell if there is a specific use case you're digging for (if so, make that a part of your question), or just want a survey of big data tools and happen to know Scala a bit and want to start there.


Relative feature importance w.r.t hyperparameters Could changing the hyperparameters of a model change relative feature importance?

Yes. The most obvious example is when using a Lasso regression : for an increasing $\alpha$ parameter you will have more and more coefficients set to zero. This resulting in a smaller set of features and thus a bigger share of feature importance for remaining features.


Choose-many categorical features: alternatives to one-hot encoding? I'm building a model to predict the lifetime value of a client based on the relational data we have on them.  The user table has a bunch of one-to-many child tables that might be predictive.  Grossly simplified, the child features boil down to things like:

a list of item categories that they've bought in the past
a list of the predominant colors in ads they've clicked on
etc, etc

In each case, the obvious feature comprises a list of ~ 0-10 choices from a categorical variable.  I have several of these features, some of which have as many as ~10k discrete values, so one-hot encoding would get very wide, very fast.
Aside: if there a term of art for this kind of "list-of-tags feature" that I'm referring to as "choose many categorical", please tell me.
Question: Is there an dense encoding scheme that works with choose-many categorical features?

If your algorithm is based on gradient descent optimization, you can use embeddings, which are dense representation spaces for discrete elements.
Embeddings are supported by most deep learning frameworks such as pytorch or tensorflow.
Update: the fact that you want to have multiple discrete values does not prevent the possibility of using embeddings: you can just add all vectors together in a single value. The most straightforward approach for this would be to have a constant length for the list (equal to the maximum number of elements in all lists, or a sensible maximum value), filling with "padding" items the positions that are not needed. If you want to take the sequential appearance of the elements into account, instead of adding the vectors together you could apply convolutional layers or an LSTM over the embedded vectors.


Has anyone succeeded in finding a good Scala/Spark kernel for Jupyter? The ones I've tried so far
Almond: Works very well for just Scala, but you have to import dependencies, and it gets tedious after a while. And unfortunately can't run when using Spark with YARN instead of Local.
Spylon-kernel: Kernels connects, but gets stuck in the initializing stage.
Apache Toree: I would've loved this so much only if it worked. Lots of language support, magics, incubated by apache. However, this kernel doesn't connect. Get's stuck on the "Kernel Connecting" stage.
Are we to never run Spark with the freedom and flexibility of Jupyter?
What curse is this?

I am using both almond and toree. For executing only scala code, I am using almond. If you want to learn new features in scala (ex: 2.13), then use almond(Add Scala 2.13.4 support in almond v0.11.0). Almond might start supporting scala 3.0 once it is released.
For spark, I am using toree. Toree works fine for me. I am using toree 0.5.0 with spark 3.0.0. Toree uses scala 2.12.
First install spark 3.0.0 by following the steps provided here. After installation check if you are able to launch spark shell from the terminal.
#toree installation
#I used poetry for managing deps in virtual environment
pip install https://dist.apache.org/repos/dist/dev/incubator/toree/0.5.0-incubating-rc1/toree-pip/toree-0.5.0.tar.gz

# I installed/copied extracted spark binaries to /opt/spark
jupyter toree install --user --kernel_name=toree --spark_home=/opt/spark

With these steps, you should find the new kernel added to ~$HOME/.local/share/jupyter/kernels/. Launch jupyter notebook and you should be able to find the toree kernel listed when trying to create a new notebook. You could also execute jupyter kernelspec list to see if the new kernel is listed.
If you are able to create a new notebook, then you should have the kernel initialized without any issues. Try executing println("Hello world") and it should be successful.
I have captured installation summary here


Removing # from header I have a $(418,2)$ matrix and I want to convert it to csv. so I write:

np.savetxt('titanic1.csv', Sol, fmt='%.2f', delimiter=",",header="PassengerId,Survived")

But the header I get is "# PassengerId" instead of "PassengerId" which is what I want. How can I solve this?

The documentation of numpy says you can add the parameter comments='', but warns that np.loadtxt() might not work on the resulting file.


What is the best way to read SQL dataset in to Tensorflow? What is the best way to read SQL database in to Tensorflow?

Currently, I am using Postgres on server and developed DL algorithm on Tensorflow using Jupyter Lab. How can I import data into Jupyter Lab using tf.data API? I do not want to store the data in the disk and keep running the algorithm when the new data arrives.

It seems like tf.data.experimental.SqlDataset only support for sqlite.

(NOTE: I did not upgrade my Tensorflow, so, I am using  tf.contrib.data.SqlDataset() for the minimal working example.)

I migrated the data from PostgreSQL to SQLite3 and using

#Ignore the warnings
import warnings
warnings.filterwarnings("ignore")

import tensorflow as tf
#To start an input pipeline, you must define a source
dataset = tf.contrib.data.SqlDataset("sqlite", "/home/musara1/musara_dev.sqlite3",
                                 "SELECT * FROM basetable LIMIT 10",
                                 (tf.string, tf.int32, tf.int32, tf.int32, tf.int32, tf.int32, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.int32, tf.int32, tf.int32))

iterator = dataset.make_one_shot_iterator()
next_element = iterator.get_next()
# Prints the rows of the result set of the above query.
sess=tf.InteractiveSession()

print(sess.run(next_element))

I can print the next element. However, there are other transformations I need to do on the dataset. such as splitting into training/validation/testing and getting rid of some columns et cetera. However, the output of tf.contrib.data.SqlDataset() is for me
<SqlDataset shapes: ((), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), (), ()), types: (tf.string, tf.int32, tf.int32, tf.int32, tf.int32, tf.int32, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.float64, tf.int32, tf.int32, tf.int32)>

I have 25 columns and tf.contrib.data.SqlDataset() creates 25 different tensorflow.python.framework.ops.Tensor. How can I bring them together? Therefore, I can use tf.data.Dataset.from_tensor_slices()?

You use the methods on SqlDataset to manipulate the data. For example, create a train/test split with:

test_dataset = dataset.take(1000)
train_dataset = dataset.skip(1000)

I would get rid of unneeded columns in the SELECT statement to reduce the size of the data as early as possible.


How can i find trend time for my articles? our article is time-based, that means is my article search more in a specific time.
as you can see in under chart this article search more in specific period time.

if my dataset looks like this(it can be a json,csv,xlsx) :

How can I find the trend time of my article (time between the red line in image 1)

I need to know that period of time to be ready for next year.

  (I need excel solution for that, but if you can solve this problem
  with other ways and programming language it's ok)

Based on your comment and what I can understand from your question, you could identify the "trend" you are talking about as a series of Outliers. From Wikipedia, an outlier is:

  In statistics, an outlier is an observation point that is distant from
  other observations. An outlier may be due to variability in the
  measurement or it may indicate experimental error; the latter are
  sometimes excluded from the data set.

How can you do this in excel? There is a nice tutorial in this link. In short, based on the data you added, you need to do

Calculate the 1st and 3rd Quartile with the formula =QUARTILE(B2:B9,1) and =QUARTILE(B2:B9,3) respectively
Then get the IQR which is the difference of the above two =F3-F1
Calculate the L Bound as =G2-(1.5*G4), where G2 is the 1st Quartile and G4 the IQR
Calculate the U Bound as =G2-(1.5*G4)
Then next to the column with the data, add this =OR(B2<$G$5,B2>$G$6) which check if your data is lower or higher the L and U Bound. If it is, that row is marked as Outlier.

You need to play around with the multiplier in 3rd and 4th step to get you the right results in your test data. With 0.8, the last 3 rows are marked as outliers and you can say that the desired outlier or "trend" as you called it period is on 17 to 19th January 2018.

There are more sophisticated ways to do it in Python, but I am not aware of a similar one in Excel. You could look for Anomaly Detection in time series with Python.


Having trouble figuring out how loss was calculated for SQuAD task in BERT paper The BERT Paper

https://arxiv.org/pdf/1810.04805.pdf

Section 4.2 covers the SQuAD training.

So from my understanding, there are two extra parameters trained, they are two vectors with the same dimension as the hidden size, so the same dimensions as the contextualized embeddings in BERT. They are S (for start) and E (for End).

For each, a softmax is taken with S and each of the final contextualized embeddings to get a score for the correct Start position. And the same thing is done for E and the correct end position.

I get up to this part. But I am having trouble figuring out how the did the labeling and final loss calculations, which is described in this paragraph

"and the maximum scoring span is used as the prediction. The training objective is the loglikelihood of the correct start and end positions."

What do they mean by "maximum scoring span is used as the prediction"?

Furthermore, how does that play into "The training objective is the loglikelihood of the correct start and end positions"?

From this Source:

https://ljvmiranda921.github.io/notebook/2017/08/13/softmax-and-the-negative-log-likelihood/

It says the log-likelihood is only applied to the correct classes. So the we are only calculating the softmax for the correct positions only, Not any of the in correct positions.

If this interpretation is correct, then the loss will be

Loss = -Log( Softmax(S*T(predictedStart) / Sum(S*Ti) ) -Log( Softmax(E*T(predictedEnd) / Sum(S*Ti) )

From your description it sounds like for every position $i$ in the input text the model predicts $$p_S(i) = \mathbb P(\text{correct start position is } i)$$ and $$p_E(i) = \mathbb P(\text{correct end position is } i).$$
Now let $\hat s = \arg\max_i p_S(i)$ and $\hat e = \arg\max_i p_E(i)$ be the most probable start and end positions (according to the model).

Then by "maximum scoring span is used as the prediction" they just mean that they output $(\hat e, \hat s)$ when predicting.

Then “The training objective is the loglikelihood of the correct start and end positions” means that if the correct start and end positions are $s^*$ and $e^*$, they try to maximize the predicted probability of $s^*$ and $e^*$. If the start and end positions are independent then this is equal to $p_S(s^*) p_E(e^*)$ and taking the negative log the loss becomes
$$L(e^*, s^*) = -\log p_S(s^*) -\log p_E(e^*).$$


Skewness in response variable Why is it a problem that your response variable is skewed in regression? Is taking logarithms the only way to solve it?

Taking logarithms is not a one-size-fits-all approach - what if your response is negatively skewed?

"Simple" linear regression assumes that the response variable is normally distributed. If you'd like to perform this kind of regression, then you should probably transform your response variable to be as normal as possible.

There are other approaches / model types to deal with skewed responses - check out the Tweedie family.


Entropy in a closed box I have not understood why the maximum number of binary questions needed to reduce
uncertainty is essentially log(T).
If i have a ball in a box and 10 possible classes, is it enough to ask log(10) binary questions to know which is the class of the ball ? Sure ???

The correct notation would be math.ceil(log(T)). Also what you are asking is/can formally be solved by Binary Search Trees. You can easily traverse it by if-else conditions and arrive at the correct result in math.ceil(log(T)) steps.

It looks something like:

Just place the colour of the balls in the leaf nodes (last nodes). Your first question would be does the ball fall in the first 4 leaf nodes or the second group of 4 nodes.Then you can progress to level 2 and so on.

BST reduces the entropy by maximum amount. You can easily construct other unbalanced trees but then you will require more steps to arrive at a determinate result (in some cases) and less steps in some other cases. BST gives an absolute number of steps you must travel in order to reach the result.

More theory here.


How to normalize data of a different nature? I am working a price prediction LTSM model for the stock market. I am using multiple features: Open, Close, High and I would like to add the Volume.

The 3 first features are of the same nature but the volume presents much higher values.

What would be the safest way to keep consistency during the normalization process?

You would usually just scale all of them to be within the same range.

You can do this by using something like the Scikit-Learn MinMaxScaler, or just a simple function like this:

def scale(data, new_min=-1, new_max=1)
    """Scale values of data to be within the range [new_min, new_max]
    data must be a numpy array or a Pandas Dataframe/Series"""

    return (data - data.min()) / (data.max() - data.min()) * (new_max - new_min) + new_min

Between -1 and +1 is just nice, as the data is centered around zero. You could play with those values.

You can think about and perhaps experiment to see if you should scale all variables together (meaning one global min and max in the dataset), or whether to scale the individual columns/features of your dataset, so each one lies in the given range.

A tip for financial data is to use the log returns - that means to take your raw prices, compute the logarithm of those values, then take the difference between the closing prices of each day.

The reason for this is to because the resulting values are normally distributed, which is an underlying assumption of many models you will subsequently use (Boosting, ARIMA, GARCH for volatilities etc.). There are also other reasons of convenience - check out this article


A dataset has skewness = 1 with missing data. Standard deviation around median is 1.5. How much data will be unaffected? There is no other description about the data, if it is univariate, bivariate, etc. neither the type of distribution is given.

I recently came across this question, I would like to know how skewness affects unaffected data percentage

This is not answerable in general. You don't even know the skewness or standard deviation around the median without missing data, so you can't compare those quantities. Without more information about the distribution there is no way to figure out how much data is missing, or where in the distribution the missing data is located.


Q-Learning: Target Network vs Double DQN I am having a hard time understanding difference between Target Network and Double DQN
From this blog:
Target Network generates the target-Q values that will be used to compute the loss for every action during training. The target network’s weights are fixed, and are frequently but by small amounts updated towards the primary Q-networks values.
Double DQN: instead of taking the max over Q-values when computing the target-Q value for our training step, we use our primary network to chose an action, and our target network to generate the target Q-value for that action
That sounds very similar to me, and the equation makes me lost too :(
qTarget = r + γQ(  s’,  argmax(Q(s’,a,ϴ),ϴ’)  )

From whaat I know, Target Network Approach allows us to do the
following:

using Online Network, get scores for each action as the response for the current state. Pick the most appropriate action $a$ and take a
note of its score $q$
using that action, "travel" to the next state and get scores from Target Net for even further actions.
Select the most appropriate action from step 2) and call it $A$. also, take a note at its score $Q$
Punish OnlineNetwork for any difference between $q$ and $Q$.

Note: the gradient flows only through $a$ component of the "output
vector" $\{a, b, c, d ...\}$

Please correct me if these steps are wrong

Ok, it's simple!

Just the "Target network approach":

  Select an item from Memory Bank
  Using Target Network, from $S_{t+1}$ determine the index of the best action $A_{t+1}$ and its Q-value
  Do corrections as usual

The "double DQN approach"":

  Select an item from Memory Bank
  Using Online Network, from $S_{t+1}$ determine the index of the best action $A_{t+1}$.
  Using Target Network, from $S_{t+1}$ get the Q-value of that action.
  Do corrections as usual using that Q-value

As seen from 3, the Online Network gives us the action that it thinks is highest.  However, we let the Target Network have the final say on the score of such an action.

Were we to let Target Network select a best action, it could have very well selected some other index.


Using BERT for co-reference resolving, what's the loss function? I'm working my way around using BERT for co-reference resolving. I'm following this highly-cited paper BERT for Coreference Resolution: Baselines and Analysis (https://arxiv.org/pdf/1908.09091.pdf). I have following questions, the details can't be found easily from the paper, hope you guys help me out.
What’s the input? is it antecedents + parapraph?
What’s the output? clusters <mention, antecedent> ?
More importantly What’s the loss function?
For comparison, in another highly-cited paper by [Clark .et al] using Reinforcement Learning, it's very clear about what reward function is. https://cs.stanford.edu/people/kevclark/resources/clark-manning-emnlp2016-deep.pdf

NER is approached as a sequence-labeling problem and the end of the network, there is a categorical distribution estimated by softmax that is trained using cross-entropy loss.
The paper you are specifically asking about claims to be based on the End-to-end Neural Coreference Resolution paper that does something more tricky. They explicitly consider all spans to the entities spans and compute the probability for that. Nevertheless, when they get the probabilities they still do in principle the cross-entropy loss (cf. their code).


How to get predictions with predict_generator on streaming test data in Keras? In the Keras blog on training convnets from scratch, the code shows only the network running on training and validation data. What about test data? Is the validation data the same as test data (I think not). If there was a separate test folder on similar lines as the train and validation folders, how do we get a confusion matrix for the test data. I know that we have to use scikit learn or some other package to do this, but how do I get something along the lines of class wise probabilities for test data? I am hoping to use this for the confusion matrix.

To get a confusion matrix from the test data you should go througt two steps:

Make predictions for the test data

For example, use model.predict_generator to predict the first 2000 probabilities from the test generator.

generator = datagen.flow_from_directory(
        'data/test',
        target_size=(150, 150),
        batch_size=16,
        class_mode=None,  # only data, no labels
        shuffle=False)  # keep data in same order as labels

probabilities = model.predict_generator(generator, 2000)

Compute the confusion matrix based on the label predictions

For example, compare the probabilities with the case that there are 1000 cats and 1000 dogs respectively.

from sklearn.metrics import confusion_matrix

y_true = np.array([0] * 1000 + [1] * 1000)
y_pred = probabilities > 0.5

confusion_matrix(y_true, y_pred)

Additional note on test and validation data

The Keras documentation uses three different sets of data: training data, validation data and test data. Training data is used to optimize the model parameters. The validation data is used to make choices about the meta-parameters, e.g. the number of epochs. After optimizing a model with optimal meta-parameters the test data is used to get a fair estimate of the model performance.


Scipy curve_fit and method "dogbox" I am trying to duplicate this papers feature engineering for user activity. They take 14 days of accumulated user activity and keep the parameters (2 parameters) that fit a sigmoid to it. I would like to do the same except with 7 days of activity. http://hanj.cs.illinois.edu/pdf/kdd18_cyang.pdf

They use the formula below and keep the parameters x0 and k as features.

from scipy.optimize import curve_fit
import numpy as np

def sigmoid(x, x0, k):
y = 1 / (1 + np.exp(-k*(x-x0)))
return y

I used scipy curve_fit to find these parameters as follows

ppov, pcov = curve_fit(sigmoid, np.arange(len(ydata)), ydata, maxfev=20000)

When I had a user that had the values below, I had the following error:

ydata1 = [0,0,0,0,0,91,91]

RuntimeError: Optimal parameters not found: gtol=0.000000 is too small func(x) is orthogonal to the columns of the Jacobian to machine precision.

I noticed that if I add the method 'dogbox' I know longer get the error.

ppov, pcov = curve_fit(sigmoid, np.arange(len(ydata1)), ydata1, maxfev=20000, method='dogbox')
print(ppov[0], ppov[1])
5.189237217957538 11.509279446215949

However, I played around with other values and noticed that the resulting parameters can have very different values.

For example. If I have values for that are

ydata2=[0,3,5,30,34,50,91]

ppov, pcov = curve_fit(sigmoid, np.arange(len(ydata2)), ydata2, maxfev=20000)
print(ppov[0], ppov[1])
-24.681668846480264 118.77183210605865

However, if I add the method='dogbox' I get very different k and x0 parameter values.

ppov, pcov = curve_fit(sigmoid, np.arange(len(ydata2)), ydata2,  maxfev=20000, method='dogbox')
print(ppov[0], ppov[1])
0.28468096463676695 8.154477352500013

Can anybody help me with 2 things:

I read the doc about 'dogbox' and don't really understand it. Can anybody explain it more simply?
The curve_fit scipy function is looping through about 100,000 users and I need to set the parameters of the curve_fit so it does not throw an error. Is using the 'dogbox' method okay for my purposes knowing that the parameter results seem very different between the 'dogbox' and default 'lm' method? Or, are there other arguments in the curve_fit function that I could set instead that will help me get past this error?

I can't speak to the dogbox algorithm,  but the sigmoid only has range (0,1), so fitting to your example data is sure to be bad. The paper you reference presumably scales the input first.

The first example you give has a best fit that's a step function  which can be approximated by the sigmoid with parameters going to infinity; so it's no surprise the algorithm wouldn't converge.

EDIT: Maybe you should try increasing the tolerances (passed as kwargs through curve_fit to least_squares); your error message mentions gtol specifically:  https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.least_squares.html#scipy.optimize.least_squares
Or, if things are converging enough for your purposes, just catch and handle that error?


Are there free cloud services to train machine learning models? I want to train a deep model with a large amount of training data, but my desktop does not have that power to train such a deep model with these abundant data.

I'd like to know whether there are any free cloud services that can be used for training machine learning and deep learning models?

I also would like to know if there is a cloud service, where I would be able to track the training results, and the training would continue even if I am not connected to the cloud.

There are no unlimited free services*, but some have starting credit or free offers on initial signup. Here are some suggested to date:

AWS: If specifically deep learning on a large data set, then probably AWS is out - their free offer does not cover machines with enough processing power to tackle deep learning projects.
Google Cloud might do, the starting credit offer is good enough to do a little deep learning (for maybe a couple of weeks), although they have signup and tax restrictions.
Azure have a free tier with limited processing and storage options.

Most free offerings appear to follow the "Freemium" model - give you limited service that you can learn to use and maybe like. However not enough to use heavily (for e.g. training an image recogniser or NLP model from scratch) unless you are willing to pay.

This best advice is to shop around for a best starting offer and best price. A review of services is not suitable here, as it will get out of date quickly and not a good use of Stack Exchange. But you can find similar questions on Quora and other sites - your best bet is to do a web search for "cloud compute services for deep learning" or similar and expect to spend some time comparing notes. A few specialist deep learning services have popped up recently such as Nimbix or FloydHub, and there are also the big players such as Azure, AWS, Google Cloud.

You won't find anything completely free and unencumbered, and if you want to do this routinely and have time to build and maintain hardware then it is cheaper to buy your own equipment in the long run - at least at a personal level.

To decide whether to pay for cloud or build your own, then consider a typical price for a cloud machine suitable for performing deep learning at around \$1 per hour (prices do vary a lot though, and it is worth shopping around, if only to find a spec that matches your problem). There may be additional fees for storage and data transfer. Compare that to pre-built deep learning machines costing from \$2000, or building your own for \$1000 - such machines might not be 100% comparable, but if you are working by yourself then the payback point is going to be after only a few months use. Although don't forget the electricity costs - a powerful machine can draw 0.5kW whilst being heavily used, so this adds up to more than you might expect.

The advantages of cloud computing are that someone else does the maintenance work and takes on the risk of hardware failure. These are valuable services, and priced accordingly.

* But see Jay Speidall's answer about Google's colab service, which appears to be free to use, but may have some T&C limitations which may affect you (for instance I doubt they will be happy for you to run content production of Deep Dream or Style Transfer on it)


Detecting over fitting of SVM/SVC I am using 3-fold cross validation and a grid search of the C and gamma parameters for a SVC using the RBF kernel I have achieved a classification score of 84%.

When testing against live data the accuracy rate is 70% (1500 samples used). However, when testing against an un-seen hold out set the accuracy is 86% (8800 samples, 20% of the original dataset).

The training and holdout data set have even distribution of the 3 classes.

What could be the cause of this large discrepancy? It does not seem to be over fitting judging by the performance of the model with the hold out set?

EDIT:

How did you split the data set?
The data was originally in sequential order. I wrote a script to randomly split each sample between the train and hold out set, making use of a CSPRNG. Then at the end a report was automatically generated to display the distribution of each class in each set. The distribution very near equal.

How did you do the grid search?
Through the SKlearn SVC grid search method (GridSearchCV).

Is there any overlap between the data points used during grid search and the un-seen hold out set?
No overlap, they are all from unique time stamps in the initial set.

Does the live data come from the same distribution as the other?
Yes the live data comes from the same source and the distribution is roughly the same.

How do you know?
I have a script to count up the occurrences of each class in the data set.

It seems likely that the live data is different somehow from your other data.  Cross-validation shows a 84% accuracy, and accuracy on the held-out set is 86%, which is pretty consistent and does not indicate overfitting.  Accuracy on the live data is 70%, which is significantly different.  That suggests that live data is somehow different in ways that are important to the classifier.  Perhaps concept drift has occurred.


How should ethics be applied in data science There was a recent furore with facebook experimenting on their users to see if they could alter user's emotions and now okcupid.

Whilst I am not a professional data scientist I read about data science ethics from Cathy O'Neill's book 'Doing Data Science' and would like to know if this is something that professionals are taught at academic level (I would expect so) or something that is ignored or is lightly applied in the professional world. Particularly for those who ended up doing data science accidentally.

Whilst the linked article touched on data integrity, the book also discussed the moral ethics behind understanding the impact of the data models that are created and the impact of those models which can have adverse effects when used inappropriately (sometimes unwittingly) or when the models are inaccurate, again producing adverse results.

The article discusses a code of practice and mentions the Data Science Association's Code of conduct, is this something that is in use? Rule 7 is of particular interest (quoted from their website):

  (a) A person who consults with a data scientist about the possibility
  of forming a client-data scientist relationship with respect to a
  matter is a prospective client.

  (b) Even when no client-data scientist relationship ensues, a data
  scientist who has learned information from a prospective client shall
  not use or reveal that information.

  (c) A data scientist subject to paragraph (b) shall not provide
  professional data science services for a client with interests
  materially adverse to those of a prospective client in the same or a
  substantially related industry if the data scientist received
  information from the prospective client that could be significantly
  harmful to that person in the matter

Is this something that is practiced professionally? Many users blindly accept that we get some free service (mail, social network, image hosting, blog platform etc..) and agree to an EULA in order to have ads pushed at us.

Finally how is this regulated, I often read about users being up in arms when the terms of a service change but it seems that it requires some liberty organisation, class action or a senator to react to such things before something happens.

By the way I am not making any judgements here or saying that all data scientists behave like this, I'm interested in what is taught academically and practiced professionally.

I think ethics in Data Science is important. There is a fundamental difference in using user data to better their experience and show relevant ads and using user data to trick people into clicking on ads for the sake of monetary profit. Personally I like ads that give me relevant information like deals on things I would buy anyway. However, showing me weight loss ads because I got dumped is creepy and unethical. As my friend Peter always says, "don't be creepy with data".


Time Series Forecasting Seasonal type As we all know there are two types of seasonal types, additive and multiplicative, but I have trouble telling them apart. To my understanding, in multiplicative seasonality, the magnitude of a seasonality would grow with the trend. For example with an upward trend, the magnitude of a season would grow bigger. But what if the magnitude of a seasonality grows but the trend stays the same. In other words, the minimum value of one season always seem to be on one horizontal line but the maximum value keeps increasing. What kind of seasonality is this? What model can I use to describe this data series?

Edit:

This is the formula given in Forecasting, Principles and Practice. From what I read it seems that the trend and the level are not multiplied together in the multiplicative seasonality case. Also in the statsmodels library, both the trend and the seasonality can be chosen as either multiplicative or additive. Can anyone clarify more what kind of additive mode we should be talking about here?

Each time series can be decomposed in at least three elements:

Trend
Seasonal component
Noise

An additive model can be explained as:

y = Trend + Seasonal + Noise

while a multiplicative component as:

y = Trend * Seasonal * Noise

The main difference is that in the case of additive models all three components impact your dependent variable independently. In case of multiplicative models instead, the impact of each component is dependent from the others.

The choice of the "right" model depends on the problem at hand. If you assume that the Seasonal effect is independent from the level of the Trend, then use an additive model. If you think the value of the Trend is going to affect the magnitude of the Seasonal effect, then go for the multiplicative one.

Hope this helps, otherwise let me know.


how to optimize the weights of a neural net when feeding it with multiple training samples? My question is:

Let's say we have a 3 by 1 neural net similar to that one in the image (an input layer with 3 neurons and an output layer with one neuron and no hidden layer), I find no problem calculating the outputs of the forward and back propagation when feeding the neural net with one training sample (namely feature1, feature2, feature3 inputs) and I know exactly how my initial weights get optimized, the problem I find is when feeding the NN with multiple training inputs each time, here, I don't know exactly how the initial weights get optimized.

For example we have training inputs of 3 × 3 Matrix.

[[195, 90, 41],
[140, 50, 30],
[180, 85, 43]]

the first column is the height, 2nd: the weight, 3rd: shoe size, we feed the NN with the first row then the second and the third row.
We know that to calculate the new weights when feeding the NN with one training sample we rely on this formula:
New_weights = Initial_weights - learning_rate × (derivative of the loss function wrt the weights).

But when we feed the NN with more than one training example then which formula do we use ? Do we calculate the average of all dw (derivative of the loss function wrt the weights) or do we sum all of'em then multiply by the learning rate and substract them from the initial weights or what?
I'm a bit confused here.

I would be grateful if any of you could explain how the initial weights get modified when feeding the NN with multiple training inputs.

But when we feed the NN with more than one training example then which formula do we use ? Do we calculate the average of all dw (derivative of the loss function wrt the weights) or do we sum all of'em then multiply by the learning rate and substract them from the initial weights or what ?

This is batch gradient optimisation, so you average the gradients over all training examples in your batch.

Side note: In principle, you could use several training epochs (= several iterations over your entire dataset) and update the weights once per epoch. However, for large datasets (like ImageNet), you divide your dataset into batches and update the weights once per batch. You could also further subdivide each batch into minibatches and update once per minibatch. The subdivision is entirely up to you and is kind of ad hoc, but a recent paper suggests that consistently higher performance can be obtained with small minibatches of size between 2 and 32. Also, your learning rate should be scaled appropriately depending on the batch size.


How to show progress of sklearn.multioutput.MultiOutputRegressor and XGBRegressor? Is it possible to show the training progress of the MultiOutputRegressor in sklearn? When a huge dataset is processed, my program runs a long time and I have no clue how long it will take. I have shortened my program to a minimal working example below.

import numpy as np
from sklearn.multioutput import MultiOutputRegressor
import xgboost as xgb

df = np.arange(50).reshape(10,5)
train = df[:8]
test = df[8:]
X_train = train[:,0:-2]
X_test  = test[:,0:-2]
y_train = train[:,-2:]
y_test  = test[:,-2:]

eval_set = [(X_test, y_test)]
multioutputregressor = MultiOutputRegressor(xgb.XGBRegressor(eval_set=eval_set, verbose_eval=True))
multioutputregressor.fit(X_train, y_train)
predictions = multioutputregressor.predict(X_test)
print(predictions)

You can consider modifying the code of MultiOutputEstimator or XGBModel to introduce some debugging output. These seem to be the classes that implement the fitting logic of the two libraries. The corresponding source files should also be provided in the installations on your local hard drive.

For example, you can print information of when the separate threads are starting and stopping in MultiOutputEstimator.fit (inherited and thus reused in MultiOutputRegressor), lines 167-169. Also, consider the tip in the documentation of the n_jobs parameter:

If 1 is given, no parallel computing code is used at all, which is useful for debugging.

You can use a similar approach to expand the debugging in XGBModel.fit (the basis of XGBRegressor).


Decision tree on big categorical dataset I would like to use algorithm ID3 in order to find a decision tree of my dataset. I would like to see which of the attributes and values lead to the different value of rating (1<= x <= 5). Do you think it is possible to do it? I got stucked but I don't know if it is because it is not a possible thing to do or it is my fault as a beginner. Can you give me some advise or hints please?

You can see this dataset as a regression problem and not a classification one. The label is ordered, so predicting 2.8 is not a mistake, it is just that it is close to 3 (satisfactory but not there).

I can suggest to start with a regression tree and then round the results to have integers.

Also you can consider this a multiclassification task, but as a starter I would go with regression.


How to choose the random seed? I understand this question can be strange, but how do I pick the final random_seed for my classifier?

Below is an example code. It uses the SGDClassifier from SKlearn on the iris dataset, and GridSearchCV to find the best random_state:

from sklearn.linear_model import SGDClassifier
from sklearn import datasets
from sklearn.model_selection import train_test_split, GridSearchCV

iris = datasets.load_iris()
X, y = iris.data, iris.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)

parameters = {'random_state':[1, 42, 999, 123456]}

sgd = SGDClassifier(max_iter=20, shuffle=True)
clf = GridSearchCV(sgd, parameters, cv=5)

clf.fit(X_train, y_train)

print("Best parameter found:")
print(clf.best_params_)
print("\nScore per grid set:")
means = clf.cv_results_['mean_test_score']
stds = clf.cv_results_['std_test_score']
for mean, std, params in zip(means, stds, clf.cv_results_['params']):
    print("%0.3f (+/-%0.03f) for %r" % (mean, std * 2, params))

The results are the following:

Best parameter found:
{'random_state': 999}

Score per grid set:
0.732 (+/-0.165) for {'random_state': 1}
0.777 (+/-0.212) for {'random_state': 42}
0.786 (+/-0.277) for {'random_state': 999}
0.759 (+/-0.210) for {'random_state': 123456}

In this case, the difference from the best to second best is 0.009 from the score. Of course, the train/test split also makes a difference.

This is just an example, where one could argue that it doesn't matter which one I pick. The random_state should not affect the working of the algorithm. However, there is nothing impeding of a scenario where the difference from the best to the second best is 0.1, 0.2, 0.99, a scenario where the random_seed makes a big impact.

In the case where the random_seed makes a big impact, is it fair to hyper-parameter optimize it?
When is the impact too small to care?

TL:DR, I would suggest not to optimise over the random seed. A better investment of the time would be to improve other parts of your model, such as the pipeline, the underlying algorithms, the loss function... heck, even optimise the runtime performance!  :-)

This is an interesting question, even though (in my opinion) should not be a parameter to optimise.

I can imagine that researchers, in their struggles to beat current state-of-the-art on benchmarks such as ImageNet, may well run the same experiments many times with different random seeds, and just pick/average the best. However, the difference should not be considerable.

If your algorithms has enough data, and goes through enough iterations, the impact of the random seed should tend towards zero. Of course, as you say, it may have a huge impact. Imagine I am categorising a batch of images, into cat or dog. If I have a batch size of 1, and only 2 images that are randomly sampled, and one is correctly classified, one is not, then the random seed governing which is selected will determine whether or not I get 100% or 0% acuracy on that batch.

Some more basic information:

The use of a random seed is simply to allow for results to be as (close to) reproducible as possible. All random number generators are only pseudo-random generators, as in the values appear to be random, but are not. In essence, this can be logically deduced as (non-quantum) computers are deterministic machines, and so if given the same input, will always produce the same output. Have a look here for some more information and relative links to literature.


Data augmentation: rotating images and zero values A lot of people rotate images to create a larger training set for neural networks. For most nets, all of the inputs have to be the same size so the image rotation function has to crop the newly rotated images to match the input size. So, say you have $32\times32$ resolution images and do 45 degree rotations. Some of the output images will look diamond shaped with black (zero values) in the corners. So, my question is: should you leave these zero values alone or change them in some way and if so, how?

Keeping the values as zeros will introduce some bias to your network. Given, you have this corner effect for the majority of your dataset, you do not want the network to identify a high probability of making the corners black. Thus, you should fill them, you can extend the edge, do a reflection, wrapping. You can also do some more complex function, like take average of a few patches in your image then place them in the missing areas.

Keras has a very nice function that can do all this for you.

import numpy as np
from keras.datasets import mnist
from keras.preprocessing.image import ImageDataGenerator
import matplotlib.pyplot as plt
%matplotlib inline

# load data
(X_train, y_train), (X_test, y_test) = mnist.load_data()
X_train = X_train.astype('float32')

# set up your data generator
datagen = ImageDataGenerator(
    featurewise_center=True,
    featurewise_std_normalization=True,
    rotation_range=15,
    width_shift_range=0.2,
    height_shift_range=0.2,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True,
    vertical_flip = False)

# Fit the generator using your data
datagen.fit(X_train.reshape((len(X_train), 28, 28, 1)))

# Black images
image = X_train[5]
plt.imshow(image,  cmap='gray')
plt.show()

plt.figure(figsize=(12,12))
plt.subplot(4, 4, 1)
plt.imshow(image.reshape((28,28)),  cmap='gray')

for j in range(15):
    augmented = datagen.random_transform(image.reshape((28,28,1)))
    plt.subplot(4, 4, j+2)
    plt.imshow(augmented.reshape((28,28)),  cmap='gray')
plt.tight_layout()
plt.show()

# White images
image = -1*X_train[5]
plt.imshow(image,  cmap='gray')
plt.show()

plt.figure(figsize=(12,12))
plt.subplot(4, 4, 1)
plt.imshow(image.reshape((28,28)),  cmap='gray')

for j in range(15):
    augmented = datagen.random_transform(image.reshape((28,28,1)))
    plt.subplot(4, 4, j+2)
    plt.imshow(augmented.reshape((28,28)),  cmap='gray')
plt.tight_layout()
plt.show()


Over fitting and association with regularization Heard and read lot about regularization helps in reducing over fitting.
But I'm not sure how exactly regularization works in reducing over fitting issue and whats the maths behind it?
Appreciate if can somebody provide a concrete example?

The short answer is that regularization penalizes model complexity. It does this by adding some function of your parameters to your basic cost function as a regularization term. This forces a compromise between a good fit (low error) on your training data and model simplicity by driving the parameters closer to zero. This reduces overfitting because large parameters (in terms of magnitude) are the main culprit behind overfitting.

For example, L1 regularization uses the sum of the absolute values of your weights, while L2 regularization uses the squares of your weights. More detailed comparison here.

For mathematical details, have a look at this answer.


Pre-processing on MRI images I have MRI images of brain tumors collected from a hospital (not a benchmark dataset). And I am planning to use them to predict/classify tumour types using a typical machine learning approach: texture analysis for feature extraction to build a classification module.
My question is, how to decide whether these images needs pre-processing or not, such as noise removal by using median filter for example? In other words, how to determine if there is noise in this data?

It depends on how did you collect these MRI images from hospital and how do you want to use these MRI images.

If you want to use these data in your deep learning model for segmentation or detection, it is possible to skip preprocessing.
If you want to apply a classical data science approach, you can do some processing like feature extraction.
Lastly, from another perspective, if you want to use also patient history with images, preprocessing can be necessary.


numpy array center of mass I'm wondering how does ndimage.measurements.center_of_mass calculate the center of mass values for a two dimensional numpy array.
For example let's say we have an array m:
m = numpy.array([[1,2,3,1],[4,4,4,1],[5,6,7,2]])

Then one would get the following for the coordinates of the center of mass (x, y):
x, y = ndimage.measurements.center_of_mass(m)

x = 1.325  and  y = 1.3

So, how are those values of x and y mathematically derived (actual derivation)?
When I follow the steps in the function definition center_of_mass, I don't get that result.

If you go to the source page for this method you can find the function:
def center_of_mass(input, labels=None, index=None):
    """
    Calculate the center of mass of the values of an array at labels.
    Parameters
    ----------
    input : ndarray
        Data from which to calculate center-of-mass.
    labels : ndarray, optional
        Labels for objects in `input`, as generated by `ndimage.label`.
        Only used with `index`.  Dimensions must be the same as `input`.
    index : int or sequence of ints, optional
        Labels for which to calculate centers-of-mass. If not specified,
        all labels greater than zero are used.  Only used with `labels`.
    Returns
    -------
    center_of_mass : tuple, or list of tuples
        Coordinates of centers-of-mass.
    """
    normalizer = sum(input, labels, index)
    grids = numpy.ogrid[[slice(0, i) for i in input.shape]]

    results = [sum(input * grids[dir].astype(float), labels, index) / normalizer
               for dir in range(input.ndim)]

    if numpy.isscalar(results[0]):
        return tuple(results)

    return [tuple(v) for v in numpy.array(results).T]

Whereby sum is (sum - centre)^2 of the input labels


Procedure for Designing CNNs Are there any standard procedure for designing a CNN?

I wrote some Python code for classifying speech signals using the 1D convolutional model in the Keras environment, but I can't meet the accuracy that I expect from it. In the training process the model reaches 98% UAR, but in the test process it overfits and remains at 58%. I used regularizer and dropout and early stopping tricks to overcome overfitting, but unfortunately, I couldn't do it. For more information, I can't augment the database or change the feature vector.

I don't know if this is what you are looking for, but Andrej Karpathy has a good blog article about his method for training networks in general: A Recipe for Training Neural Networks

I will put the bullet points of his recipe here, but there is way more practical advice in the actual blog.

The recipe

Become one with the data
Set up the end-to-end training/evaluation skeleton + get dumb baselines
Overfit
Regularize
Tune
Squeeze out the juice

I think the advice that best fit your situation is from step 2:

  At this stage it is best to pick some simple model that you couldn’t
  possibly have screwed up somehow - e.g. a linear classifier, or a very
  tiny ConvNet. We’ll want to train it, visualize the losses, any other
  metrics (e.g. accuracy), model predictions, and perform a series of
  ablation experiments with explicit hypotheses along the way.

You usually want to start with something simple that will not allow you to have issues with overfitting like you have. Then you will progressively work your way into a more complex model that can overfit in step 3.


AI that maximizes the storage of rectangular parallelepipeds in a bigger parallelepiped As you can see in the title, I'm trying to program an AI in Java that would help someone optimize his storage.

The user has to enter the size of his storage space (a box, a room, a warehouse etc...) and then enter the size of the items he has to store in this space. (note that everything must be a rectangular parallelepiped) And the AI should find the best position for each item such that the space is optimized.

Here is a list of what I started to do :

I asked the user to enter the size of the storage space (units are trivial here except for the computing cost of the AI later on I'm guessing), telling him that the values will be rounded down to the unit
I started by creating a 3-dimensional array of integers representing the storage space's volume, using the 3 values taken earlier. Filling it with 0s, where 0s would later represent free space and 1s occupied space.
Then, store in another multidimensional array the sizes of the items he has to store
And that's where the AI part should be starting. First thing the AI should do is check whether the addition of all the items' volumes doesn't surpass the storage space's volume. But then there are so many things to do and so many possibilities that I get lost in my thoughts and don't know where to start...

In conclusion, can anyone give me the proper terms of this problem in AI literature, as well as a link to an existing work of this kind ?
Thanks

This looks very much like a variation of the bin packing problem. The bad news is that this is mathematically a hard problem, NP-hard in fact (which means that no polynomial-time solution is possible). The good news is that it is heavily studied, with multiple approaches to solving it, or at least optimising solutions to within reasonable bounds.

Which approach to optimisation will work best for you will depend on how physically accurate you want your model to be. Your voxel approach already makes some decisions in that regard, but you may also want to consider issues such as gravity, stability of any piles of structures you create, and access to items. These things add additional constraints that make modelling the problem and implementing an optimiser harder, so you might want to start with more unrealistic model initially.

One simple approach might be to use a simple heuristic e.g. pack largest objects first, fill close to edges first etc. Then attempt to swap or revise the order of packing of a few pieces and see if it makes an improvement. You can make those changes greedily, or look into making them more randomly but with a system that rewards better improvements. Approaches such as simulated annealing or genetic algorithms can be used as stochastic approaches, many others are possible. There are a very large number of global optimisers that might help with such a combinatorial problem.

Here is an example of solving a bin packing problem using Simulated Annealing.

Here is an example of solving a bin packing problem using a Genetic Algorithm


Relationships between groups of features against independent variables I have several groups of features that I'd like to test against independent variables. The idea is to find which groups tend to be associated with a specific value of an independent variable.
Let's take the following example where s are samples, f are features, i are independent variables associated with each s.
        s1    s2    s3     s4 ....
    f1  0.3   0.9   0.7    0.8
    f2  ...
    f3  ...
    f4  ...
    f5  ...
    i1  low   low   med    high
    i2  0.9   1.6   2.3    10.5

Features f1, f2, f3 belong to group1 and f4,f5 belong to group2. If I wanted to find whether a given feature tended to be associated with a given independent variable, I could regress each feature vs i2 or an encoded i1 and test whether there is an association between feature and independent variable.
But now I'm wondering, is it possible to test whether a group of features tends to be associated with an independent variable? I'm not sure how to approach this problem.
One idea is to test each independent variable against all features in each group using multilinear regression. The model to regress would contain only features in each group separately, so in this case we would have $2*2$ models in total (for group1 and group2, and for 2 independent variables).
I have a feeling that this could also be formulated as a classification problem, but not really sure how.

One example would be to use ANCOVA (analysis of covariance) technique to capture association between continuous and categorical variables.
See here for more details.


How do I best visualize this voltage data for a science project I'm helping my son with his 7th grade science project. We've had a good deal of fun with our experiments with Solar Arrays and charging 12 volt UPS batteries! But, I am not sure how to interpret the data!
Our original hypothoesis was that the lenght of the wire between the solar panel and the battery would effect the voltage charge the most. We do NOT think that was true!
We just need to get some good charts out of the data to show something that we learned!

As per my observation, you need to visualize different configurations. WEKA can be used to visualize this type of things where it automatically stack multiple Graphs side by side to find trends. There are other features as well.

Yo can further explore here

enter link description here


Comparing rows in Data Studio I am using Data Studio for a project and I am connecting to my BigQuery table.
My table contains following columns:

Date
Store_name
Footfall

I'd love to compare the footfall of two stores using Data Studio but apparently I can't do that!
Any hints or should I just switch to another viz tool?

Welcome to the site! I think that you should be asking yourself "why" you can't do this in Data Studio. The likely answer is that Data Studio is a visualization tool while you are looking to do a transform of your existing data.

Most visualization tools are really only good at handling data in it's final form. It sounds like you still need to work on the data a little more to get to that stage. You should consider doing some SQL and/or scripting work (python, R, Julia, whatever) on your data to get it ready in the right format prior to importing it into Data Studio. Good luck!


Dimensionality reduction for feature extraction when missing some feature values I have two questions:
1-Which method is appropriate for dimensionality reduction for feature extraction when missing some feature values?
2-Which textbook is the best source for the answer in (1)?

Denoising autoencoders will be your bet in this case. I don't have a book handy for this case. They are good at reconstruction and calculate a good latent representation.  Just replace your missing feature with the mean or a fixed value.


How is the salary for Data Scientist different among countries? I would like to know the salary for Data Scientists by countries, but failed to find any resources on it.

I know that US gives data scientists an insanely high amoung of money (well, it might not be "insanely high" on the standard of US), but I rarely hear about any other countries, possibly except UK, Canada, and Australia in some cases.

Is there any research or survey or something that shows the diversity of salary for Data Scientists by countries, something like for software engineers?

The biggest one I know is O'Reilly's survey (here the last available from 2016). There are also polls on KDNuggets (older version) and other websites. But, in general, most of them are not very representative.

You can also go look at the national averages on glassdoor.com (example India) or other job sites.


When to choose character instead of factor in R? I am currently working on a dataset which contains a name attribute, which stands for a person's first name. After reading the csv file with read.csv, the variable is a factor by default (stringsAsFactors=TRUE) with ~10k levels. Since name does not reflect any group membership, I am uncertain to leave it as factor.

Is it necessary to convert name to character? Are there some advantages in doing (or not doing) this? Does it even matter?

Factors are stored as numbers and a table of levels. If you have categorical data, storing it as a factor may save lots of memory.

For example, if you have a vector of length 1,000 stored as character and the strings are all 100 characters long, it will take about 100,000 bytes. If you store it as a factor, it will take about 8,000 bytes plus the sum of the lengths of the different factors.

Comparisons with factors should be quicker too because equality is tested by comparing the numbers, not the character values.

The advantage of keeping it as character comes when you want to add new items, since you are now changing the levels.

Store them as whatever makes the most sense for what the data represent. If name is not categorical, and it sounds like it isn't, then use character.


R code that gives results like Wolfram Alpha for the expectation of a function of a random variable? When I ask Wolfram Alpha to calculate $E[f(X)]$ where $f(x) = e^{-x^2}$ and $X \sim \mathcal{N}(1,4)$, it gives the result
$$
E[f(X)] = \frac{1}{3\sqrt[9]{e}} \approx 0.29828,
$$
and the following plot which appears to be based on taking a number of simulations:

How can I generate the same type of plot using R?

You can simulate one of these lines like this:
library(ggplot2)
X<-rnorm(1000, 1, sqrt(4))
f<-function(x) exp(-x*x)
df<-data.frame(sample_size=seq_len(1000), sample_means=cumsum(f(X))/seq_len(1000))
ggplot(df, aes(x=sample_size, y=sample_means)) +
geom_line() + theme_minimal() + labs(x=NULL, y=NULL)


SVR is giving same prediction for all features I'm creating a basic application to predict the 'Closing' value of a stock for day n+1, given features of stock n using Python and Scikit-learn
A sample row in my dataframe looks like this (2000 rows)
       Open     Close    High     Low      Volume
0      537.40   537.10   541.55   530.47   52877.98

Similar to this video, where he uses 'Dates' and 'Open Price'. In this example, Dates are the features and Open price is the target.
Now in my example, I don't have a 'Dates' value in my dataset, but instead want to use Open, High, Low, Volume data as the features because I thought that would make it more accurate
I was defining my features and targets as so
features = df.loc[:,df.columns != 'Closing']
targets = df.loc[:,df.columns  == 'Closing']

Which would return a  df looking like this
features:
       Open      High      Low      Vol from
29     670.02    685.11    661.09   92227.36

targets:
       Close
29     674.57

However I realised that the data needs to be in a numpy array, so I now get my features and targets like this
features = df.loc[:,df.columns != 'Closing'].values
targets = df.loc[:,df.columns  == 'Closing'].values

So now my features look like this
[6.70020000e+02 6.85110000e+02 6.61090000e+02 9.22273600e+04
  6.23944806e+07]
 [7.78102000e+03 8.10087000e+03 7.67541000e+03 6.86188500e+04
  5.41391322e+08]

and my targets look like this
[  674.57]
[ 8042.64]

I then split up my data using
X_training, X_testing, y_training, y_testing = train_test_split(features, targets, test_size=0.8)

I tried to follow the Scikit-Learn documentation, which resulted in the following
svr_rbf = svm.SVR(kernel='rbf', C=100.0, gamma=0.0004, epsilon= 0.01 )
svr_rbf.fit(X_training, y_training)
predictions = svr_rbf.predict(X_testing)
print(predictions)

I assumed that this would predict the Y values given the testing features, which I could then plot against the actual y_testing values to see how similar they are. However, the predictions is printing out the same value for each X_testing feature.
[3763.84681818 3763.84681818 3763.84681818 3763.84681818 3763.84681818

I've tried changing the value of epsilon, c and gamma but that doesnt seem to change the fact that the predictions always gives the same value
I know that it might not be accurate to predict stock prices, but I must have done something wrong to get the same value when applying the model to various different test data

There are a couple of parts that I think changing will help.

First, a general one for all model building: I would suggest you scale your data before putting it into the model.

It might not directly solve the problem of receiving the same predicted value in each step, but you might notice that you predictions lie somewhere in the ranges of your input values - as you are using unscaled volume, that is making things difficult for the model. It is essentially have to work on two different scales at the same time, which is cannot do very well.

Have a look at the StandardScaler in sklean for a way how to do that.

Next a few suggestions of things to change, specifically because you are working with stock prices:

I would normally predict the value of the stock market tomorrow, and not the closing prices on the same data, where you are using open/high/low/volume. For me that only make sense if you were to have high-frequency (intraday) data.
Given this, you would need to shift your y value by one step. There is a method on Pandas DataFrames to help with that, but as you dont have a date column and you only need to shift by one timestep anyway, you can just do this:

features = df.loc[:-1, df.columns != 'Closing'].values    # leave out last step
targets = df.loc[1:, df.columns  == 'Closing'].values     # start one step later

You could then even then predict the opening price of the following day, or keep closing data in the features data, as that would not introduce temporal bias.

Something that would require more setup, would be to look at shuffling your data. Again, because you want to use historical values to predict future ones, you need to keep the relevant hsitory together. Have a look at my other answer to this question and the diagram, which explains more about this idea.

EDIT

You should also scale y_train and y_test, so that the model knows to predict within that range. Do this using the same StandardScaler instance, as not to introduce bias. Have a look at this short tutorial. Your predictions will then be within the same range (e.g. [-1, +1]). You can compute errors on that range too. If you really want, you can then scale your predictions back to the original range so they look more realistic, but that isn't really necessary to validate the model. You can simply plot the predictions against ground truth in the scaled space.

Check out this thread, which explains a few reasons as to why you should use the same instance of StandardScaler on the test data.


Which hyperparameters of a neural network can be tunned independently? The hyperparameter search is computationally expensive. I am wondering if one can tune the hyperparameters independently: tune one hyperparameter for a fixed value of other hyperparameters. For example, let's say we have two hyperparameters, A and B. We search for best value of A at fixed random B, then we search for best value of B at fixed best value of A.

This make sense if the the other hyperparameters does not interfere with the ordering of the validation loss for the hyperparameter we want to tune. In that sense, the number of units and the number of layers cannot be tuned independently. According to the Y. Bengio's paper (link), at some point the mini-batch size can be tuned independently (page 9, right column, The Mini-Batch Size).

But what about the other ones? Learning rate, activation function, dropout, ... which one can be tuned independently?

Your question is indeed a little bit broad, but I will try to explain to give you an overview on the importance and on some specific topics.

Indeed, multiple hyper-parameters exist in the context of deep learning. At the same time, just like Andrew Ng mentions in his courses, some are of an bigger importance than the others.

For instance, if you see that your training progresses very slowly (i.e. your convergence is relatively slow), you may want to fine-tune your learning rate.

Learning rate is a quintessential example of hyper-parameter that is more important than the number of neurons in a FullyConnected layer or to change the Dropout rate on a layer from 0.3 to 0.5.

At the same time, there exist two well known techniques for hyper-parameter search: grid search and random search. While the former behaves exactly like you explained (keeping the values of N-1 hyper-parameters fixed and iterating over some specific values of the N hyper-parameter), the random search has proven to have a better positive impact, as it slightly modifies all your hyper-parameters after a search step; although it may not be intuitive at first sight, this can yield better and earlier results than the grid search.


How do I use TF*IDF scores for my machine learning model? I have applied TF*IDF on the 'Ad-topic line' column of my dataset.
For every ad-topic line, I get the same output:

Firstly, I am unable to make sense of the output. The TF*IDF values are mentioned to the right, but what exactly are the numbers in brackets?

I plan to use these for my logistic regression model for classification. How exactly do I feed these values to the algorithm?

The numbers on the left are also important, they are basically the indexes in the following format

(document_number, token_number)

TF-IDF is computed for all the unique tokens in all the documents.

Let me know if you have any further doubt.

Vote me if i was able to help ;)


How many models shall be used in ensemble modelling? I wish to make an ensemble of deep models to solve a classification problem. I want to know how many models shall be used to create that ensemble to ensure unbiased results. I have head 30+ models shall be used to determine the skill of the model but cannot find anything relating to ensemble modelling.

There are no restrictions/guidelines on the number of models. You can start even from 3 models. You can keep the number of models as a hyperparameter if the training cost is less. Typically, you will observe the slanted 'L' shaped curve for MSE vs # of models plot. You can take the elbow point as the final # of models.