Trying to find the correlation between inputs and output I have tried the pandas code for trying to find out the correlation between the output and the inputs I am feeding. Here is the code:

dataframe.corrwith(dataframe['output']).plot(kind='barh',figsize=[20,10], legend=True,grid=True)

I got the following image:

I was trying to understand which column will affect the result more in the positive or negative or neutral way.

The above image I got, I am not able to conclude what exactly it means.
Can someone please tell me whether I am on the right direction of what I am trying to achieve? What is the meaning of the above image?
Let me know. Here is the link to the sample data set:  Training.csv

In a correlation framework above, the biggest driver of the output is the input which has the greatest absolute correlation value.

Correlation lies in the range [-1,1], and:

Negative correlation (correlation < 0) implies that the input and output move in opposite directions - i.e. as the input increases, the output decreases (and vice versa).
Nil correlation (correlation == 0) implies that the two variables are completely unrelated.
Positive correlation (correlation > 0) implies that the input and output move in the same direction - i.e. as the input increases, the output increases (and vice versa).

In the chart above, it looks like all-but-one of the inputs are negatively correlated with the output. This implies that as these inputs increase, the output decreases and vice versa.

A few things about your approach:

There is more than one type of correlation - Spearman (rank) and Pearson (linear) correlation are two examples. Be mindful of which you are using.
It would be helpful to rank/sort the result before plotting it. It would be easier for you to visually identify the drivers of the output if your chart was sorted.
You may also want to drop output from the chart. A variable's correlation with itself is always 1, and so this does not add any value to the graphic.


To learn machine learning which one is good? Until now I have implemented linear and logistic regression myself. I have not used any library other than numpy and matplotlib. But in the internet every example is solved using libraries such as sklearn, pandas and etc. My question is, which one is good to learn for machine learning, implementing algorithm yourself or using libraries (sklearn or ...)?

Thanks.

I can think of the following pros and cons for each. As for learning to code your own machine learning algorithms such as logistic regression:

You will definitely learn more about specific algorithms, their details, role of different parameters and etc. That is also a good practice of coding itself. Then you can validate your implementation by benchmarking it against other packages and implementations.
You will have more freedom in controlling different aspects of your method. You can add functions and modules as you wish and do not necessary have to deal with predefined variables, methods and etc.

On the other hand, implementing algorithms when it is not necessary and you can just use existing packages is like reinventing the wheel. It normally takes a lot of time and you have to verify your results for each one of them. Packages like sklearn are popular because of the following:

A group of people are working on those, constantly making them up to date, testing the  methods in different situations for different needs. That makes packages like sklearn very dependable and usually scalable.
If you have a question about them, there are tons of resources out there; documentation, forums, source code, communities like StackOverflow where thousands of people are eager to help you literally for any error you face while running your code.
Another important feature is automated hyperparameters tuning. Most of machine learning algorithms have a series of hyperparameters that need to be optimized in order to achieve the best performance. Packages like sklearn efficiently search for the optimal tuning parameters (or "hyperparameters") for your machine learning model in order to maximize its performance.
Still if you are interested in implementing machine learning algorithms and like the coding, you can always contribute to the existing packages. They usually have Github repositories where you can raise an issue, ask for a new feature or provide help improving them.

All in all, if you have enough time and you are keen to learn low level details about those models, go ahead and give implementation them a shot. That is certainly fun. However, if you need to get to the results as soon as possible and looking for a reliable package where a huge group of people both in industry and academia are already using, sklearn, pandas and others are you options.

Hope this is helpful and good luck.


Price optimization for tiered and seasonal products Assuming I can collect the demand of the purchase of a certain product that are of different market tiers. Example: Product A is low end goods. Product B is another low end goods. Product C and D are middle-tier goods and product E and F are high-tier goods.

We have collected data the last year on the following
1. Which time period (season - festive? non-festive?) does the different tier product reacts based on the price set? Reacts refer to how many % of the product is sold at certain price range
2. How fast the reaction from the market after marketing is done? Marketing is done on 10 June and the products are all sold by 18 June for festive season that slated to happened in July (took 8 days at that price to finish selling)

How can data science benefit in terms of recommending
1. If we should push the marketing earlier or later?
2. If we can higher or lower the price? (Based on demand and sealing rate?)

Am I understanding it right that data science can help a marketer in this aspect? Which direction should I be looking into if I am interested to learn about it.

You should be able to use linear regression to find correlation between the factors which cause your products to sell better (or worse).

There are many correlations you can test against in this data set. Some examples are:

If a product has been marketed aggressively, does it sell more quickly?
If a low tier item is available, do fewer high-tier items sell?
If multiple high-tier items are available, are fewer sold of each item?

Keep in mind that correlation does not necessarily imply causation. Always think about other factors which may cause sales to go up and down. For example, you may sell more high tier items in a season one year than another year. But, this could be due to changes in the overall economy, rather than changes in your pricing.

The second thing you can do is perform A/B tests on your product sales pages. This gives you clear feedback right away. Some example tests could be:

Show the user one high-tier product and one low-tier product (A). Show the user two high-tier products and no low-tier products(B). Which page generates more revenue?
Send out marketing emails for a seasonal sale 5 days in advance to one group of users (A). Send the same email to a different set of users 1 day in advance (B).

There are many possibilities. Use your intuition and think about previous knowledge you have about your products.


how to handle different categorical values in train and test dataset? I have a dataset in which if i do
train_df["era"].value_counts()

then it will show 120 different type of categorical values
and then if i do
test_df["era"].value_counts()

It do have 310 totally different categorical values like
train_df["era"].value_counts() intersection test_df["era"].value_counts() is {}

So if i will convert it to onehot encoding then it will give me 430 new features and datasize will explode. So my question is

if
train_df["era"].value_counts() intersection test_df["era"].value_counts() is {}

Then can we remove that feature ? Because how will model relate i
can think for continuous variable but this is categorical and if it
do affect performance then how ?

how to handle this number of different values in categorical
data ?

Whether to remove the dimension or not would wholly depend on the kind of data you're working with and how important is the feature for your modelling task.
What you need to ask is are the labels predefined? And by that I mean is there any particular upper bound to the kind of labels you're receiving? If yes, you could simply use label encoding from sklearn. After which I'd recommending doing some feature analysis and getting the feature importance score for that label and then decide if you want to discard it or not.
Regarding your second question, you need to encode the labels before you train your model.


Error when running RGFClassifier I recently tried to run Regularized Greedy Forest algorithm (rgf Classifier) from this package https://pypi.python.org/pypi/rgf_python.

I did run pip install rgf_python, but when I tried to run the example this error pops out in spyder:

  File "C:\Users\me\Anaconda3\lib\site-packages\rgf\sklearn.py", line 111, in <module>
    "config flag 'exe_location' to RGF execution file.".format(_EXE_PATH))

Exception: C:\Users\me\rgf.exe is not executable file. Please set config flag 'exe_location' to RGF execution file.

What should I change in order for config the flag 'exe_location'? May sound beginner but can't find it anywhere.

I had the same problem, and I'm not proud of the very "hacky" solution, but the bottom line is it's working now. I'm working on a Mac, so I'm sure your solution will be a bit different.

The normal pip install wasn't working for me, so to install I did...

git clone https://github.com/fukatani/rgf_python.git
cd rgf_python
python setup.py install

I was then able to find the executable file your error message is complaining about in the new rgf_python directory, specifically...

rgf_python/include/rgf/bin/rgf

From there, I tried to change the exe_location in my .rgfrc file to point to the above file, but it didn't work so I went to the /site-packages/rgf/sklearn.py file mentioned at the top of your error message and changed a line in the "_get_paths" function.

For me it was line 64/66. I suspect yours may be 54/56 if you're on Windows. I commented out the old declaration of "exe" and changed it to point to the rgf executable file I found earlier.

I realize how messy this is, so I hope someone can find a better solution, but in the meantime, this did the trick for me. Good luck, and let me know if you have any questions.


How exactly does DQN learn? I created my custom environment in gym, which is a maze. I use a DQN model with BoltzmannQPolicy.
It trains fine with the following variables:

position of the agent
distance from the endpoint
position of the endpoint
which directions can it move to

So I don't give it an image or anything. If I train and test it in the same environment (the same maze, without changing the position of walls) it can solve it easily. But if I introduce it to a completely different environment (maze) without the training then it doesn't know what to do. I don't know if the problem is with my code, or DQN is just for solving the same environment.
Which algorithm should I use instead?

The way you have set your DQN up, it is designed to solve just one maze at a time. It has not (and cannot) learn to solve mazes in general, because it has no access to data about the layout of the maze, and a basic DQN agent has no capability to memorise layout seen so far.
You could view the training process as general algorithm for "solving the maze". That is, given a new maze, you can take your written agent, run it for a while in training mode and it will produce a policy that solves that maze. You may be able to tune or adjust the algorithm so that it does it the most efficiently in terms of time or number of steps. It will not be able to do so as well as a hard-coded maze-solving algorithm, because DQN and Q-learning is a very generic learning algorithm that takes no advantage of consisent logic seen in mazes.
If you want to train a more generic maze-solving agent, this becomes a harder problem to solve using reinforcement learning, but it is achievable. There are two key - and linked - assumptions of the theory behind Q learning that you would need to address before you could make a general maze solver that coped with variations such s moving the walls, but without needing to be re-trained:

The Markov property is assumed for the state variables, which means that the state description contains all the relevant information about future state transitions and rewards. Without some way to know where the walls are, or at least note the position of walls it has seen before, the agent does not have access to a state representation with the Markov property.

There is at least one deterministic optimal policy. If you are intending to withold information from the agent (and not give it capabilities such as memory in order to construct missing information), then there may not be a deterministic optimal policy.

So it is possible to use reinforcement learning to teach an agent about solving mazes in general, and gain the ability to attempt to navigate out of a new unseen before maze. If that is of interest to you, you need to first decide what capabilities and knowledge about the environment the agent is allowed to observe. E.g. what specifically interests you about writing a maze solver?
Which approaches you could use to make a more general maze solver depends critically on how you want to frame the problem. For instance it is entirely valid to give the solver an image of the maze as input if you want the agent to learn to solve mazes in a similar manner to a human solving a printed puzzle.
There is one common factor that you will need for training a more generic maze solver - you will want to train on many example mazes taken from the population of all possible mazes. Training on just one maze as an example will typically be about as successful as training a supervised classifier or regression task on one example. Adding many more mazes, or better a maze generator, and training on many of them is likely to make training time much longer than before.


benchmark Result for MovieLens dataset? I am looking for a benchmark result or any kaggle competition held using MovieLens(20M or latest) dataset.
Similar question has been asked here but, provided links are dead so re-raising the question.

One result for MovieLens 20M using Factorization Machine can be found here. They got MAE: 0.60 and RMSE: 0.80.

Another result for MovieLens 20M using Autoencoders can be found here. They got RMSE: 0.81.


In XGBoost would we evaluate results with a Precision Recall curve vs ROC? I am using XGBoost for payment fraud detection.  The objective is binary classification, and the data is very unbalanced.  One out of every 3-4k transactions is fraud.

I would expect the best way to evaluate the results is a Precision-Recall (PR) curve, not a ROC curve, since the data is so unbalanced.

However in the eval_metric options I see only area under the ROC curve (AUC), and there is no PR option.  https://github.com/dmlc/xgboost/blob/master/doc/parameter.md

Also the documentation recommends AUC  http://xgboost.readthedocs.io/en/latest/how_to/param_tuning.html

Does it make sense to not use a Precision-Recall (PR) curve?

I am answering my own question, almost two years later.  XGBoost now has a new eval metric aucpr.

https://xgboost.readthedocs.io/en/latest/parameter.html#learning-task-parameters


What is the difference between Latent and Explicit Semantic Analysis I'm not quite sure what "latent" refers to in this context.

In Computing Semantic Relatedness using
Wikipedia-based Explicit Semantic Analysis they say

''Our semantic analysis is explicit in the sense that we manipulate
 manifest concepts grounded in human cognition, rather than
 'latent concepts' used by Latent Semantic Analysis''.

What does that mean in simple terms?

I had a look at the related Wikipedia articles, ( Latent (wikipedia.org/wiki/Latent_semantic_analysis), Explicit wikipedia.org/wiki/Explicit_semantic_analysis) ), and I wasn't able to make heads or tails of it.

Perhaps someone with a better appreciate of the nuance involved here might be able to provide me with a clear indication of the similarities and differences, the pros and cons between these two methods for accessing document/text fragment relatedness to a particular concept.

The difference is that with ESA, the concepts are already known and labeled (hence, "manifest concepts"), whereas in LSA the concepts are latent (they are undefined and need to be discovered).

Note the following statement from the ESA Wikipedia page:

The name "explicit semantic analysis" contrasts with latent semantic analysis (LSA), because the use of a knowledge base makes it possible to assign human-readable labels to the concepts that make up the vector space.[3][1]

In short, ESA uses prior knowledge of relationships between words and concepts (as well as labels for those concepts), whereas LSA does not.

References cited by Wikipedia for the quote above:

[1] Ofer Egozi, Shaul Markovitch and Evgeniy Gabrilovich (2011). "Concept-Based Information Retrieval using Explicit Semantic Analysis" (PDF). ACM Transactions on Information Systems 29 (2). Retrieved January 3, 2015.

[3] Gabrilovich, Evgeniy; Markovitch, Shaul (2007). Computing semantic relatedness using Wikipedia-based Explicit Semantic Analysis (PDF). Proc. 20th Int'l Joint Conf. on Artificial Intelligence (IJCAI). pp. 1606–1611.


is final fit with X,y or X_train , y_train? I split the dataset with

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

and the fit

from sklearn.metrics import log_loss
clf.fit(X_train, y_train)
clf_probs = clf.predict_proba(X_test)
score = log_loss(y_test, clf_probs)
print(score)

is final submission with

clf.fit(X,y) or clf.fit(X_train,y_train)???

Training dataset is something that you use to train your
dataset(data    which the model learns upon).
Validation dataset is that part of your data which is not used to
train your model, but to check the model's performance and optimize
it further. Here, as the model is optimized based on its performance
on the validation dataset, so according to experts, your model though
indirectly, has 'seen' the validation data as you might unknowingly
create a bias towards the validation data by optimizing the model to
perform better on the validation data only.
The stage of testing the models, comes after all the optimization(by
that time your model has processed (but not trained on) the validation
data). So, once you think your model is
optimized (you simply can't know it). Now you can and should use validation data also to train
the model along with the training data.
The testing data on the other hand is that part of your data which is
completely unknown to model(that is neither it is used to train nor
to validate the model). It is the completely unbiased performance
check of your model.

Usually in case of competitions, the data you are given, since it has labels, it should be used to train and optimize you model. But testing should always be done only after the model has been trained on all the labeled data, that includes your training(X_train, y_train) and validation data(X_test, y_test).

Hence you should submit the prediction after seeing whole labeled data :- Hence clf.fit(X, Y)

I know this long explanation was not necessary, but one should know why you do what you do.

Hope it helps, thanks!!


Generation of institution names I have found a number of parsers for the automatic extraction of institution names from texts (e.g. this one). My task is in a sense the inverse one: I want to automatically generate reality-like institution names, with a possibility to differentiate them by type (private-held, public, educational, etc.) and by branch.
Is there any algorithm / application / papers to be found? Alternatively, is there any (free access) database with such data?

Conditional on you having data, yes, you can. Check out Generative Adversarial Networks and/or Reinforcement Learning for text generation. This paper is a good starting point: https://openreview.net/forum?id=rJedV3R5tm.
Also, here's a tool that might help you. What you can do is generat these institution names without differentiating by type, and then build another model to classify them.


A new comer in this field - How to start? To be honest I find entire DS field so vague that I can't find actual roadmap of learning. I am actually a programmer for 10+ years and pretty much interested to get into Data related field. My stats and math suck at this moment.

My question is, is it Ok that I use my programming skills and jump right onto Machine Learning/Analysis tools like WEKA, Python Scikit etc and use their algorithm for understanding and keep studying/learning theory in parallel. Right now in terms of Data, I am already involved in:

Data Processing(RDBMS)
Data Scrapping(Python)

What I am thinking the logical steps are:

Data Mining
Data Analysis (sentiment analysis etc)
Data Presentation/Visualization( D3 etc?)
Machine Learning Kit/Libraries

I have given myself a 6 month time to get into this field.

Thanks

I would suggest you taking a Machine Learning and Statistics (Coursera, Udacity etc.) course straight away, since you have a profound experience in programming, so mastering any library wouldn't be of a big issue. Understanding of theoretical approaches is way more important than a particular library.

Main reason for that is the fact, that tutors of courses usually provide you with data big enough to understand the purpose of each algorithm/approach in the session/assignment so you will not only become proficient in some ML approach, but also understand which type of data can be then handled by a particular algorithm.

Also, I suggest you select a particular field of data science / machine learning, since data science, machine learning, statistics and so on, are just powerful tools and their application in computer vision, statistical modelling, natural language processing, fraud detection and other fields share something in common, but focusing on one field will allow you to progress faster.


Multivariate time series prediction with binary target I have an electronic component whose sensors record temperature, current and voltage values of various sub-elements. These readings are taken at regular intervals of time and I organized them as records of a dataset. In addition to these features, the dataset has a column with value 1 or 0 if at that moment the component is experiencing a malfunction or not. Here is an example of the dataset structure:

My goal is to build a model that is able to predict the occurrence of a malfunction with some time advance, so I think this is a multivariate time series prediction problem.

What approaches could I use to achieve this goal? I have read that it is possible to use Recurrent Neural Networks / LSTM but I do not have an adequate GPU.

I recently trained some multivariate LSTM regressions on a relatively slow, CPU-only laptop, and it worked. Training time didn't take too much, it's definitely feasible.

About your problem, I would definitely use some RNN architecture. LSTM are the most powerful RNN cells up to now, they have more memory and are able to learn longer sequences than GRUs, but they are slower than them. Choose based on you preference/circumstances.

The final layers must be Dense(), and the final output layer should have two nodes with softmax activation, to perform the binary classification. The most appropriate loss would then be some crossentropy measure.


Exploring variables to guide xgboost tuning In short: How to think about the type and distribution of my variables when choosing parameter values for xgboost?
Context: I have a dataset which I want to classify using the binary:logistic objective (I am using the R implementation). I wonder if visually inspecting variables in my dataset could inform selection of parameters values, and if so then how does one generally go about it.
Example: A dataset with a binary outcome variable and a mixture of binary/continuous predictors.
Outcome variable (one value per observation):

A set of predictors called flavours (~150 variables) and textures (~80 variables) have binary values 0, 1. Each observation has one or multiple flavours and textures (number of f/t per observation on the horizontal axis):

Some flavours/textures occur much more likely than others (frequency on horizontal axis):

I added some quantitative variables to summarise flavour/texture for each observation. I calculated a frequency for each f/t (= what is the percentage of observations where this f/t occurs) and then calculated min (most unique f/t in the given observation) max (most common f/t) and mean (average frequency).

Apart from that, there are three more continuous variables:

What conclusions regarding model parameters one might draw from this, if any?

A tip: Dont
A trick:Dont
The reason? Machine learning scientific methodology is based on cross-validation. Almost all papers (and i put the almost because of yes) select everything based on cross-validation and not in previous knowledge.
Xgboost is particularly more complicated because it has a lot of math involved.
For a simpler case, lets say that you have a problem with 3 features and you want to use a Lasso regression. Which hyperparameter will you choose? Even in this really simple case you don't know. You will need to do CV and select them there.
For xgboost the only tip that I can provide based on my experience is that the only important hyperparameter is the number of trees on the ensembling. The rest seems not to be so important.
My recommendation: only tune number of trees in the xgboost and select the best in cross-validation. Unless you want to find the absolute minime, that then, you are going to explore the whooooole hyperspace. Good luck!


ValueError: Input 0 of layer conv1_pad is incompatible with the layer: expected ndim=4, found ndim=3. Full shape received: [None, 224, 3] I am trying to make a gender classifier. I am using MobileNet from Tensorflow with input shape as (224,224,3). After training the model, I tried to check if the model was working by passing an image to the model to predict, but it throws the error in the title. I tried debugging it and google the error and tried everything I could, but the error still persists. Please help me to understand why the error.
model = tf.keras.applications.mobilenet.MobileNet(input_shape = (224,224,3), include_top = False, pooling = 'avg')
#model.summary()

x = Dropout(rate=0.4)(model.output)
x = Dense(3)(x)
x = Softmax()(x)
model= Model(model.inputs, x)

for layer in model.layers[:-3]:
    layer.trainable = False

model.compile(
    optimizer=Adam(lr=0.001),
    loss='categorical_crossentropy'
)

datagen = tf.keras.preprocessing.image.ImageDataGenerator(preprocessing_function = tf.keras.applications.mobilenet.preprocess_input,
          shear_range = 0.2, zoom_range = 0.2, horizontal_flip = True, validation_split = 0.1)
training = datagen.flow_from_directory(
    '...\Pictures\Model',
    target_size=(224, 224)
)

validation = datagen.flow_from_directory(
    '...\Pictures\Model',
    target_size=(224, 224)
)

batch_size = 32

history = model.fit_generator(generator = training, steps_per_epoch = training.samples//batch_size, epochs = 10,
                              validation_data = validation, validation_steps = validation.samples//batch_size, verbose = 2)

man = np.array(tf.keras.preprocessing.image.load_img('...\\Pictures\\Detected Faces\\0156224224_face.jpg', target_size = (224,224)))

model.predict(man) #Here the model throws an error

You need to make sure to add an extra dimension for the batch size, if you are passing in a single image the batch size would be 1. You can use np.expand_dims to add the extra dimension.


Ingestion of periodic REST API Calls into Hadoop We are planning to use REST API calls to ingest data from an endpoint and store the data to HDFS. The REST calls are done in a periodic fashion (daily or maybe hourly).

I've already done Twitter ingestion using Flume, but I don't think using Flume would suit my current use-case because I am not using a continuous data firehose like this one in Twitter, but rather discrete regular time-bound invocations.

Please I would like to hear suggestions / alternatives (if there's easier than what I'm thinking right now) about design and which Hadoop-based component(s) to use for this use-case. If you feel I can stick to Flume, then kindly give me also an idea how to do this.

You can use Kafka to ingest data into HDFS or any other cloud storage like S3 or Google storage.
 And you can use Gobblin to schedule your kafka consumer to write into HDFS.

Kafka Producers => Kafka Consumer => Gobblin (Monthly/weekly/Daily/hourly/minutes) => HDFS

Hope this will work for you


How to get p-value and confident interval in LogisticRegression with sklearn? I am building a multinomial logistic regression with sklearn (LogisticRegression). But after it finishes, how can I get a p-value and confident interval of my model? It only appears that sklearn only provides coefficient and intercept.

Thank you a lot.

One way to get confidence intervals is to bootstrap your data, say, $B$ times and fit logistic regression models $m_i$ to the dataset $B_i$ for $i = 1, 2, ..., B$. This gives you a distribution for the parameters you are estimating, from which you can find the confidence intervals.


Xgboost - How to use feature_importances_ with XGBRegressor()? How could we get feature_importances when we are performing regression with XGBRegressor()?

There is something like XGBClassifier().feature_importances_?

from xgboost import XGBClassifier
model = XGBClassifier.fit(X,y)

# importance_type = ['weight', 'gain', 'cover', 'total_gain', 'total_cover']
model.get_booster().get_score(importance_type='weight')

However, the method below also returns feature importance's and that have different values to any of the "importance_type" options in the method above. This was raised in this github issue, but there is no answer [as of Jan 2019].

model.feature_importances_


On Creating an Interoperable Network Matrix Complete self taught novice here.
Question
Are there any popular standards, or commonly used formats for representing/saving/sharing/serializing a network?
Background
I built a simple but very large (5.5m nodes, 16.7m edges) network matrix using my own python code. But if I wanted to use this matrix with other data science tools what format should I use?
Currently I have it in a giant csv file with headers and "1" representing edges. Which I think is called a binary matrix.
Goal
I want to make use of any benefits present in using binary files, or formatted flat files that would help me learn other tools.

There is no single universally accept standard, however, many packages use edgelist for sparse (un)directed (un)weighted graphs. Regarding storing and sharing, I usually see compressed text files (in ascii format) instead of binary ones, due to binary compatibility problems.

Maybe you should take a look at this draft: Unravelling Graph-Exchange File Formats.


Text classification- What to do when train and test data have different features I am performing binary text classification. I have to classify a tweet 0 if neutral and 1 if hate speech.

So as general thumb rule i preprocessed my data. create term document frequency and After removing sparse terms i divide my data into train and test.
I train my model using random forest and logistic regression and it worked fine.

set.seed(123)
tweetRand = randomForest(label ~ ., data = train_sparse, importance=TRUE, nTree=500 )
randPridct = predict(tweetRand, newdata = test_sparse)
table(test_sparse$label,randPridct >=0.5)

Its is working fine on test data which divided from raw content.
But when i am running it on a new unseen data it is throwing an exception.

> predicrRand_test=predict(tweetRand, newdata=sparse_4testing)
Error in eval(predvars, data, env) : object 'run' not found

My understanding is that 'run' is a feature present in training but not in unseen test data and during my model training 'run' was included in tdm. In preprocessing of test , run was not in test tdm.

SO how should i deal with these situation. I am new to data science. Please help.

It's perfectly fine to have different features in your training and testing partitions. If the two groups are randomly selected from the same population, one would expect, if there are many possible features, as is frequently the case in text classification, that differences would be observed. To handle this, the standard practice in the field is to train your model on the training partition, and then evaluate on the testing partition with any new features in that grouping being discarded. If you encounter the edge case of a test observation with no features after this procedure, it is common practice to use a heuristic to classify it, bypassing the model entirely (e.g., an observation with no features is classified as neutral).


The difference between data science and algorithm development I see a lot of job opportunities in the field of data science but I'm not sure the difference between a data scientist and deep learning algorithm developer.
Can someone explain that to me?

A data scientist has many general skills, such as

data collection
software developer
data engineering and cleansing
algorithm development, selection and refining **
statistical analysis
data visualization
etc.

A deep learning algorithm developer would be more specialized, concentrating more on that one aspect of the data science field, and less on the other skills.
Typically an algorithm developer will be part of a data science team, consisting of members having different skills, but overall the team will possess all of them.


Medical Image Analysis What are some good starting points for learning medical image analysis and combining it with deep learning?

I would like to analyze images with bone cancers but not sure what is proper way to preprocess them and prepare for the model.

Check out this blog post. There you find a summary of two fairly new papers from the field, which might start you off. One about semantic segmentation, the other about a classification problem.


Best practice with cross validation I have done a 10 fold Cross Validation on my data and have selected the best model from the results. With cross validation, I will have 10 models trained from different folds of the data. For the final model to use, should I take the average of the models or just fit a model for the entire training set?

Typically you would use the best model parameters and then re-run the model with the portion of the data set aside for training to come up with a new 'best' that you can run against your test set.


CNN strategy in recognizing spinned images I wrote my CNN code from scratch with some convolution kernels. But my CNN can't recognize flipped/spinned images correctly when there are only a few convolution kernels (3*3). My convolution kernels change very little during training. Why?

When there are over 10 convolution kernels, my CNN starts to recognize fipped images. So more kernels help. However it also starts to make wrong recognition.

How will the resolution of images affect the result compared to the convolution kernel size? The higher the resolution, the higher the dimension of this fitting problem

Learning for a CNN depends on the width and depth of a network. Wider and deeper networks can learn more complex data structures, including data augmented images. Increasing the width and depth of a network increase the capacity of the model to learn features.

Width is generally associated with the number of features. The wider a network is the more features it is able to learn. Imagine a network that is very thin. It would only learn the single feature that is correlated with high performance on the task.

Depth is will increase the capacity of the model better weigh lower-level features. Each successive layer is a combination of the previous layer. The lower layers learn simple features which are combined to create complex features in higher layers. In the case of faces, lower layers learn lines of different orientations. Higher layers learn to combine those lines to form eyes and noses.

Increasing the number of kernels is one way to increase the width of the network.

Increasing the number of layers (i.e., depth) is generally more useful.


How to use hashing trick with field-aware factorization machines Field-aware factorization machines (FFM) have proved to be useful in click-through rate prediction tasks. One of their strengths comes from the hashing trick (feature hashing).

When one uses hashing trick from sci-kit-learn, one ends up with a sparse matrix.

How can then one work with such a sparse matrix to still implement field-aware factorization machines? SKLearn does not have an implementation of FFM.

EDIT 1: I want to perform feature-hashing/hashing-trick for sure in order to be able to scale FFM to millions of features.

EDIT 2: Pandas is not able to scale to many fields. I also want to convert an arbitrary CSV (containing numerical and categorical features) into LIBFFM (field:index:value) format and perform hashing trick at the same time (preferably without using Pandas). Pandas2FFM does not scale even after performing the Hashing Trick.

One option is to use xLearn, a scikit-learn compatible package for FFM, which handles that issue automatically.

If you require feature hashing, you can write a custom feature hashing function:

import hashlib

def hash_str(string: str, n_bins: int) -> int:
    return int(hashlib.md5(string.encode('utf8')).hexdigest(), 16) % (n_bins-1) + 1


Student-Teacher matching problem The problem I want to solve is as follows:
I have data about how many teaching hours different students spend with their teacher in order to pass their exams. The teaching varies from 25-55 hours. In the school there are approximately 15 teachers and given a set of 10 student-teacher features that describe personality characteristics such as, age, sex, education, nationality, etc., I want to develop a student-teacher match model in order to minimize the teaching hours a student needs to pass the exams.

Any idea of what sort of algorithms I should use, or any examples I can look at?

This seems to be a recommendation problem. You can find a nice overview here.


Gumbel Softmax vs Vanilla Softmax for GAN training When training a GAN for text generation, i have seen many people feeding the gumbel-softmax from the generator output and feed into the discriminator. This is to bypass the problem of having to sample from the softmax which is a non-differentiable process and hence prevents training.

My question is though, why not just feed the regular softmax (no argmax!) from the generator directly into the discriminator? What is the benefit of using the gumbel-softmax?

Thanks.

Passing directly the output of the softmax is also common (among the few textual GANs out there), e.g. see the improved Wasserstein GANs (WGAN-GP).

With hard Gumbel-softmax (+ straight-through estimator), you pass one-hot encoded vectors, which is the same as what you have with real data. If you pass the output of the softmax, the discriminator should be able to more easily tell apart real data (one hot) from fake data (non-one hot).

That being said, in my opinion neither of the two approaches seems very promising nowadays. There seems to be far more REINFORCE-based textual GANs.


Tensorflow ArgumentError: argument --model_dir: conflicting option string: --model_dir I am trying to execute this Tensorflow tutorial https://github.com/tensorflow/tensorflow/blob/r0.12/tensorflow/examples/learn/wide_n_deep_tutorial.py but I have the following error:

ArgumentError: argument --model_dir: conflicting option string: --model_dir

that appears in line:

flags.DEFINE_string("model_dir", "", "Base directory for output models.")

Do you know how I can solve it?

Thanks in advance,

Keira

The example you linked is from version 0.12 but the current tensorflow version is 1.2. The example underwent a lot of changes from 0.12 to 1.0. Actually, the very line you are referencing is no longer there in the last version.

This way, it's possible that you are using a version of tensorflow that is no longer compatible with the APIs used by the example. Ensure that the version of tensorflow you are using is the same as the example.


Why does adding a dropout layer improve deep/machine learning performance, given that dropout suppresses some neurons from the model? If removing some neurons results in a better performing model, why not use a simpler neural network with fewer layers and fewer neurons in the first place? Why build a bigger, more complicated model in the beginning and suppress parts of it later?

The function of dropout is to increase the robustness of the model and also to remove any simple dependencies between the neurons.
Neurons are only removed for a single pass forward and backward through the network - meaning their weights are synthetically set to zero for that pass, and so their errors are as well, meaning that the weights are not updated.
Dropout also works as a form of regularisation, as it is penalising the model for its complexity, somewhat.
I would recommend having a read of the Dropout section in Michael Nielsen's Deep Learning book (freely available), which gives nice intuition and also has very helpful interactive diagrams/explanations. He explains that:

Dropout is a radically different technique for regularization. Unlike L1 and L2 regularization, dropout doesn't rely on modifying the cost function. Instead, in dropout we modify the network itself.

Here is a nice summary article. From that article:

Some Observations:

Dropout forces a neural network to learn more robust features that are useful in conjunction with many different random subsets of the other neurons.

Dropout roughly doubles the number of iterations required to converge. However, training time for each epoch is less.
With H hidden units, each of which can be dropped, we have
2^H possible models. In testing phase, the entire network is considered and each activation is reduced by a factor p.

Example
Imagine I ask you to make me a cup of tea - you might always use your right hand to pour the water, your left eye to measure the level of water and then your right hand again to stir the tea with a spoon. This would mean your left hand and right eye serve little purpose. Using dropout would e.g. tie your right hand behind your back - forcing you to use your left hand. Now after making me 20 cups of tea, with either one eye or one hand taken out of action, you are better trained at using everything available. Maybe you will later be forced to make tea in a tiny kitchen, where it is only possible to use the kettle with your left arm... and after using dropout, you have experience doing that! You have become more robust to unseen data.


how to use standardization / standardscaler() for train and test? At the moment I perform the following:

estimators = []
estimators.append(('standardize', StandardScaler()))
prepare_data = Pipeline(estimators)

n_splits = 5
tscv = TimeSeriesSplit(n_splits = n_splits)

for train_index, val_index in tscv.split(df_train):
    X_train, X_val = prepare_data.fit_transform(df_train[train_index]), prepare_data.fit_transform(df_train[val_index])

X_test = prepare_data.fit_transform(df_test)

Now I would like to know if this is correct. My concern is that X_train and X_test are transformed separately. While in the first instance I thought this is how it should be I'm about to change my mind as I think I have to use the mean and std of the train set to use within the test set?

The recommended way (see 'Elements of Statistical Learning', chapter 'The Wrong and Right Way to Do Cross-validation') is to calculate the mean and the standard deviation of the values in the training set and then apply them for standardizing both the training and testing sets.

The idea behind this is to prevent data leakage from the testing to the training set because the aim of model validation is to subject the testing data to the same conditions as the data used for the model training.


LSTM Target Is Also One of It's Inputs? I have two input arrays that include both historical and forecasted data, and one input array that is only historical. I'm trying to predict (or "forecast") the latter array given the forecasts of the first two arrays.
Inputs:

Rainfall (historical and forecast)
Air temp (historical and forecast)
Water volume (historical)

Training target:

Water volume (historical)

Desired model output:

Water volume (forecast)

I'm still a bit of a noob, and lost as to how this would work. None of the examples I've found take the target's previous state as input. The crux of the problem is that I could do a vanilla multivariate LSTM with only rainfall and air temp as inputs, but it would not take into account the starting condition of water in the container.

Doing some further searching, I found a nice post by Jason Brownlee that has helped me better understand the problem and one potential solution:

a multi-input model with an LSTM input for the historical data and a
vector input for expected conditions.

Furthermore, Keras' Functional API (https://keras.io/guides/functional_api/) will help me build such a model.


Training machine learning models from log files Im starting to learn about ml and most of examples show very simple examples on how to train your ml modell. For me this looks more like statistic calculations than actual ml. These kind of examples could very easily been written in sql server etc.

What Im looking for is the ml to find pattern within data that i dont know about. For example supose i have a lot of devices which is logging temperature, humidity, hours used and much more. At some point these devices fails for some unknown reason and Will be marked as "failed"

So what om looking for is if there is a way to use ml to analyze all the log files and find common patterns to why these devices failed, without me have to "tell it what to look for first"  in this way I Will be able to predict feature failures based on log files.

Would this be something I can use ml for and would Azure be a good starting point for this?

Hi and welcome to the community!

Sure you can use ML. I would start with brainstorming to formulate the problem. So one idea could be Sequence Learning:

Injecting a unique and recognizable number instead of F and training a NN-based model (RNNs are used in this case e.g. ESNs or LSTMs) will learn the dynamic of the data themselves as they model memory. But for training such models you need lots of data actually (to be more precise, you need long Time-Series). In this solution, the task is to see that unique and recognizable number in the prediction and output is a failure. See this for more info.
Hidden Markov Models are also designed for such tasks.

Another idea could be:

Segment each time series into intervals ending in a failure. For example:

  ts1 = [1,2,1,2,F,1,2,3,4,F1,2,F]

Becomes

> ts1_1 = [1,2,1,2]
> ts1_2 = [1,2,3,4]
> ts1_3 = [2]

Now you have a set of training samples (positive samples only!) for Time-Series 1 (e.g. Temperature). Now produce some negative samples from those sequence segments which do not result in "F" and train a model for e.g. classifying. Then on real-time prediction, you need to make a prediction after receiving every new point (temp for instance) and see if it is going to result in a failure.

DISCLAIMER: This way is not what you really do in practice but a nice starting idea for a newbie learner in the field!

I recommend having a look at this answer, this tutorial and this video as well.

Hope it helps!


Free a bit of RAM space Here is Colab Notebook

After 1500 episodes if batch_size=256, the RAM crashed. With Colab, I have the equivalent of 25.5 gigs of RAM. Is it normal? Or I don't have enough ram? I thought at a certain point, the RAM must free a bit of space. How can I fix that?

If RAM memory is a problem during training, and you have run for 1500 episodes, it would suggest to me that you are saving some information in each episodes (e.g. appending to a list). That will slowly add up over time and cause a crash.

After a quick look at your code, it looks like the scores and ep_history grow with every loop. You could consider writing that information to disk e.g. to a json, file or pickle object, with something like this

import pickle

if episode % 1000 == 0:
    with open(f"./results_dir/scores_{episode}.pkl", "wb") as file_handle:
        pickle.dump(scores, file_handle)

You should of course retain the amount of history required for your print() statements and computing average scores over time.

You rarely need to worry about freeing up some RAM yourself, as Python (and then afterwards, the operating system itself) will start to allow memory to be overwritten. The results is the same, they are essentially being deleted.


Using Heuristic Methods for AB Testing I've just started reading about AB testing, as it pertains to optimizing website design.  I find it interesting that most of the methods assume that changes to the layout and appearance are independent of each other.  I understand that the most common method of optimization is the 'multi-armed bandit' procedure.  While I grasp the concept of it, it seems to ignore the fact that changes (changes to the website in this case) are not independent to each other.

For example, if company is testing the placement and color of the logo on the website, they find the optimal color first then the optimal placement.  Not that I'm some expert on human psychology, but shouldn't these be related? Can the multi-armed bandit method be efficiently used in this case or more complicated cases?

My first instinct is to say no.  On that note, why haven't people used heuristic algorithms to optimize over complicated AB testing sample spaces?  For an example, I thought someone might have used a genetic algorithm to optimize a website layout, but I can find no examples of something like this out there. This leads me to believe that I'm missing something important in my understanding of AB testing as it applies to website optimization.

Why isn't heuristic optimization used on more complicated websites?

If I understand you question correctly, there are two reasons why genetic algorithm might not a good idea for optimizing website features:

1) Feedback data is coming in too slow, say once a day, genetic algorithm might take a while to converge.

2) In the process of testing genetic algorithm will probably come up with combinations that are 'strange' and that might not be the risk the company wants to take.


1d time series to time series approximation using deep learning I have a basic question about choosing right architecture of my deep learning task.

I have input signal $X(t)$ as a function of time when it is fed to the system, it generates output $Y(t)$ which is also a function of time.

I have a bunch of experiments performed with many input signals $X_1, X_2, X_3 ,...$ and many output signals $Y_1,Y_2,Y_3,...$.

Instead of performing more experiments I would like to generate neural net where I can feed input signal X (t) and get output Y (t).
I want to use already measured data as training data.

To me, this looks like like regression problem but involves time series data.
One of the neural network architectures that come to mind is RNN LSTM OR CNN.
People mainly use LSTM for forecasting problem (knowing history to predict the future) not regression.

So can I use CNN, use 1D filters and then some pooling and fully connected layers?
Will this work? What kind of set up should I use?

Yes, you can use one-dimensional convolutions but you have to consider the fact that the size of your inputs should be the same. In Tensorflow you can exploit tf.nn.conv1d for your purpose. RNN networks like LSTMs and GRUs are also other architectures that can be used. If your input has a patter that may happen different times in a single signal try to use conv1d. If you want to use RNNs you have to find which task you have such as a one-to-one or one-to-many-relation. For convolutional architecture, take a look at here and here, and take a look at [here] for RNNs.


How to best visualize data when outliers lead to lack of contrasting colors for the rest of the plot? I have plotted the following map of Philippines according to the population in its regions, using matplotlib.pyplot in Python 3:

As can be seen, there's one province (in red) that is an outlier. When looking at the data, it is clear:

 population = [16.892118644067804,
 27.13259740259741,
 27.325479452054793,
 25.502352941176472,
 29.62087999999999,
 25.593870967741932,
 27.649384615384616,
 25.38908450704225,
 26.924931506849322,
 26.636250000000004,
 27.317105263157895,
 26.58338345864662,
 29.48689393939393,
 28.283986013986016,
 26.563118279569903,
 23.037142857142857,
 24.2674]

This is making the plot "unattractive" - there would have been a lot more contrasting colors for the other regions had this one region's data not been an outlier. What options do I have to rectify this?

Remove this region's data. In that case, this region would appear white (uncolored), and the rest will have contrasting colors. But for whatever reason, that's not an option for me currently.
Manually tweak that one data point to "blend in" with the other data points (something like a mean of all the other data points, or even a handcrafted value that is reasonably close to the other data points, like 22, or whatever). I tried this, and this does give more contrasting colors. The trouble is, this is misrepresentation of data. So, if possible, I would like to avoid it.
Stretch the yellow further down in the colormap, so that only a little space is given to red. That is, instead of the 2 colors starting to transition from one to the other around 24 in the colorbar, it should do so around, say, 19. Anybody knows how to do that?

Any other ideas? What are some of the best practice approaches to tackle this issue?

With Wolfram Language you may use GeoRegionValuePlot with the ColorFunction option to customise the colour scaling.

With some "AdministrativeDivision" "Population" data for the Philippines.

popPH = EntityValue[
   Entity["Country", "Philippines"]["AdministrativeDivisions"],
   "Population",
   "EntityAssociation"];

First@Normal@popPH

Then GeoRegionValuePlot with and without a custom ColorFunction.

linear = GeoRegionValuePlot[popPH,
  ColorFunction -> ColorData["SolarColors"],
  PlotLabel -> "Linear ColorFunction",
  PlotLegends -> Placed[Automatic, Below]
  ];

log = GeoRegionValuePlot[popPH,
  ColorFunctionScaling -> False,
  ColorFunction -> (ColorData["SolarColors"][
      Rescale[Log@#, Log@QuantityMagnitude@MinMax[popPH]]] &),
  PlotLabel -> "Log ColorFunction",
  PlotLegends -> Placed[Automatic, Below]
  ];

Row[{linear, log}]

Hope this helps.


How to avoid confusion between Percentage and Percentage Points? Whenever I have to communicate a change in a numerical metric, I found myself struggling if I should use Percentage or Percentage Points. Especially, when the metric is a percentage itself, it can be really tricky.
For example, if we track the Retention Rate of users in a platform and the percentage increased from 10% to 30%, we can either say that we had an increase of 20p.p. or we had an increase of 200%.
Another example, if we classify the risk of a business decision to 5% of revenue drop and an alternative choice has 7% revenue drop. We can say 2p.p increase or 40% increase.
Things to consider

Not every team member has a math/statistical background to understand the difference of the two options without giving an explanation. The difference is not common in the "average-person".
Not everyone has English as his/her first language and the right choice of words might not have a big effect. For example, using "increase by .." or "increase to.." might not have any difference at all.
The use of p.p. or % can be either verbal in person or through an online, video call, an official report or a short message in a chat app (slack, Skype etc)

Why it matters?

The number per se, might have a different impact to others. For example, an increase of 20p.p could "feel" low, while the increase of 200% "feels" enormous.
Using % only ends up to misunderstandings. For example, I found myself in a situation where a team member understood an increase of 50% to a 10% retention rate was ended up to 60% retention rate.

What to do?
How to choose the way I communicate such an increase in the following cases

Verbal in person or video call
An official report or documentation
A short, informal message like in Slack or Skype

Use whatever conveys the message more clearly.
This usually means to use percentages only for absolute metrics and use %-points for metrics which are presented as percentages (e.g. "retention rate increased by 5%-points" in your example).
From my experience you can expect almost everybody to know the difference. The only exception would be highly operational staff, e.g. blue collar staff on the shop floor, cashiers in a supermarket etc.
In the latter cases, or if there are language barriers (e.g. audience is not fluent in English) or technical issues (e.g. a bad connection), make the changes explicit. For example:

Retention rate increased from 10% to 15%
Potential revenue drop of 7% compared to 5% in the first scenario


Logistic regression in python I have done linear and multivariate regression so I understand what probability, cost and gradient descent functions are. I do not understand what the last 3 formulas mean and how they relate to each other. I don't expect anyone to explain them to me but could someone please share a link that explains them in simple terms? I've tried looking online but the formulas used often look different from the ones below.

Sorry for posting a picture like this but I don't know how to format formulas.

Logistic Regression is rather a hard algorithm to digest immediately as details often are abstracted away for the sake of simplicity for practitioners. To explain the idea behind logistic regression as a probabilistic model, we need to introduce the odds ratio, i.e. the odds in favor of a particular event. The odds ratio can be written $\frac{p}{1-p}$, where p stands for the probability of the positive event. We can then further define the logit function, which is simply the logarithm of the odds ratio (log-odds). The logit function takes input values in the range 0 to 1 and transforms them to values over the entire real number range, which we can use to express a linear relationship between feature values and the log-odds:

$logit \frac{p}{1-p} = w_{0}x_{0} + w_{1}x_{1}...w_{n}x_{n}= w^{T}x $ (1)

In Logistic Regression we are interested in predicting the probability that a certain sample belongs to a particular class, which is the inverse form of the logit function. It is also called the logistic function, sometimes simply abbreviated as sigmoid function:

$p(Y=1|x) = \phi(z=W^Tx) = \frac{1}{1+e^{-z}}$ (2)

$ p(Y=0|x) = 1 - p(Y=1|x) $ (3)

Here it comes the tricky part, the parameter estimations i.e. w (see eq. 1). It basically states that a solution to a Logistic Regression problem is the set of parameters w that maximizes the likelihood which is defined via maximum likelihood estimator (recall that we are going to define a loss function going from here). As logistic regression predicts probabilities, rather than just classes, we can fit it using likelihood:

$ L(w) = P(Y_{i}|w;x) = \prod_{i=1}^{n} \phi(z)^{Y_{i}}(1-\phi(z))^{1-Y_{i}}$ (4)

In practice, instead of maximizing this likelihood, it is easier to maximize the (natural) log of this equation, which is called the log-likelihood function:

$log(L(w)) = \sum\limits_{i=1}^n [Y_{i}log\phi(z)+(1-Y_{i})log(1-\phi(z))]$ (5)

Or to make it mathematically more convenient, we take the negative of this function, so that it can be minimized using gradient descent:

$ cost = - log(L(w)) = l_{i}(w) = \sum\limits_{i=1}^n [- Y_{i}log\phi(z)-(1-Y_{i})log(1-\phi(z))]$ (6)

The cost would look like this (taken from here):

Here then we take derivative with respect to parameters w (see eq. 1), rather than z (in your photo it is with respect to $\alpha$). Those derivations would be like what you showed. Still I recommend checking out this blogspot that goes through these steps in more details, or parts of implementation in python here, if needed.


How to train a neural network on multiple objectives? I have a multi-class neural network classifier that has K classes(products). For every row, only one of the classes will be 1 at a time. Now, this approach works fine if I have only 1 objective to optimize i.e Which of these N products was "clicked" by the user.
But how will I solve this problem if I need to optimize on a 2nd objective i.e Which of these N products was "purchased" by the user?
A purchase event is always preceded by a click event. I can obviously solve this problem by training two separate models - 1 for click and the other for purchase. But purchase data is very low as compared to the click data. And we ran the purchase model on production. It did not perform well.
So how do I take both the click and purchase data and frame my problem as "Which of these N products will be clicked and possibly purchased by the user"? and train a single model.
Any sources or papers in this direction will be really helpful.

What you're referring to is called multi-task learning, where your goal is to have a single network learn multiple tasks (in your case "click" and "purchase").
The benefit of having a single model learn both tasks is that the network can use information extracted for one task to improve its performance on the other.
Technically you need your network to have two output layers (i.e. one for predicting "clicks" and one for "purchases") and likewise two loss functions (one for each objective). These loss functions usually contribute equally towards the total loss, unless you don't want to consider both tasks of equal importance.
If you're using keras an example for multi-output models can be found here.


How to choose an optimal threshold for binary discretization We know that we usually do discretizations to continuous features to remove extra information and unwanted regularities, which makes the model robust and well-predicted.
But I am wondering except based on the context of feature how to choose an optimal threshold for data discretization, binary or multi.

Inspecting the histogram chart of that feature by different numbers of bins can be helpful. the maximum width of the bins in which the distribution of the data does not change noticeably can be a suitable choice.


How to handle features which are not always available? I have a feature in my feature vector that is not always available respectively sometimes (for some samples) it makes no sense to use it. I feed a sklearn MLPClassifier with this feature vector. Does the neural network learn by itself when the feature make sense to use for its decision or do I have to add a flag in the feature vector which says for example “1” if it makes sense or “0” if it does not make sense.

If the feature doesn't make sense in a subset of the samples, doesn't this mean that this is (or should be) a separate dataset, that needs a second model? That's one approach I'd think about.

The second would be to work with the data (feature) itself. It's probably best to use neutral value.

In case of numerical value:

try using a mean or median value, calculated on all entries
try using an extreme value, e.g. -1 if your feature has only positive values. This should indicate that the feature is missing and the network should be able to handle it.

In case of textual value, e.g. word embeddings, replace the value with a placeholder like N/A that doesn't have an embedding


How to treat patients without events in time-to-event analysis? I'm working with longitudinal data for a series of patients. Duration of followup on a patient-level is non-uniform.
Patients can either experience a discrete event (e.g., a heart attack) or never experience the event. This feature is of course binary. Additionally, patients that have experienced an event (e.g., the first heart attack) can also continue to experience more events (e.g., subsequent heart attacks). Each event is anchored to an event date which will be compared to when the patient was diagnosed with their primary, chronic condition (e.g., multiple myeloma).
I'd like to append to my dataset a derived column - TimeSinceLastEvent. Derivation of this value for first events would be calculated by (FirstEventDate - DiagnosisDate)/365 and subsequent events would be calculated by (SecondEventDate - FirstEventDate)/365, (ThirdEventDate - SecondEventDate)/365, etc.
How should I code this derived column for patients that never experience the event? I can't insert NA/NaN for these patients because downstream analyses require non-NA and finite data; so they would be imputed incorrectly anyway. One thought I had was setting these values to something drastically different, but standardized, such as -1 or 9999. Is this a valid and reasonable approach? If not, what have you used?

Generally, I would pick a value that is vastly outside the range of values that are non-imputed in this column since that'd probably cause more extreme (noticeable) errors in case they get included. So rather 9999 than -1. However, as you probably already attempted, the "most correct" solution would be to set it to NaN since the column is just not defined in these cases. If at any step during your analysis, some computation is done on the TimeSinceLastEvent column, anything but NaN would technically lead to incorrect results.
What is your downstream analysis strategy? Can you maybe split the data into two subsets (one with heart attacks, one without) and analyse them separately?


Classification model to classify large number of classifiers? Hello I am  very new into the field of machine learning/deep learning , and I am finding it hard to select the right model for my research.

What I am trying to build is a model to classify which subway route a user have used based on travel time and transfer time given the origin station and destination station.

Here is a description of my data set.

   BSEC BSTN ASTN1 BSTN2 ASTN2 BSTN3 ASTN3 BSTN4 ASTN4 BSTN5 ASTN TFtime Ttime
 69551 1001  1703  1703     0     0     0     0     0     0 1003    399  2933
 69664 1001  1703  1703     0     0     0     0     0     0 1006    399  2284
 66606 1001  1703  1703     0     0     0     0     0     0 1701    118  1750
 66600 1001  1703  1703     0     0     0     0     0     0 1701    118  1750
 66601 1001  1703  1703     0     0     0     0     0     0 1701    118  1750
 69434 1001     0     0     0     0     0     0     0     0 1703      0  1005

ASTN1,BSTN2,ASTN2...BSTN5 refers to via stations
BSTN ASTN refers to boarding and arrival stations.

I have a another data set of route information labeled.

The problem starts here.

I am trying to build a model that can classify which route a user have used given BSTN, ASTN, and time info BSEC, TFtime, Ttime.
There are too many labels of routes because the routes all differs for each pair of Origin Station and Destination Station.

Below are number of routes per origin station and destination station

   BSTN  ASTN trips
  <dbl> <dbl> <int>
1   150   152     3
2   150   153     7
3   150   154     2
4   150   156     2
5   150   157     2
6   150   158     4

as described there are already 20 different routes for only 5 Origin Destination pair. There are total of 109,425 pairs of origin and destination
and 236,213 number of routes. I could not give label to every 236,213 routes for the model to classify.

i tried making random forest model for every pair of Origin Destination pair.
But I was not able to tune or interpret them because there are too many type of models.

What would be a proper model for my situation?
Would there be a way the model could interpret given OD pair, and then perform classification within the Origin Destinatnion pair assembly?

I would really appreciate some advice or help.

It looks like a very difficult problem, since there are many possible classes and very little information in the features to distinguish them. For the record, the reverse problem of estimating the travel time based on the route would probably be more feasible.

So you can't expect great performance on a problem like this, the goal will be to design the problem in a way which makes things as simple as possible for the classifier to do a decent enough job. Here are some suggestions:

Start with training a model specific to a pair BSTN,ASTN.
Discard the least likely routes, i.e. routes which are rarely used for the pair BSTN,ASTN (for instance routes with frequency lower than 10).
Inspect the data to see if the features allow a distinction between the (main) classes. For instance you can plot the distribution of BSEC, TFtime, Ttime for different routes: if the distributions are close there's little chance the classifier will succeed. You can also train a decision tree and inspect it manually so see what happens.


If A and B are correlated and A and C are correlated. Why is it possible for B and C to be uncorrelated? Let's say
A and B are correlated
A and C are correlated
B and C is uncorrelated
How is it possible for B and C to be uncorrelated when they are both correlated to A?

Imagine a random point on a plane with coordinates $(x, y)$, where $x, y \in [-1, 1]$.

A = both $x$ and $y$ are positive

B = $x$ is positive

C = $y$ is positive

It is clear A is correlated with both B and C, which are not themselves correlated (assuming uniform distribution).


Conditional Function in R I have an R function for (i in 1:10) that reads in data and performs some cleaning. Is it possible to write a flexible statement to execute certain parts of the code conditionally, for example have different steps for i = 2 and 8?

You can add conditions inside the loop if you want:

for (i in 1:10){
  if (i<=2){print("1-2")}
  if (i>2){print(" >2")}
}


Changing categorical data to binary data is not reflected on the dataset I am working through the Titanic competition.  This is my code so far:

import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

train = pd.read_csv("https://raw.githubusercontent.com/oo92/Titanic-Kaggle/master/train.csv")
test = pd.read_csv("https://raw.githubusercontent.com/oo92/Titanic-Kaggle/master/test.csv")

train['Sex'].replace(['female', 'male'], [0, 1])
train['Embarked'].replace(['C', 'Q', 'S'], [1, 2, 3])

# Fill missing values in Age feature with each sex’s median value of Age
train['Age'].fillna(train.groupby('Sex')['Age'].transform("median"), inplace=True)

linReg = LinearRegression()

data = train[['Pclass', 'Sex', 'Parch', 'Fare', 'Age']]

# implement train_test_split
x_train, x_test, y_train, y_test = train_test_split(data, train['Survived'], test_size=0.2, random_state=0)

# Training the machine learning algorithm
linReg.fit(x_train, y_train)

# Checking the accuracy score of the model
accuracy = linReg.score(x_test, y_test)
print(accuracy*100, '%')

This line previously looked like this: data = train[['Pclass', 'Parch', 'Fare', 'Age']], which ended up giving me an accuracy score of 19.5%.  I realized that I didn't include sex so I went ahead and did this:

data = train[['Pclass', 'Sex', 'Parch', 'Fare', 'Age']]

Then, I got the following error:

ValueError: could not convert string to float: 'female'

Here I realized that the changes that I've done to my train['Sex'] and train['Age'] did not reflect on the training and the testing of the model, which seems to be the reason why my model performed at 19.5%.  How do I come across this problem?

You need to transform your independent variables into numeric values. Normally for binary variables, we use the 0-1-encoding. Introduce a new variable called is_female. If the observation is a male person then give the variable the value $0$ and if the observation is a female person give the observation the value $1$.

If you have categorical variables like a city with three categories you will need to create additional variables. Imagine we have the possible values ["New York", "London", "Moskau"]. Then you create three variables is_new_york, is_london, is_moskau.
If we have an observation from New York this will result in is_new_york=1, is_london=0, is_moskau=0. If we have an observation from London then the values will be is_new_york=0, is_london=1, is_moskau=0 and if we have an observation from Moskau then the values will be is_new_york=0, is_london=0, is_moskau=1. This type of encoding is called one-hot encoding. You can also have multiple cities. For example, if a person lives in Moskau and London then you can use is_new_york=0, is_london=1, is_moskau=1.


Some suggestion for career in data science or predictive modeling I am looking to choose my career in the area of decision science or predictive modeling and I am aware that this is kind of opinion based but I would like to have some suggestion from experts that I can use it to build my career in correct path. What are the tools should I know like R, SAS or any other. What are the thinks I should know to work in a data science or machine learning or predictive modeling. For me I am having problem in identifying steps that I should follow. Please suggest me some steps to follow.

First of all I should say you question probably is an off-topic and will be closed soon.

Discussed at this SE site

Anyway I can target you to similar questions discussed at this SE site already:

Statistics + Computer Science = Data Science?
Starting my career as Data Scientist, is Software Engineering experience required?

Cross Validated SE

A set of relevant questions at Cross Validated Stack Exchange:

https://stats.stackexchange.com/questions/tagged/careers

Data scientist map

This is good infographics of data science knowledge you might need to start a career (Link to image):

Careers SE

Also, simple "data scientist" querying of Careers SE site
http://careers.stackoverflow.com/jobs?searchTerm=data+scientist&location= will lead you to the following knowlege "tags":

R
bigdata
data-visualization
hadoop
mapreduce
scala
python
matlab

etc.


Confused about how to apply KMeans on my a dataset with features extracted I am trying to apply a basic use of the scikitlearn KMeans Clustering package, to create different clusters that I could use to identify a certain activity.  For example, in my dataset below, I have different usage events (0,...,11), and each event has the wattage used and the duration.

Based on the Wattage, Duration, and timeOfDay, I would like to cluster these into different groups to see if I can create clusters and hand-classify the individual activities of each cluster.

I was having trouble with the KMeans package because I think my values needed to be in integer form. And then, how would I plot the clusters on a scatter plot?  I know I need to put the original datapoints onto the plot, and then maybe I can separate them by color from the cluster?

km = KMeans(n_clusters = 5)
myFit = km.fit(activity_dataset)

       Wattage        time_stamp       timeOfDay   Duration (s)
    0    100      2015-02-24 10:00:00    Morning      30
    1    120      2015-02-24 11:00:00    Morning      27
    2    104      2015-02-24 12:00:00    Morning      25
    3    105      2015-02-24 13:00:00  Afternoon      15
    4    109      2015-02-24 14:00:00  Afternoon      35
    5    120      2015-02-24 15:00:00  Afternoon      49
    6    450      2015-02-24 16:00:00  Afternoon      120
    7    200      2015-02-24 17:00:00    Evening      145
    8    300      2015-02-24 18:00:00    Evening      65
    9    190      2015-02-24 19:00:00    Evening      35
    10   100      2015-02-24 20:00:00    Evening      45
    11   110      2015-02-24 21:00:00    Evening      100

Edit:
Here is the output from one of my runs of K-Means Clustering.  How do I interpret the means that are zero?  What does this mean in terms of the cluster and the math?

print (waterUsage[clmns].groupby(['clusters']).mean())
          water_volume   duration  timeOfDay_Afternoon  timeOfDay_Evening  \
clusters
0             0.119370   8.689516             0.000000           0.000000
1             0.164174  11.114241             0.474178           0.525822

          timeOfDay_Morning  outdoorTemp
clusters
0                       1.0   20.821613
1                       0.0   25.636901

For clustering, your data must be indeed integers. Moreover, since k-means is using euclidean distance, having categorical column is not a good idea. Therefore you should also encode the column timeOfDay into three dummy variables. Lastly, don't forget to standardize your data. This might be not important in your case, but in general, you risk that the algorithm will be pulled into direction with largest values, which is not what you want.

So I downloaded your data, put into .csv and made a very simple example. You can see that I am using different dataframe for the clustering itself and then once I retrieve the cluster labels, I add them to the previous one.

Note that I omit the variable timestamp - since the value is unique for every record, it will only confuse the algorithm.

import pandas as pd
from scipy import stats
from sklearn.cluster import KMeans
import matplotlib.pyplot as plt
import seaborn as sns

df = pd.read_csv('C:/.../Dataset.csv',sep=';')

#Make a copy of DF
df_tr = df

#Transsform the timeOfDay to dummies
df_tr = pd.get_dummies(df_tr, columns=['timeOfDay'])

#Standardize
clmns = ['Wattage', 'Duration','timeOfDay_Afternoon', 'timeOfDay_Evening',
         'timeOfDay_Morning']
df_tr_std = stats.zscore(df_tr[clmns])

#Cluster the data
kmeans = KMeans(n_clusters=2, random_state=0).fit(df_tr_std)
labels = kmeans.labels_

#Glue back to originaal data
df_tr['clusters'] = labels

#Add the column into our list
clmns.extend(['clusters'])

#Lets analyze the clusters
print df_tr[clmns].groupby(['clusters']).mean()

This can tell us what are the differences between the clusters. It shows mean values of the attribute per each cluster. Looks like cluster 0 are evening people with high consumption, whilst 1 are morning people with small consumption.

clusters  Wattage     Duration   timeOfDay_Afternoon  timeOfDay_Evening timeOfDay_Morning
0         225.000000  85.000000             0.166667           0.833333  0.0
1         109.666667  30.166667             0.500000           0.000000  0.5

You asked for visualization as well. This is tricky, because everything above two dimensions is difficult to read. So i put on scatter plot Duration against Wattage and colored the dots based on cluster.

You can see that it looks quite reasonable, except the one blue dot there.

#Scatter plot of Wattage and Duration
sns.lmplot('Wattage', 'Duration',
           data=df_tr,
           fit_reg=False,
           hue="clusters",
           scatter_kws={"marker": "D",
                        "s": 100})
plt.title('Clusters Wattage vs Duration')
plt.xlabel('Wattage')
plt.ylabel('Duration')


How to prove that a function is sigmoidal Given a function, how to prove that it is sigmoidal in nature. So far, my approach has been to verify if the properties of sigmoidal functions hold:

1)That it is monotonic

2)That it is constrained by a pair of horizontal asymptotes

3)That it has a first derivative that is "bell" shaped

4)That it is convex for values less than 0 and concave for values more than 0

5)That a Sigmoid function and its affine compositions can posses multiple optima.

For example this is one of the functions that I am trying to verify:

Trying to prove all the properties seems like time consuming especially in exam scenarios. I suppose there is a better approach that exists and I should follow. Please suggest me what other approach could be used.

Thank You.

I'm not so sure about how close this is to what you want but one way to prove if a function $y = f(x)$ is the sigmoid equation, is to check if it's a solution of the ordinal differential equation:

$$
\frac{dy}{dx} = y \; (1-y)
$$

with initial condition $y(0) = 1/2$.


Modern reference on general Deep Learning theory In the present, Deep Learning is experiencing lightspeed growth, with plethora of new architectures and radiant ideas emerging each month. Since the last few years several influential ideas, applied in different papers have been developed - among which most important are Transformer, EfficientNet, MobileNets. Is there some modern reference, in the spirit of the famous https://www.deeplearningbook.org/, in some way, generalizing the most prominent advances in Deep Learning?

For the latest practices of deep learning architectures, I follow the Kaggle latest Competition's notebooks.
Say, for example, In the recent finished Cassava Leaf Deasese Computer Vision Competition, People are sharing the experimental notebooks on different State of the art Architectures Like Vision Transformer(Various versions pretrained  - vit_large_patch16_384, vit_base_patch16_384 etc. models), Facebook's Data-efficient Image Transformers , EfficientNet (Noisy Student version, Imagenet Version) and etc.
 https://www.deeplearningbook.org/ this book undoubtedly one of the great book. But for the latest practices if you follow the Kaggle notebooks you get a good explanation with codes.
I suggest you,  Explore this Notebooks Section of the Casava Leaf Disease Classification Contest to get more Models and Various recent new released Loss Functions and so on.


Agglomerative Clustering Stopping Criteria I am trying to implement section 3.4 of paper Predicting Important Objects for Egocentric Video Summarization where they have created a distance matrix of frame histograms.

In short, let say Ω is mean of distances between all frames,DVis distance matrix.

I didn't understand what is meant by this:

  We next perform complete-link agglomerative clustering
  with DV , grouping frames until the smallest maximum interframe
  distance is larger than two standard deviations beyond
  Ω

Can this be achieved by setting the cutoff value to 2Ω in Matlab's clusterdata function ?

Clearly, 2 standard deviations beyond Omega is not the same as twice the mean.

Apparently, their process is this:

compute the distance matrix
compute the mean
compute the standard deviation
compute hierarchical clustering with maximum linkage
cut the tree at mu+2*sigma

Because complete linkage is in O(n^3), this approach will not scale to longer videos or higher frame rates.


What is an appropriate approach to sampling for probability of default using a classification model? If we have a loan book and want to train the data to predict the probability of default, what is an appropriate way to sample the historical data to train the model, given that each account is open over a period of time and not just at a single snapshot?

For example - for predicting defaults of new customers on a loan book, it is easy to select data to train a model at an equivalent point in time - i.e. when each customer is new, we may look at the FICO score, age of customer, home State etc at the point of loan application.

But once a loan has been made and we wish to build a model for likelihood of default given ongoing repayment history, what is the statistically appropriate sample to take?  A random sample within the life to date of the loan picking each record at a single random point in time, choosing all loans at a specified point in time (e.g. at exactly 6 months in from origination) or a sample of dates covering a period?

Or does it require something else - is a classification model actually appropriate for this type of scenario (assuming a binary outcome), or should some sort of time series or survival model be considered?

You should have a look at this Kaggle competition on fraud detection. In this kernel the winning team explains their approach.

This could be a time series approach (same as yours) but in this case:

  We are not predicting fraudulent transactions. According to the competition host Lynn here. Once a client (credit card) has fraud, their entire account is converted to isFraud=1. Therefore we are predicting fraudulent clients (credit cards).

For this team that won the challenge the solution was:

  We did a CV GroupKFold using month as the group.

It seems that perhaps you could use the same validation strategy.

https://www.kaggle.com/c/ieee-fraud-detection/discussion/111284


Convert Lasagne to Keras code (CNN -> LSTM) I would like to convert this Lasagne code:

et = {}
net['input'] = lasagne.layers.InputLayer((100, 1, 24, 113))
net['conv1/5x1'] = lasagne.layers.Conv2DLayer(net['input'], 64, (5, 1))
net['shuff'] = lasagne.layers.DimshuffleLayer(net['conv1/5x1'], (0, 2, 1, 3))
net['lstm1'] = lasagne.layers.LSTMLayer(net['shuff'], 128)

in Keras code. Currently I came up with this:

multi_input = Input(shape=(1, 24, 113), name='multi_input')
y = Conv2D(64, (5, 1), activation='relu', data_format='channels_first')(multi_input)
y = LSTM(128)(y)

But I get the error: Input 0 is incompatible with layer lstm_1: expected ndim=3, found ndim=4

I know the difficult part is to connect an LSTM to the output of Conv2D. Maybe using TimeDistributed?

You need to reshape the output of the Conv2d before you feed into the lstm. The output of Conv2d is 4d whereas the input for LSTM required is 3d.

Just print the output of Conv2d first, that will give you an idea on how to reshape the tensor as input to LSTM.


Looking for animals dataset for Deep Learning classification Do you know any datasets that contain animals and their accurate classifications? I am looking for any dataset that categorizes animals.
For example:

a dataset with insects, with an image of an insect and its scientific name as label
a dataset with dogs of different breeds, their scientific names and images.
a dataset with Myriapodes (like centipedes or millipedes), images and their names as labels
mammals
birds
reptiles
...

Generally, I am looking for datasets of all animals worldwide, classified as good as possible.
I would very much appreciate it!
Here a list of possible classes (superclasses, groups, etc.).

Unfortunately, a single dataset with all animals does not seem to exist (perhaps you can make one :D ), but there are plenty of datasets with a subset of animal species. Here are a few I can think of:

There are many datasets on Kaggle. Searching for "species", "animal", or some other smarter keyword should give some options. I found the 10 Monkey Species, STL-10, bird species classification, and Animals-10 datasets fairly quickly, for example.
Most large-scale datasets like OpenImages, CIFAR, ImageNet, the Visual Genome, and COCO have animals as some of the categories (among non-animal ones).
The iNaturalist dataset is a large scale species classification dataset (see the 2018 and 2019 competitions as well).
The Nature Conservancy Fisheries Monitoring dataset focuses on fish identification.
The Caltech-UCSD Birds-200-2011 is a standard dataset of birds.
The animals with attributes 2 dataset focuses on zero-shot learning (also here).
The Snapshot Serengeti dataset covers 40 mammals from the African Savannah.
The Oxford-IIIT Pet Dataset is good for fine-grained cat and dog classification.
A survey on image-based insect classification by Martineau et al has references to a number of image datasets for insect identification (though I am not sure how many are public).
The group at the Labeled Information Library of Alexandria: Biology and Conservation (LILA BC) track a large set of animal conservation and ecology datasets, including many involving image-based species recognition here.


Correlation as an evaluation metric for regression I'm dealing with a regression prediction challenge where the evaluation metric is (pearson) correlation.
However, I have the impression that this metric is kind of arbitrary.
While I can keep the RMSE stable the correlation can have great variety.

Could someone please explain this metric and how to optimise for it?

The organizers might deem the direction of predicted change more important than the magnitude, i.e., it is more important that your prediction is high when the known value is high (and vice versa) than to get as close to the known value as possible. The measurements might be noisy anyway.

One, fairly robust way of optimizing for it would be by grid-searching for the local optimum, like in this QA.

However, you should also take note that algorithms tweak ther internal parameters when fitting according to some loss function. Some algorithms accept custom cost functions and derivatives, yet some implementations don't. Information-theoretic measureas are standard in classification while MSE in regression.

Theoretically, you should be able to tell your Random Forest (or another algorithm accordingly) what the optimal split is as a function of pearson correlation.


Intuition behind using non-hypercubic kernels in density estimation Suppose that we perform density estimation in m-dimensional space: we estimate the value $p(a)$ for some point $a$ given observations $\{x_1, \dots, x_n \}$.

It is known that if region $A \subset \mathbb{R}^m$ is "small" enough to consider density being constant on points from $A$ then we can make the following estimate:
$$ p(a) \approx \frac{k / n}{|A|} $$
where $k$ is the number of observations that lie in $A$ and $|A|$ is Lebesgue measure of $A$.

Let parameter $h$ be small enough to consider density as constant inside hypercube centered at $a$ with side length equal to $h$. The volume of this hypercube is equal to $h^m$ and point $x$ lies inside this hypercube iff $K(\frac{x-a}{h}) = 1$ where
$$K(u) =\cases {
      1\text{, if $|\frac{u^k - a^k}{h}| \leq \frac{1}{2}, k = 1,\dots,m$}\cr
      0\text{, otherwise}
}$$
It's easy to see that the number of observations inside this hypercube equals to
$$k = \sum_{i = 1}^{n} K(\frac{x-a}{h})$$
and so the estimation described above gets the following form:
$$p(a) \approx \frac{1}{n h^m} \sum_{i = 1}^{n} K(\frac{x-a}{h}) $$

We can interpret $K$ as "weight" given to particular observations and one of the drawbacks of hypercubic approach is that all observations lying inside hypercube have equal weights despite having different distances from $a$. Yet another drawback is that the resulting estimate is not continuous. That's what i understand to be the main reason of using non-hypercubic kernels such as gaussian kernel which give more weight to points close to $a$ and yields continuous estimate.

But i have troubles with interpreting the usage of such kernels. The sum $\sum_{i = 1}^{n} K(\frac{x-a}{h})$ is no longer equal to $k$ so we can't justify the usage of these kernels by formula $p(a) \approx \frac{k / n}{|A|} $. Finally here are my questions: how do we justify the usage of smooth kernels? how can one interpret this usage?

Thank you for any ideas.

Histograms and methods based on binning have a number of well-known problems. Different anchor points etc. can introduce artificial patterns that make interpretation unreliable. Smooth kernels don't use a grid and thus smooth out the noise.

This also has the advantage that it makes it easier to get a single overall picture of the data because it takes into account neighboring points and smooths the data into areas where no data is observed.

Smooth kernels can also be justified by their favorable statistical properties. Popular methods like fastKDE use the fact that one can find "an empirical kernel that is optimal in the sense that the integrated, squared difference between the resulting KDE and the true PDF is minimized."


Ordinal classification with xgboost I am working in the problem where the dependent variables are ordered classes, such as bad, good, very good.

How could I declare this problem in xgboost instead of normal classification or regression?

Thanks

You can run 2 xgboost binary classifiers

1 classifier  classifies if sample is (good or very good)
2 classifier classfies if sample is very good
if both true on unseen data classify as very good
if only 1st one true, second false  classify as good
 both false=> classify as bad


Is the interval variable considered as a type of numerical variable or ordinal variable? I have a fundamental question about interval variable and I have searched about it in different tutorials but still not sure.
"An interval scale is one where there is order and the difference between two values is meaningful." from graphpad.com
"The interval variable is a measurement variable that is used to define values measured along a scale, with each point placed at an equal distance from one another. It is one of the 2 types of numerical variables and is an extension of the ordinal variable." from formpl.us
My question is about the last sentence above, which states that interval variable is numerical and an extension of ordinal. If we have an atribute 'age' as follows:

Then based on the given definitions above, "age" must be an interval attribute. However, doesn't it seem to be more ordinal than numerical?

This is the formal definition of an interval (from Wikipedia):

In mathematics, a (real) interval is a set of real numbers that
contains all real numbers lying between any two numbers of the set.
For example, the set of numbers x satisfying 0 ≤ x ≤ 1 is an interval
which contains 0, 1, and all numbers in between.

This definition doesn't tell us what to do to represent an interval for ML purposes... and that's normal :)
For ML purposes we need to decide a suitable representation, and there's rarely a mathematically pure or unique way to represent a mathematical object. Your confusion comes from the fact that the sites you mention propose their own representation, i.e. they consider a particular definition of interval, presumably because it suits their goal. The second one in particular is clearly defining an "interval variable" as an object in a particular language, with constraints which have nothing to do with the above definition.
Now to answer your question: an interval is not atomic, since it's not a single value. So strictly speaking it's neither a numerical nor an ordinal.
But you're probably not very satisfied with this answer, since you still want to be able to use this kind of feature. How? As usual, it depends:

Most of the time an interval can be converted to an ordinal value (note that it's a conversion, i.e. we simplify the data to make it usable). The conversion to ordinal is typically done by numbering the possible intervals (following their order) with integers. If the variable is going to be a label in a classification task, it can even be converted to a categorical variable.
However there are cases where an ordinal would not be a good representation, in particular if the intervals are not equal and the ML method involves distances. In such cases it's probably safer to convert the interval to a numerical value, typically the mean of the interval. This way both order and distances are preserved.


What value can I gain by doing exploratory data analysis on features (and thus data) before doing clustering? This might not be a very good question, but I would still ask if it's beneficial to do EDA before running a clustering algorithm?

I understand that EDA helps us generate good and helpful insights into the data, which is crucial in data understanding. If we leave aside standard checks and manipulations like - removing outliers, scaling, removing constant value columns, removing null/'zero' value columns, etc. and if we have 20-30 features.
How will EDA help me in producing good and sensible clusters? Is it even necessary to do the EDA before clustering?

Note: I am using k-means

How would you know you have to do cluster analysis before looking at your data ?

Setting aside data quality questions (which you should never do), a bare minimum of EDA will help you :

Know if it's relevant to do a clustering analysis (rarely imo)
Know if K-means is the best clustering tool (rarely imo)
Get an idea of the number of the clusters

Then you should do some EDA after, to understand what are the clusters you have selected.

Edit: Basically, it will help you answers this kind of question : How do I interpret my result of clustering?


Mean Absolute Error increasing with more correlated factors I am using Microsoft Azure Machine Learning Studio to predict stock market prices. We have the variables- Index price(target-to be predicted),Low price,High price,dates and days. We use split of 0.7 and run Linear regression. We get Mean absolute error of 109. We then try to add more variables(macroeconomic factors which positively effect the index prices) which are correlated with the target variable and should improve the predictions- we find that the Mean Absolute error increases to 110.I have attached the pics for your reference. Are we interpreting wrong or what's wrong in we are doing?
PS:We tried Boosted Tree regression as well-but the same problem as described above is observed.
Errors

Any reason why you are using MAE instead of MSE? The reason for using MSE is that you get a parabola which would be more accurate for the optimizing algorithm to find the local minima

Did you check the relation between predictors and target variable? did you check multi collinearity?


Formulate multivariate multistep time series forcasting using traditional machine learning, NOT deep learning How do you represent multivariate multistep data using traditional machine learning? I know this seems like a tailored problem for RNN/LSTM, but I am wondering what the alternative machine learning solution would be.

Ex.

time, feature_1, feature_2, feature_3
1     25         150        0.7
2     25.3       147        1.1
3     25.5       145        0.8
...
10    26.7       165        1.2

Now I want to predict feature_1 at time 10 using the first 3 time steps of all features, and get a dataset like this

feature_1,       feature_2,       feature_3,       target
[25,25.3,25.5]   [150,147,145]    [0.7,1.1,0.8]    26.7

You can always reduce the forecasting problem to a tabular regression problem and then apply any tabular regression algorithm you like. Here's a good explanation of how this works.


Linear optimization problem of $argmin$ Consider a vector $a \in R^n$.

I want to know how I can find analytically the solution of the following optimization problem: $x^* = argmin_{x \in R^n} f(x)$, where

$f(x) = ||x-a||_{2}^2 + \lambda ||x||_1$
$\lambda > 0$ and
$||.||_p$ is the p-norm in $R^n$.

Thanks in advance.

In response to the comments suggesting subgradient descent methods: the problem in the question does not contain any sum or similar terms, therefore the variables are independent of each other and there is a closed-form solution:
$$ x^* =
\begin{cases}
    a - \frac{\lambda}{2},& \text{if } a > \frac{\lambda}{2}\\
    a + \frac{\lambda}{2},& \text{if } -a > \frac{\lambda}{2}\\
    0,              & \text{otherwise}
\end{cases}
$$