Google Colab conda packages permanently installation I want to install some conda packages on Google Colab. Installation done successfully, but it wont detect it when we restart the new Colab session.

I followed the link "#https://stackoverflow.com/questions/55253498/how-do-i-install-a-library-permanently-in-colab", but still no luck.

Please let me know how to install python packages using conda + pip permanently on Google Colab.

There is no way to permanently install packages because the code is run on virtual machines that are sometimes recycled. You'll need to install it each time.
From the FAQ:

Where is my code executed? What happens to my execution state if I close the browser window?
Code is executed in a virtual machine dedicated to your account. Virtual machines are recycled when idle for a while, and have a maximum lifetime enforced by the system.


Why my Keras CNN model isn't learning My project have to decide if a image is 'pdr' or 'nonPdr', and I have 391 images (22 of PDR class, and the 369 of nonPdr)..

In my first model i was trying this: https://stackoverflow.com/questions/57663233/my-keras-cnn-return-the-same-output-value-how-can-i-fix-improve-my-code
.. and my return was always the same...

Now I made some changes in my model file:

TRAIN_DIR = 'train_data/'
#TEST_DIR = 'test_data/'

def ReadImages(Path):
    LabelList = list()
    ImageCV = list()

    # Get all subdirectories
    FolderList = os.listdir(Path)

    # Loop over each directory
    for File in FolderList:
        if(os.path.isdir(os.path.join(Path, File))):
            for Image in os.listdir(os.path.join(Path, File)):
                # Convert the path into a file
                ImageCV.append(cv2.imread(os.path.join(Path, File) + os.path.sep + Image))
                # Add a label for each image and remove the file extension
                classes = ["nonPdr", "pdr"]
                LabelList.append(classes.index(os.path.splitext(File)[0]))
        else:
            ImageCV.append(cv2.imread(os.path.join(Path, File) + os.path.sep + Image))
            # Add a label for each image and remove the file extension
            classes = ["nonPdr", "pdr"]
            LabelList.append(classes.index(os.path.splitext(File)[0]))

    return ImageCV, LabelList

model = Sequential()
model.add(Conv2D(64, kernel_size=(3,3), padding="same",activation="relu", input_shape=(605,700,3)))
model.add(MaxPooling2D((2, 2)))
model.add(Conv2D(128,  kernel_size=(4,4), padding="same",activation="relu"))
model.add(MaxPooling2D((2, 2)))
model.add(Flatten())
model.add(Dense(2, activation='sigmoid'))
model.compile(optimizer='RMSprop', loss='binary_crossentropy', metrics=['accuracy'])

data, labels = ReadImages(TRAIN_DIR)

data = np.array(data, dtype="float") / 255.0

le = LabelEncoder()
labels = le.fit_transform(labels)
labels = np_utils.to_categorical(labels, 2)

model.fit(data, labels, epochs=8, batch_size=20)

model.save('model.h5')

... but running this code give me a Loss = 8.0 and a acc = 0.50

What can I do? I appreciate any answer..

UPDATE I forgot that I reduce my train imgs to 20/20

It seems that you have an output size of 2 in your final layer, while you should rather have size 1 (because of your sigmoid output and binary cross entropy loss).

Also, don’t use the to_categorical transformation as you only have two classes so no need to one-hot encode.

Try to change this and see if training improves.


Need a little help Understanding how to build model's in Keras I am trying to make a CNN in Keras, and to test the validity of my model i am trying to get it to train on MNIST dataset, so i am sure that everything is working fine, but unfortunately model is barely training and i suspect that nothing updating.
My model is :

model=Sequential()

#conv1_1
model.add(Conv2D(128,kernel_size=3, strides=1,
                 padding='SAME', use_bias=False,
                 activation='relu',name='conv1_1',input_shape=(28,28,1)))
#conv1_2
model.add(Conv2D(128, kernel_size=3, strides=1,
                 padding='SAME', use_bias=False,
                 activation='relu',name='conv1_2'))
model.add(MaxPooling2D(pool_size=2,strides=2))

#conv2_1
model.add(Conv2D(64, kernel_size=3, strides=1,
                 padding='SAME', use_bias=False,
                 activation='relu',name="conv2_1"))

#conv2_2
model.add(Conv2D(64, kernel_size=3, strides=1,
                 padding='SAME', use_bias=False,
                 activation='relu',name='conv2_2'))
model.add(MaxPooling2D(pool_size=2,strides=2))

model.add(Flatten())
model.add(Dense(1024, activation='relu',name='Dense1'))
model.add(Dropout(0.2))
model.add(Dense(512, activation='relu',name='Dense2'))
model.add(Dense(10, activation='softmax',name='output'))

Compiled with:

model.compile(loss='categorical_crossentropy', optimizer='adadelta', metrics=['accuracy'])

model.fit(X_train,y_train,batch_size=10,validation_split=0.2,epochs=10)

My X_train and y_train look like:

plt.imshow(X_train[0].reshape(28,28))
plt.show()

y_train[0]
array([0., 0., 0., 0., 0., 1., 0., 0., 0., 0.])

Here are the Results of first 3 epochs:

Epoch 1/10
48000/48000 [==============================] - 45s 927us/step - loss: 14.2813 - acc: 0.1140 - val_loss: 14.4096 - val_acc: 0.1060
Epoch 2/10
48000/48000 [==============================] - 44s 915us/step - loss: 14.2813 - acc: 0.1140 - val_loss: 14.4096 - val_acc: 0.1060
Epoch 3/10
48000/48000 [==============================] - 44s 924us/step - loss: 14.2813 - acc: 0.1140 - val_loss: 14.4096 - val_acc: 0.1060
Epoch 4/10
48000/48000 [==============================] - 45s 930us/step - loss: 14.2813 - acc: 0.1140 - val_loss: 14.4096 - val_acc: 0.1060

This is my first Keras Model, and i think i am missing something important here.

I have implemented your model to the astonishment, there is a very minute error that is hard to notice.

The way, I was able to get better accuracy is by changing the optimizer to "SGD" or "ADAM".

As you have used "ADADELTA" which is an extension of "ADAGRAD" optimizer. In "ADAGRAD" has good performs on sparse data & while training a large scale neural network. Its monotonic learning rate usually proves too aggressive, stops learning too early.

Refer to this link for understanding on optimizers


Training Machine Learning Model - Neural Network - Islands Problem I was working on the following leetcode problem:

  Given a 2d grid map of '1's (land) and '0's (water), count the number
  of islands. An island is surrounded by water and is formed by
  connecting adjacent lands horizontally or vertically. You may assume
  all four edges of the grid are all surrounded by water.

  Example 1:

  Input: 11110 11010 11000 00000

  Output: 1

  Example 2:

  Input: 11000 11000 00100 00011

  Output: 3

And wrote up the following explicit solution:

from collections import OrderedDict

class Solution:
    def numIslands(self, grid: List[List[str]]) -> int:
        if len(grid) == 0:
            return 0

        land = ones(grid)
        parcels = len(land)
        islands = 0

        while parcels > 0:

            unexplored = []
            start = land.popitem()[0]
            parcels = parcels - 1

            print(start)

            up = (start[0] + 1, start[1])
            down = (start[0] - 1, start[1])
            left = (start[0], start[1] - 1)
            right = (start[0], start[1] + 1)

            if up in land:
                unexplored.append(up)
                del land[up]
            if down in land:
                unexplored.append(down)
                del land[down]
            if left in land:
                unexplored.append(left)
                del land[left]
            if right in land:
                unexplored.append(right)
                del land[right]

            while len(unexplored) > 0:

                start = unexplored.pop()
                parcels = parcels - 1

                up = (start[0] + 1, start[1])
                down = (start[0] - 1, start[1])
                left = (start[0], start[1] - 1)
                right = (start[0], start[1] + 1)

                if up in land:
                    unexplored.append(up)
                    del land[up]
                if down in land:
                    unexplored.append(down)
                    del land[down]
                if left in land:
                    unexplored.append(left)
                    del land[left]
                if right in land:
                    unexplored.append(right)
                    del land[right]

            islands += 1

        return islands

def ones(grid):
    ones = OrderedDict()
    height = len(grid)
    width = len(grid[0])

    for i in range(0,height):
        for j in range(0,width):
            if grid[i][j] == "1":
                ones[(i,j)] = 1

    return ones

But, my intuition said that finding the number of islands in a grid of ones and zeroes should be a good machine learning problem, associated with image detection.

So, I wrote the following code to generate test cases:

from random import randint

def make_grid():

    width = 30
    height = 30
    grid = []

    for i in range(0,height):
        row = []
        for j in range(0,width):
            if randint(0,1) == 0:
                row.append("0")
            else:
                row.append("1")
        grid.append(row)

    return grid

And, I then generated 60,000 cases and scored them using my explicit solution. I divided them into 50,000 train cases and 10,000 test cases.

I then tried to feed the train cases into a Keras network that was 64 nodes per layer, 10 layers deep using relu and a softmax activation function at the end. I trained the model for 250 iterations.

I got low accuracy statistics on recognizing the number of islands in the "images" on both the train and test cases.

I then tried to feed them into a Keras network using two convolutional layers, the first one 2x2 and the second one 4x4 with 64 nodes each followed by 2 dense layers with 64 nodes per layer. I trained the model for 3 iterations.

I got low accuracy statistics at predicting the number of islands in an "image" on both the train and test cases.

Is this just a bad problem for neural networks? Am I using the wrong kind of networks? Are my networks too small? Do I need to do a larger number of test cases?

Any input appreciated!

For a start I would begin with simpler tests cases because they look rather complex (see an exemple of what you seems to simulate below). May be start with a 10x10 grid and generate 1 to 5 islands of different size.


How to get K most different rows in csv? We have boring CSV with 10000 rows of ages (float), titles (enum/int), scores (float). How to select 1000 most different rows? I look for a general solution that would work for more than one case.
What do I mean by different:

We have N columns each with int/float values in a table.
You can imagine this as points in ND space
We want to pick K points that would have maximised distance between each other.

So if we have 100 points in a tightly packed cluster and one point in the distance we would get something like this for three points:
 or this

It looks like an ND point cloud "triangulation" with a given resolution yet not for 3d points... So how to select K most distant rows (points) from N (with any complexity)?

To select the most different rows, you would need to define first what you consider different. For ages and scores, subtracting values would work, for example:
Row1

Age is 38
Score is 0.2

Row2

Age is 87
Score is 1.0

Difference by numeric feature:

Age Diff is 49
Score Diff is 0.8

Those values could be normalized or weighted to account for different importances between features.
For titles, you’d probably need to use a text similarity metric, such as tf-idf with cosine similarity or embedding vector distance. The most different ones are the ones with higher score/age difference and lower title similarity.


Classification with millions of records, thousands of categories - keep memory use efficient? I have about 20 categorical variables to predict another categorical variable. One of the variables can have as many as 12000 levels (O/P will be one of the 12000) and another one can have about 8000 levels. Rest will have less than 100 levels. Random Forest was the first thing that came to my mind, but Python’s Random Forest implementation doesn’t support categorical variables. If I one-hot encoded all these variables, then I will end up with thousands of variables for Millions of records which will throw me into memory issues.

What are my options?

I assume you are using the sklearn implementation of random forests. If you are not bound to use sklearn, you could try another implementation, like the one from h2o, that supports enum categorical variables.

Alternatively, you could first determine if all of the 12000 levels are relevant: for each of the 12000 levels, you could compute something like information gain, which will tell you if that level provides any information relevant to predicting the class. You may then use a threshold on the information gain to discard all levels that provide no information, reducing the number of levels and, perhaps, allowing you to use one-hot encoding.

Basically, if $H$ is the entropy (of the class label) for a set of examples, given a dataset $D$, you could discard any level $v$ of an attribute $a$ for which the following is below a certain threshold (to be determined given how many levels you want to keep):

$$
H(D) - \frac{|D_v|}{|D|} \cdot H(D_v)
$$

where $D_v = \{x \in D|value(x, a) = v\}$


What algorithm can predict structured outputs of arbitrary size? I have a collection of graph objects of variable size (input) which are each paired to another graph of variable size (output). The task is, given an input graph, produce the most likely output graph.

I have been looking at 'structured output' techniques such as SSVM but as far as I can tell, they all act on outputs of fixed size (or at least a size that matches the input size, i.e. sequence tagging). Are there any tools that can map an input to a structured object of arbitrary size?

You better try graph embedding and for that I propose going through Horst Bunke from University of Bern who has been doing it for years. Just search him in google scholar and go through his publication network (co-authors,cited papers, citing papers,etc) for example this and this as more classic papers or this as an exact answer to your question. I am pretty confident you will find your solution as it's exactly his research direction. I am on mobile so referencing is difficult now but I can update my answer later.

The other point that I would like to mention is that graph embedding has two kinds:

Embedding the vertices (nodes) of one graph into a n-dimensional space i.e. each vector will be a node of the graph. (see this)
Embedding a dataset of graphs into a n-dimensional space in which each vector will be an entire graph.

You need the second one.

The other thing worth to mention is looking at the problem from Network Science point of view in which the statistical and structural of a graph is widely studied. There are plenty of graph/network measures which are less or more tolerante to the number of nodes e.g. Clustering Coefficient. Using them as features can be fed to a classifier to see how it works.

The last but not the least is what you actually asked! What you are looking for is called Graph Kernel. It is nothing but the definition of the ML-known concept Kernel but on graphs. So kernel methods can get benefit of that by skipping the feature extraction step (like what I suggested above) and directly work with structural inputs. For this purpose I would suggest that you equip yourself with the graph kernels e.g. this paper.

Good luck!


best python library for finding sequential rules mining? I want to do sequence learning for that I want to find frequent sequential rules ,this rule consider the order of occurrence

Probably you could try Conditional Random Fields (https://github.com/scrapinghub/python-crfsuite) or a Long Short term Memory RNN (Keras+Tensorflow backend)


Probability Calibration : role of hidden layer in Neural Network I try a simple Neural Network (Logistic Regression) to play with Keras.
In input I have 5,000 features (output of a simple tf-idf vectorizer) and in the output layer I just use a random uniform initialization and an $\alpha = 0.0001$ for $L_{2}$ regularization with a sigmoid activation function (so something pretty standard).

I use an Adam optimizer and the loss function is the binary cross-entropy.

When I display the probability for both classes I end up with something like this :

After that I try to add a hidden layer with a $Relu$ activation function, 64 nodes and again the same parameters for regularization & initialization.
-- EDIT --
Here for the output layer I keep exactly the same parameter (and $sigmoid$ activation function) as in the previous NN.
-- END OF EDIT --

But now when I plot the probability distribution for both classes I end up like this :

And cleary I didn't get why by adding a hidden layer the probability are push to 0 or 1 ?

Have you any references that I can read so I can understand the math behind ? It could be great!

Moreover sometimes (with a more "deeper" neural network for another application) I get the same plot as the second one but this time the predicted probabilities are between $[0.2; 0.8]$. The probabilities are push to some "values" but those values are more "centered". Not sure it is clear. And here again I don't understand what is the phenomena behind this ? How can I "debug" it to see in my architecture of my Neural Network what is the cause ?

In addition How can I "tweak" my ANN to have a "perfect" calibration plot (like in the scikit webpage : https://scikit-learn.org/stable/modules/calibration.html) ?

Thank you in advance for every answer that can enligthen me :)

It makes sense that a network with ReLU activations produces worse probabilities than a sigmoid activation when you think about how these activation functions interact with the unit in the output layer of the network. Sigmoid scales the output to be in the range $[0, 1]$, so the values going into the final sigmoid classification unit will be fairly low. Meanwhile, the ReLU-based network has the potential for very high activation magnitudes, which will skew the estimated probabilities to either end of the spectrum.

Having said that, calibration of neural networks is not very well understood in general. A paper was published last year at ICML (https://arxiv.org/pdf/1706.04599.pdf) which showed that often neural nets are poorly calibrated, but it doesn't really explain why this is the case. However, they did investigate factors that affect calibration, and show that deeper, wider networks can often be less well calibrated than their shallower, narrower counterparts. They also show that applying $L_2$ regularisation can improve calibration.

Given this information, I'm surprised that your (fairly shallow and narrow) network is so poorly calibrated. It might be overfitted to NLL, so I would try applying heavier regularisation or training for fewer epochs. If this doesn't work, try implementing temperature scaling, which is described in the paper I linked above. They provide a PyTorch implementation here, so it shouldn't be too difficult hopefully to translate into Keras.


How to create a big data frame in Python I have a sparse matrix, $X$, created by TfidfVectorizer and its size is $(500000, 200000)$. I want to convert $X$ to a data frame but I'm always getting a memory error.
I tried
pd.DataFrame(X.toarray(), columns=tokens)
and
pd.read_csv(X.toarray().astype("float32"), columns=tokens, chunksize=...).
And it seems that when I convert $X$ to a numpy array using  X.toarray(), I get an error.
Can someone tell me what is an easy solution for this? Is there anyway I can create a sparse dataframe from $X$ without memory error?
I have been running my codes on Google Colab Pro and I think it provides me less than 100 GB Ram.

You can use pandas.Dataframe.sparse.from_spmatrix.  It will create a Dataframe populated by pd.arrays.SparseArray from a scipy sparse matrix.
Pandas used to have explicit sparse dataframes, but in more modern versions there is no such concept.  Only normal pd.Dataframe populated by sparse data.


Difference between MultiDiGraph and DiGraph This post was originally posted to stackoverflow (https://stackoverflow.com/questions/57315017/best-way-to-include-weights-in-a-directed-acyclic-graph) sorry for long link, goo.gl was unable to shorten it.

Someone recommended I post it in this forum as it seemed more suitable.

I am working on my thesis, which consists of writing a Tabu Search algorithm to solve a Flexible Job Shop Problem. To solve this problem, I am using graphs as a problem/solution representation (the problem is an Undirected Graph, which must be turned into a Directed Acyclic Graph).

In short, I have a start (0) and end (*) node, which are 'dummy' operations whose weight/duration is 0. Nodes represent operations and paths represent the machines on which operations are processed, so the length of an edge (u,v) is the setup time incurred when swapping between operations u and v on a given machine. The weight of a node should be the processing time of that operation on the machine it's assigned. In a Directed Acyclic Graph, the length of the longest path from 0 to * represents the makespan (maximum production time) of the solution.

To implement this, I am using the networkx library. It's been going well, but I have come across some issues, namely when computing the makespan.
First of all, since the functions I found (dag_longest_path() and dag_longest_path_length()) don't account for node weights, I transfered node weights to the edges preceding them. That seemed to solve the problem.
However, I just noticed that these functions have different outputs in case the graph is either a DiGraph or a MultiDiGraph, as shown below.

Before presenting the code, forgive my noobness here, first time poster. The instance I am using consists of 30 operations and 8 machines, but for simplicity purposes I am presenting a simple example with 5 operations and 3 machines.

import networkx as nx

operations = (1,2,3,4,5)
G=nx.DiGraph()
G.add_node(0,pos=(0,0)) #starting node
G.add_node('*',pos=(1,0)) #end node

G.add_nodes_from(operations)

G.add_edge(0,1, weight = 500)
G.add_edge(1,2, weight = 200)
#these edges are for one machine

G.add_edge(0,3, weight = 150)
G.add_edge(3,4, weight = 177)

#these are for a second machine

G.add_edge(0,5, weight = 999)
#this edge is for the last machine

G.add_edge(2, '*', weight = 0)
G.add_edge(4, '*', weight = 0)
G.add_edge(5, '*', weight = 0)
#edges connecting final operation of each machine to the end node

print(nx.dag_longest_path(G))
print(nx.dag_longest_path_length(G))
#printing the longest path and corresponding length

The previous code outputs the following:

[0, 5]
999

This is sort of expected, as the path going through node 5 is indeed the longest path. However, it does not show the end node '*'.

If instead of G=nx.DiGraph() I do G=nx.MultiDiGraph(), the output is the following:

[0, 1, 2, '*']
3

Now the '*' node is included, rightfully so, but the path returned is the one with most nodes instead of the one with the longest edges.

What am I missing here? MultiDiGraph() seems to be the correct graph type here, as nodes 0 and '*' both have multiple edges leaving/entering them.

I need a way for the output to be:

[0, 5, '*']
999

Thank you in advance!

PS: feel free to correct any errors in formatting and presentation as this is my first post

The first way (with DiGraphs) is correct, but the two paths [0, 5] and [0, 5, '*'] both have the same (weighted) length, and the method just returns one of them.  Which one depends on how your version of python orders dictionaries, but I think since vertex 5 appears first in the topological ordering it's rather likely to appear first.

You could hack it by making the edges into '*' have weight 1, then subtract that from the longest path length.

dag_longest_path doesn't appear to be built to work for MultiDiGraphs.  For a vertex's predecessors' .items(), a DiGraph has just the dictionary of attributes, but the MultiDigraph has a dictionary of edge-attributes pairs.  dag_longest_path doesn't try to take that into account, so never sees any weight attribute to use.


How to interpret the sample_weight parameter in MiniBatchKMeans? I am using scikit-learn MiniBatchKMeans to do text clustering.
In the fit() function there is a parameter sample_weight described as follows:

The weights for each observation in X. If None, all observations are
assigned equal weight (default: None).

How should I interpret it exactly?
For instance, if I assigned greater weight to certain observations, will the centroid be nearer to that observations?

Yes, the parameter is available in the vanilla K-Means too.

The algorithm supports sample weights, which can be given by a parameter sample_weight. This allows assigning more weight to some samples when computing cluster centers and values of inertia. For example, assigning a weight of 2 to a sample is equivalent to adding a duplicate of that sample to the dataset X.

scikit-learn.org


How to handle potential interactions when one-hot encoding? Let's say I have two categorical features: Movie, Director. I one-hot encode both the Movie and Director features for use in a linear regression model.

The problem is that two or more movies may be directed by the same director. i.e. a particular director's bit may be on for two or more different movies. Would this be a problem? Should I be combining movie and director into one feature?

You could combine these features before using one-hot encoding, and see if the performance is improved. But keep in mind, that it really depends on the problem each time.

Generally, is a good thought to combine these type of features. CatBoost, a very good gradient boosting library, create such combinations and the results are pretty good most of the time. I would give it a go if I were you.


Visualize Softmax values in CNN prediction What is the most convenient way to visualize Softmax values after calling the CNN prediction function? Do I have to collect different probability values and feed them to the matplotlib or are there any more convenient ways/libraries to do this?

Below is one example what I mean:

Fastest and easiest would be seaborn. It's built on top of matplotlib and you can plot your stuff with little code. Once you become to hate matplotlib's paradigm or need something fancier, there's plotly.


How to find similar phrases I have the following problem:
I have created a customized Dictionary for getting used in some NLP tasks. I want to enhance my dictionary by finding phrases similar to the phrases I have in my dictionary.
For example:

Lets say I have a phrase:
- take responsibility
then, I should be able to find phrases like:
- hold accountable for
-responsible for

I have just started learning NLP and I tried Word2Vec model over my dictionary but it only gives similar words that are already present in the dictionary.
Is there any other algorithm that helps me find similar phrases using multiple sources.

One option is Word Mover’s Distance (WMD). WMD is an algorithm for finding the distance between phrases. WMD is based on word embeddings (e.g., word2vec) which encode the semantic meaning of words into dense vectors.

  The WMD distance measures the dissimilarity between two text documents as the minimum amount of distance that the embedded words of one document need to "travel" to reach the embedded words of another document.

For example:

Source: "From Word Embeddings To Document Distances" Paper

For your problem, you would have to generate candidates then use WMD to find the best matches for a given phrase.


Dimension Reduction - After or Before Train-Test Split Should one apply dimensionality reduction methods to the data set before or after train-test splitting?
Anyway, in case of training a model with preprocessing by dim-red, one should apply the same dim-red to the future instances to be predicted, right? But how could you reduce the dimensionality of one single instance in the same way?

It depends on the type of algortithm you use for dimensionality reductions. In case you use PCA, you should build your PCA on your train set. Then you need to set your principal components to transform your points in test set into the same space. This way you can then use train and test set in the same reducted space.


Python to clean miswritten words with repetitive letters such as "wwwwooorrrrddss" to "words" When cleaning text-data in Python3 for NLP, are there are any 'common practices' when it comes to addressing repetitive letters in words such as "wwwwooorrds" to "words", or "fffunnnyyyyyy" to "funny"?
The source of the miswritten words is an OCR and I am not able to address the issue upstream, and thought I would check if there was anything that I can do downstream to fix this.
Thanks!

A simple two part solution from this site
remove any letter sequences longer than two (probably not good for welsh)
def reduce_lengthening(text):
    pattern = re.compile(r"(.)\1{2,}")
    return pattern.sub(r"\1\1", text)

print(reduce_lengthening( "finallllllly" ))

Then using pattern.en to check spelling.
from pattern.en import spelling

word = "amazzziiing"
word_wlf = reduce_lengthening(word) #calling function defined above
print word_wlf #word lengthening isn't being able to fix it completely

correct_word = spelling(word_wlf)
print(correct_word)

NLTK is another common toolkit that can help with this


Can accuracy become worse on the training set with more epochs? I know that overfitting occurs when the accuracy on the training set improves but the accuracy on the validation set decrease. So, we must stop the training. I would like to know if this is a rule that more epochs always decrease the accuracy on the training set. In other words, is the accuracy function over epochs a non-ascending function? If not, then why?

Yes, accuracy measured on the training set over epochs is not monotonically decreasing.

Possible reasons:

Software bug
Nan due to division by zero / log (0) / overflow
Too big learning rate


Why isn't my model learning? import numpy as np
from keras.datasets import cifar10
from keras.layers import Dense, Activation
from keras.optimizers import SGD
from keras.utils import np_utils
from keras import backend as k
from keras.models import Sequential

(Xtr,Ytr),(Xte,Yte)=cifar10.load_data()

Xtr = Xtr.astype('float32')
Xte = Xte.astype('float32')

Xtr = Xtr.reshape(50000, 3072)
Xte = Xte.reshape(10000, 3072)

Ytr = np_utils.to_categorical(Ytr, 10)
Yte = np_utils.to_categorical(Yte, 10)

model=Sequential()
model.add(Dense(100, input_shape=Xtr.shape[1:]))
model.add(Activation('relu'))
model.add(Dense(10))
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy', optimizer="sgd", metrics=['accuracy'])

model.fit(Xtr, Ytr, batch_size=200, epochs=30, shuffle=True,verbose=1)

scores = model.evaluate(Xte, Yte, verbose=0)
print("Accuracy: %.2f%%" % (scores[1] * 100))

And the result:

I am trying to creat a pretty basic 2 layer NN on cifar-10. I know that the data is not preprocessed. But that can't be the reason for learning nothing. Where am I making the mistake?

I'll go through an example that will help you get started. It should get approximately 50% accuracy.

So I keep the code the same as yours for loading the data. The only difference is that I normalize the data to lie between 0 and 1. This is usually recommended to bound the weights more tightly.

import numpy as np
from keras.datasets import cifar10
from keras.layers import Dense, Activation
from keras.optimizers import SGD
from keras.utils import np_utils
from keras import backend as k
from keras.models import Sequential
import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D
from keras.callbacks import ModelCheckpoint
from keras.models import model_from_json
from keras import backend as K

(x_train, y_train), (x_test, y_test)=cifar10.load_data()

  Downloading data from https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz 170500096/170498071 [==============================] - 71s 0us/step

(x_train, y_train), (x_test, y_test)=cifar10.load_data()

x_train = x_train.astype('float32') / 255.
x_test = x_test.astype('float32') / 255.

x_train = x_train.reshape(50000, 3072)
x_test = x_test.reshape(10000, 3072)

# The known number of output classes.
num_classes = 10

# Channels go last for TensorFlow backend
x_train_reshaped = x_train.reshape(x_train.shape[0], x_train.shape[1],)
x_test_reshaped = x_test.reshape(x_test.shape[0], x_test.shape[1],)
input_shape = (x_train.shape[1],)

# Convert class vectors to binary class matrices. This uses 1 hot encoding.
y_train_binary = keras.utils.to_categorical(y_train, num_classes)
y_test_binary = keras.utils.to_categorical(y_test, num_classes)

Now let's make our model. Here I do make the hidden layers have more neurons per layer.

model = Sequential()
model.add(Dense(32,
                 activation='relu',
                 input_shape=input_shape))
model.add(Dense(128, activation='relu'))
model.add(Dense(num_classes, activation='softmax'))
model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])

Now we can train the model

epochs = 4
batch_size = 128
# Fit the model weights.
model.fit(x_train_reshaped, y_train_binary,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          validation_data=(x_test_reshaped, y_test_binary))

  50000/50000 [==============================] - 3s 58us/step - loss: 1.6716 - acc: 0.4038 - val_loss: 1.6566 - val_acc: 0.4094


Overfitting Question Would you consider that overfitting?

No it this isn't overfitting.

First of all the AUC is exactly the same between train and validation sets. The losses might have a gap but since the validation loss is still dropping (even if slowly) you are OK.


Accuracy value constant even after different runs I am using the neural network toolbox of Matlab to train a network. Now my code is as follows:

x = xdata.';
t = target1';
% Create a Pattern Recognition Network
hiddenLayerSize = 10;
net = patternnet(hiddenLayerSize);
net.input.processFcns = {'removeconstantrows','mapminmax'};
net.output.processFcns = {'removeconstantrows','mapminmax'};
net.layers{2}.transferFcn = 'softmax';
net.divideFcn = 'dividerand';  % Divide data randomly
net.divideMode = 'sample';  % Divide up every sample
net.divideParam.trainRatio = 60/100;
net.divideParam.valRatio = 20/100;
net.divideParam.testRatio = 20/100;
net.trainFcn = 'trainscg';  % Scaled conjugate gradient
net.performFcn = 'mse';
net.performParam.regularization = 0.5;
%net.performParam.normalization = 0.01;
net.plotFcns = {'plotperform','plottrainstate','ploterrhist', ...
  'plotregression', 'plotfit', 'plotconfusion'};

% Train the Network
[net,tr] = train(net,x,t);
% Test the Network
y = net(x);
e = gsubtract(t,y);
tind = vec2ind(t);
yind = vec2ind(y);
percentErrors = sum(tind ~= yind)/numel(tind);
performance = perform(net,t,y)
% Recalculate Training, Validation and Test Performance
trainTargets = t .* tr.trainMask{1};
valTargets = t  .* tr.valMask{1};

Now I am supposed to get different accuracies with different run of data since the sampling (division of dataset into train test and validation set) is random. But I am getting a constant accuracy (89.7%). The variable 'xdata' contains only those features selected by a feature selection algorithms. Is there any reason why my accuracy value is constant?
I have trained an SVM too with the same dataset. There too I am getting the a stale accuracy even with multiple run (94%)
The output y contains 2 values. What do those values signify?

I see that you have set a Random Number Generator(rng) seed in the following line in your code: rng(1).

So, this splits the data in the same way no matter whichever run. So, that is the reason why you are getting the same error values.

Try removing the line. Then, the data shall be slit randomly (as there is no seed now). You shall get different error then, depending on how the split is done (which is, randomly).

A model which generalizes well should be robust to the choice of seed


Predicting most likely application to be opened Background

I'm currently preparing for a paper in which I will discuss the ability to predict the most likely application to be opened by the user at the given time.

The application will collect information and learn to predict which application that is most likely to be run based on the learned patterns.

Proposed method

I'm collecting the following features for any given program run on the machine:

Time of day (To predict if the program is being run)
Day of week (To learn when it is used)
Most recently opened program (To see connections between programs)
CPU load (Read below)
GPU load (Read below)
Memory usage (To connect programs to heavy usage)
Screen use (To see if the program has connections with an active screen)
Last mouse movement (To see if the program is actively used when it is
run)
Last keyboard usage (Same as above)
App running (Whether or not the application is running)

The mentioned features should allow for some seasonality to be present and should therefore yield some interesting results.

When the data has been collected, An ANN will be trained with the normalized inputs and each application as a separate output representing the possibility of the program being run. (0-1)

Questions

Is the proposed method adequate for such a task? I've read about using Bayesian classifiers but it doesn't seem like it's what I'm after.

I've also read about this being a "time series" task - something I couldn't quite wrap my head around but I assume I am exploiting through the proposed method. Is this a time series? Does it have to be treated as such?

Should each application be present as a separate output or should the current application be fed in as an input, with the output presenting the possibility of the given app being run?

How do you plan to predict the exact app?  Are you thinking of a user level model (you can't predict that a user will open app X if she doesn't have it installed in the first place) Or maybe you want to predict apps at a segment level( instead of predicting individual apps)
A neural network should work well if you have enough data since it has good "capacity"
The time series part comes from the fact that the probability of opening app depends on what app was opened before it. This is called "structured prediction" and is quite common in language models etc.
The output will be a distribution over the apps and you can use a soft max layer for the final output


Scalable way to train graph embeddings I have a use-case to train graph embeddings, looking for a way to do it in PyTorch and TensorFlow. The restriction is the methodology should incorporate edge weights in calculating transition probability. Can someone please advise the best framework/library to do? The node2vec implementation by the authors is in spark only, however, my current ML pipeline is more PyTorch/TF friendly.

Don't know if this is what you need but I know of the Ampligraph library:

Python library for Representation Learning on Knowledge Graphs


To remove Chinese characters as features - I have created document-term matrix using TfIdfVectorizer, but just noticed the feature contains Chinese characters. Is it possible to remove them using Python's regex?

I believe these characters are one of reason for lower prediction accuracy of my model.

Currently I use the below for pre-processing my data-

   # Pre-processing the data
    def text_preprocess( data ):
        # Changing to lower case
        data = data.lower()
        # Removing special characters
        data = re.sub("(\\d|\\W)+"," ",data)
        return data

Also, please note I used stopwords='english' in my TfidfVectorizer.

Please let me know if any information required. (New here, still learning)

If you want to remove non-English characters then this regex will work, by selecting characters not in a given ASCII range (0 to 122, you can adjust this since it will allow some special characters):

([^\x00-\x7A])+

So to remove those characters:

data = re.sub("([^\x00-\x7F])+"," ",data)


Sentiment analysis for multiple entry in one text I would like to do sentiment analysis on a set of financial news from the S&P 500 for given entities (organization names). However, each news (rows in my dataset) may have more than one entity and I have to do analysis for each entity separately the same news. First, my algorithm should find which entities are present in a specific news and then do sentiment analysis for them.

I would appreciate if someone could help me.

Example of output:

A Ensemble of "Target based sentiment" models worked for a similar use-case that I worked on.

Some of the models that were part of the solution :

http://sentic.net/sentic-lstm.pdf
https://hal.archives-ouvertes.fr/hal-01537896/document
https://arxiv.org/pdf/1802.05818.pdf


Is it domain adaptation? Let's say I trained a classifier to assign a shop department to a product, e.g.: ALGIDA Cow milk->Diary. It did it on a domain of official product names.
When I applied pre-trained classifier to names of products in a shopping list, I found it doesn't perform that well as in the source domain because people tend to make spelling errors, abbreviate product names, do not include brand names, etc. in their shopping lists.
Simply speaking, the classifier does not generalize well to another domain.
I tried a few techniques where each alone improved classifier accuracy in a target domain:

Pre-processed product names in a shopping list by correcting spelling errors and expanding abbreviations
Pre-processed product names in a source domain before training by removing brand names.
Stemmed product names before constructing TF/IDF vectors
Embedded product names using pretrained USE (Universal Sentence Encoder) model.

Each of those techniques makes phrases in one domain more similar to phrases in another domain, or features build on top of those phrases more similar across domains.
Would you call any of those techniques domain adaptation?

Not sure if it is domain adaptation as it has been described in literature, because the performance of your classifier in the target domain depends more in the alignment between the two domains due to the mistaken spelling etc. instead of measuring the divergence between the distribution of the two domains. If the distribution of the data between source and target is different then after the alignment you should use any appropriate method in order to tackle this difference and use the available information from the source domain.
P.S. Have you used the "levinstein distance" in order to align the two domains? (Match together the most similar ones etc...)


Is it bad practice to use multi-class over multi-label classification? I have a multi-label classification problem-- millions of records that potentially could hold more than one label. I'm running into issues related to lack of research/examples online, and am unable to create a strong model. Would it be advised against to build a multi-class classification model, and take any label with a predicted probability over 0 to be considered a potential label? Even with problem transformations, like binary relevance and classifier chains, the accuracy is terribly low (~12%) as opposed to using Naive Bayes and getting ~57% accuracy.

Perhaps I understood the question wrong, but multi-label and multi-class problems are fundamentally different, as the multi-class is exclusive (no instance can be classified into two classes a the same time), while the multi-label relaxes this condition. As Wikipedia on multi-label states:

  in the multi-label problem there is no constraint on how many of the classes the instance can be assigned to.

So, using multi-class or multi-label depends on the problem itself and not on how you solve it.


To continue training an existing tensorflow model do I need to restore the session? I'm trying to work out the best way to integrate with EC2 spot instances that can be started and stopped.

Do I need to store the tf.session or can I just do load_model('myfile.h5') and continue with mymodel.fit(....checkpoint=...) which saves the model every epoch back to myfile.h5?

I managed to get it workng with just load_model and then continue with model.fit(...). My model is quite simple, using just Keras Input, LSTM and Dense layers with no variables or anything complex.


Changing reference class in imbalanced data drastically affects the error rate Working on a binary classification problem that tries to predict customer churn, the data set is imbalanced with 2000 observations of non-churn cases vs 600 observations of churn cases.

On using GLM I see that when the majority class[Non-churn] is the reference level I get ~40 % error rate[confusion matrix] on both the levels [churn non-churn] but when the minority class is set as the reference level I get 100% error rate in predicting the minority class or in a way everything gets predicted as non-churn case.

After balancing the data using SMOTE the same trend continues, how should I interpret this behaviour. ?

Is it in a way saying that the non-churn population has users who have similar behaviour as the churners and hence the high error rate, but at the same time the non-churn users have a subset which are quite different than the churners in their behaviour and hence lower error rate when the reference class is the majority or the non-churn class.

Outcome on test data when majority class is set as the reference class:
Confusion Matrix (vertical: actual; across: predicted) for F1-optimal threshold:
             0   1    Error      Rate
    0      268 419 0.609898  =419/687
    1       46 168 0.214953   =46/214
    Totals 314 587 0.516093  =465/901

Outcome on test data minority class is set as the reference class:
Confusion Matrix (vertical: actual; across: predicted) for F1-optimal threshold:
                   1   0    Error      Rate
            1      3 211 0.985981  =211/214
            0      1 686 0.001456    =1/687
            Totals 4 897 0.235294  =212/901

Decision function of GLM itself does not depend on the choice of "reference level". What can probably depend is your threshold, and I guess it is chosen poorly.

For the problem of churn prediction, you probably shouldn't use error rate or confusion matrix at all. You can read here why ROC AUC or other metrics can be preferred to accuracy. Or if you still use error rate, choose your threshold in a wiser way than letting an algorithm maximize F1 metric for a single class.

What you need in the end is to decide for each client whether to treat her as ready-to-churn (it would certainly cost you $c_1$) or leave her alone (but if she churns, you lose $c_2$). If it is the case, you will profit from your anti-churn measures iff the probability of churn is higher than $\frac{c_1}{c_2}$. This is the natural threshold for your classification problem, and you can use the corresponding cost function to measure your success. Or if you don't know exact losses $c_1$ and $c_2$ in advance, use ROC AUC, which averages all the possible thresholds. And yes, ROC AUC is not affected by class balance/imbalance.


Bayesian Decision Tree I was looking to learn about Bayesian theory in decision tree and how it avoids overfitting but couldn't find any tutorials for someone just starting. Do you know any resources to learn about it?

Regarding the being new to decision trees and wanting to get off the ground, I wrote a tutorial on decision trees that will help.

Regarding methods to avoid overfitting:  The game for any model is to limit its complexity to what is reasonable given the data you have.  Complexity in decision trees is manifested as adding new decision boundaries, so any limit in complexity is a limit in the decision boundaries it can draw.  Two common ways to do this is to place constraints on when a new decision can be created (a minimum of data in a leaf, significant increase in information, etc) or more simply to limit the max depth of the tree.


Transfer learning within Tensorflow's inception model I want to freeze the layers, except the first three layers, in the Inception v3 model with TensorFlow in Python 3, and modify the weights of these three layers to be able to re-initialize and re-train only the three first layers of the network.

If this can't be done within the inception model, is there any other network (in TensorFlow) with which this could be done?

To freeze the lower layers during training, the simplest solution is to give the optimizer the list of variables to train, excluding the variables from the lower layers:

train_vars = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES,
                               scope="hidden[34]|outputs")
training_op = optimizer.minimize(loss, var_list=train_vars)

The first line gets the list of all trainable variables in hidden layers 3 and 4 and in the output layer. This leaves out the variables in the hidden layers 1 and 2.


Independence of Features assumption in Naive Bayes How do we know if your features in my dataset are independent before applying Naive Bayes? Basically I want to know is it possible for us to get an idea before training our model if Naive Bayes will give decent results on it.

Statistical independence is a pretty straightforward thing. If $$p(A\cap B) = p(A) p(B)$$ then $A$ and $B$ are independent (in other words if marginal distributions are equal to conditional). If you want, you could even check that on your data. Though it would be easier to check:
$$p(A|B) = p(A) \ \text{and} \  p(B|A) = p(B)$$ instead of constructing a joint distribution. The latter is easy, if your features are categorical then you could estimate  $p(A)$, $p(B)$ , $p(A|B)$, $P(B|A)$ as sample frequencies. If one of A or B is categorical computations are also simple. If both A and B are numeric, you need to fit a KDE (kernel density estimation) model to all probability distributions.
However, on practice it is simpler and faster just to fit a Naive Bayes and check its performance on a test set.


Is Flatten() layer in keras necessary? In CNN transfer learning, after applying convolution and pooling,is Flatten() layer necessary?
I have seen an example where after removing top layer of a vgg16 ,first applied layer was GlobalAveragePooling2D() and then Dense().
Is this specific to transfer learning?
This is the example without Flatten().
base_model=MobileNet(weights='imagenet',include_top=False) #imports the mobilenet model and discards the last 1000 neuron layer.

x=base_model.output
x=GlobalAveragePooling2D()(x)
x=Dense(1024,activation='relu')(x) #we add dense layers so that the model can learn more complex functions and classify for better results.
x=Dense(1024,activation='relu')(x) #dense layer 2
x=Dense(512,activation='relu')(x) #dense layer 3
preds=Dense(3,activation='softmax')(x) #final layer with softmax activation

This example is with Flatten().
vgg = VGG16(input_shape=IMAGE_SIZE + [3], weights='imagenet', include_top=False)

# don't train existing weights
for layer in vgg.layers:
  layer.trainable = False

  # useful for getting number of classes
folders = glob('Datasets/Train/*')

# our layers - you can add more if you want
x = Flatten()(vgg.output)
# x = Dense(1000, activation='relu')(x)
prediction = Dense(len(folders), activation='softmax')(x)

# create a model object
model = Model(inputs=vgg.input, outputs=prediction)

What is the difference if both can be applied?

With GlobalAveragePooling2D, only one Feature per Feature map is selected by averaging every elements of the Feature Map.
e.g. if your global average pooling layer input is 220 x 220 x 30 you will find 1x1x30 output.
It means that you are finding a global representative feature from every slice. That is Global Average Pooling.
No, this isn't specific to transfer learning. It is used over feature maps in the classification layer, that is easier to interpret and less prone to overfitting than a normal fully connected layer.
On the other hand,
Flattening is simply converting a multi-dimensional feature map to a single dimension without any kinds of feature selection.


End-to-end ML project walkthrough I am new to machine learning and have finished Andrew Ng's course on Coursera. I have just begun to tackle my first "real" ML problem - which is a binary classification problem. I was wondering if the steps I have taken is the "usual" way to work through a problem.

I have gathered my data which is in this case is some images (positive and negative images)

Preprocess the data - in my case read the images and convert these to grayscale images in order to simplify the data. I have then set up my arrays in terms of input and labels. No feature scaling is needed in my opinion or anyother preprocessing.

After this I split the data in training and test set (80-20 ratio). I used stratified split.

Model selection and evaluation. In this case I used three models: logistic reg., SGDClassifier and a decision tree.

Train the different models and evaluate them using k-fold cross validation sets. I evaluated the mean accuracy on the CV set and also the F1-score on the CV set. Based on this I selected the best modell (the scores for all the modells on the training set was 1 and on the CV sets 0.91 (log.reg) , 0,57 (dec.tree) and 0.73 (SGD))

NOTE : Shouldn't I plot some learning curves too see if there is a overfitt or underfitt problem (based on the result from the CV sets it seems like log.reg model is a good fit). Is there anything else to study or is it enough based on the scores?

Fine tune the selected model.

Evaluate on the test set.

Depending on the images you used but I would not necessarily convert them to greyscale as there could be vital information in the colors of the image when it comes to making a prediction.

Feature scaling is not necessary for any of those algorithms but they could perform better with it.

Yes you should plot the Receiver Operating Characteristic and the Area Under the Curve. This is a metric you should be using as your problem is about Binary Classification.

The rest looks good.


What is the dimensionality of the bias term in neural networks? I am trying to build a neural network (3 layers, 1 hidden) in Python on the classic Titanic dataset.
I want to include a bias term following Siraj's examples, and the 3Blue1Brown tutorials to update the bias by backpropagation, but I know my dimensionality is wrong. (I feel I am updating the biases incorrectly which is causing the incorrect dimensionality)
The while loop in the code below works for a training dataset, where the node products and biases have the same dimension, but once I pass in a test example into the predict function, the dimensions do not match up and get an error. I have commented my code with the dimensions of the calculations of dot products between nodes and inputs.
Can someone help me understand what the dimensionality of the bias term should be, both in this particular case and in general, and how it should be added (row-wise, column-wise)?
Code:
def sigmoid(x, deriv=False):
    """
    Activation function
    """
    if(deriv==True):
        return (x*(1-x))
    return 1/(1+np.exp(-x))

# learning rate, hidden layer dimension, error threshold, dropout rate
alpha, hidden_size, threshold, drop_rate = (0.035,32,0.1,0.5)

# x_train and y_train are the training dataset and corresponding classes

# syn0 and syn1 are the synapses, weight matrices between layers (3 layers, 2 synpases)
syn0 = 2*np.random.random((x_train.shape[1],hidden_size)) - 1 # NxH
syn1 = 2*np.random.random((hidden_size,1)) - 1 # Hx1
b1 = np.random.random((x_train.shape[0],hidden_size)) # MxH
b2 = np.random.random((x_train.shape[0],1)) # Mx1

layer_2_error = 100*np.abs(np.random.random((y_train.shape[0],1))) - 1 # Mx1
avg_err = []
count = 0

while np.mean(np.abs(layer_2_error)) > threshold:

    # Forward
    layer_0 = x_train # training dataset
    A = np.dot(layer_0,syn0) + b1 # MxN X NxH + MxH ~ MxH
    layer_1 = sigmoid(A)
    # drop out to reduce overfitting
    layer_1 *= np.random.binomial([np.ones((len(x_train),hidden_size))],1-drop_rate)[0] * (1/(1-drop_rate))

    B = np.dot(layer_1,syn1) + b2 # MxH X Hx1 + Mx1 ~ Mx1
    layer_2 = sigmoid(B)

    # Backprop
    layer_2_error = layer_2 - y_train # Mx1
    layer_2_delta = layer_2_error * sigmoid(layer_2,deriv=True) # Mx1 * Mx1 ~ Mx1

    layer_1_error = np.dot(layer_2_delta,syn1.T) # Mx1 X 1xH ~ MxH
    layer_1_delta = layer_1_error * sigmoid(layer_1,deriv=True) # MxH * MxH ~ MxH

    # update weights
    syn1 -= alpha*np.dot(layer_1.T,layer_2_delta) # HxM X Mx1 ~ Hx1
    syn0 -= alpha*np.dot(layer_0.T,layer_1_delta) # NxM X MxH ~ NxH

    # update biases
    b2 -= alpha*layer_2_delta # Mx1
    b1 -= alpha*layer_1_delta # MxH

    avg_err.append(np.mean(np.abs(layer_2_error)))
    if count % 500 == 0:
        print("Error after",count,"iterations:",np.mean(np.abs(layer_2_error)))

    count += 1

def predict(x, w0, w1, b1, b2):
    """
    Function to predict an output given a data x, weight matrices w1 & w1 and biases b1 & b2
    """
    A = np.dot(x,w0) + b1 # mXN X NxH (+ MxH) ~ mxH
    layer_1 = sigmoid(A)
    B = np.dot(layer_1,w1) + b2 # mxH X Hx1 (+ Mx1) ~ mx1 (preds)
    layer_2 = B
    return (sigmoid(layer_2) > 0.5).astype(int)

As per the general case, the bias vector must have the same dimensions as the output vector.

Please, have a look at this excellent presentation:

In this example by M.Görner, there are 10 classes, so is bias dimension. Once inputs are multiplied by weights, the bias is added pointwise (it is 'broadcasted'). And that's pretty much it.


how to generate sample dataset for classification problem I am a newbie to data science. I have a 'short text' categorization problem where input variables are either unstructured texts (names, definition, description etc) or categorical. There is not much semantic to the fields as they are product names, territory name, sales order type etc. Issue is I do not have any sample data set from which I can derive training, test, validation set or divided it into k-fold for cross validation. So how should I generate sample data? I have about 20 target classes. I can classify some dataset using regex or lucene rule based matches and manually verify them and make sure each class have equal amount of samples. But I am open to other suggestion.

I know this isn't answering the question that you actually asked, but I suggest that you NOT generate data for your 'short text' categorization problem.

Generated data can work for certain cases when data scientists who are very familiar with an algorithm want to demonstrate a specific feature, but there is a hokeyness that may lead you astray as someone new to data science and machine learning.

I suggest you avail yourself of some free and open data here, or an entire stack exchange dedicated to open data.

Real data will give you experience with less contrived problems and you can often try out similar ML models on very different data sets in order to gain experience with the variations and pitfalls of data science. Be creative... you might not find exactly what you want, but you can subset the data by projecting out a few text fields and using them to classify another field.

Addendum update:

With your recent edit, it is now apparent that you are seeking to turn your unsupervised training data into supervised training data in order to train a supervised learning classification model.  The method that you suggested, "classify some dataset using regex or lucene rule based matches and manually verify them", is a deterministic unsupervised method without the human verification step.

Without the human verification of the target data that you have created, I would not consider feeding the derived targets back into the classification algortihm as your results will only be as strong as the derived target data and predictions will show similar errors. Instead, you should think about a semi-supervised learning method where you perhaps employ a clustering algorithm and then label the clusters with the target variable.

With human verification of the target data that you have created, this will work just fine as training data for a classification model. The only issue is that this can become tedious. There are mechanical turks (humans paid to perform repetitive tasks) that you can hire to perhaps perform the labeling for you, which may be a more scalable option.

Hope this helps!


What could be a right datascience approach of doing fuzzy string matching in large amount of short text data? I have two lists to compare. List 1 contains strings which represent the strings in List2. Sometimes, they are direct matches, some times, one word match, sometimes there are acronyms etc. An example is as below:

   List1           |  List2
  ------------------------
   great hall town | Great Hall
   grt_town        | Great Hall
   gHall           | Great Hall
   great town      | Great Hall
   mind great town | Great Hall

There are two main problems:

List2 is very large ( millions of records, all short in length)
List1 records are very unrelated sometimes. There are longer prefixes, suffixes which are not in List2. Some words in List2 are reduced to a letter.

I vaguely have some thoughts around clustering List2 and trying to find distance between centroids and the elements from List1. Do you think this as a starting point?
Thanks

You could try this:

Take distinct records from List2
Clean List1 of any extraneous characters or symbols
Use a fuzzy logic package like Python's fuzzywuzzy. It has partial matching options.


Logistic or Random Forest? I have data on various standardized tests that certain students took and whether or not they ended up passing a specific class in college. Using the pass/fail as the dependent variable and the different scores on the test for the independent variables, I would like to figure out a metric that best predicts the students success in that class; i.e., Student gets these scores on these tests hence the probability that they pass is so and so.

Logistic regression is my first instinct but I was wondering if there are other, possibly better, ways to approach this problem.

In general, it's never a bad idea to use logistic regression as a first stab at a classification problem. If it doesn't work great, it at least gives you a baseline. Random forest would probably work well here as well. Generally, random forests aren't considered very interpretable, but they're actually pretty interpretable if you're interested in understanding the decision process for a single prediction, which it sounds like would be sufficient for your needs here (i.e., the model can tell you a student is likely to perform a certain way because of their behavior on specific tests). A random forest will probably also handle missing data better, which might be useful to you since I'm guessing not all students took the same tests.

Another option you can try here which handles null data extremely well and would also be simple for inference, is naive bayes.


How can a GRU perform as well as an LSTM? I think I understand both types of units in terms of just the math.

What I don't understand is, how is it possible in practice for a GRU to perform as well as or better than an LSTM (which is what seems to be the norm)? I don't intuitively get how the GRU is able to make up for the missing cell content. The gates seem to be pretty much the same as an LSTM's gates, but with a missing part. Does it just mean that the cell in an LSTM is actually nearly useless?

Edit: Other questions have asked about the differences between GRU and LSTM. None of them (in my opinion) explain well enough why a GRU works as well as an LSTM even without the memory unit, only that the lack of a memory unit is one of the differences that makes a GRU faster.

GRU and LSTM are two popular RNN variants out of many possible similar architectures motivated by similar theoretical ideas of having a "pass through" channel where gradients do not degrade as much, and a system of sigmoid-based control gates to manage signals passing between time steps.

Even with LSTM, there are variations which may or may not get used, such as adding "peephole" connections between previous cell state and the gates.

LSTM and GRU are the two architectures explored so far that do well across a wide range of problems, as verified by experiment. I suspect, but cannot show conclusively, that there is no strong theory that explains this rough equivalence. Instead we are left with more intuition-based theories or conjectures:

GRU has less parameters per "cell", allowing it in theory to generalise better from less examples, at the cost of less flexibility.
LSTM has a more sophisticated memory in the form of separating internal cell state from cell output, allowing it to output features useful for a task without needing to memorise those features. This comes at the cost of needing to learn extra gates which help map between state and features.

When considering performance of these architectures in general, you have to allow that some problems will make use of these strengths better, or it may be a wash. For instance, in a problem where forwarding the layer output between time steps is already a good state representation and feature representation, then there is little need for the additional internal state of the LSTM.

In effect the choice between LSTM and GRU is yet another hyperparameter to consider when searching for a good solution, and like most other hyperparameters, there is no strong theory to guide an a priori selection.


Tensorflow Calculate error for a single neuron I'm required to be able to calculate the error on a given neuron in a neural network using Tensorflow.

Using this :

mse = tf.reduce_mean(tf.squared_difference(out, Y))
opt = tf.train.GradientDescentOptimizer(0.1).minimize(mse)

i just get the updated weight, how i can get or calculate the error for each neuron in my network.

Thanks for the replies.

Easy, you have already done it.

mse = tf.reduce_mean(tf.squared_difference(out, Y))

This command means you are taking the mean of all the neurons over all the examples.

error = tf.squared_difference(out, Y)

This command will give you the error over all neurons over all examples. But make sure you calculate against the correct axes.


Pre-train using sigmoid and train using ReLU? Using RBMs to pre-train a deep net as in this example RBM, the activation function is sigmoid and makes the math much easier.

What are the implications after the initial weights are learned using sigmoid activation functions to switch to ReLU for the train phase?

I suppose that using tanh in either phase (pre-train or train) and sigmoid or ReLU in the other would cause great problems, but since ReLU and sigmoid are similar for small values, would it still render the pre-train phase useless?

The question can be more general in the sense of how much knowledge can you transfer from a neural network using sigmoid activation functions to one identical in structure, but using ReLU activation functions.

Since RBM has only one layer of weights, why you bother changing sigmoid to ReLU in 1-layer net? Gradient vanishing is very unlikely to happen in such a shallow net.

You can also train Gaussian-Bernoulli or Gaussian-Gaussian RBM (more here), which has identity activation function, which is closer to ReLU than sigmoid, and what is more justified if you have real-valued data, not binary. However, these types of networks are a bit less stable to train because of this unconstrained activation.


Loaded model predicts well in colab but gives same label and accuracy when downloaded I have developed a Recurrent Neural Network to perform sentiment analysis on tweets using the Kazanova/sentiment140 dataset in Kaggle.
The model looks like this:
def scheduler(epoch):
  if epoch < 10:
    return 0.001
  else:
    return 0.001 * tf.math.exp(0.1 * (10 - epoch))

callback1 = tf.keras.callbacks.LearningRateScheduler(scheduler)
callback2 = tf.keras.callbacks.ReduceLROnPlateau(monitor='val_loss',patience=10, verbose=0, mode='auto',min_delta=0.0001, cooldown=0, min_lr=0)
callback3 = tf.keras.callbacks.EarlyStopping(monitor='val_accuracy', min_delta=0, patience=3, verbose=0, mode='auto',baseline=None, restore_best_weights=True)

model = tf.keras.Sequential([
    tf.keras.layers.Embedding(vocab_size+1, embedding_dim, input_length=max_length, weights=[embeddings_matrix], trainable=False),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Conv1D(64, 5, activation='relu'),
    tf.keras.layers.MaxPooling1D(pool_size=4),
    tf.keras.layers.LSTM(64),
    tf.keras.layers.Dense(1, activation='sigmoid')
])
model.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])
model.summary()
num_epochs = 50

training_padded = np.array(training_sequences)
training_labels = np.array(training_labels)
testing_padded = np.array(test_sequences)
testing_labels = np.array(test_labels)

history = model.fit(training_padded, training_labels, epochs=num_epochs, validation_data=(testing_padded, testing_labels), verbose=2,callbacks=[callback1,callback2])

print("Training Complete")

model.save('sentiment_final.h5')

The model runs fine and predicts output perfectly when loaded from colab itself
The loaded colab code:
load_model= tf.keras.models.load_model('sentiment_final.h5')
#load_model.summary()

def decode_sentiment(score):

    if score < 0.5:
        return "NEGATIVE"
    else:
        return "POSITIVE"

def predict(text):

    x_test = pad_sequences(tokenizer.texts_to_sequences([text]), maxlen=16)

    score = load_model.predict([x_test])[0]

    return {"label": decode_sentiment(score), "score": float(score)}

predict("I love this day") #Outputs -> {'label': 'POSITIVE', 'score': 0.793081521987915}
predict("I hate this day") #Outputs -> {'label': 'NEGATIVE', 'score': 0.38644927740097046}
predict("I shouldn't be alive") #Outputs -> {'label': 'NEGATIVE', 'score': 0.12737956643104553}

But If I load the model in VSCode , the output is the same for all the models.
VSCode Implementation:
import tensorflow
from tensorflow.keras.models import load_model
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences
import os

tokenizer=Tokenizer()
model = load_model('sentiment_final.h5')

def decode_sentiment(score):

    if score<0.5:
        return "Negative"
    else:
        return "Positive"

def predict_score(text):

    x_test=pad_sequences(tokenizer.texts_to_sequences([text]),maxlen=16)
    score=model.predict([x_test])[0]
    return {"label":decode_sentiment(score),"score": float(score)}

def call_predict_function(text):
    return predict_score(text)

print(call_predict_function("I love this day")) #Outputs -> {'label': 'POSITIVE', 'score': 0.793081521987915}
print(call_predict_function("I hate this day")) #Outputs -> {'label': 'POSITIVE', 'score': 0.793081521987915}
print(call_predict_function("I shouldn't be alive")) #Outputs -> {'label': 'POSITIVE', 'score': 0.793081521987915}

Where am I going wrong? Can somebody resolve this problem?

As far as I am aware you also need to save and load the tokenizer you used. The tokenizer is not fitted/trained and therefore is outputting nothing sensible for the model to predict on.


Curve Fitting By Predict polynomial degree with ANNs My Science Fair Project is on Curve fitting with neural nets as an alternative to polynomials. I know this is a widely done procedure in datascience but I wanted to do my own research on it. Currently the biggest problem is training time, and to solve that I had an idea. What if I I trained an ANN to predict the degree of the polynomial that best fits the model then use standard polynomial regression to curve fit. This would make the training only happen once so I could train for as long as I wanted, and thus it would increase speed by a lot. What are your thought in this? Has anyone done this before? Is it plausible?

Interesting idea :)

The things I would consider:

Problem definition

Should it be classification or regression? If you define it as classification, you should select the number of classes, hence the maximum and minimum polynomial degree (e.g. 1,2,3,4,5,6,7,8,9 and 10 could be polynomial degrees which you recognize).

If you select regression, you don't need to define the maximum number of classes, but the output layers could be trickier to define.

Dataset

How do you define your samples? I guess that it's easy to generate samples (select some polynomial with degree $n$, sample points from it and then all those points define one sample for class (or regression value) n).

It would be interesting to create the image out of the sample (plot of points) and then maybe use a convolutional network to learn the classification (or regression).

Also, consider how dataset big should be. It's hard to say, you will have to test it and see how big it should be.

Losses

For classification probably cross entropy and for regression probably mean squared error.

Let me know the results.


Torch and conceptual question about neural nets backpropagation First a theoretical question and then a practical one.

Neural nets backpropagation is the computation of the weights derivatives or the computation of the new weights (that is, the original weights minus  the weight derivatives times the learning rate - simplified)?

It may well be a semantics issue but important nevertheless.

Also, if anyone is familiar with Torch, nn class

gradInput = module:backward(input, gradOutput)

is gradinput the weight set for the next forward pass or is it the derivatives of the weights of the previous forward pass?

Thanks!

I have been using torch for a few months now but I will give it a go (apologies if incorrect).

Yes a weight $w$ is updated as follows;

$$ w_{new} = w_{old} - \gamma \partial E/ \partial w_{old}  $$

where $ \gamma $ is your learning rate and $E$ is the error calculated using something like criterion:forward(output,target). The criterion could be, for example, nn.MSECriterion().

To calculate $\partial E/\partial W$  you need $\partial E/\partial y$ gradOutput = criterion:backward(output,target)(gradient respect to output) as well as the input input to the net i.e. your $X$ (e.g. image data) to generate the recursive set of equations which multiply with gradOutput.

model:backward(input, gradOutput) therefore serves to update the weights so that they are ready for the next model:forward(input) as it generates a big derivative tensor $dE/dW_{old}$.

This is then combined with a optimiser such as optim.sgd using optimMethod and the old weights $W_{old}$ to generate the new weights in the first equation. Of course you can just update the weights without an optimiser with model:updateParameters(learningRate) but you miss useful stuff like momentum, weight decay etc.

Got a bit side tracked there but hope this helps.


Understanding declared parameters in my Conv2d layer of my convolutional neural network I am trying to understand the architecture of my keras model implemented by the sequential model.
Here is a piece of the code :
model = Sequential([
    #block1
    layers.Conv2D(nfilter,(3,3),padding="same",name="block1_conv1",input_shape=(64,64,3)),
    layers.Activation("relu"),
    layers.BatchNormalization(), .......

My question is why the two parameters input_shape and name are declared in the layer conv2D while they are not included in the set of defined parmeters for Conv2D() in this link https://keras.io/api/layers/convolution_layers/convolution2d/

Bot the name and input_shape come from the Layer class which Conv2D inherited. In the doc you provide, they are implicitly in **kwargs


What is the difference between the 3 architectural components of Microsoft R Server? Lately, I've been experimenting with Microsoft R Server(MRS) and SQL Server R Services. I've come across that MRS is built on architectural components like ScaleR algorithms optimized for fast parallel execution, DistributedR parallel computing framework for managing compute resources, and ConnectR for versatile connections to data sources.

I've also learnt (and got confused) that

ScaleR algorithms are basically functions of the RevoScaleR package built on the DistributedR framework.
ConnectR component of the architecture is also a bunch of RevoScaleR functions / adapters to access a variety of data sources.

My question is:
What is RevoScaleR in relation to DistributedR and ConnectR?

Where does DeployR fit into all this?

In general, I want to know the high level differences between
RevoScaleR package
DistributedR
ScaleR
ConnectR
DeployR

RevoScaleR package & ScaleR are same (I think after MS acquisition its became scaleR)
A new file format especially designed for large files such as :
XDF File Format (block based data format)
RxDataSource (designed to support the use of external memory algorithms with .xdf files.)
PEMA  based Algorithms
External memory implementations of the statistical algorithms most commonly used with large data sets such as : rxSummary, rxCrossTabs, rxLinMod. rxKMeans, rxGlm etc.
An extensible programming framework that allows R programmers to write their own external memory algorithms that can take advantage of ScaleR Enterprise’s new Big Data capabilities.

DistributedR for supporting big data framework
ConnectR It has all connectors to interact with other data sources
DeployR is used to integrate R with any application programming


How is the evaluation setup for YouTube faces of FaceNet? The YouTube Faces database (YTF) consists of 3,425 videos of 1,595 different people. Given two videos, the task for YTF is to decide if they contain the same person or not. Having $n$ comparisons, the classifier might get $c \leq n$  right. Then the accuracy would be $\frac{c}{n}$.

FaceNet is a CNN which maps an image of a face on a unit sphere of $\mathbb{R}^{128}$. It was evaluated on YTF. How did they decide which person is in the video?

(I can imagine several procedures how this could be done, but I couldn't find it in the paper. One example, how it could be done, is by evaluating all images $x_i^{(k)}$ with $i = 1, \dots, \text{length of video }k$ and averaging the results - but I would like to know what they did / how this is usually done.)

The objective function they use to train the CNN minimizes the squared L2 distance (i.e. the squared Euclidean distance) between two similar (positive) images and simultaneously maximizes the distance between two different (negative) images.
That means, the (squared) Euclidean distance between two representations is a measure of their similarity.
Then, recognizing a face in a new image is as simple as 1) running it through the CNN and 2) finding its nearest neighbors with a KNN algorithm.

The last paragraph was only about images - in the Youtube Faces DB, we are handling videos of different persons.
In section 5.7 of the paper, they describe how they evaluate performance:

  We use the average similarity of all pairs of the first one hundred frames that our face detector detects in each video.

So, you were partially right: they just average the independent results over video frames. Probably for performance reasons, they chose to average the first 100 frames. They do describe that increasing this to the first 1000 frames increases performance from 95.12% to 95.18%, which is not significantly more.


Support vector machine margin term, norm or norm squared? $$ \lambda||\hat w||^2 +(1/n)\sum   max(0,1-y_i(\hat w \hat x_i -b))   $$

we know that $2/||\hat w||$ is the width of the margin.

The second term penalizes a misclassified point for how far away it is from the margin relative to the width of the margin. E.g., suppose there is a misclassified point $x_0$:
$$ 1-y_0(\hat w \hat x_0 -b)=3 $$
That means $x_0$ is $3/||\hat w||$ away from $1-y_i(\hat w \hat x_i -b)=0$ and is penalized for $3$.

The first term penalizes for the inverse of the width of the margin squared. I find it hard to reconcile with the second term - they seem to be of different scales. Is there any reason (intuitively) why $||\hat w||^2$ is used instead of just $||\hat w||$?

PS: Perhaps one reason is that  $||\hat w||^2 $ is easier computation-wise (quadratic programming)? Or perhaps norm squared assumes sample noise to be Gaussian? I am not sure. Has anyone seen the use of $||\hat w||$ instead of $||\hat w||^2$?

It is indeed for computational tractability. You would not lose the all-important sparsity, since that is provided by the second term. The original formulation of SVM, which you can find on Wikipedia actually uses $\lVert w \rVert$:

Minimize $\|{\vec {w}}\|$ subject to ${\displaystyle y_{i}({\vec {w}}\cdot {\vec {x}}_{i}-b)\geq 1}-\xi_i$

Obviously, minimizing $\lVert w \rVert$ is the same as $\lVert w \rVert^2$. A quantative difference arises when introducing the constraint using the Lagrangian function, but qualitatively it is the same.


sckit-learn Cross validation and model retrain I want to train a model and also perform cross validation in scikit-learn, If i want to access the model (For instance to see the parameter's selected and weights or to predict) i will need to fit it again.

I believe and according to my understanding if CV is training the model for performing k-fold validation, do we need to train again as any of k-models are not trained on the whole training data.

I have seen cross_val_predict method and it is not exactly what i need as it is for predicting score and i want model object.

My main confusion is that for hyper parameter tuning and in grid search we can have best paramters,but in cross validation only do we need to call the fit method after CV step

"I want to train a model and also perform cross-validation in scikit-learn"

By this, I assume you meant saying, "I want to assess different model performace using K-fold Cross-validation approach and based on the performance, I want to select a model fitting"

Primarily, K-fold CV will divide the training data into a number of folds k, and then will fit and evaluate the model. Reference from chapter 5 in ISLR:

  Cross-validation can be used to estimate the test error associated
  with a given statistical learning method in order to evaluate its
  performance, or to select the appropriate level of flexibility.

I try to explain with a sample code:

from sklearn import svm
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score

# Training Data
feature_cols=['X1', 'x2', 'x3']
X = train_data[feature_cols]
y = train_data.target

#Build a svm classifier model
model_svm = svm.SVC(C=100, gamma=1)

#Initialise the number of folds k for doing CV
kfold = KFold(10, False, 2)

#Evaluate the model using k-fold CV
cross_val_scores = cross_val_score(model_svm, X, y, cv=kfold, scoring='accuracy')

#Get the model performance metrics
print("Mean: " + str(cross_val_scores.mean()))

You can do k-fold CV for different model parameters say (different C and gamma values in SVM) or even different models altogether (say Logistic regression) and choose the best model.

Once you chose the best performing model you then can proceed to fit the model for making the prediction.

#Fit the model
model_svm.fit(X,y)

#Prediction on test set
model_svm.predict(test_data)

"I believe and according to my understanding if the CV is training the model for performing k-fold validation, do we need to train again as any of k-models are not trained on the whole training data."

In the above sample code, the number of folds k=10. So CV will be done 10 iterations with each training set containing (k − 1)n/k = 9(n/10) observations and n/10 observations for testing. So no, you don't need to train again as K-fold CV does this procedure for you. From ISLR:

  K-fold CV approach involves randomly dividing the set of
  observations into k groups, or folds, of approximately equal size. The
  first fold is treated as a validation set, and the method is fit on
  the remaining k − 1 folds. This procedure is
  repeated k times; each time, a different group of observations is
  treated as a validation set.

"My main confusion is that for hyper parameter tuning and in grid search we can have best paramters,but in cross validation only do we need to call the fit method after CV step"

Again, the k-fold CV method is used for model assessment. What you can do is to use the grid search for selecting the best parameters and say the GridSearch method to use K-fold Cross-validation to selecting the best parameters. Also in a way Yes, you will call the fit method after CV because you would have chosen the best performing method and be proceeding to fit the model.

    from sklearn import svm
    from sklearn.model_selection import KFold
    from sklearn.model_selection import GridSearchCV

    # Training Data
    feature_cols=['X1', 'x2', 'x3']
    X = train_data[feature_cols]
    y = train_data.target

    #Build a svm classifier model
    model_svm = svm.SVC(C=100, gamma=1)

    #Initialise the number of folds k for doing CV
    kfold = KFold(10, False, 2)

    Cs = [0.001, 0.01, 0.1, 0.3, 1, 3, 10, 100]
    gammas = [0.001, 0.01, 0.1, 0.3, 1, 3, 10, 100]

    param_grid = {'C': Cs, 'gamma' : gammas}
    grid_search = GridSearchCV(model_svm, param_grid, cv=kfold, scoring='accuracy')
    grid_search.fit(X, y)
    grid_search.best_params_

I hope this clears thing up.


What are the features to distinguish two short term signals (time series) Imagine that I have a set of short term time series data (signals) with different shapes and parameters. I would like to extract/select a set of features (parameters/functions/indeces) to identify each sequence and distinguish between them. Such features are simple features e.g. max and min value, mean, variance, start and end value, etc. or more complex ones e.g. entropy, no. peaks, power spectrum, etc.
I am looking for more "representative" features to express the signals and distinguish them. What features do you know (in signal processing or data mining fields) that can represent a parameter in signals?
For example, in the following image, there are two signals with the same parameters such as mean, min-max values. But what are the features that differ them (for instance no. peaks, but generally what else?)

The examples that you provided look similar to one another in duration and amplitude, so normalization might not be a problem.  But consider whether you will always be comparing the same duration and whether you want to compare the relative change in the time series or the absolute change.  The relative change can be measured by normalizing the series so that the first value (starting point) is set to 0.  Depending on your problem, this might be worth considering.

Second, think about some sort of smoothing algorithm to get rid of some of the jitter in your data.  Examples include an Exponentially Weighted Moving Average or a second order Holt-Winters EWMA.

The suggestions in the comments are good ones... both Fourier Transform via FFT or moving to a wavelet formulation are interesting, but they are still a one-to-one in terms of the amount of data in (in temporal space) being equal to the amount of data out (in frequency space).

I would suggest first including the raw data or a down-sampling of the raw data with some additional features like area under the curve, max, min.  I would stay away from procedures which amplify the jitter of your data (like derivatives) in favor of those which decrease the jitter (like integrals).

A nice option, which implicitly smooths would be to take a running integration of your data and then plot the running integral rather than the values themselves.  This wouldn't require apriori smoothing and is still deterministic.

Following this, I suggest doing a whole bunch of feature extraction/feature engineering and then trying a LASSO regression to pick out the most useful features.

Some feature engineering ideas include:

Fourier Transform via FFT
Possibly only keeping the n (~10) largest Fourier Modes from above rather than the full set.
Integrals of your data.
Min.
Max.
Number of zero crossings.

Suggestions to look at the number of changes in the second derivative are tough with time series data due to its stochastic nature.  The slopes (derivatives) are very wild and depend largely on how much smoothing has been applied and how the data is being sampled.


Data dashboard for SQL server database I am working on developing a data dashboard / app for a Microsoft SQL database. Currently, I am developing the dashboard using Shiny and R. The app is mostly for exploratory analysis to allow people to filter out some subset of data, build some plots, and export data / plots.

My question is what are some other options for creating this type of dashboard? I am aware of Tableau.

You could consider PowerBI. It offers a couple of useful licensing scenarios (bundled with Office 365 etc.) and also has connectors to source data that already works well with SQL Server.

You did not say if SQL Server is running on premises or in the cloud, but PowerBI should work in both cases.


Relationship between batch size and the number of neurons in the input layer Regarding LSTM neural networks, I am unable to understand the relationship between batch size, the number of neurons in the input layer and the number of "variables" or "columns" in the input. (Assuming that there is a relationship and despite seeing examples to the contrary, I cannot understand why there is no relationship)

For the sake of clarity, I am going to use an example to formulate my query.

Let's assume that the dataset contains three columns of input and one column of output. So it will be something like
input variable 1  input variable 2  input variable 3  output variable 1

From what I understand, the input layer of the LSTM network has to have 3 neurons corresponding to each of the input variables. It cannot be less or more. Even though I have seen examples like this answer (which appears to be very well described but unfortunately, I am not able to comprehend it.)

Now let us say that we have 50 rows of the above 4 columns.

That is essentially to me means that we have 50 samples.
Now if the batch size is 5, then how many input neurons we have? Is the number of neurons in the input layer independent of the batch size? The way I understand batch size is that it is the number of samples that the neural network will see before updating its weights. So let's assume if we have only three neurons in the input layer, then we will be passing the first row of input variable, followed by the second row of input variable and repeating it until the fifth row before we update the weights. How is that going to be any different than just passing the fifth row?

In general I think you've understood both concepts. I'll try to address both of them in more detail.
Input layer size
In Neural Networks (NN), the size of the input layer is always equal to the number of variables (or features as we usually call them in Machine Learning) in your data.
Batch size
This refers to how many samples of data the network will see before updating its weights.
Normally, when performing gradient descent (i.e. the algorithm that trains the NN), the network would be input all of the samples in order to calculate the loss and update the weights. However, especially in Deep Neural Networks (DNNs) this is infeasible, as the size of the dataset is too big and even if the network had enough memory the computation of the gradients would be inefficient. What we do instead is pass a few samples (called a batch) at a time and train the model on them. Then another batch is passed, and so on. The number of samples we input to the model at each iteration is called the batch size.
Relationship between the two
In theory, if you have many features (in the thousands) you can't use a large enough batch size and the more features you have the less of a batch size you can use. However, in practice I have found the two to be practically unrelated! In Deep Neural Networks, the batch size is primarily governed by the size of the model. An easy (but not always accurate) way of estimating this size is through the number of trainable parameters in a model. More parameters, mean more memory for the model (and longer gradient computations), leading to a smaller batch size. In DNNs, that have multiple layers, only a fraction of the parameters are those of the input layer. Thus, the number of features plays a minor part in the selection of the batch size.
Example
In your example you say you have 50 rows and 4 columns, out of which 3 are the input variables.
The number of neurons in the input layer will be 3, to match the number of input variables (or features). This we cannot change, even if we wanted to!
The batch size is something we can change. Let's say we use a batch size of 5.
1st iteration:
In the first iteration, the NN will be given the first 5 rows of data. For each, the first three columns will be passed to the first layer of the network, which will calculate an output. After all 5 outputs are generated, they will be compared with the first 5 rows of the 4th column, to calculate a loss. Using this loss, the network will compute the gradients and perform the updates. Again, this loss corresponds to the first 5 samples, not just the 5th.
2nd iteration:
The rows 6-10 will be passed to the network, which will do the same procedure as in the first step.
10th iteration:
The rows 45-50 will be passed to the network. This marks the point where the NN has seen every sample of data once. In the context of Machine Learning, we call this an epoch. At this point, the data are shuffled and the 1st iteration of the 2nd epoch begins.
Note:
In most frameworks the samples in each batch are processed in parallel by the network and not sequentially. This makes no difference to the result, I just wanted to point this out.


Difference between Bernoulli and Multinomial Naive Bayes Here is my understanding of the difference between a Bernoulli and a Multinomial Naive Bayes:

Bernoulli explicitly models the presence/absence of a feature, whereas Multinomial doesn't.

Is there something that I am missing?
Can someone explain why this difference matters intuitively? Perhaps using an example where you would obviously use one instead of the other

On the wikipedia page, they say:

  Note that a naive Bayes classifier with a Bernoulli event model is not the same as a multinomial NB classifier with frequency counts truncated to one.

Why is that?

Bernoulli models the presence/absence of a feature. Multinomial models the number of counts of a feature. Here's a concise explanation.

Wikipedia warns that

  Note that a naive Bayes classifier with a Bernoulli event model is not
  the same as a multinomial NB classifier with frequency counts
  truncated to one.

To understand why, we should note that, as this page nicely explains,

  Whereas the binomial distribution generalises the Bernoulli
  distribution across the number of trials, the multinoulli distribution
  generalises it across the number of outcomes, that is, rolling a dice
  instead of tossing a coin.

What does this imply for us? Multinomial NB cares about counts for multiple features that do occur, whereas Bernoulli NB cares about counts for a single feature that do occur and counts for the same feature that do not occur.

This means that, for example, Multinomial NB will classify a document based on the counts it finds of multiple keywords; whereas Bernoulli NB can only focus on a single keyword, but will also count how many times that keyword does not occur in the document.

So they do model slightly different things. If you have discrete multiple features to worry about, you have to use Multinomial NB. But if you only have a single feature to worry about, then you can make a modelling choice based on the above.


Quick start using python and sklearn kmeans? I started tinkering with sklearn kmeans last night out of curiosity with the goal of clustering users into groups to see what kind of user groups I can derive.  I am lost when it comes to plotting the results as most examples have nice (x,y) coordinates.  For example, the iris data set has pedal width and pedal length.  From my experimentation, I don't seem to have anything that displays very nice.  Is this assumption correct / does anyone have tips, pointers, learning resources that they could offer?

import pandas as pd
import pprint
import numpy as np
from sklearn.preprocessing import normalize
from sklearn.preprocessing import LabelEncoder
from sklearn.cluster import KMeans
from collections import defaultdict
import matplotlib.pyplot as plt
from matplotlib import style
style.use('ggplot')

I normalized the data as it had a wide variance...again, not sure if this is a correct assumption to make

X = np.array(normalize(data, axis=0, copy=False))

kmeans = KMeans(n_clusters=3)
pred = kmeans.fit_predict(X)
labels = kmeans.labels_
cent = kmeans.cluster_centers_

plt.scatter(X[:, [4]], X[:, [6]])
plt.scatter(cent[:, [4]], cent[:, [6]], marker="x", s=150, linewidths=5, zorder=10)
plt.ylabel('Count')
plt.xlabel('Department')
plt.show()

Any pointers are appreciated, I will include sample data below.  Thanks!

Sample Data:

emp_type,title,work_country,director_userid,dept_name,business_unit_name,UserCNT
0,9,7,29,20,2,2
0,13,7,8,14,6,5
0,4,3,56,29,8,3
0,15,3,36,32,2,3
0,4,3,32,16,2,0
0,4,1,40,13,6,0
0,4,3,62,12,4,1
0,13,7,61,5,13,4
2,1,3,70,35,15,2
0,4,3,64,4,13,0
2,1,3,43,43,2,0
0,13,7,50,17,16,0
2,1,3,31,26,2,1
2,1,3,65,58,17,0
0,4,3,57,63,12,0
2,1,6,7,45,18,2
2,1,3,43,42,2,0
1,1,7,65,58,17,0
2,1,3,32,16,2,0
2,1,3,29,20,2,0
0,4,0,50,17,16,2
0,5,3,20,23,9,0
0,9,3,32,16,2,2
0,4,3,5,51,12,0
2,1,7,51,53,7,0
0,13,7,37,55,12,0
2,1,4,19,62,13,0

Example clustering using entire data set:

This has been answered in other places e.g. here.

You could run Principal Component Analysis (or other dimensionality reduction techniques) and plot the cluster for the first two principal components.
You could plot the results for two variables at a time.
You could encode third or fourth variables using standard visualization techniques like color coding, symbols or facetting.
There are ways to visualize the quality of the fit e.g. silhouette analysis or elbow test for determining the number of cluster etc.
Have a quick look at this link