<jupyter_start><jupyter_text>**Sustainable Software Development, block course, March 2021**  *Scientific Software Center, Institute for Scientific Computing, Dr. <NAME>* Analysis of the dataImagine you perform a "measurement" of some type and obtain "scientific data". You know what your data represents, but you have only a vague idea how different features in the data are connected, and what information you can extract from the data.You would start first with going through the data, making sure your data set is complete and that the result is reasonable. Imagine this already happened.In the next step, you would inspect your data more closely and try to identify structures. That is the step that we are focusing on in this unit.In the `data` folder, you will find several data files (`*.t` and `*.dat`). These are data files generated through some "new approach" that hasn't been used in your lab before. No previous analysis software exists, and you are going to establish a protocol for this "new approach" and "publish your results".The data can be grouped into two categories: 1. data to be analyzed using statistical methods;2. data to be analyzed using numerical methods.In your hypothetical lab, you are an "expert" in one particular "method", and your co-worker is an "expert" in the other. Combined these two methods will lead to much more impactful results than if only one of you analyzed the data. Now, the task in this course is to be solved collaboratively with your team member working on one of the analysis approaches, and you working on the other. You will both implement functionality into the same piece of "software", but do so collaboratively through git.As you do not know yet which analysis is most meaningful for your data, and how to implement it, you will start with a jupyter notebook. You and your team member will work on the same notebook that will be part of a github repository for your project. This is the task for today. Discuss with your team members who will work on the statistical and who on the numerical analysis.  Step 1Generate a github repository with the relevant files.  Step 2Clone the repository to your local machine.  Step 3Start working on task 1 for your analysis approach.   Step 4Create your own branch of the repository and commit your changes to your branch; push to the remote repository.  Step 5Open a `pull request` so your team member can review your implementation. Likewise, your team member will ask you to review theirs.  Step 6Merge the changes in your branch into `main`. Resolve conflicts.  Step 7Repeat working on task; committing and pushing to your previously generated branch or a new branch; open a pull request; merge with main; until you have finished all the tasks in your analysis approach. Delete obsolete branches.  Start of the analysis notebook**Author : <NAME>**  *Date : 11.03.2021*  *Affiliation : Heidelberg University, IWR*  Place the required modules in the top, followed by required constants and global functions.<jupyter_code>import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sb
import numpy as np
data_path = "../../data/"
files = ['expec.t', 'npop.t', 'table.dat']
threshold = 1e-5
def reader (file):
    return pd.read_csv(data_path + file, delimiter = '\s+')<jupyter_output><empty_output><jupyter_text>Statistical analysisFind correlations in the data sets. Analyse the data statistically and plot your results.  Here we would want to do everything with pandas and leave the data in a dataframe. The files that are relevant to you are `expect.t`, `npop.t` and `table.dat`.  Task 1: Read in expect.t and plot relevant data<jupyter_code>df = reader(files[0])<jupyter_output><empty_output><jupyter_text>We can discard the entries norm, \, and \ as these are mostly constant.<jupyter_code>def drop_col (df):
    return df.drop(df.var()[df.var() < threshold].index.values, axis = 1)

df1 = drop_col(df)
#df1 = df.drop(['norm', '<x>', '<y>'], axis = 1)
#df1<jupyter_output><empty_output><jupyter_text>Task 2: Create plots of the relevant data and save as .pdf.<jupyter_code>df1.plot(x='time')
plt.savefig('task2.pdf')<jupyter_output><empty_output><jupyter_text>Task 3: Read in file `npop.t` and analyze correlations in the data<jupyter_code>df2 = reader('npop.t')
df2 = drop_col(data)
df2.plot(x = 'time')<jupyter_output><empty_output><jupyter_text>Plot the remaining columns. Seaborn prefers "long format" (one column for all measurement values, one column to indicate the type) as input, whereas the cvs is in "wide format" (one column per measurement type).<jupyter_code>sb_df = df2.set_index('time')
sb.lineplot(data=sb_df, )<jupyter_output><empty_output><jupyter_text>Quantify the pairwise correlation in the data- negative correlation: y values decrease for increasing x - large values of one feature correspond to small values of the other feature- weak or no correlation: no trend observable, association between two features is hardly observable- positive correlation: y values increase for decreasing x - small values of one feature correspond to small values of the other featureRemember that correlation does not indicate causation - the reason that two features are associated can lie in their dependence on same factors.Correlate the value pairs using Pearson's $r$. Pearson's $r$ is a measure of the linear relationship between features:$r = \frac{\sum_i(x_i − \bar{x})(y_i − \bar{y})}{\sqrt{\sum_i(x_i − \bar{x})^2 \sum_i(y_i − \bar{y})^2}}$Here, $\bar{x}$ and $\bar{y}$ indicate mean values. $i$ runs over the whole data set. For a positive correlation, $r$ is positive, and negative for a negative correlation, with minimum and maximum values of -1 and 1, indicating a perfectly linear relationship. Weakly or not correlated features are characterized by $r$-values close to 0.Other measures of correlation that can be used are Spearman's rank (value pairs follow monotonic function) or Kendall's $\tau$ (measures ordinal association), but they do not apply here. You can also define measures yourself.<jupyter_code># print the correlation matrix
df3 = df2.drop(['time'], axis=1)
cm = df3.corr(method='pearson')
cm<jupyter_output><empty_output><jupyter_text>This tells us that each value is perfectly correlated with itself. We are not interested in the diagonal values and also not in the correlation with time. We also need to get rid of redundant entries. Finally, we need to find the value pairs that exhibit the highest linear correlation. We still want to know if it is positive or negative correlation, so we cannot get rid of the sign.<jupyter_code># get rid of time column, lower triangular and diagonal entries of the correlation matrix
# sort the remaining values according to their absolute value, but keep the sign
upper_triang = cm.where(np.triu(np.ones(cm.shape), 1).astype(np.bool))
upper_triang = upper_triang.stack().reset_index()
upper_triang.columns = ['row','col','val']
upper_triang.sort_values(by=['val'])<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup