
%------------------------------------------------------------------------------
%
%	BEGIN: Paper root
%
%------------------------------------------------------------------------------

%------------------------------------------------------------------------------
%
%	BEGIN: Preamble
%
%------------------------------------------------------------------------------
\documentclass[
	12pt,
	letterpaper,
	oneside,
	notitlepage
]{book}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage[english]{babel}
\usepackage[
	left=1in,
	right=1in,
	top=1in,
	bottom=1.5in
]{geometry}
\usepackage{import}
\usepackage[
	backend=biber,
	sorting=nyt,
	safeinputenc,
	pagetracker,
	style=apa,
	dashed=false,
	ibidtracker=constrict
]{biblatex}
\usepackage[normalem]{ulem}
\usepackage{xcolor}
\usepackage{csquotes}
\usepackage{listings}
\usepackage{times}
\usepackage{lscape}
\usepackage{placeins}
\usepackage{graphicx}
\usepackage{xparse}
\usepackage{fancyhdr}
\usepackage{lipsum}
\usepackage{etoolbox}
\usepackage{setspace}
\usepackage{cancel}
\usepackage[
	unicode=true, 
	bookmarks=true, 
	bookmarksnumbered=false, 
	bookmarksopen=false, 
	breaklinks=true, 
	hidelinks,
	colorlinks=false,
	linkbordercolor={white},
]{hyperref}
\usepackage[
	toc,
	acronym
]{glossaries}

%------------------------------------------------------------------------------
%
%	Custom commands
%
%------------------------------------------------------------------------------
\newcommand{\see}[1]{(See \ref{#1})}
\newcommand{\fsee}[1]{(See Figure \ref{img:#1})}
\newcommand{\lsee}[1]{(See Listing \ref{lst:#1})}
\newcommand{\im}[0]{[emphasis added] }
\newcommand{\obj}[1]{\framebox{{\small\textbf{\texttt{#1}}}}}
\newcommand{\poscite}[1]{\citeauthor{#1}'s (\citeyear{#1})}
\newcommand{\inspire}[1]{{{\small\textit{inhale} (#1)} \par}}
\newcommand{\img}[4]{
\begin{figure}[!htbp]
\centering
\includegraphics[width=#2\textwidth]{/Users/federicocamarahalac/Documents/fd_work/text/waves/bin/img/#1.png}
\caption{#4}
\label{img:#1}
#3
\end{figure}
\FloatBarrier
}


%	MAKE = INTO \LEFTARROW INSIDE LISTINGS

\lstset{columns=fullflexible,
        mathescape=true,
        literate=
               {=}{$\leftarrow{}$}{1}
               {==}{$={}$}{1},
        morekeywords={if,then,else,return}
        }

%	DOUBLE SPACING

\doublespacing

%	ADD SPACE BETWEN PAGE NUMBERS AND LAST LINE

\setlength{\footskip}{1.5\baselineskip}
\setlength{\parindent}{4em}

%	MAKE ALL itemize, quote AND lstlisting ENVIRONMENTS SINGLESPACED

\makeatletter
\AtBeginEnvironment{itemize}{
	\if@nobreak\vspace*{-\topskip}\fi
	\singlespacing}
\makeatother

\makeatletter
\AtBeginEnvironment{quote}{
	\if@nobreak\vspace*{-\topskip}\fi
	\singlespacing}
\makeatother

\makeatletter
\AtBeginEnvironment{lstlisting}{
	\if@nobreak\vspace*{-\topskip}\fi
	\singlespacing}
\makeatother


%------------------------------------------------------------------------------
%
%	GLOSSARY & ACCRONYM ENTRY:	https://en.wikibooks.org/wiki/LaTeX/Glossary
%
%------------------------------------------------------------------------------
\import{/Users/federicocamarahalac/Documents/fd_work/text/waves/bin/glossary/}{definitions.tex}

%------------------------------------------------------------------------------
%
%	Bibliography (bitlatex-biber)
%
%------------------------------------------------------------------------------
\addbibresource{main-paper.bib}

%------------------------------------------------------------------------------
%
%	APA style with ibid (apa-ibid.tex)
%
%------------------------------------------------------------------------------
%https://tex.stackexchange.com/questions/449249/getting-ibid-for-apa-style-citations-from-biblatex

\makeatletter
\providecommand*{\mkibid}[1]{#1}

\newbibmacro*{cite:ibid}{%
  \printtext[bibhyperref]{\bibstring[\mkibid]{ibidem}}}

\renewbibmacro*{cite}{%
  \ifthenelse{\ifciteibid\AND\NOT\iffirstonpage}
    {\usebibmacro{cite:ibid}}
    {\iffieldequals{fullhash}{\cbx@lasthash}
     % Multiple cites in one command
      {\setunit{\compcitedelim}%
       \usebibmacro{cite:plabelyear+extradate}}%
     % Single cite
      {\ifthenelse{\ifnameundef{labelname}\OR\iffieldequalstr{entrytype}{patent}}
     % No author/editor
         {\usebibmacro{cite:noname}%
          \setunit{\printdelim{nameyeardelim}}%
          \usebibmacro{cite:plabelyear+extradate}%
          \savefield{fullhash}{\cbx@lasthash}}
     % Normal cite
        {\ifnameundef{shortauthor}
           {\printnames[labelname][-\value{listtotal}]{labelname}}%
           {\cbx@apa@ifnamesaved
             {\printnames{shortauthor}}
             {\printnames[labelname][-\value{listtotal}]{author}%
              \addspace\printnames[sabrackets]{shortauthor}}}%
         \setunit{\printdelim{nameyeardelim}}%
         \usebibmacro{cite:plabelyear+extradate}%
         \savefield{fullhash}{\cbx@lasthash}}}}%
   \setunit{\multicitedelim}}

\renewbibmacro*{textcite}{%
  \iffieldequals{fullhash}{\cbx@lasthash}
   % Compact cite - more than one thing for same author
    {\setunit{\compcitedelim}%
     \usebibmacro{cite:plabelyear+extradate}}
   % New cite
    {%
    \ifbool{cbx:parens}
      {\bibcloseparen\global\boolfalse{cbx:parens}}
      {}%
      \setunit{\compcitedelim}%
      \ifthenelse{\ifnameundef{labelname}\OR\iffieldequalstr{entrytype}{patent}}
    % No author/editor or patent
       {\iffieldundef{shorthand}%
    % Cite using title
         {\usebibmacro{cite:noname}%
          \setunit{\ifbool{cbx:np}%
                   {\printdelim{nameyeardelim}}%
                   {\global\booltrue{cbx:parens}\addspace\bibopenparen}}%
          \usebibmacro{cite:plabelyear+extradate}}
    % Cite using shorthand
         {\usebibmacro{cite:shorthand}}}
    % Normal cite with author/editor
    % Normal full cite
       {\ifnameundef{shortauthor}%
    % Normal full cite
         {\printnames[labelname][-\value{listtotal}]{labelname}}
    % Cite using short author
         {\cbx@apa@ifnamesaved
           {\printnames{shortauthor}}
           {\printnames[labelname][-\value{listtotal}]{author}}}%
    % Year
        \setunit{\ifbool{cbx:np}
                  {\printdelim{nameyeardelim}}
                  {\global\booltrue{cbx:parens}\addspace\bibopenparen}}%
    % Put the shortauthor inside the year brackets if necessary
        \ifnameundef{shortauthor}
         {}
         {\cbx@apa@ifnamesaved
           {}
           {\printnames{shortauthor}\setunit{\printdelim{nameyeardelim}}}}%
    % Actual year printing
        \ifthenelse{\ifciteibid\AND\NOT\iffirstonpage}
          {\usebibmacro{cite:ibid}}
          {\usebibmacro{cite:plabelyear+extradate}}%
    % Save name hash for checks later
        \savefield{fullhash}{\cbx@lasthash}}}}
\makeatother

%------------------------------------------------------------------------------
%
%	CANCEL DATE COMMAND
%
%------------------------------------------------------------------------------
\date{}

%------------------------------------------------------------------------------
%
%	BEGIN: Document
%
%------------------------------------------------------------------------------
\begin{document}
\mainmatter
\pagenumbering{arabic}

%------------------------------------------------------------------------------
%
%	BEGIN chapter: Databasing Sound: Applications Of Databases In Sound ----
%
%------------------------------------------------------------------------------
\chapter{Databasing Sound: Applications Of Databases In Sound}
\label{section-3}

\glsresetall
\img{mir_comp_sonif_interaction}{1}{
	The arrows between databases (cylinders) and computers (squares) represent data flow. Left: the database is `visibly next' to the computer, as is the case with \gls{mir}; the two bottom arrows indicate the intervention of the human operator. Right: the database is `visibly below' the computer as is the case with Sonification; the database feeds the computer from an external source (right arrow). Middle: the database is `invisibly behind' the computer, within the softwares used for (and as) music works. The arrows in between the practices represent interdisciplinary feedback.
}{Database performance and interdisciplinary feedback.}

Having discussed the current state of new media theory and the theory of databases and data structures, in this section I theorize the use of databases in relation to sound. To a certain extent, ever since the first computers were used to make music the database has been an invisible partner in the music literature. I argue that by shedding some light on this inherent aspect of computers we can arrive at a clearer notion of how databases sound. Particularly, by placing the database along a visibility continuum, we may find a reverse relation with audibility: the more invisible the database, the more present its sound. By this I do not argue in favor of either loudness or quietness. I am only addressing the different possibilities that come from multiple access points to computers. Here I will use the words `database' and `computer' somewhat interchangeably. This decision comes from the fact, as I described in earlier sections, that computers cannot exist without databases. From this, we can further ask ourselves if all computer music is database music. 

As I demonstrate, there are overt and covert uses of the database, but the database is ubiquitous in all computer practices. The various disciplines at the intersection of music and computers take each a different approach to databases and, thus, to database performance. In this sense I describe and discuss the scope of actions that comprise database performance within three practices using computers and sound: \gls{mir}, sonification, and computer music. While there may be different ways of representing the position of the database in relation to each practice, I provide a set of figures throught this section that can be used metaphorically to point to the differences with which each practice interacts with databases \fsee{mir_comp_sonif_interaction}. Furthermore, these positions are only fixed within a graph in the form of snapshots. Thus, in no ways I mean to fix a certain position or to priorize one in relation to another. The purpose of these graphs is to describe a motility inherent to database practice that suggests furhter a way of thinking database performance in a choreographic way. By coreography, I mean the movement that we as `computer musicians,' as `sonificators,' and as music information `retrievers' (in sum, as databasers), perform in the continuous dance of our practices. 
%------------------------------------------------------------------------------
%
%	END chapter: Databasing Sound: Applications Of Databases In Sound
%
%------------------------------------------------------------------------------

%------------------------------------------------------------------------------
%
%	BEGIN section: Music Information Retrieval ----
%
%------------------------------------------------------------------------------
\section{Music Information Retrieval}
\label{mir}

\glsresetall
%you might want to include at some point the idea that databases in general are used to teach computers about the world, and to get them to to do things that humans can do, but not computers.

In \gls{mir}, the database is \textit{in front} of the programmer, \textit{next} to the computer. This practice combines \gls{ir} with Music Theory, and it has been present in academia for a while, most generally within Electrical Engineering departments. The objective of \gls{mir} is to obtain useful information from the analysis of sound signals. That is, \gls{mir} seeks to represent a complex signal with a small number of data points, thus defining a a navigable `information space,' which is, quite literally, the discretized space of the database.

\img{mir}{0.3}{
	The database is visibly next to the computer, and the two bottom arrows indicate the intervention of the human operator.
}{Diagram of database performance in \gls{mir} practices.}

For instance, out of sound file containing millions of samples, information space reduces these points to a database of few `descriptors' that point to certain `features' of the sound file. A descriptor is, in essence, a small amount of data that identifies other larger data. In this case, a feature descriptor relates to the values of a certain characteristics of the analyzed audio file, such as spectral centroid, brightness, flatness, etc. 

Over the 18 years of the \gls{ismir} conference, more than thirty databases of this sort have been publicly created and released, as a means to classify millions of songs and musical genres. This type of database navigation has been used to perform automatic tasks such as categorization for recommendation systems \parencites{Tza02:Mus}{DBLP:journals/corr/abs-0812-4235}{asmita_poddar_2018_1422565}, track separation or instrument recognition, and score transcriptions, among other uses (see below). A recent emphasis in open source database creation has gained momentum \parencite{DBLP:conf/ismir/FonsecaPFFBFOPS17}, such as the \gls{freesound} or \gls{looperman} databases, or \gls{cmam}'s \gls{telemeta}, both collaborative database systems: the first two for general sound file sharing and classification, the latter for ethno-musicological purposes. Audio databases such as \gls{freesound} or \gls{looperman} have been growing exponentially, as well as their use within live performances and interactive systems \parencite{nuno_n_correia_2010_849729}. Automatic audio description and clustering among these databases automatic have improved greatly their usability \parencite{gerard_roma_2012_850102}. \textcite{collins_2015} created open-source software implementing \gls{mir} techniques for navigation, analysis, and classification of the electronic music archive within \gls{ubuweb}. 

Before sound and audio descriptor databases, however, music notation databases have been developed with a variety of file formats \see{applications:notation}. Some examples of these notation databases can be the Polish folk song database in the \gls{esac} format, the electronic library for musical scores \gls{musedata}, the \gls{rism} database, the \textit{Kern Scores} database,\footnote{\url{http://kern.ccarh.org/}} among others. In turn, these databases have been a fruitful area of exploration in Computational Musicology \parencite{DBLP:conf/iciso/Yokl11}, for which toolkits such as \gls{mit}'s \gls{music21} have been developed. Two examples of widely used libraries for audio analysis, classification, and synthesis are \gls{marsyas} \parencite{tzanetakis_cook_2000} and the \gls{essentia} \parencite{DBLP:conf/ismir/BogdanovWGGHMRSZS13}. For a more general overview of \gls{mir} software, see \parencite{DBLP:conf/ismir/BogdanovWGGHMRSZS13}. The different applications of databases are endless and so varied that would extend the scope of this study.\footnote{For example, consider the \gls{mimo} database, a project dedicated to the cataloguing of musical instruments, and how it was used for the statistical tracking of the evolution of the violin based on pattern recognition of its shapes \parencite{2018arXiv180802848P}} Some specific uses that \gls{mir} has given to databases have been:

\begin{itemize}


\item for audio classification and clustering \parencites{ilprints489}{DBLP:conf/ismir/HomburgMMMW05}{marcelo_queiroz_2018_1422585}
\item for genre recognition and classification \parencites{Tza02:Mus}{DBLP:conf/icmc/XuZY05}{DBLP:conf/ismir/SillaKK08}{icmc/bbp2372.2010.003}{DBLP:journals/corr/abs-1803-04652}{DBLP:journals/corr/WangH17a}{DBLP:journals/corr/MitraS14}{2010NJPh:12e3030C}{DBLP:journals/corr/abs-0812-4235}

\item to describe performance expression \parencites{DBLP:conf/ismir/HashidaMK08}{mitsuyo_hashida_2017_1401963}{mitsuyo_hashida_2018_1422503}
\item for emotion recognition and color associations in the listener \parencite{DBLP:conf/ismir/PesekGPSGSPM14}
\item for multimodal mood prediction \parencites{DBLP:journals/corr/abs-1809-07276}{xiao_hu_2014_850795}{humberto_corona_2015_851021}


\item for multi-instrument recognition \parencite{DBLP:conf/ismir/HumphreyDM18}
\item for the evaluation of multiple-source fundamental frequency estimation algorithms \parencite{DBLP:conf/ismir/YehBR07}
\item for contextual music listening pattern detection using social media \parencite{DBLP:conf/ismir/HaugerSKT13}
\item for melody \parencites{ioannis_karydis_2007_849469}{DBLP:conf/ismir/BittnerSTMCB14} or singing voice \parencite{DBLP:journals/corr/abs-1711-00048} extraction


\item for structural analysis \parencite{DBLP:conf/ismir/SmithBFRD11}
\item for schenkerian analysis \parencite{DBLP:conf/ismir/Kirlin14}
\item for harmonic analysis \parencite{DBLP:conf/ismir/DevaneyACN15}
\item for melodic similarity \parencite{goffredo_haus_2005_849297}
\item for forensic analysis as a complement of video analysis \parencite{serizel:hal-01393959}
\item for the evaluation of tempo estimation and key detection algorithms \parencite{DBLP:conf/ismir/KneesFHVBHG15}
\item for tonal music analysis using \gls{gttm} \parencite{DBLP:conf/ismir/HamanakaHT14}
\item for counterpoint analysis \parencite{DBLP:conf/ismir/AntilaC14}


\item to train models for phoneme detection \parencite{DBLP:conf/ismir/ProutskovaRWC12} and music source separation \parencite{marius_miron_2017_1401923}
\item for training and evaluating chord transcription algorithms \parencite{DBLP:conf/ismir/EremenkoDBS18}
\item for training querying methods \parencites{mark_cartwright_2012_850060}{DBLP:journals/corr/Brzezinski-SpiczakDLP13}{DBLP:journals/corr/NagaviB14}{DBLP:journals/corr/abs-1301-1894}{icmc/bbp2372.1999.355} 

% proposed a \gls{mir} system aimed at query-by-content navigation of a musical collection based on melodic segmentation. In their research, they implemented a \gls{lbdm} to perform the automatic segmentation of melodic information taken from \gls{midi} files of Baroque, Classical and Romantic music. They then proceeded to normalize and index the melodic phrases into separate files for querying.

\item for adversarial audio synthesis \parencite{2018arXiv180204208D}
\item for orchestration \parencite{DBLP:conf/ismir/CrestelEHM17}
\item for modeling carnatic rhythm generation \parencite{carlos_guedes_2018_1422615}

\item to create digital libraries \parencite{DBLP:conf/ismir/Dunn00}
\item to store music notation \parencite{DBLP:conf/ismir/Good00}
\end{itemize}

For further reference, the following citations point to different audio databases which have been created over the years: \textcites{DBLP:conf/ismir/GotoHNO02}{DBLP:conf/ismir/GotoHNO03}{DBLP:conf/ismir/WustC04}{DBLP:conf/ismir/MaxwellE08}{DBLP:conf/ismir/Bertin-MahieuxEWL11}{DBLP:conf/ismir/Karaosmanoglu12}{Jaimovich:2012}{Mital:2013}{bbortz:2015}{jjaimovich:2015,Nort2016}{ DBLP:conf/ismir/DefferrardBVB17}{DBLP:conf/ismir/VigliensoniF17}{DBLP:conf/ismir/Meseguer-Brocal18}{DBLP:conf/ismir/DonahueMM18}{DBLP:conf/ismir/XiBPYB18}{DBLP:conf/ismir/WilkinsSWP18}.
%------------------------------------------------------------------------------
%
%	END section: Music Information Retrieval
%
%------------------------------------------------------------------------------

%------------------------------------------------------------------------------
%
%	BEGIN section: Sonification ----
%
%------------------------------------------------------------------------------
\section{Sonification}
\label{sonification}

\glsresetall
\img{sonif}{0.3}{
	The database is visibly below the computer, and it feeds the computer from an external source represented by the right-most arrow.
}{Diagram of database performance in sonification practices.}

The database is the ground floor of sonification. The sonified data is very likely to be digital,\footnote{There are cases where sonification is entirely analog, such as the first sonification tool ever created: the Geiger counter} which means that data needs to be stored in a structured way for fast access by computers, and the role of the sonifier is to acoustically translate the database's inner relationships \parencite[9]{WalkerNees2011-TOS}.

According to \textcite{WalkerNees2011-TOS} there are three types of sonification: event-based, model-based, and continuous. I see these types of sonification as ways of performing a database. Continuous sonification (audification) consists of directly translating waveforms of periodic data into sound, that is, reading non-audio data as if it were audio data \parencite[17]{WalkerNees2011-TOS}. Model-based sonification consists of distributing data points in such a way that enables data exploration. Generally, these models are interactive interfaces with which users navigate the database to find relationships \parencite[17]{WalkerNees2011-TOS}. Event-based (parameter mapping) sonification is aimed at representing changes in a database as acoustic saliences or tendencies. In this sense, dimensions of the data need to be translated (mapped) into acoustic parameters (frequency, periodicity, density, etc.), so as to listen how the generated sound behaves over time and interpret these changes within the database \parencite[16]{WalkerNees2011-TOS}.

Sonification depends on databases, on the interaction between databases, and on their traversing, but also on the human body's perceptual limits. In sonification, the data comes first, and it needs to be pre-processed so that it can be adapted to the sound synthesis engines of choice. Sonification is a subset of auditory display techniques, and it belongs to the broader scope of information systems and visualization practices \parencite[10]{WalkerNees2011-TOS}. Therefore, since sonification belongs to the process of information, as a practice it has taken into account the auditory system's ability to extract biologically relevant information from the complex acoustic world \parencite{Carlile2011-P}. What this emphasis on sound perception and cognition abides to, however, is the fact that there is no one-to-one correspondence between sound parameters (frequency, amplitude, spectral content) and how these are perceived (pitch, loudness, timbre). Therefore, the success of a sonification is the result of the play between, on the one hand a rigid link between data and sound, and on the other, the perceived acoustic relations. From this interplay of relations is how information can be obtained from data. In other words, in sonification practices there is no communication unless the data has been acoustically shaped, and perceived as information (\textit{in}-formed) by the listener. 

In what follows, I present some instances of sonification practices as described by their authors.

\subsection{Parameter mapping}
\label{sonification:parametermapping}

\paragraph{DOW}
\textcite{icmc/bbp2372.1996.085} sonified the Dow Jones financial stock market data with Csound. Since the Csound program depends on two separate files (orchestra and score), they implemented another program to control the data flow. Within this second program, the Csound score was automatically generated based on a `configuration' file which was used to map the `data file' holding the stock market data, as it was read in separate window frames into the Csound-formatted score.\footnote{Other examples of stock market sonification include Ciardi's set of tools for downloading and sonifying real-time data, see \textcite{icmc/bbp2372.2004.124}; and <NAME>'s research on telematic performance, see \textcite{icmc/bbp2372.2014.046}}

\paragraph{Medical Images}
\textcite{DBLP:conf/icmc/CadizCMMATI15} proposed a sonification approach based on statistical descriptors of \gls{roi} selected from medical images. In their study, they focused on enhancing breast cancer symptom detection in mammograms by mapping statistical descriptors, such as mean, minimum, maximum, standard deviation, kurtosis, skewness, among others, to different synthesis techniques in various ways. They then surveyed the usefulness and pleasantness of the sonifications to different subjects in order to better adjust the technique to the task. What is novel of their approach is on the creative use of statistical curves obtained from pixel distributions within computer music techniques.

% \paragraph{Geolocations}
% \citeauthor{icmc/bbp2372.2010.002} \parencite{icmc/bbp2372.2010.002} created an interface called \gls{compath} which enabled users to draw paths on a map, sonifying data points which represented information (e.g., traffic, weather, culture events) along the points of the path. They used major commercial web services such as Amazon, Google, and Yahoo that offered public \glspl{api} so that each geo-location inputted on the path acted as a database query. The returned data from these services was then mixed together and mapped with a virtual synthesized via \gls{midi}.

\subsection{Model-based sonification}
\label{sonification:model}

\paragraph{Space}
One example of model-based sonification is the \textit{Data Listening Space} installation by the \gls{qcd-audio} project at the \gls{iem} of the University of Music and Performing Arts in Graz \parencite{icmc/bbp2372.2012.096}. Within this installation, they proposed a three dimensional, navigable space holding a Monte Carlo simulation of the theory of \gls{qed}. Within this \gls{qed} \textit{lattice}, a walking participant holding sensors ---$x$, $y$, and $z$ coordinates--- could explore the simulated data by way of sonification.

\subsection{Artistic sonification}
\label{sonification:artistic}

\paragraph{Wolves}
\textcite{Kle98:The} composed a piece called \citetitle{Kle98:The}, using a set of recordings she took along the Bays Mountain Park in Kingsport, Tennessee, for a period of six months. In this period she researched the sonic activity of a pack of wolves, and in her recordings she achieved a level of intimacy with the pack that translated into the recordings, and resulted in a strong animal rights activism \parencite{Kle17:Lec}. Therefore, her compositional choice was to treat the sound file in a non-destructive and non-intrusive way: ``for the composition I used the Csound computer music language. All of the sounds came from the recordings, in unaltered or slightly modified form as the source material in musical settings and transitions'' \parencite{Kle98:The}. Thus, by analyzing spectral contours of extremely precise frequency bandwidths of the data and resynthesizing into the soundscape in almost unnoticeable ways, she sonified a space in between the wolves. This space invites the listener into a space of action, and to reflect on human activity itself and how it always returns to resonate with the wolves.

\paragraph{Selva}
\textcite{icmc/bbp2372.2000.123} composed an electroacoustic work called \citetitle{Bar20:Viv} \parencite{Bar20:Viv} using 14-hour long recordings taken with an array of four microphones from a a biological field station called \textit{La Suerte} in Costa Rica. From these recordings, she extracted location (by difference in arrival time) and timestamps (by manual logging) of different animal sounds, and long-term energy distribution in various frequency bands, to describe various environmental sounds such as airplanes, wind, insects, etc. While the spatio-temporal data of the animal sounds was used for sound spatialization of sounds within the electroacoustic work, the long-term energy distribution was scaled down to 20 minutes so as to constitute the form of the piece.

\paragraph{Ocean}
\textcite{icmc/bbp2372.2002.056} sonified ocean wave conditions of the USA Pacific coast obtained by the \gls{cdip} since 1975. The database until 2002 contained over 50 \gls{gb} of spectral and directional content of the wave-driven motions at the location of the sensing buoys. By scaling to hearable range and then performing an \gls{ift} of the data, Sturm composed a piece called \textit{Pacific Pulse}, on which frequency sweeps indicate storms beginnings (rising) and endings (falling).

\paragraph{Molecules}
% \paragraph{Rivers and Molecules}
Falk \textcite{icmc/bbp2372.2016.002} composed \textit{Spin Dynamics} using molecular sonification by two audification processes (direct audification and via a straightforward additive synthesis process) applied to the \gls{hmdb}, a database holding \gls{nmr} spectroscopies of molecules.

% \footnote{\url{https://soundcloud.com/falk-morawitz/spin-dynamics-stereo-reduction}}

% \citeauthor{icmc/bbp2372.2014.065} \parencite{icmc/bbp2372.2014.065} sonified river data as a multimedia collaboration.

\paragraph{Gender Distribution}
Emma \textcite{Fri17:Son} derived a database of gender distribution by applying the python module \texttt{genderize} to author names in three main computer music conference proceedings databases: \gls{icmc}, \gls{nime}, and \gls{smc}. By assigning polar frequency ranges for each group (male and female), her sonification emphasizes the significant inequality of gender in the resulting acoustic stream segregation into male background (continuous drone-like sound) and female foreground (fewer and sparser sounds). Her conclusion, therefore, is that ``there is a need for analysis of the existing environments and social relations that surround music technology and computer music. If we identify the challenges that women are facing in our research community, we will be able to create more initiatives towards changing practices'' \parencite[238]{Fri17:Son}.

\subsection{Sonification Installations}
\label{sonification:installations}

% 
% Interesting example. Ballora has also worked with a rock musician
%  on sonifications for musical applications. 
%  I forget the project details at the moment, but the collabortion is ongoing.
% 

\paragraph{IP-based soundscape}
\textcite{icmc/bbp2372.2010.117} sonified a database of \gls{http} requests at Penn State's \gls{nc2if}. This database contained entries with four fields such as timestamp, location (latitude-longitude), \gls{ip} address, and response type. Using parameter mapping, Ballora controlled rhythm and spatialization with the first two, and pitch and timbre with \gls{ip} data. However, the latter ranged from the more concrete (\gls{ip} to frequency) to the more abstract (\gls{ip} as formant and high-pass filters for brown noise), thus resulting in a soundscape with different but simultaneous sonifications of the data. This multi-layered approach to sonification stems from his PhD dissertation on cardiac rate sonification \parencite{Ballora/2000/phdthesis}.

\paragraph{Earthquakes}
\textcite{icmc/bbp2372.2017.033} sonified real-time earthquake data as a sound sculpture. Within \citetitle{icmc/bbp2372.2017.033}, he used data from the \gls{iris} Data Services, which transmits seismographic data packets updated every thirty minutes from multiple observation sites. He spatialized this data using coordinates of the events and using a four-speaker array located at the center of the gallery space, and mapped the rest of the data to \gls{fm} synthesis parameters.

\paragraph{GPU-based waveforms}
\textcite{icmc/bbp2372.2016.056} proposed a novel way to generate waveforms by populating an array using vertex data obtained from the \gls{gpu}. In order to carry this out, they used the Metal API\footnote{Apple's built-in framework to interface with the \gls{gpu}. See \url{https://developer.apple.com/documentation/metal}}, and intervened on the processing pipeline to output \gls{cpu} accessible data. The audio engine running on the \gls{cpu} was able to interpret as waveforms the values of the vertex and fragment shaders, thus sonifying the position data related to a rendered shape and the pixel values respective to its display. Therefore, they obtained simultaneous visualization and audification of the rendered three dimensional shape. In their installation \textit{The Things of Shapes}\footnote{\url{https://vimeo.com/167646306}}, they used the generated waveforms as a database, composing each waveform together with their visual generators as a collage.

\paragraph{Uncanny Faces}
\label{hearing_the_self}
\textcite{fdch/installation/spectral} designed \textit{Hally}, an installation based on face tracking and real-time sonification of spectral features present in both pixel information containing the face, and the $x$ and $y$ coordinates of the moving data points of the face mesh used for tracking. Furthermore, by video-based audio convolution, \textit{Hally} aims to simulate a theory of perception based on \gls{ift} \parencite{connes:shapes}. Parting from previous work by \textcite{Sch07:How} on simultaneous sonification and visualization, \textit{Hally} explores the role of both sound and image in the definition of the self, by immersing the participant in an uncanny spectrality \parencite{fdch/papers/spectral}. Among other sounds, there is one drone-like sound produced throughout the installation that comes out of a set of programming decisions which bring into surface certain aspects of the software used in the piece. The \gls{dsp} within Pure Data is scheduled in blocks of samples that the user can set. A certain combination of block size, overlap, and sample rate results in a frequency, in this case, of about 43 Hz \parencite[3]{fdch/installation/spectral}. In fact, these combinations around block size result in a set of octave-spaced pitches that are tethered to Pure Data's only possible block sizes ($2^n$, at least in audio computations). In this sense, when making these pitches audible, the listener can recognize a certain `sound' proper to the internal memory management and computation rate of the software. In this installation, it is the human participant that enters as a filter to the spectrum of this `sound' by the position of their faces on the camera sensing area. Thus, participants change this `sound' with the unique ways in which they behave in front of the sensor.

% \subsection{Affective Sonification}
% \label{sonification:affective} 

% \paragraph{Telematic Electroacoustic Music}
% <NAME> \parencite{icmc/bbp2372.2014.046} carried out an embodied approach to data sonification. For Whalley, the role of the human body when it concerns telematic electroacoustic concerts has been traditionally left out. Because of this disembodied limitation of telematic concerts, he derived a set of mapping rules out of a set of embodied responses in listening tests. Thus, he designed parameter mappings for sonification of real-time stock market data basing parameter constraints on actual human performers ---musicians who were used as baseline for an affective sonification mapping. 
% 
% 



\subsection{Sonification Software}
\label{sonification:software}

\paragraph{SonArt}
Originally intended for sonification purposes, \gls{sonart} \parencite{icad/2002/ben-tal} was an open-source platform that enabled users to map parameters to sound synthesis, and later \parencite{icmc/bbp2372.2004.128} to obtain cross-correlated image and sound synthesis. In other words, users were able to easily translate a database into sound parameters, or image and sound data into one another. The program acted in a modular way, that is, it was networked with other software via \gls{osc} connections. This software enabled \textcite{DBLP:conf/icmc/YeoB05} to generate novel image sonifications, by combining two methods of sonification into one interface: sonified data in a fixed, non-modifiable order (\textit{scanning}) and sonified selected data points (\textit{probing}).

\paragraph{DataPlayer}
In his \gls{caddc} environment called \textit{DataPlayer} programmed as a standalone \gls{max/msp} application, \textcite{icmc/bbp2372.2015.072} sonified data from the \gls{aflowlib}. His sonification intent was aimed towards data navigation by means of a unique mapping that would convey an overall trend (a gist) of each material compound. Furthermore, this environment allowed for artistic remixing and exploration of the sonification procedures, simultaneously touching on the scientific and the artistic uses of the environment.

\paragraph{madBPM}
\textcite{icmc/bbp2372.2017.087} devised \gls{madbpm}, a data-ingestion engine suitable for database perceptualization, that is, sonification and visualization. This modular C++ software platform enables data loading from \gls{csv} files, multiple mapping via tagging, several traversing algorithms and units, and networked connectivity to SuperCollider for sound and \gls{ofx} for visual output. Their approach is innovative since they provide features for database behaviors. By `behavior' they mean ways of structuring, traversing and perceptualizing the database. These behaviors define the dual purpose of the software: finding relationships among the inputted data and interpreting them artistically. Furthermore, users can structure and re-structure potentially any type of data set \parencite[504]{icmc/bbp2372.2017.087}. However, in order to design new behavior objects the user needs to implement them in the source code and compile them. Thus, besides real-time data streaming and networking functionality, in their future work the authors aim at designing a \gls{dsl} that would enable extending the functionality of these behaviors in real-time.

For further sonification software, see \gls{sondata} and the following references: \textcites{Wil96:Lis}{Pau04:ATo}{Lod98:MUS}{Bei09:Aes}{Her14:Aso}{DBLP:conf/icad/2007/Worral}{DBLP:conf/icad/2003/Walker}{domenico_vicinanza_2006_849321}


%------------------------------------------------------------------------------
%
%	END section: Sonification
%
%------------------------------------------------------------------------------

%------------------------------------------------------------------------------
%
%	BEGIN section: Computer Music ----
%
%------------------------------------------------------------------------------
\section{Computer Music}
\label{computer_music}

\glsresetall
Computer music software is computer music's playground. Composing and programming blend into different forms of play that can be understood by a closer look at the playground's design. A key aspect of software design is delimiting constraints to data structures. The first choice is generally the programming language, after which the database tree unfolds its way up to the leaves. Among these leaves is where computer music programs reside. At this level of `leaves' software users are certainly aware that there is a `tree' in front of them. However, their awareness does not necessarily extend to the branches, trunk, or roots of the tree. There is endless music that can be made with leaves just as it can with paper. However, neither music quantity nor music quality are the point here. My argument is that working with data structures changes how we think and perform music making. I claim that composers using these leaves of computer music software are working indirectly with data structures, and unless they engage with programming, they remain unaware of data structures and their constraints. `Indirectly,' because the twigs and branches connect the leaf to the trunk, but these connections become invisible to the non-programmer composer \textit{by design}. Like a phantom limb of the tree, the database remains invisibly \textit{behind}. In this section, I present different approaches from composers and programmers that show how music concepts change with the presence and performance of the database. By database performance I mean neither the quality of musical output, nor the dexterity of the programming activity. Database performance in music composition is the activity of the databaser: databasing to make music.

\img{comp}{0.2}{
	The database is invisibly behind the computer, within the softwares used to create musical works. 
}{Diagram of database performance in computer music practices.}

\subsection{Hierarchical environments}
\label{computer:sssp}

\begin{quote}
	One of the most important aspects in the design of any computer system is determining the basic data types and structures to be used\dots we have been guided by our projection of the interaction between the tool which we are developing, and the composer. \parencite[119]{icmc/bbp2372.1978.012}
\end{quote}

\paragraph{Reducing cognitive burden}
In <NAME>'s survey of computer music practices \parencites{Bux77:Aco}{icmc/bbp2372.1978.012}{DBLP:conf/icmc/BuxtonPRB80}, he distinguished between \textit{composing programs} and \textit{computer aided composition}, arguing that they had both failed as software, the former on account of their personalization and formalization, and the latter on their lack of interactivity. On his later interdisciplinary venture called \gls{sssp}, he focused on \gls{hci} ---a field in its very early stages in 1978---\footnote{<NAME> is now considered a pioneer in \gls{hci}, and he is now a major figure in the Microsoft Research department.}. Buxton's concern throughout his work on \gls{sssp} was to address the``problems and benefits arising from the use of computers in musical composition'' \parencite[472]{DBLP:conf/icmc/BuxtonFBRSCM78}. His solution to the problems was to reduce the cognitive burden of the composer, who ``should simply not have to memorize a large number of commands, the sequence in which they may be called, or the order in which their arguments must be specified'' \parencite[474]{DBLP:conf/icmc/BuxtonFBRSCM78}. He argued that reducing the amount of information given to composers helped them focus on music making. Therefore, in \gls{sssp}, the composer's action was reduced to four main selection tasks: timbres, pitch-time structure, orchestration, and playback. Timbres were assigned by defining waveforms for the table lookup oscillators, and pitch-time structure consisted on pitches and rhythms on a score-like \gls{gui} program called \gls{scriva} \parencite{youtube/buxton10}. Orchestration consisted in placing the previously chosen timbres on the score, and playback meant running the score or parts of it. With this simple but very concise structure, stemming from a somewhat dated programming philosophy in relation to audio software, Buxton delimited the scope of action of the composer

\paragraph{A Hierarchical Representation}
\textcite{DBLP:conf/icmc/BuxtonFBRSCM78} based their research on differing approaches to composition: <NAME>'s score-as-entity approach in his 1971 \textit{Formalized Music}, an unpublished 1975 manuscript by <NAME> at \gls{mit} studio for Experimental Music, where Buxton found a note-by-note approach, and <NAME>'s computer music systems \parencite{Tru73:The} which was, for Buxton, located somewhere in between the first two but did not provide a solution for ``the problem of dealing with the different structural levels of composition ---from note to score'' \parencite[120]{icmc/bbp2372.1978.012} \see{computer:balance}. Buxton, however, condensed these different approaches into what he called a ``chunk-by-chunk'' composition, where a `chunk' represented anything from a single note to an entire score, and thus reframed the question of a compositional approach as one of scale. For Buxton, ``the key to allowing this `chunk-by-chunk' addressing lies in our second observation: that the discussion of structural `levels' immediately suggests a hierarchical internal representation of scores'' \parencite[120]{icmc/bbp2372.1978.012}. That is to say, his solution for the scalability problem relied on a hierarchical representation of scores. 

In Buxton's \gls{sssp}, the hierarchical design depended on a data structure called \textit{symbol table}, which he subsequently divided into two objects called \texttt{score} and \texttt{Mevent} (musical events). The \texttt{score} structure had a series of global fields (variables) together with pointers to the first (head) and last (tail) \texttt{Mevent}s. In turn, \texttt{Mevent}s had local fields for each event together with pointers to the next and previous \texttt{Mevents}, so as to keep an ordered sequence \see{computer:linked} and enable temporal traversing of the tree. In turn, \texttt{Mevents} could have two different types: \texttt{MUSICAL\_NOTE} and \texttt{Mscore}, the former relating to terminal nodes editable by the user ---what he referred to as `leaves' of the tree structure---, and the latter consisting of nested \texttt{score} objects that added recursivity to the structure. Buxton's model was thus hierarchic (a tree structure) implemented in nested and doubly-linked symbol tables.

\textcite{icmc/bbp2372.1978.012} gave a detailed exposition of the data structures and their functionality. Buxton's general purpose in his \gls{hci} philosophy was to make the software work in such a way that it became invisible or transparent to the user. This is also known as a black-box approach. His innovations in this and other projects have had enormous resonances in computer science, and the concept of reducing cognitive burden of the user has developed as a standard of \gls{hci} \parencite{youtube/buxton16}.

\paragraph{Black-boxing}

Media theorist Vílem \textcite{Flu11:Int} proposed the term `envision' to describe a person's power to visualize beyond the surface of the image, and to bring the technical image into a concrete state of experience. The `image,' in Flusser's case is the television screen in its abstract state of ``electrons in a cathode ray tube.'' Therefore, he argues, ``if we are asking about the power to envision, we must let the black box remain ---cybernetically--- black'' \parencite[35]{Flu11:Int}. By seeing past the abstract quality of media we bring an image into experience. The black box allows envisioning to take place. In a similar way, by seeing past the hidden complexities of the software, composers are able to create music with unrestrained imagination. However, as I have shown before, Hansen makes a divergent point claiming that the virtuality inherent in the body is the creative potential of image \textit{in-formation} \see{embodiment}. 

Understanding the process of information as the experience of technical images, it follows that virtuality and envisioning can be considered complementary. On one hand, there is the technical device, whose multidimensionality is as complex as it is hidden from the envisioner. On the other, the human body with its capacity to create and embody. Flusser's point is, however, paradoxical: ``The envisioner's superficiality, to which the apparatus has \textit{condemned} him and for which the apparatus has \textit{freed} him, unleashes a wholly unanticipated power of invention'' \im \parencite[37]{Flu11:Int}. Therefore, the black-box is what condemns and frees the envisioner to a state of superficiality. However, Flusser continues, ``envisioners press buttons to inform, in the strictest sense of that word, namely, to make something improbable out of possibilities'' \parencite[37]{Flu11:Int}. In other words, Flusser justifies the invisibility of the technological device in favor of its most useful consequence, that is, its ability to make the user create something ``out of possibilities.'' Composers, therefore, are often given these possibilities to create, at the cost of a restricted creation space.

\paragraph{Generality and Portability}
\label{computer:free}

\begin{quote}
	Music data structures must be general enough so that as many styles of music as possible may be represented. This implies that the data structures (or the application's interface to them) should not enforce a musical model (such as equal temperament) that is inappropriate for the musical task at hand. \parencite[318]{icmc/bbp2372.1987.046}
\end{quote}

The \gls{sssp} lasted until 1982 due to lack of funding, and in the mid-1980s its research re-emerged with the work of \textcites{DBLP:conf/icmc/FreeV86}{icmc/bbp2372.1987.046}{DBLP:conf/icmc/FreeV88}, under Helicon Systems' \gls{camp}. Free's programming philosophy called for generality, portability, and simplicity. Due to \gls{sssp}'s many hardware dependencies, the code had to be completely re-written \parencite{DBLP:conf/icmc/FreeV86}. A crucial aspect of Free's programming concerns was portability \see{portability}, which moved him to create higher levels of software abstractions, so that software continued to live on in newer hardware. Free also developed \gls{scriva}, \gls{sssp}'s \gls{gui} program into extensible data structures for music notation arguing that software had to be general enough so that composers could work in multiple styles. The larger implication in Free's argument is that enforcing musical concepts in data structures limits the style that the program can achieve. Therefore, if the program fails to provide a certain level of generic functionality, the composer's output will be modelled by the data structure. On the one hand, it can be argued that this implication is simultaneously overestimating the agency of the database and underestimating that of the composer. In any case, the database works for the composer by taking care of the more tedious task. The cost of this, nonetheless, is that by working for the composer, the database guides the composer through certain paths while hiding other paths.

\paragraph{Simplification}
\label{computer:vanilla}

Hardware-independence led Free to imagine a general purpose, or \textit{vanilla} synthesizer, with which students in ``a music lab with multiple users on a networked computer system'' \parencite[127]{DBLP:conf/icmc/FreeV88} could seamlessly use the timbre world offered by various synthesizers made by different manufacturers. Free created a database that enabled simultaneous interaction among different types of hardware. The \textit{Music Configuration Database} consisted of an intermediate program between the physical \gls{midi} input devices (such as the Yamaha DX7 or Casio CZ101), and the computers in the network, so that ``rather than have the user tediously specify the \gls{midi} device properties for each synthesizer'' \parencite[133]{DBLP:conf/icmc/FreeV88} (channel management, control mapping, etc), these processes were handled by an intermediary database. Free's approach, in comparison to Buxton's, was not entirely black-boxed, since the database was open to modification by a specific set of commands provided to the user. The user could edit the database with a library of database access subroutines such as open/close, create/delete items, querying fields/keys, and loading/storing property items. With this library, Free simultaneously simplified user's interaction and reduced the ``chance of corrupting the database'' \parencite[137]{DBLP:conf/icmc/FreeV88}. 

\paragraph{Balance}
\label{computer:balance}

\img{truax_generality_b}{0.7}{
	<NAME>' ``Inverse Relation Between Generality and Strength'' \parencite[51]{Tru80:The}. Another version of this graph can be found in \parencite[38]{laske_otto_1999}.
}{Generality vs. Strength}

\begin{quote}
	\dots all computer music systems both \textit{explicitly and implicitly embody a model of the musical processes that may be inferred from the program and data structure of the system}, and from the behavior of user working with the system. The inference of this model is independent of whether the system designer(s) claim that the system reflects such a model, or is simply a tool. \im \parencite[230-231]{Tru76:ACo}
\end{quote}

\textcites{Tru73:The}{Tru76:ACo}{Tru80:The}[Chapter~8]{Emm86:The} often compared grammatical structures of natural language to the structures of computer music systems, claiming that in both cases one can find certain constraints and facilitations for thought \parencite[156]{Emm86:The}. Arguing for balance between generality of applicability and strength of embedded knowledge within models for computer music systems \fsee{truax_generality_b}, he writes:

\begin{quote}
	In a computer music system, the grouping of data into larger units such as a sound-object, event, gesture, distribution, texture, or layer may have a profound effect on the composer's process of organization. The challenge for the software designer is how to provide powerful controls for such interrelated sets of data, how to make intelligent correlations between parameters, and how to make such data groupings \textit{flexible according to context}. \im \parencite[157]{Emm86:The}
\end{quote}

Truax's notion of balance speaks of a `meeting halfway' between the system and the user regarding the programmer's capability to embed a more complex conception of hierarchy in the system. What provides this balance is a certain flexibility among data structures which would enable them to adapt to the different hierarchical contexts with which music is understood. That is to say, since data structures can embody models of musical processes, they have an effect on the composer's overall performance of the database, and by extension, on the resulting music. 

\subsection{Music Notation Software}
\label{applications:notation}

Music representation has occupied an important area of research within the programming community. Formats and specifications such as \gls{midi}, \gls{musicxml}, the Humdrum \texttt{**kern} data format \parencite{DBLP:conf/ismir/Sapp05}, \gls{guido}, to name a few, have appeared over the years in conjunction with music engraving software. An extensive guide on musical representations can be seen in \textcite{Selfridge-Field:1997:BMH:275928}. In this section, I point to certain aspects of music notation software development that reveal different approaches towards data structures, and the possibilities that arise henceforth.

\paragraph{DARMS and SCORE}
Two major programs were developed during the 1960s and 1970s: <NAME>berg's \gls{darms} project for music engraving which started in 1963 \parencites{icmc/bbp2372.1983.002}{10.2307/30204239}, and <NAME>'s  \gls{score} \parencite{smith1971}. Both of these programs worked first in mainframe computers and were used for music printing and publishing. At first, \gls{score}'s character scanner was designed to interpret complex musical input into \gls{music-v} output, thus acting as an link between music notation and computer music synthesis. However, with the appearance of vector graphics in the 1970s it shifted solely to music printing.  With the appearance of the PostScript format in the 1980s, it became commercially available thus becoming one of the earliest music engraving softwares still in use today by major publishing houses \parencite{scoremus}. 

\paragraph{From Staves to Speakers}
Other programming approaches stemming from \gls{darms} and \gls{score} were developed during the 1980s. \textcite{icmc/bbp2372.1980.020} joined together the \gls{darms} data structures with those used in \gls{music-v} in a first attempt to obtain sonic feedback out of a notation system. Clements' attempt was nonetheless overshadowed by \gls{score}'s success. Later, \textcite{icmc/bbp2372.1987.045} worked on an interface to the \gls{darms} language called the \textit{Note Processor}, which became one of the earliest commercially available music notation systems. Dydo's data structures, however, were not publicly released when he presented his software at the \gls{icmc} in 1987. He later released it commercially in the early 1990s at a significantly lower price than other notation software such as \textit{Finale} which is still available today by MakeMusic, Inc. \parencites{10.2307/941442}{10.2307/940555}. \textcite{icmc/bbp2372.1981.018} modeled the \gls{score} input format into \textit{Score-11}, adapting it to <NAME>'s \gls{music-11}. Written in Pascal, \textit{Score-11} used circular linked lists traversed by an interpreter to produce \gls{music-11}-formatted output. The user creates a text file with blocks dedicated to individual instruments and specifies parameters such as rhythm, pitch, movement (glissandi, crescendo), amplitude, etc. These parameters are then re-formatted to fit the less musically-oriented notation of the \gls{music-n} programs. Brinkman argued that such a software would result in faster and less arduous performance on the composer's end: ``a crescendo over several hundred very short notes requires several hundred different amplitude values representing the increasing volume. \textit{Typing in several hundred note statements each with a slightly larger amplitude number would take forever}. If the computer could be instructed to gradually increase the amplitude value over twenty seconds then \textit{life would be much simpler}'' \im \parencite{score11manual}. Brinkman emphasized on the program's extensibility by users, inspiring <NAME>'s recent \textit{nGen} program \parencite{csoundMethods}, a version of Brinkman's program for the current Csound. \textcite{icmc/bbp2372.1983.002} later designed an interpreter for the \gls{darms} language, which became useful for obtaining computable data structures for automated music analysis \parencite{icmc/bbp2372.1984.033}. Another approach to music notation was carried out at \gls{ccrma}, when \textcites{icmc/bbp2372.1988.020}{10.2307/3680043} devised a ``pure structure'' devoted to the ``hierarchical organization of musical objects into musical scores:'' the \textit{TTree} \parencite[184]{icmc/bbp2372.1988.020}. Stemming from his PhD research on formal languages in music theory \parencite{diener1985}, this data structure was based in the hierarchic structures of the \gls{sssp} project. The change Diener introduced to these structures was their capability of sustaining links between not only the previous and the next data records, but to the `parent' or `child' data records to which it was related. This is known as `inheritance,' and it enabled ``any event in the [structure] to communicate with any other event'' \parencite[188]{icmc/bbp2372.1988.020}. While Diener implemented this data structure in the object-oriented programming language \gls{smalltalk}, he later developed it into \textit{Nutation} \parencite{DBLP:conf/icmc/Diener92}, a visual programming environment for music notation. \textit{Nutation} was written in \gls{objective-c}, and it combined the previously developed \textit{TTree} structure with glyphs and a music synthesis toolkit called \textit{Music Kit} that the \gls{next} computer provided. This resulted in an extremely malleable \gls{cac} environment, which enabled fast manipulation and sonic feedback at the cost of limiting timbre to a predefined, hardware-specific set of digital instruments. 

\paragraph{Theoretical Performance}
What notation software is most often criticised for is the way in which sonic feedback often comes to be equiparated to (human) music performance. When <NAME> presented \gls{score} as ``not a `performer's' instrument, but rather a `musician's' instrument,'' for example, he claimed that ``theoretically, any performance, clearly conceived in the mind, can be realized on [the computer]'' \parencite[14]{smith1971}. It is indeed a fact that computers can offer automated tasks to an unimaginable extent. However, to translate this type of automation into music composition and performance, results in a disembodied music conception. In other words, an algorithmically generated stream of notes may result in physically impossible tasks for a performer, or for the listener. This is the point of inflexion when envisioning goes beyond the threshold of embodiment. It can be argued, however, that further developments in musical performance techniques can be achieved by pushing the limits of bodily skills. Nonetheless, what I am stressing here is the extent to which music composition can be reconfigured by the possibilities data structures have brought to the field. Furthermore, what is at stake with notation-based music software is yet another musical concern that governed most of music software development during the 1980s: style.

\subsection{Enter Objects}

\img{realtime}{0.9}{
A bodiless abstract published at the \gls{icmc} (1981) stating that a real-time version of \gls{music-11} was ``near completion'' by a group at MIT \parencite{DBLP:conf/icmc/PucketteVS81}.
}{A real-time version of \gls{music-11}.}

\paragraph{Max}
\label{computer:real-time}

Faster, cheaper, and portable microcomputers with real-time capabilities for audio processing began to appear onstage within institutions such as \gls{mit} and \gls{ircam}, and a growing interest among composers and programmers circled around real-time computer music software \fsee{realtime}. Towards the end of the 1980s, after the proliferation of \gls{midi} \parencite{Loy85:Mus}, composers were already incorporating real-time techniques within musical instruments and software \parencites{Ver84:The}{Puc91:Som}. This is the context for Miller Puckette's development of \gls{max} for the 4X real-time audio processor at \gls{ircam} \parencite{DBLP:conf/icmc/Puckette86}. With an emphasis on time and scheduling, Puckette devised a new approach towards complexity in computer music software:

\begin{quote}
	\dots complexity must never appear in the dealings between objects, only within them. Three other features currently in vogue seem unnecessary. First, there is no point in having a built-in notion of hierarchy; it is usually a hindrance. Second, I would drop the idea of continuously-running processes; they create overhead and anything they do can be done better through [input, output] related timing. Third, there should be few defaults. Rather than hide complexity I would keep it visible as an incentive to avoid it altogether. \parencite[43]{DBLP:conf/icmc/Puckette86}
\end{quote}

Puckette keeps complexity ``visible'' within the concept of the programming \textit{object}. Furthermore, he removes the notion of hierarchical programming proposing a light-weight, on-the-spot programming practice based on discontinuous processes: ``the scheduler keeps the runnable-message pool in the form of a separate queue for each latency'' \parencite[46]{DBLP:conf/icmc/Puckette86}. In other words, the structure of the database was placed \textit{horizontally} along the time axis, and Puckette's efforts were dedicated to optimizing the internal timing of audio processes. Specifically, linked lists are used to keep track of the order of processes that are run, and each process is scheduled according to its own temporality (latency). Thus, the entire network of processes that can be run is maintained in a dynamic list (stack) that can be changed at any time by adding or removing elements (push/pop). The way in which these processes (methods) are called is by messages that can be sent (input/output) by the user or objects themselves.\footnote{``The scheduler always sends the first message in the lowest-latency non empty queue. When the associated method returns the scheduler sends another message and so on. The only situation in which we need to interrupt a method before it is done is when I/O (including the clock) causes a lower-latency message to appear\dots In this case the scheduler causes a software interrupt to occur by pushing a new stack frame onto the stack and executing the lower-latency method. When this method returns \dots we pop the stack back to the prior frame at latency \(d_2\) and resume the associated method'' \parencite[46]{DBLP:conf/icmc/Puckette86}.} In sum, the object-oriented paradigm was thus applied to the scheduling system, resulting in a ground-breaking implementation that changed the real-time computer music performance scene: ``\dots rather than a programming environment, \gls{max} is fundamentally a system for scheduling real-time tasks and managing intercommunication between them'' \parencite{DBLP:journals/comj/Puckette02}.

\paragraph{Kyma}
\label{computer:kyma}

Another powerful example of an object-oriented language for non-real-time music composition is <NAME>' Kyma, developed at the University of Illionis' \gls{cerl} \parencite{DBLP:conf/icmc/Scaletti87}. It is designed as an interactive composition environment for the Platypus digital signal processor. Scaletti's language was hierarchical in its structure, enabling data records to be linked vertically and horizontally. Together, these data structures formed objects, enabling the composer to treat any set of sounds within the composition, and even starting from the composition itself as an object. In such a way: ``\dots the composer could create a `sound universe,' endow the sound objects in this universe with certain properties and relationships, and explore this universe in a logically consistent way'' \parencite[50]{DBLP:conf/icmc/Scaletti87}. Given the ``vast amounts of data required for sound synthesis'' \parencite[50]{DBLP:conf/icmc/Scaletti87}, Kyma's objective was to fit timbre creation and temporal event lists into the same traversable database underlying the program. Like Puckette and Free, Scaletti's design was aimed at a language that ``itself would not impose notational or stylistic preconceptions'' \parencite[50]{DBLP:conf/icmc/Scaletti87}.

On one hand, Scaletti based her research on <NAME>'s \gls{hmsl}, another ``non-stylistically based'' music composition environment that was ``not fundamentally motivated by a desire to imitate certain historical compositional procedures'' \parencite[224]{DBLP:conf/icmc/RosenboomP85}. Polansky's focus was on a language that would ``reflect as little as possible musical styles and procedures that have already been implemented ---like conventional music notation---'' \parencite[224]{DBLP:conf/icmc/RosenboomP85}. On the other hand, the ``notational bias'' \parencite[49]{DBLP:conf/icmc/Scaletti87} that Scaletti recognized in languages such as \gls{music-v} and FORMES \parencites{DBLP:conf/icmc/RodetBCP82}{DBLP:conf/icmc/BoyntonDPR86}, prescribed a very clear division between composition and synthesis which, in turn, was a very difficult and time-consuming ``wall'' she had to ``circumvent'' \parencite[49]{DBLP:conf/icmc/Scaletti87}. Therefore, she imagined a language in which the composer could ``choose to think in terms of notes and keyboards and staves but in which this structuring would be no easier and no harder to implement than any of countless, as yet uninvented, alternatives'' \parencite[49]{DBLP:conf/icmc/Scaletti87}. Both \gls{hmsl} and Kyma are still in used today, the former with a further Java version by Didkovsky and Burk \parencite{DBLP:conf/icmc/DidkovskyB01}, and the latter embedded into a commercially available workstation.\footnote{\url{https://kyma.symbolicsound.com/}}

\paragraph{Pure Data}
\label{computer:puredata}

Ten years after \gls{max}, \textcite{icmc/bbp2372.1997.060} moved on to Pure Data. The commercially available \gls{max/msp} \parencite{DBLP:conf/icmc/Zicarelli98} presents, like Pure Data, the \gls{max} programming paradigm \parencite{DBLP:journals/comj/Puckette02}. In resonance with the neutrality of the 1980s, Puckette introduced more data structure flexibility as a means to provide a musical instrument without stylistic constraints. Data structures became a more accessible feature for the user to define and edit:

\begin{quote}
	The design of \gls{max} goes to great lengths to avoid imposing a stylistic bias on the musician's output. To return to the piano analogy, although pianos might impose constraints on the composer or pianist, a wide variety of styles can be expressed through it. To the musician, the piano is a vehicle of empowerment, not constraint. \parencite{DBLP:journals/comj/Puckette02}
\end{quote}

Puckette, therefore, aims to a certain stylistic neutrality, which he represents by the way in which the user opens the program: a blank page: ``no staves, time or key signatures, not even a notion of `note,' and certainly none of instrumental `voice' or `sequence''' \parencite{DBLP:journals/comj/Puckette02}. While acknowledging that even the `blank page' is a culturally loaded symbol referring to the use of paper in Western Art Music (much in the same way that it is favoring complexity altogether), Puckette reconfigured computer music design, composition, and performance by considering the way in which the structure of the program resonates aesthetically.

\paragraph{Graphic Scores}
\label{graphic_scores}
In order to include graphic scores for electronic music within the Pure Data, Puckette implemented a data structure deriving from those of the C programming language, which can be used in relation to any type of data: ``the underlying idea is to allow the user to display any kind of data he or she wants to, associating it in any way with the display'' \parencite[184]{DBLP:conf/icmc/Puckette02}. Puckette's philosophy, as I have mentioned earlier, was aimed at detaching music software from music concepts, leaving these aesthetic decisions to the user. To this end, anything within the canvas can be customizable, and there is no notion of time assigned to canvas coordinates. However, Puckette provided the user with a sorting function, ``on the assumption that users might often want to use Pd data collections as $x$-ordered sequences'' \parencite[185]{DBLP:conf/icmc/Puckette02}. In fact, this is the only sorting function within Pure Data, and it is the same function that sorts the patch `graph,' only now made accessible to the user. A common and elementary database routine (\texttt{sort}) that emerged to the program's surface because of traditional music notation practices.

Puckette contextualized his research with the \textit{Animal} project by Lindemann and <NAME> which allowed users to ``graphically draw pictures which define complex data objects'' \parencite{DBLP:conf/icmc/Lindemann90a}, three cases of graphic scores used to model electroacoustic music: Stockhausen's \textit{Kontakte} and \textit{Studio II}, Yuasa's \textit{Towards the Midnight Sun}, and Xenakis' \textit{Mycenae Alpha}, and, most interestingly, the \gls{sssp}'s user-defined features for graphical representations. Although in the \gls{max} papers Puckette does not quote Buxton's research, the latter's numerous publications at \gls{icmc} towards the end of the 1970s suggests that they reached the scope of \gls{mit}'s Experimental Studio where Puckette studied with Barry Vercoe. Furthermore, in Puckette's later introduction of graphic scores to Pure data \parencite{DBLP:conf/icmc/Puckette02} \see{graphic_scores}, he references the \gls{sssp} quoted by <NAME> (1985) as one source of inspiration, indicating that at least in 2002 Puckette was aware of Buxton's research. In any case, both Buxton's and Puckette's approaches can be considered musical resonances that go beyond geographical limits, reaching the level of data structures in computer music software.

An interesting point in common, however, between much of the interactive composition programs that emerged during the 1980s is that stylistic neutrality became a leitmotif. Computer music software designers were interested in providing stylistic freedom by user-definability. This became a programming need that stemmed from earlier computer music software implementations, and their experimentation. This shift in the course of computer music programs can be understood from two perspectives. On the one hand, by experiencing first-hand the extent to which data structures can indeed structure musical output, the composer-programmers of the 1980s took charge on data structure design and devised new approaches to music-making software. On the other hand, the novel flexibility allowed by the object-oriented model within the programming world made its way to the community by the younger generation of composer-programmers. In any case, the database was moving, expanding through computer music networks, institutions, and softwares.

\paragraph{OpenMusic}
In the same \gls{icmc} 1997 where Pure Data was presented, two object-oriented languages appeared: \gls{rtcmix} \parencite{DBLP:conf/icmc/GartonT97} and OpenMusic \parencite{DBLP:conf/icmc/AssayagAFH97}. While neither real-time nor a synthesis engine, the strength of OpenMusic resides in its ability to provide the composer access to a variety of sound analysis tools for composition \parencites{icmc/bbp2372.2004.004}{icmc/bbp2372.2010.129}, as well as the possibility to generate algorithmic streams that output directly into a traditionally notated score. For example, OpenMusic introduced the concept of a \textit{maquette}, which is a graphic canvas upon which a heterogenous set of elements as varied as audio waveforms, scores, or piano-roll type notation can be displayed. The \gls{lisp}-based graphic language developed as a collaboration at \gls{ircam} held music notation as a focal point, distinguishing it from other stylistically neutral software. 

\paragraph{Heaps and Nodes}
\textcite{DBLP:conf/icmc/GartonT97} presented \gls{rtcmix}, a real-time version of <NAME>'s \gls{cmix} \parencite{DBLP:conf/icmc/Lansky90}. What they described as innovative in this project was, in a similar way to the data structures for time management that Puckette presented, the scheduling capabilities of the program. In contrast to the \gls{cmix} language, which assumes a non-real-time access of objects, ``event scheduling is accomplished through a binary tree, priority-queue dynamic heap\dots '' \parencite{DBLP:conf/icmc/GartonT97}. A heap is a tree-based data structure where both keys and parent-child relationships follow a hierarchical logic. Garton and Topper thus introduced hierarchy into the scheduler. What this allowed, in turn, was ``scheduling-on-the-fly,'' that is, ``allowing notes to be scheduled at run-time (usually triggered by an external event, such as a \gls{midi} note being depressed)'' \parencite{DBLP:conf/icmc/GartonT97}. The real-time problem became once again a scheduling problem of computational tasks, and it was solved differently with yet another element: instruments instantiated ``on-the-fly'' could also establish their own \gls{tcp/ip} connection sockets in order to allow for networked access to the individual synthesizers \parencite{DBLP:conf/icmc/GartonT97}. That is to say, whenever a new instrument appears, it has the potential to enter into networked communication with earlier and future nodes. This means that synthesizer nodes could enter and leave the scheduler at any time, always in communication with each other. A musical equivalent would be for a violin player to enter in an out of the orchestra at will, while being able to lend the violin to any other player, and also play any other instrument except the conductor. In a similar networked way, SuperCollider \parencites{DBLP:conf/icmc/McCartney96}{DBLP:conf/icmc/McCartney98} is a high-level language that provides the user with a different paradigm to handle audio processes scheduling. The innovation that this language implemented, however, is the ``garbage collection'' of each process. McCartney took the hierarchic structure of the object-oriented paradigm and defined `nodes' in a tree-like structure, each with its own capability of nesting groups of other nodes, but most importantly, with its own initiation and expiration times. In other words, in contrast to Pure Data and \gls{max/msp}'s constantly running audio processes, SuperCollider only consumes \gls{cpu} resources whenever it needs to.

Both \gls{rtcmix} and SuperCollider meant a step forward towards networked musical environments that have resulted in recent forms of music making such as laptop orchestras and live coding, along with new music software such as ChucK \parencite{DBLP:conf/icmc/WangC03}. The literature on computer music software for composition alone would extend beyond the scope of this dissertation. For further reference in other sound synthesis data structures, see: the Diphone synthesis program \parencites{DBLP:conf/icmc/RodetDP88}{Rodet1989}{DBLP:conf/icmc/DepalleRGE93}{DBLP:conf/icmc/RodetL96}{DBLP:conf/icmc/RodetL97}; the Otkinshi system \parencite{icmc/bbp2372.2002.039}. For an overview of existing audio software up to 2004, see Xamat's PhD Dissertation \parencite[Chapter~2]{Amatriain/2004/phdthesis}. See also the Integra project \parencites{Bullock2009}{Bullock2011}, and Ariza's work on python's data structures \parencite{Ari05:Ano}. %, and Rowe's work on interactive music systems \parencite{Row92:Int, Row01:Mac}






%------------------------------------------------------------------------------
%
%	END section: Computer Music
%
%------------------------------------------------------------------------------

%------------------------------------------------------------------------------
%
%	BEGIN section: Intersections ----
%
%------------------------------------------------------------------------------
\section{Intersections}
\label{applications}

\glsresetall
The computer music software race that took place at the level of data structures has moved from music to media in an attempt to generalize applicability by maximizing stylistic potentials. To a certain extent, this motion can be understood as an axis between sound and music data structures. On one hand there is music tradition with its notational baggage. On the other, sound synthesis and programming, with its multi-stylistic promise grounded on the more general use of media. In any case, the shape that this motion takes is given by the composer-programmer's needs, ideas, and implementations. The computer music scene today builds on these struggles, and continues to propose novel approaches that reconfigure the practice.

In this section, I provide a glimpse of the many shapes that this reconfiguration has taken. I focus on artistic ventures, program extensions, and innovative research that has appeared under four main aspects of database performance: corpus-based approaches, querying methods, traversing methods, and resource sharing. These examples point only to some moments in which data structure design changed computer music.

\subsection{Corpus-based Approaches}
Modern uses of databases in computer music take the general form of a corpus of sounds from which descriptors are obtained and then used to create sounds. These are known as corpus-based approaches, also known as data-driven approaches. Their difference is a matter of scale. These approaches have emerged in opposition to rule-based ones, highly useful still in many applications. In what follows, I show some implementations of the corpus-based model in sound.

\paragraph{Concatenative Synthesis}
<NAME> developed the concept of data-driven concatenative sound synthesis in his PhD thesis at \gls{ircam} \parencites{Schwarz2000}{icmc/bbp2372.2003.099}{Sch06:How}. By segmenting a large database of source sounds into units, a selection algorithm is used to find any given target by looking for ``units that match best the sound or musical phrase to be synthesised'' \parencite{Sch06:How}. In contrast to rule-based approaches in which sound synthesis is arrived at by models of the sound signal, concatenative synthesis is data-driven, or corpus-driven (when referring to larger databases). That is to say, by joining together recorded samples, Scwharz obtained a model for sound synthesis that preserves even the smallest details of the input signal. Schwarz later contextualized `information space' as a musical instrument in itself \parencites{diemo_schwarz_2009_849679}{Schwarz:2012}.

\paragraph{Other approaches}
The variety of applications of corpus-based or data-driven is still a fruitful research area. I present here only some data-driven cases that arrive at other ways to generate sounds than sample concatenation. \textcite{icmc/bbp2372.2003.052} used a database of \gls{stft} analysed sounds in an original way. Upon calculating the distances between the results of these analysis he was able to define a database of similarity between his original database which he then re-synthesized. \textcite{DBLP:conf/icmc/Collins07} developed an audiovisual concatenative synthesis method where ``databases tagged by both audio and visual features, then creating new output streams by feature matching with a given input sequence'' \parencite[1]{DBLP:conf/icmc/Collins07}. A recent case in which concatenative synthesis was applied to rhythm can be found in \textcite{Nuannicode225in2016}. \textcite{icmc/bbp2372.2003.030} was able to implement a model for heterophonic texture by pitch-tracking the highly ornamented music of the Csángó\footnote{``The Csángó, in some cases a Szekler ethnic group, are found in eastern Transylvania (Kalotaszeg), the Gyimes valley, and Moldavia'' \parencite{icmc/bbp2372.2003.030}.} music into a database that enabled him to present a data structure of the ornament. The implementation of analysis and subsequent algorithmic rule extraction can be thought of as a form of analysis-based sound generation: by inputting a sound file, a dataset of rules was obtained to approach a model for the ornament. Therefore, a rule-model was obtained by means of a data-driven approach.	This relates to \textit{Orchidée} \parencite{gregoire_carpentier_2006_849343}, a computer-aided orchestration tool based on database input-matching and a series of candidate orchestration targets. The data-driven approach is combined with a highly dense corpus of instrumental techniques, in order to concatenate orchestral targets.

\paragraph{Software Libraries}
One of the central concepts of the object-oriented programming is extensibility. The list of objects that can be added to the main program tends to grow exponentially as a function of its use. A list covering all extensions would require a research project of its own. However, I would like to focus on those extensions that enable further and more specific use of databases in the context of music composition. \textcite{Stu04:Mat} developed \textit{MATCONCAT}, a concatenative synthesis library for \textit{Matlab}. \textcite{Sch06:Rea} designed \gls{catart} as a concatenative synthesis toolkit both as a standalone application and as a \gls{max/msp} external. Another concatenative synthesis library is <NAME>'s python module \gls{audioguide}. <NAME>'s research on timbre analysis developed into a timbre description library for Pure Data called \obj{timbreID} \parencite{icmc/bbp2372.2010.044}. Within this library, users are able to analyze sound files using most available timbre descriptors. Since Brent's library enables users not only to analyze sounds and store the resulting descriptors in a database, but also to cluster them within the database, it allows for a variety of applications of which only one of them is concatenative synthesis.

\subsection{Querying Methods}

\paragraph{Query-by-content}
One of the innovations that brought forth \gls{mir} is high-level audio feature analysis. This enabled computers to understand keywords such as `bright', `sharp', `dark', `metallic', etc., that would describe timbral content of audio files. When applied to database querying, these keywords enable `query-by-content' searches. Many online databases such as \gls{freesound} or \gls{looperman} have this type of querying. The \gls{cuidado} project at \gls{ircam} consisted of a database system aimed at content based querying of sound files \parencites{DBLP:conf/ismir/VinetHP02}{DBLP:conf/icmc/VinetHP02}{DBLP:conf/icmc/Vinet05}. This project enabled \glspl{dj} to browse through files, apply beat-synchronized transitions between them, among other automated tasks during performance. \gls{cuidado} later developed into the \textit{Semantic Hi-Fi} project and influenced subsequent software. \textcite{icmc/bbp2372.2007.117} enabled users generation of personalized audio description databases that could also be queried by content in \textit{Data Jockey}.

\paragraph{Similarity-based}
\textcite{Frisson2015} provides an overview of multimedia browsing by similarity. Real-time audio analysis moved users beyond descriptive keyword, with sound based input by singing or by providing a sample array. These systems calculate the spectral similarity between the incoming signal to obtain a match from a sound database. In this sense, a different type of performativity was enabled with systems with query-by-content in live contexts. For example, \textit{SoundSpotter} \parencite{DBLP:conf/icmc/CaseyG07} was dedicated to real-time matching of audio-visual streams by using audio input as feed for a shingling algorithm based on \glspl{lfcc}.\footnote{``Audio Shingling is a technique for similarity matching that concatenates audio feature vectors into a sequence of vectors, and matches the entire sequence'' \parencite{DBLP:conf/icmc/CaseyG07}. ``Shingles are a popular way to detect duplicate web pages and to look for copies of images. Shingles are one way to determine if a new web page discovered by a web crawl is already in the database'' \parencite{DBLP:conf/ismir/CaseyS06}.}

Querying a database by similarity appeared in contexts other than performance workstations. Based on both the \textit{Semantic Hi-Fi} and \textit{SoundSpotter} projects, \textcite{Price2008} developed an installation with an interface to a relational database of percussive sounds.\footnote{In their project, they used a \gls{max/msp} library called \textit{net.loadbang-SQL} to query and import data for the communication with \gls{sql} databases.} This database contained description data of the beginning of each analyzed sound file. Thus, participants were able to query a bank of percussion timbres based on brightness, noisiness, and loudness. 

Concatenative synthesis uses similarity for the purpose of sample concatenation at the analysis frame level. In this sense, concatenative synthesis can be understood as a real-time query-by-content engine feeding a granular synthesis engine. Therefore, the difference between concatenative synthesis and content-based-queries is a matter of scale (samples as opposed to sound files) and in the use (new sample combinations as opposed to previously stored sound files). 


\paragraph{Hybrid Queries}
\label{applications:hybrid_queries}
Some authors have managed to conjugate disparate database uses by hybridizing the queries. The following modal translations represent only some of the many examples in the literature. \textcite{icmc/bbp2372.2001.103} used audio analysis to obtain gesture features from the non-audio signals obtained from the \textit{Radio Drum}, an variant of Max Mathews's \textit{Radio Batton} \parencite{DBLP:conf/icmc/Boie89}. \textcite{icmc/bbp2372.2001.103} searched for peak detection in the incoming signal to determine mallet (air) strokes. At \gls{ccrma}, \textcite{icmc/bbp2372.2001.071} managed to invert the concept of physical modeling by estimating violin bow position, pressure, and speed using \gls{lpc} coefficients of violin audio recordings. \textcite{DBLP:conf/icmc/OliverJ08} presented a controller composed of an elastic head suspended along the rim of an empty drum shell. The player presses the head making different shapes with the hand, fingers, or mallets, and ``these shapes are captured by a video camera that sends these images to the computer, which analyzes them and outputs the tracked parameters'' \parencite[1]{DBLP:conf/icmc/OliverJ08}. This instrument enabled a possibility for ``dissociating gesture with sound'' \parencite[1]{DBLP:conf/icmc/OliverJ08}. That is to say, gestures whose sound could be visually anticipated (the hitting of a drum) were extended by micro-gestures only visible to the camera sensor. Further, in contrast with acoustic drums, with the silent drum ``one can manipulate continuous sounds through a new gestural vocabulary \parencite[1]{DBLP:conf/icmc/OliverJ08}. <NAME> developed the sensing algorithm into a \gls{gem} external called \obj{pix\_drum}, and later moved on to \obj{pix\_mano}, where he removed the fabric and focused on what he calls ``direct hand-tracking'' \parencite{DBLP:conf/icmc/OliverJ10} \see{inoperativity}. \textcite{Caramiaux2011} proposed gestural input for the query-by-content method. They used gesture-to-sound matching techniques based on the similarities of temporal evolution between the gesture query and the sound target. Another example of hybrid querying is \textcite{mcartwright:2014}, where a database of computer synthesis parameters was queried by vocal input, enabling users to mimic sounds with their voices in order to obtain parameter settings (presets) that would approach the analyzed vocal sound.	

\subsection{Traversing Methods}

Given that querying methods have resulted in novel ways to approach information space within databases, many authors have proposed their own approaches towards navigating this space. Like browsing, or surfing the Internet, database traversing is a form of navigation across the \textit{n}-dimensional space that databases have to offer. Despite their differences, the approaches I refer to now point to the hybrid qualities that data can take when used in performance, specifically in terms of the mixed use of data coming from multiple sensing mechanism, and the networked quality that reconfigures music performance and composition.

\paragraph{Sensorial Networks}
Insook \textcite{icmc/bbp2372.2000.146} presented an interactive installation \parencite{Cho00:Voi} at the Dorsky Gallery in NYC where a `sensorial network' made from a sound database of speeches by famous leaders was distributed along the installation space. \citeauthor{icmc/bbp2372.2000.146} implemented a motion tracking computer vision algorithm enabled sounds to be modulated as a function of the different `clouds' of pixel data where values gradually changed as participants moved across the sensing area: ``pixels do not switch on and off, they fade in and out forming clusters in the 2D camera plane according to the degree of movement projected from the corresponding floor positions'' \parencite[4]{icmc/bbp2372.2000.146}. In this sense, participants were able to walk the database itself: ``Traversing the [sensorial network] can be thought of as rotating its shadow such that one moves through a semantic neighborhood which includes sound synthesis and residual tuning as well as speech acts'' \parencite[3]{icmc/bbp2372.2000.146}	In addition to this tracking system, however, she included hysteresis within the system. Thus, the recorded history of the participant's interaction with the system enabled condition-dependent events to occur as participants' interaction lasted longer. Within this installation, the artist prototyped a ``sensory information retrieval system where the acquisition of information is an acquisition of an experience'' \parencite[1]{icmc/bbp2372.2000.146}.

\paragraph{Involuntary Navigation}
Bioinformatic data taken from galvanic skin sensors attached to a cellist's toes within a live performance environment is the point of departure for a complex network for performance \parencite{icmc/bbp2372.2006.123}. The \gls{gsr} activity was correlated with intervallic distance between adjacent musical notes in a database of `cell nodes' previously written by the composer. However such score acted as a ``filter for the autonomic control signals generated by the performer'' \parencite[601]{icmc/bbp2372.2006.123}. What this means is that the music fragment database, involuntarily navigated by the performer, becomes a parameter for a live-generated score. The performer is thus embedded within a convoluted networked loop that goes through voluntary and involuntary agents that intertwine composition, interaction, and performance.

\paragraph{Networked Collaborations}
Among the many cases of network performances with multiple players that exist in the literature, I would like to point to one case where the rules of 16th century counterpoint demanded a relational database \parencite{Nakamoto2007}. By implementing a database system such as \gls{mysql}, to store and retrieve vocal parts, \citeauthor{Nakamoto2007} enabled performers to sing together in canon form from distant locations. Going beyond any notion of anachrony, what is interesting about this approach is the fact that by ``using a PC and database server with the internet'' two or more performers can engage seamlessly in musical performance \parencite{Nakamoto2007}. Telematic performances have spawned ever since Internet connectivity enabled networked audio and video feeds. \textcite{icmc/bbp2372.2014.046} considers that the listener's body within telematic electroacoustic concerts has been traditionally left out. Therefore, in he devised a set of parameter constraints within these performances, based on musicians who were used as baseline. His argument was grounded on an affective approach towards networked performance, and it is aimed at addressing the limitations that arise from the separation between performer and listener, specifically within telematic electroacoustic performances.

\paragraph{Mobile Devices}
The mobility that networks enabled can be represented in the work of 
\textcite{Liu:2013}, who created an audiovisual environment for live data exploration that implemented simultaneous sonifications and visualizations of networked database queries made by participants using \gls{ios} devices. \textcite{btaylor:2014} implemented centralized database systems to include user-defined interfaces to be saved and shared within their mobile device platform. \textcite{Rya17:OnT} presented a work that gives each member of the audience their own instrument through their cell phones. By accessing a website that loads custom synthesizers made with the Web Audio \gls{api}, the audience becomes the performer in an innovative way. While the title of the work (\citetitle{Rya17:OnT}) refers to the potentials and the ubiquity of small transducers, the `score' (source code) of the work lives on a server and travels wirelessly to the audience to become a (mobile) instrument.

These are some of the many examples that point to the many shapes that traversing a database can take. These shapes have given different resonances within the concert and installation spaces, as well as within the performativity of the music involved. Further, the possibilities of these reconfigurations can be seen in terms of a need for sharing resources and experiences through networks.

\subsection{Resource Sharing}

Sharing resources can be interpreted in many ways. On one end, it points to networked environments on which multiple client users connect to a server that provides shared data flow among the network. This is the case of live coding, where multiple users share the same network. Another definition pertains to the data itself, the way that it is formatted, and how to access or edit it: the file format, where users can read the same data. Lastly, the activity of sharing relates to publishing results like in research or academic communities. This is the case of the multiple datasets that exist.\footnote{`Dataset' differs from `database' in terms of scale: multiple datasets may reside in a single database.} In any case, what is common between these forms of sharing is an entropic and endless plurality.

\paragraph{Multimodal Datasets}
Among the many datasets that are available \see{mir}, a research interest has been growing among gesture datasets. This is the case of a hand drumming gesture dataset that uses data from a two-dimensional pressure sensor that could be compared to the membrane of a drum \parencite{DBLP:conf/icmc/JonesLS07}. \citeauthor{DBLP:conf/icmc/JonesLS07} aimed to provide physical model designers with a collection of six techniques of hand drumming, recorded as matrices at a slow rate (100 Hz) suitable for non-real-time synthesis by way of interpolation into a model for physical modeling of wave propagation called `waveguide mesh.' <NAME> \parencite{icmc/bbp2372.2009.005}, stemming from the research at \gls{cnmat} on the \gls{osc} format, proposed a real-time application for efficient storage and retrieval of gestural data using the relational model offered by the \gls{postgresql} \gls{dbms}. The motivation behind these datasets, besides research is mostly to provide open access to any user with a computer and an Internet connection. \textcite{Young2007} created web-accessible databases of gestural and audio data concerning violin bow strokes. \textcite{Hochenbaum2010} developed a gestural and audio joint database that enabled identification of a given performer between a group of performers, gaining insight on musical performance itself. These joint databases combining more than one sensing mode are called `multimodal.' Multimodal databases can be extremely focused ---combining different blowing profiles on recorder flutes (along with their sound) \parencite{Garcia2011}---, or radically plural: listening subjects asked to move as if creating a sound \parencite{fvisi:2017}. 

\paragraph{Formats}
While the purpose of a format is to store as much information as possible, using as little space possible, and in an efficient way so that read and write operations occur seamlessly, formats are the equivalent of database models within files: they can be implemented in endless ways, and they are contingent upon programming decisions \parencite[8]{Ste12:MP3}. One way to categorize formats is based on human readability. Readability of the format is a function of the task at hand and the quantity of the data involved. In cases where the data is very little, for example, a \texttt{.pd} file (Pure Data), \gls{metrixml} \parencite{Amatriain/2004/phdthesis}, \gls{json}, \gls{yaml}, or \texttt{.bib} (\LaTeX{} bibliography file), data structures can be stored in (text) characters, and thus be readable by humans. In this sense, data does not need to be highly structured. For example, within the \textit{Integra} project, programmers implemented a data format called \gls{ixd}, capable of containing sequences, tags and meta-data, and presets, for shared use among different multimedia environments. Their argument for a semi-structured model resided in the semantic richness that can be allocated in opposition to the binary format only readable by machines. To this end, they implemented \gls{ixd} using the \gls{xml} language \parencite{icmc/bbp2372.2009.012}. In other cases, data is large enough to justify the need for binary format with a simple header such as \texttt{.timid} (\obj{timbreID}). At this level, by structuring the format and sacrificing human readable semantic richness, faster write and read times are achieved, and less resources are used. However, in the case of larger media files such as audio, image, or video, and also multimodal gesture data, these demand high-performance compression algorithms that reproduce data in `streams.' Some formats for sound and gesture analysis were standardized in recent years, as is the case of \gls{sdif} and \gls{gdif}, which are widely used in audio analysis software like \gls{spear} and OpenMusic \parencites{icmc/bbp2372.2004.004}{kristian_nymoen_2011_849865}. In one case revealing the extent to which data can reside in multiple combinations, the \gls{sdif} format was used for audio spatialization data \parencite{icmc/bbp2372.2004.004}. There is still little format standardization within datasets, and in general, the plurality of formats demands database creators to either implement routines to interpret as many formats as possible, or to rely on external libraries for transcoding. In any case, the plurality of formats is almost as great as that of datasets and, to a certain extent, almost as numerous as there are software developers.

\paragraph{Live Coding}
Live coding has now a long history and it occupies a fair portion of the computer music scene today. In terms of database performance, the practice of live coding in audio or in video exposes both computer technology and art performance in simultaneity to the cutting edges of both worlds. For a more general overview on live coding, see \parencites{nickcollinsphd}{Col03:Liv}{Nilson2007}{Zmo15:Liv}. In this brief section, I would like to point to the work of \cite{croberts:2014}, who implemented within a real-time live coding web-based environment called \textit{Gibber} a centralized database for the storage and quick access of digital instruments that can be prototyped in the environment. This type of on-the-spot database system enables shared access to sound files that have potential use throughout the performance. By means of a networked database, two or more players can grab and record sounds from different locations. Another case of networked situations in live coding is a system that incorporates content based searches (query-by-humming, query-by-tapping) of various \gls{cc} sound databases such as \gls{freesound} or to user-defined databases \parencite{nime18-Xambo-b}.


\subsection{Closing Remarks}
The many shapes that database performance has taken over the years can be approached with what I have shown so far. Since many applications of the database in music continues to grow, I have only selected a few areas in which the database has had some agency. One area that I have not included above is that of artificial intelligence for music applications, where databases have been used for training models, and other forms of machine learning. For example, in interactive music systems \parencite{Row92:Int}, in improvisation systems \parencites{DBLP:conf/icmc/AssayagDD99}{DBLP:conf/icmc/BlochD08}, to model \gls{edm} patterns \parencite{rvogl:2017}, analog synthesizer parameter settings \parencite{Loviscach2008}. Multimodal datasets have also been used in training \parencite{DBLP:conf/icmc/SchonerCDG98}. Notwithstanding the multiple gaps and omissions that these lines reveal, I believe the plurality of shapes speaks for itself. As I have shown, from bytes to terabytes, from data structures and files to datasets and databases, has had different positions in relation to sound practices. These positions can be summarized in a three-dimensional diagram \fsee{intersections} where the database can be placed along three axes. Visibility of the database can be represented by the sign: positive values indicate visibility and negative indicate invisibility. `Negative' in this context relates only to the sign of the value, and not to any `judgment' whatsoever. If anything, this graph is intended as a metaphor. As I have mentioned earlier, the database is generally the grounds of sonification, so it can be represented by the $y$-axis. In \gls{mir}, the database is next to the databaser, that is, in the $x$-axis. Finally, I mentioned that the database in computer music is behind the databaser, therefore the $z$-axis seems appropriate.

\img{intersections}{0.5}{
	A snapshot indicating a possible position of the database in terms of visibility among \gls{mir}, sonification, and computer music. Positive values indicate visibility and negative indicate invisibility. This graph is only one of multiple frames in which we can capture the visibility of the database in relation to the different practices. Despite the fact that the location in which the database can be placed within this graph is reduced to a single point, the database can occupy multiple points simultaneously. Thus, the more complex coreography inherent in databasing escapes the bi-dimensionality of this graph, but it can nonetheless be hinted by it.
}{Intersection space.}

The simplicity of this diagram is intentional, to avoid any attempt to quantize the actual value that the database represents in the plurality of shapes that I have discussed. There is no percentage that can be drawn from how visible a database can be. Therefore, when practices begin to intersect, as I have shown here, the visibility of the database can thus be understood as in constant motion along these axes. Database performance, in this sense, provides a key to understand the motion of this intersection. Furthermore, there is one dimension not contemplated within this diagram: time. The intersections referenced here are always moving in time, which indicates that the diagram that I have shown here is but one frame in the movement of this database choreography. At each point in time the databaser can pause for a second, analyze the frame, and perhaps describe the motion that the database has taken thus far, and the position that it occupies at that point. And further, perhaps the database occupies two positions at once. The dimensionality reduction carried out when graphing the position of the database should not misguide the reader into thinking that the database is fixed, or that it may have only one position. Quite the contrary, the database is in constant motion and incessant fluidity, and databasers engage in a form of dance that sometimes brings the database to a certain visibility, and other times might veil it beyond layers of code. In any case, this description has been my task until now, and it is safe to say that we have glanced at the database dance. In what follows, I will change the focus and approach the database from a different perspective, one not guided by light, but immersed within sound.

%------------------------------------------------------------------------------
%
%	END section: Intersections
%
%------------------------------------------------------------------------------
{
\backmatter
\singlespacing
\setglossarysection{part}
\printglossaries
\printbibliography
\addcontentsline{toc}{part}{Bibliography}
}
\end{document}
