Open science at SMPC 2019

Open science was a bit of a theme at this year’s meeting of the Soceity of Music Perception and Cognition. Around my NYU hosting duties, I got to moderate and contributed to the Symposium on Open Science, a two part affair intended to encourage open science practices in our Music Cognition community. The whole project was coordinated by Dominique Vuvan, and also involved Psyche Loui, Bob Slevc, Dave Baker, and Haley E. Kragness.

The symposium’s focus was on making open science practices accessible to music science researchers by bringing in concrete examples from our own work and discussing how open science can help interdisciplinary research. Full slides are available here.

Two points from the audience discussion stuck with me:

– Opening up project materials beyond the publication may be good for accessibility and efficiency but it can also makes researchers more vulnerable. In order to lessen this potential cost, it is important for the community to maintain a culture of respect while addressing potential problems with work that is shared. Attacking researchers for honest mistakes or different interpretations will only discourage the sharing we need for better science.

– It’s important to make space for exploratory research on empirical data along side the more controlled hypothesis testing paradigms that fit into pre-registration practices. Theory development and the honing of methods have been collapsed into experiment reporting that often then use inappropriate statistics. We need space and appropriate language for the work of developing hypotheses, instead of rewarding the practice of changing the story for the journal article. (I expect there is plenty written about this in the philosophy of science. I should go check.)

Out of this session also came the request for the conference to have an open repository for posters and talk slides. With a few clicks and a couple emails the OSF meeting repository came into being and so far has several dozen of the 360+ presentations shared at the conference.

And in the spirit of open sharing, I’ve also posted the python scripts and latex files I used to make the program and abstract publications. May they get used again one day!

Dissertation in Open on ProQuest

Finally, my PhD dissertation is posted in full on ProQuest, open access to all. It’s a bit of a behemoth at 64 MB and 441 pages, but if you want to know everything about involuntary respiratory phase alignment to music, this is the document for you.

The first half is a lot of technical details on how to get relevant timing information from respiratory sequence recordings. The second is a long analysis of when alignments arise and what that says about how our breathing is engaged by what we are listening to.

I feel lucky to have had the time to dig deep in the analysis. The patterns are investigated from the perspective of the individual musical works used as stimuli AND from the perspective of individual listeners in five case studies. And as associations arose between respiratory behaviour and musical events, the last section focuses on how these relate to known (or hypothesized) respiratory control mechanisms. By studying the details, this dissertation goes from detection of a little known phenomenon to testable hypotheses about causal mechanisms. I look forward to putting each of these to the test.

I’m really proud of this work. It brings together research from multiple fields and made use of all my formal training plus a lot I had to learn outside of the classroom. (Like everything about the respiratory system. That certainly wasn’t covered in math, or music theory, or psych classes…)

Full Abstract:

This dissertation explores the surprising phenomenon of listeners’ unconsciously breathing in time to music, inspiring and expiring at select moments of specific works. When and how the experience of hearing music might produce stimulus-synchronous respiratory events is studied through Repeated Response Case Studies, gathering participants’ respiratory sequences during repeated listenings to recorded music, and through Audience Response Experiments, responses for participants experiencing live music together in a concert hall.

Activity Analysis, a new statistical technique, supported the development and definition of discrete phase components of the breath cycle that come into coordination: the onsets of inspiration and expiration, the intervals of high flow during these two main phases, and the post-expiration pause. Alignment in these components across listenings illuminate when the naturalistic complex stimuli can attract or cue listener respiration events.

Four patterns of respiratory phase alignment are identified through detailed analysis of stimuli and responses. Participants inspired with the inspirations of vocalists and wind performers, suggesting embodied perception and imagined action may exert influence on their quiet breathing. Participants suppressed and delayed inspirations when the music was highly unpredictable, suggesting adaptation in aid of auditory attention. Similar behaviour occurred with sustained sounds of exceptional aesthetic value. Participants inspired with recurring motivic material and similar high salience events, as if marking them in recognition or amplifying their affective impact. And finally, participants occasionally breathed following structural endings, suggesting a sigh-like function of releasing the respiratory system from cortical control.

These instances of music-aligned respiratory phase alignment seemed to be stronger in participants who were typically active with heard music, but the impacts of training and expertise was not a simple condition for this behaviour. Contrasts between case study participants showed highly idiosyncratic patterns of respiratory alignment and differences in susceptibility along side moments of shared effect. In the audience experiments, alignment within phase components was measurable and significant, but rarely involved more than a quarter of participants in any given instance. These levels of concurrent activity in respiration underline the subtlety of this bodily response to music.

And if you want to know more than what you can find in the document, or borrow scripts/data that haven’t been posted yet, get in Touch!

Updated Activity Analysis, 2.1

Updated version of the Activity Analysis toolbox is up on GitHub now. The changes in functions came from working with more psychophysiological measurements and events. Now there is an option to assess the coordination score for a response collection using the distributions of local rank in activity levels, instead of the distribution of activity levels themselves. For more on Activity Analysis generally, check out this post on the paper that was published earlier this year, including a link to the full text pdf.

This addition may seem like a tiny adjustment. For many kinds of response events, it doesn’t substantially change alignment assessments. When considering the coordination of events that are fairly consistent in rate over time, like say inspiration onsets, the distributions of activity levels and of local rank give the same kinds of coordination scores. However, for response events that change their rate of occurance over the course of a piece of music, like skin conductance increases, the activity level time series distribution obscures moments of exceptional alignment in quieter times. In such cases, the local rank does a better job at capturing anomalous alignments.

This rank based coordination score appeared to be necessary when I was testing unrelated response collections of facial sEMG signals and skin conductance. The old calculation generated scores that were too low, producing insufficient numbers of false positives, while this adjustment behaved just as the statistic should.

Besides this change to the localActivityTest function outputs, a few other functions have been tweaked and the demos have been amended to work with these changes. Lastly, Demo_2 now include coordination assessments on a number psychophysiological signals recorded during 24 listenings to a fun piece of music by a single participant.

It all should work just fine in MatLab.

I’m looking forward to releasing this version in Python in the not too distant future too.

ICMPC15 – Reliable Psychophysiological Changes to Music listening

A talk and a poster discussed a number psychophysiological responses to music, specifically events measurable over repeated listenings in skin conductance, respiration rate, heart rate, and zygomaticus and corrugator contractions. The talk explains how Activity Analysis can be applied to these responses in order to identify reliable reactions to musical stimuli, and the poster shares some of the consistencies and inconsistencies we can find in different listeners’ responses through these signals.

Demo: Activity Analysis on Psychophysiological Measures of Responses to Music

Download the slides (pdf)

Abstract

Continuous measurements of responses are particularly useful for music cognition as our experience develops during the presentation of this powerful stimulus. And yet, responses can vary substantially, and psychophysiological measurements are particularly noisy. Multiple studies have identified changes in psychophysiological states with the presentation of music, but identifying when changes are triggered is not a simple task, even with rapid changing signals like skin conductance, heart rate, and respiration.

Activity Analysis is a new analysis paradigm developed specifically for music research that focuses on response events and their co-occurrence across multiple listenings to the same stimulus, whether by different listeners to a live performance or repeated listenings by a single participant to recorded music. This approach accommodates the extraneous information in continuous measurements of response and leads to new results from these complex signals, including statistical assessment of coherence between responses at specific moments in music. It supports response-led exploration of the stimuli as well as addressing questions of whether and how individual pieces are coordinating the experiences of listeners.

Aims

To demonstrate the application of Activity Analysis with the MatLab toolbox on collections of psychophysiological responses from repeated response experiments.

Main Content

This demo will introduce Activity Analysis, demonstrate the visualisation capabilities of this approach to continuous responses, and apply tests of coordination to skin conductance, heart rate, and respiration belt measurements from audience response collections and repeated response collections. Particular attention will be paid to the local coordination test, which identifies when responses are in significant alignment with the music. All to be discussed is the process of determining appropriate parameters for coordination testing of response events such as orienting responses in skin conductance, along with the implications of a lack of measurable coordination in response activity.

Implications for practice

Activity Analysis may be very useful for the study of responses to music, allowing researchers to pin point when changes in responses occur and whether the timing of changes might be expected to replicate. With the MatLab toolbox, these techniques can be readily applied to existing data sets as well as future experiments.

Value for this conference

Activity Analysis can be applied to a number experiments reported at ICMPC and conducted in labs associated with many related organizations. The relationship between music and listeners bodily responses is a long-standing but still growing area of research and tools for the exploration of experimental data are needed along side methods for testing specific hypotheses.

References

Upham, F., & McAdams, S. (2018). Activity analysis and coordination in continuous responses to music.Music Perception: An Interdisciplinary Journal, 35(3), 253-294.

Upham, F. (2016) ActivityAnalysisToolbox_2.0. GitHub repository, https://github.com/finn42/ActivityAnalysisToolbox_2.0

Felt It My Way: Idiosyncratic Psychophysiological Responses to Recorded Music

Download the poster (pdf)

Abstract

Psychophysiological responses to music have been studied from decades, but the focus has typically been on common response patterns across listeners. The Repeated Response paradigm, recording a participant’s responses to a set playlist of stimuli over multiple listenings, allows for more in depth considerations of responses typical of individual listeners. Repeated exposures to the same stimuli have been associated with desensitization (Grewe, Nagel, Kopiez, & Altenmüller, 2007) as well as sensitization in increased coordination (Sato, Ohsuga, & Moriya, 2012). Either way, there is an opportunity to capture consistencies tied to individual participants musical histories and inclinations that can easily be lost when looking for agreement across a population or audience.

Aims

Identify consistency and coordination in individual participants psychophysiological responses to music and consider the contrasts between participants’ responses to pieces of music.

Method

Five participants heard the same six pieces of music 12 times over several weeks. During these listenings, skin conductance, finger temperature, heart rate, respiration, and facial muscle sEMG (Zygomaticus, Corrugator) were recorded continuously. Using activity analysis, we evaluated first which responses showed significantly coordinated in music relevant response events per participant. When participants showed coordination, their moments of consistency were compared to see whether they aligned or contrasted.

Results

Preliminary results show that participants vary how well their responses are coordinated between listenings and which response measures show the most coordination. For example, two participants showed very high respiratory coordination but different patterns of consistency in finger temperature decreases. Besides the overall pattern of disagreement, specific results will be shared on responses to specific works, including a late Beethoven String Quartet excerpt and a Dubstep track.

Conclusions

Listeners can show some shared patterns of behaviour to music, but they also develop idiosyncratic response sequences to pieces they come to know. This is not only measurable in post-stimulus ratings and preference but also in the sensitivity, reliability, and timing of changes in their psychophysiological responses.

References

Grewe, O., Nagel, F., Kopiez, R., Altenmüller, E. (2007). Listening to music as a re-creative process: Physiological, psychological, and psychoacoustical correlates of chills and strong emotions. Music Perception, 24(3), 297-314.

Sato, T. G., Ohsuga, M., and Moriya, T. (2012). Increase in the timing coincidence of a respiration event induced by listening repeatedly to the same music track. Acoustical Science and Technology, 33(4):255–261.

ICMPC15 – The Audience’s Breath

This year’s ICMPC recorded all of the talks in support of virtual attendance. Here is my long talk (20 min + Q&A) on coordination in respiration between audience members.

The Audience’s Breath: Collective Respiratory Coordination in Response to Music

F. Upham, H. Egermann, and S. McAdams

Abstract

Performers have used respiratory metaphors to describe the reactions of the audience’s engagement with a performance. We refer to an audience holding their collective breath, or sighing with a release of tension. Significant regularities in respiratory phase have been measured in participants’ responses over multiple listenings to some recorded music (Sato, Ohsuga, & Moriya, 2012), but this fleeting alignment has not yet been measured in audiences at live concerts.

Aims

With recordings of respiration from audience members at live performances, we aim to evaluate whether there is measurable respiratory alignment between them to some or all pieces. If there is coordination, we consider which phase of the respiratory cycle shows the highest degree of alignment and how this could relate to audience members’ experience of the music performed.

Method

Respiration data from two audiences were evaluated using new techniques in respiratory phase detection and measurement of coordination. From the first audience, 40 participants sat amongst a larger group in an experiment-led concert of chamber music including three pieces of contrasting genres. The second audience was composed of 48 participants who were presented solo flute music, some recorded and some played live. Half of this group continuously reported the unexpectedness of the music while the remaining half reported their felt emotional responses through handheld devices.

Five components of the respiratory phase were evaluated for coordination using activity analysis with parameters tuned to each: Inspiration Onset, High Inspiration Flow interval, Expiration Onset, High Expiration Flow interval, and Post-Expiration Pause. These phases relate to the mechanics of respiration and the sensory consequences of air exchange.

Results

Significant coordination in respiratory phase components were observed between audience members to most stimuli, but the most coordinated phases varied from piece to piece. High Inspiratory and Expiration Flow intervals were most often significantly coordinated, compared to onsets. Post Expiratory Pauses, which would count instances of breath holding, were only coordinated in one piece. Less than half of participants engage in phase alignment concurrently, however numerous instances relate well to developing theories of respiration/cognition interactions, including differences in the alignment patterns of participants per rating task.

Conclusions

Audiences engage in measurable collective respiratory coordination with live performance and recorded music through simultaneous inspirations and expirations. However, these behaviours are performed by only a subset of participants at a time. This inter-participant difference is consistent with the results from repeated response experiments, in which only some participants have shown respiratory coordination with their own previous listenings. The fact that different phases of respiration showed coordination underlines the possibility that multiple mechanisms like embodied listening, attention, and hearing facilitation may be encouraging adjustments in audience members’ respiratory sequences for alignment.

References

Sato, T. G., Ohsuga, M., and Moriya, T. (2012). Increase in the timing coincidence of a respiration event induced by listening repeatedly to the same music track. Acoustical Science and Technology, 33(4):255–261

Materials

The slides are also available for download.