Skip to main content

Undergraduate Program

Summer REU

National Science Foundation Research Experience for Undergraduates (NSF REU)

Computational Methods for Understanding Music, Media, and Minds

How can a computer learn to read an ancient musical score? What can methods from signal processing and natural language analysis tell us about the history of popular music? Can a computer system teach a person to better use prosody (the musical pattern of speech) in order to become a more effective public speaker?

These are some of the questions that students will investigate in our REU: Computational Methods for Understanding Music, Media, and Minds. They will explore an exciting, interdisciplinary research area that combines machine learning, audio engineering, music theory, and cognitive science. Each student will work in a team with another student and will be mentored by two or more faculty members drawn from Computer Science, Electrical and Computer Engineering, Brain and Cognitive Science, the program in Digital Media Studies, and the Eastman School of Music.

Applying to the REU

You are eligible to apply if:

  • You are a 1st, 2nd, or 3rd year full-time student at a college or university.
  • You are a U.S. citizen or hold a green card as a permanent resident.
  • You will have completed two computer science courses or have equivalent programming experience by the start of the summer program.

It is not a requirement that you are a computer science major, or that you have prior research experience. We wish to recruit a diverse set of students, with different backgrounds and levels of experience. We encourage applications from students attending colleges that lack opportunities for research, and from students from communities underrepresented in computer science.

Before starting the application, you should prepare:

  • An unofficial college transcript, that is, a list of your college courses and grades, as a pdf, Word, or text file.  Include the courses you are currently taking.
  • Your CV or resume, as a pdf, Word, or text file.
  • A 300 word essay as a pdf, Word, or text file explaining why you wish to participate in this REU, including how it engages your interests, how the experience would support or help you define your career goals, and special skills and interests you would bring to the program.
  • The name and email address of a teacher or supervisor who can recommend you for the REU.

The application website does not allow you to save and resume your application before submitting, so start the application when you have time to enter all the information.

STEP 1: Apply online (application portal to open on December 1, 2018) no later than February 15, 2019.

STEP 2: After submitting online application, have the teacher or supervisor who can recommend you for the REU send a letter of recommendation to

STEP 3: Notification of acceptance will be communicated between March 15-April 15, 2018.

The REU Experience

The REU is 9 weeks long, running from May 29th to July 31, 2018.

Students accepted into the REU will receive:

  • On-campus housing
  • Meal stipend
  • A stipend of $4,500 for other expenses and to take back home
  • $540 to help pay for your travel to and from Rochester

Your experiences will include:

  • A programming bootcamp to help you learn or improve your programming skills in the language Python.
  • Working with a team of students and faculty on one of the projects.
  • Workshops on topics such as career planning and preparing for graduate school.
  • Social events, including a trip to the Rochester International Jazz Festival.

If you have questions about the REU or application process that are not answered here, please send an email to


You will be assigned to a project based on your background and goals.  Please note that you cannot specify a specific project on the application, but we will do our best to match you with one that matches your interests.  The projects evolve and change each year, but the projects from the past will give you an idea of the range of projects that will be available in upcoming years.

2018 Projects

Audio-Visual Scene Understanding

Mentor: Chenliang Xu (Computer Science)

Evaluating the role of audio towards comprehensive video understanding -  We are interested in measuring the role of audio plays in high-level video understanding tasks such as video captioning and spatiotemporal event localization. In this project, students will design novel Amazon Mechanical Turk interfaces to be used to collect audio-oriented annotations for tens of thousands YouTube videos. They will get hands on experiences on training deep learning algorithms to run on large-scale data with the focus on joint audio-visual modeling. 


Assessing the Effectiveness of a Speaker by Analyzing Prosody, Facial Expressions, and Gestures

Mentor: Ehsan Hoque (Computer Science)

Assessing the severity of Parkinson's disease through the analysis of a voice test - This project involves the analysis of two vocal tasks from the Movement Disorder Society-Unified Parkinson’s Disease Rating Scale (MDS-UPDRS) performed by people both with and without Parkinson's disease (PD). The tests include uttering a sentence and saying ‘uhh’ in front of the computer’s microphone. Our analysis will include extracting and identifying useful features from the audio recording and develop a novel machine learning technique to assess the severity level of Parkinson’s disease.


Computational Methods for Social Networks and Human Mobility

Mentor: Gourab Ghoshal (Physics and Astronomy) 

Investigating Human Mobility in Virtual and Physical Space - The student will develop the data analysis skills required to investigate complex system data, including python coding and statistics. They will then apply these skills to study the unexpected similarities between human mobility in physical and virtual space.


Computational Methods for Audio-based Noninvasive Blood Pressure Estimation

Mentor: Zeljko Ignjatovic (Electrical and Computer Engineering)

Audio Based Non-invasive Blood Pressure Estimation - With cardiovascular disease as the leading cause of death in America, constant blood pressure measurement is imperative to detect early onset symptoms. Piezoelectric sensors can be used in conjunction with a recurrent neural network in a wearable device (such as a smartwatch) to extract pulse wave velocity data and heart rate data to estimate blood pressure. The concept further expands the use of machine learning techniques and applies it to activity trackers. Although related technologies exist in the field, none of these technologies use a recurrent neural network with a piezoelectric sensor, nor is any of the said technologies achieved the status of the standard in the industry, as the field is still in its infancy. Continued research is required to develop a smartwatch which can accurately detect blood pressure; however, enough pulse wave velocity, heart rate, and blood pressure data to teach the recurrent neural network and develop a working prototype sufficient for the end of the summer. 


Music and the Processing Programming Language

Mentor: Sreepathi Pai (Computer Science)

A Framework for Developing Music-Generated Games (Erik Azzarano, Rochester Institute of Technology) - Erik is investigating a framework for developing music-generated games based on live or external audio input. He aims to create an intuitive mapping between a game’s mechanics and features of the audio input. For example, features of the audio such as frequency, amplitude, and beats, or onsets are extracted and mapped to different game parameters to drive the experience, such as when enemies spawn, their location, and how fast they move. The goal of this project is to have a finished framework with all of the appropriate mappings between game mechanics and audio features. The framework should allow the game to suitably portray any type of music or sound input.

Applying Recurrent Variational Autoencoders to Musical Style Transfer (Adriena Cribb, University of Pittsburgh) - Artistic style transfer refers to taking the style of one piece of art and applying it to another. While this problem has seen great progress in the image domain, it has been largely unexplored in the context of music. Adriena is building a single recurrent variational autoencoder that allows harmonic style to be transferred to any degree directly between two musical piece to ultimately produce deep learning methods for compositional style transfer and tools that allow musicians to explore novel modes of composition through the recombination of stylistic elements in different pieces of music.



2017 Projects

Click here to see student presentations from summer 2017.

Deep Learning of Musical Forms

Mentors: Daniel Gildea (Computer Science), David Temperley (Eastman School of Music). Train deep artificial neural networks to recognize motifs in classical music.

Reverse-Engineering Recorded Music

Mentors: Professors Mark Bocko and Stephen Roessner (Electrical and Computer Engineering) and Darren Mueller (Eastman School of Music). Use signal processing algorithms to discover how the same recordings were remastered over time.

Web-based Interactive Music Transcription

Mentors: Professors Zhiyao Duan (Electrical and Computer Engineering and David Temperley (Eastman School of Music). Building an interactive music transcription system that allows a user and the machine to collectively transcribe a piano performance.

The Prosody and Body Language of Effective Public Speaking

Mentors: Professors Ehsan Hoque (Computer Science), Chigusa Kurumada (Brain and Cognitive Science), and Betsy Marvin (Eastman School of Music). Measuring the visual (e.g. smiling) and auditory features (e.g. speaking rate) that cause a speaker to be highly rated by listeners.

Synthesizing Musical Performances

Mentors: Professors Chenliang Xu (Computer Science), Jiebo Luo (Computer Science) and Zhiyao Duan (Electrical and Computer Engineering). Using deep generative learning to synthesize video of a musical performer from audio input.

Reading Ancient Manuscripts

Mentors: Professors Henry Kautz (Computer Science) and Greg Heyworth (English).  Using deep artificial neural networks and probabilistic algorithms to transcribe Medieval Latin manuscripts.


2018 Participants

Erik AzzaranoRochester Institute of Technology (RIT)
Alexander BerryMiddlebury College
Adriena CribbUniversity of Pittsburgh
Nicole GatesWellesley College
Justin GoodmanUniversity of Maryland - College Park
Kowe KadomaFlorida Agricultural and Mechanical University
Shiva LakshmananCornell University
Connor LuckettAustin College
Marc MooreMississippi State University
Michael PeymanMesa Community College

2017 Participants

Jake AltabefRenssaleaer Polytechnic Institute (RPI)
Harleigh AwnerCarnegie Mellon University
Moses BugBrandeis University
Ethan ColeUniversity of Michigan
Adrian EldridgeUniversity of Rochester
Arlen FanUniversity of Rochester
Sarah FieldUniversity of Rochester
Lauren FowlerMercer University
Graham PalmerUniversity of Michigan
Astha SinghalUniversity of Maryland
Wesley SmithUniversity of Edinburgh (UK)
Andrew SmithUniversity of Central Florida