Undergraduate Program

National Science Foundation Research Experience for Undergraduates (NSF REU)

Computational Methods for Understanding Music, Media, and Minds

How can a computer learn to read an ancient musical score? What can methods from signal processing and natural language analysis tell us about the history of popular music? Can a computer system teach a person to better use prosody (the musical pattern of speech) in order to become a more effective public speaker?

These are some of the questions that students will investigate in our REU: Computational Methods for Understanding Music, Media, and Minds. They will explore an exciting, interdisciplinary research area that combines machine learning, audio engineering, music theory, and cognitive science. Each student will be mentored by two or more faculty members drawn from Computer Science, Electrical and Computer Engineering, Brain and Cognitive Science, the program in Digital Media Studies, and the Eastman School of Music.

Applying to the REU

You are eligible to apply if:

  • You are a 1st, 2nd, or 3rd year full-time student at a college or university.
  • You are a U.S. citizen or hold a green card as a permanent resident.
  • You you will have completed two computer science courses or have equivalent programming experience by the start of the summer program.

It is not a requirement that you are a computer science major, or that you have prior research experience. We wish to recruit a diverse set of students, with different backgrounds and levels of experience. We encourage applications from students attending colleges that lack opportunities for research, and from students from communities underrepresented in computer science.

Apply online no later than March 10, 2017.

Before starting the application, you should prepare:

  • An unofficial college transcript, that is, a list of your college courses and grades, as a pdf, Word, or text file.  Include the courses you are currently taking.
  • Your CV or resume, as a pdf, Word, or text file.
  • A 300 word essay as a pdf, Word, or text file explaining why you wish to participate in this REU, including how it engages your interests, how the experience would support or help you define your career goals, and special skills and interests you would bring to the program.

The application website does not allow you to save and resume your application before submitting, so start the application when you have time to enter all the information.

The REU Experience

The REU is 9 weeks long, running from May 30th to July 31, 2017.

Students accepted into the REU will receive:

  • On-campus housing
  • $1,800 for meals
  • A stipend of $4,500 for other expenses and to take back home
  • $540 to help pay for your travel to and from Rochester

Your experiences will include:

  • A programming bootcamp to help you learn or improve your programming skills in the language Python.
  • Working with a team of students and faculty on one of the projects.
  • Workshops on topics such as career planning and preparing for graduate school.
  • Social events, including a trip to the Rochester International Jazz Festival.

If you have questions about the REU or application process that are not answered here, please send an email to gids-reu@rochester.edu.


You will be assigned to one of the following projects. You can specify your preferences (if any) between projects on the application form.

Deciphering Ancient Musical Scores

Mentors: Professors Gregory Heyworth (English), Daniel Gildea (Computer Science), Henry Kautz (Computer Science), David Temperley (Eastman School of Music)

The Sachsische Landesbibliothek und Universitatsbibliothek in Dresden, Germany contains perhaps the most important collection of baroque music scores in Europe, much of which was damaged in the infamous fire-bombing of 1945. In 2015, Prof. Heyworth began digitizing illegible baroque musical scores, operas, and libretti using multispectral imaging. Although imaging using non-visible frequencies of light makes parts of the notation visible, manually reading the scores requires great expertise and much time. In this project, students will apply semi-supervised convolutional neural networks (also called “deep learning”) to the task of recognizing notes and their position within staves, as well as the texts of vocal pieces.

Mining the History of Music from Recordings and Wikipedia

Mentors: Professors Mark Bocko (Electrical and Computer Engineering), Henry Kautz (Computer Science), and Darren Mueller (Eastman School of Music)

The goal of this project is to understand and mimic human capabilities for recognizing musical performance style similarity, for applications such as studies of music history, music recommendation systems, and training of musicians. Starting with a database of 30,000 recordings of the weekly "Billboard Top 100 Hits" going back to 1958, students will use highly efficient batch audio processing algorithms developed in our audio engineering lab to extract expressive features from recorded musical performances, such as dynamic level, timing variation, vibrato, pitch modulation and related parameters. We will combine the audio data with information extracted from Wikipedia entries on popular music in order to create models of various musical genres.

Web-based Interactive Music Transcription

Mentors: Professors Zhiyao Duan (Electrical and Computer Engineering, David Temperley (Eastman School of Music)

Having a system that is able to transcribe music performances into music notation will make great impacts on music education (spotting errors in a piano recital), retrieval (searching music with a similar bassline), and musicological research (analyzing a jazz improvisation). Fully automatic music transcription systems have made substantial progress over the last decade, but have not achieved the accuracy level needed for practical use with polyphonic music. Manually transcribing music performances can be accurate but is extremely time consuming and requires high music expertise. In this project, we propose to build an interactive music transcription system that allows a user and the machine to collectively transcribe a piano performance.

The Prosody of Effective Public Speaking

Mentors: Professors Ehsan Hoque (Computer Science), Chigusa Kurumada (Brain and Cognitive Science), and Betsy Marvin (Eastman School of Music)

This project on the perception and computation of musicality in speech grows out of collaborative research at UR in computer science, cognitive science, and music. Psycholinguistic research has revealed acoustic features than have high impact on listeners' evaluations of a speaker's performance. For example, reduced fillers and shorter pauses are generally positively correlated with a higher rating of confidence and intelligence. Students will begin by identifying target features and constructing an algorithm that extracts information from the audio signal to predict human ratings. The predicted ratings will then be fed back to speakers in order to help them improve their performance.

Reconstruction of Live Performances

Mentors: Professors Chenliang Xu (Computer Science), Jiebo Luo (Computer Science), Zhiyao Duan (Electrical and Computer Engineering)

When attending a live musical performance, people often use smartphones or cameras to record exciting moments, and then post these clips on social media, along with their comments. The goal of this project is to build a system that can create a rich reconstruction of a live performance from fragmentary clips. Videos will be registered and combined by linking action and scene elements using algorithms developed by Prof. Xu and others. Students will help extend these methods to use correlations in the audio signal to improve video alignment and to fill in video gaps. Textual comments users include with their posts, together with video clips of the crowd, will be synchronized with the performances and analyzed using deep learning methods to predict the shifting mood of the audience. The REU students on this project will be engaged in leading-edge research on computer vision and multimedia analytics.