Correcting a Rehearsal and Training Music++
Music++ is a program that accompanies a musical soloist with an ensemble of musicians. Music++ was developed with "classical" music in mind since it assumes non-improvisatory music with considerable room for expressive nuance, though, of course, lots of music fits this general description. The program is unique in the way it listens to a live player and generates an accompaniment that actually follows the soloist. Furthermore, the program learns from past rehearsals, so as to better anticipate and follow your musical interpretation in future performances. Music++ was created by me, Christopher Raphael, over a period of many years, integrating a variety of interests of mine including musical performance, audio recognition, probabilistic modeling, artificial intelligence, and computer programming. I believe this program can make a significant contribution to the lives of musicians by providing a sensitive and tireless ensemble of musicians whose only goal in life is to provide support for your musical ideas. I hope you will find that Music++ serves well as a laboratory for musical exploration, learning new music, and, of course, enjoyment.
I continue to develop the program and welcome your polite comments, suggestions, and bug reports at musicplusplus@gmail.com. Further information about the program can be found at my web site: http://xavier.informatics.indiana.edu/~craphael/music_plus_one/
What follows is a section of short "chapters" that introduce the basic functionality of the program.
Since you are reading this documentation, I will assume that you have successfully found and run the setup program which installs Music++ onto your computer, and have managed to launch the program. When you do this you will see the program's Spectrogram form, which is the main form of Music++ and looks like this:
You also have probably located the program help: Help/HowTo, since that is, most likely, how you are reading this. To begin using the program you must first identify youself and say what piece of music you want to work on. To do this select Main/Choose Piece which brings up the Setup form:
First choose Player which will bring a form that looks like this:
Here you should add yourself to the list of players by typing your name in the New Player window, choose the appropriate instrument category, and click on the Add button. Your name should appear, highlighted, in the Player List. Choose Ok to return to the Setup form.
Next, click on Score which gives a form looking like this:
Highlight the Faure Sicilienne for Flute and Piano and click on Ok. The audio for the Faure Sicilienne, as well as a part for the soloist, has been distributed with the program to allow the program to work without further data files. The solo part can be found in the directory Program Files\Music++\scores\faure_sicilienne named faure_sicilienne.pdf. It doesn't matter if you don't play the flute, as long as you can play some instrument in the treble range.
Let's go ahead and perform a short section of the piece to see how the program works. From the Spectrogram form, choose Main/Perform Live which brings up the Live Performance form:
Before you can play with Music++, you need to tell the program what section of the piece you are going to work on. Click on Range bringing up the form:
Type “1” in the Start Measure window. A list of possible starting places in the first measure will appear in the Full Start Position window. For instance “1+0/1” corresponds to the beginning of the first measure, while “1+3/8” means 3 eighth notes into the first measure. Highlight the “1+0/1” starting position. Similarly choose “9+0/1” for the ending position, by typing "9" in the End Measure window and selecting "9+0/1" from the Full End Position window. You can name your section for future reference by typing a name such as “beginning section” in the Selection Tag window, though this is not necessary. Choose Ok to move back to the Live Performance form. You will see your chosen section at the bottom of the form.
Before you actually start playing, try clicking on Live followed by Stop several times. You should hear the beginning of the piano part and see the green light come on above the volume indicator on the right side of the form, while the indicator should fluctuate with the changes in sound level. (The first time you do this you may need to wait for a moment before anything happens). If you do not hear any audio, check the settings on your computer’s audio mixer to make sure you are allowing audio output. If you don’t see any movement in the volume indicator, most likely either your computer does not have a built in microphone or you need to unmute and/or increase the microphone volume from the Windows audio control panel. If your computer doesn’t have a built-in microphone, any simple computer microphone will do at present. See the Sound Setup section for more information about audio issues for Music++.
Now you are ready to begin playing with Music++. The simplest arrangement is to use headphones to hear the computer’s audio. Other possibilities are discussed in the Sound Setup section, though, for the sake of simplicity, just use any simple pair of headphones for now. Just in case you need a little time to get your instrument ready before the accompaniment begins, you may wish enter a number in the Delay Secs window. Music++ will wait this number of seconds before the piano begins.
Now click on Live to begin the performance and this time play your part at the appropriate time. Most people think they are bound to follow the piano when they first begin playing with the program. You should take liberties with the performance to see how much freedom you actually have – you may be surprised! Keep in mind that Music++ is trying to learn from you and cannot succeed in doing this unless you show what you want to do.
Be sure to watch the volume indicator to see that your level does not get up to the very top or spend too much time in the “red zone.” The volume indicator should hover around ½ to ¾ of the way to the top, on average. When you have finished the excerpt without interupting with the Stop button, the program will return you to Spectrogram window with a visual depiction of your performance. I’ll discuss what you should do with the Spectrogram window in a later section, but for now feel free to return to the Main/Live Performance form and try this excerpt again or other sections as well. Again, remember that to take advantage of what Music++ has to offer you must lead the performance.
My oboe teacher, Ray Still, tried to impress upon me the value of listening to recordings of my playing – “facing the music” as he called it. Music++ facilitates this process by allowing “random access” to your audio – letting you begin listening from any point in the music, as well as providing a visual depiction of your performance. Some people find they can see things in their performance that they cannot, at first, hear easily. Any previous performance can be recalled, reviewed, or mixed with orchestra. Music++ allows the result to be placed into a wave file that can be played on any media player, shared with friends, or just studied for its instructive value.
For example, using the Main/Choose Piece selection from the Spectrogram window, choose the player as “chris” and the piece as “Mozart 4tet for oboe + strings, mvmt 3.” Now choose Main/Choose Perf, giving the following form:
Highlight the performance you want to here (in this case there is just one) and click on Okay. The result will be to recall the performance into the spectrogram window, looking like this:
The spectrogram is a visual depiction of audio that shows frequency content (up and down) evolving over time (left to right). Each musical note can be recognized as a collection of equally spaced horizontal bands representing the “harmonics” or “overtones” of the musical note. The wider the spacing between the harmonics, the higher the pitch. The brightness of these bands is a measure of the strength or loudness of the various frequencies. In the above spectrogram, vertical red lines mark the note onset in the excerpt. With only a little exposure, you will quickly understand the relationship between what you see and what you hear.
Since the audio has been matched to the score by Music++, it is easy to navigate to any particular place you may wish to hear. You can move forward or backward one note by using the “f” and “b” keys, or move in larger increments with the “F” and “B” keys. As you do this, you will see the green vertical move, marking your current position in the performance. Your current position is also displayed in the status bar on the bottom of the form. For instance, 2+0/1 means the start of measure 2. You can skip to any arbitrary measure by typing the measure number in the Jump to Meas box, or simply scrolling through the audio with the “f,F,b,B” keys. To play from the current position, simply hit “p” – you can stop playing with “q” (alternatively, you can use the Play and Stop buttons. When you play the audio, you will hear clicks identifying the note onsets. We will return to the purpose of these markers in a later section. If don’t want to hear these clicks or see the vertical note markers, change the Note Markers radio button on the Spectrogram form to "Out."
You may also wish to hear a mix of your solo audio with the pure orchestral audio generated by the program. To do this, change the Accompaniment button to "In." Now when you play back audio, you will hear a mix of the two audio files. You can adjust the mix using the Options/Audio Settings option from the Spectrogram window giving the following form:
The bottom three sliders are used to control the mix.
1. Balance controls the proportions of solo and orchestra
that are mixed together. The orchestra channel is “pure orchestra”
– it will never have any contribution of the solo part mixed
into it. However, depending on how you have recorded, the solo
channel may have a contribution from the orchestra mixed in. If
you want to have two “pure” channels, you must record
hearing the orchestra with headphones.
2. Center controls the panning between the two channels.
In this case the panning position is realized by varying the overall
levels in the left and right speakers, so that, for instance,
moving the Center slider to the left puts all of the audio into
the left channel, while putting the slider in the middle gives
equal amounts of audio from both speakers.
3. Volume is, of course, the overall volume. When the volume is set in the halfway position, you will be hearing “full volume” – exactly the level of the original recording. Music++ allows you to move beyond this level, essentially amplifying the original audio. It is fine to do this, however, there is a danger of introducing undesirable “clipping” into the audio if the sample values go beyond their maximum value. The effect of clipping is usually a quite obvious degradation in audio quality.
You can save your mixer settings for future use by clicking on the Save button. If the same player revisits this piece at a later time, the saved settings will remain in effect. Once you are satisfied with the performance and the mix, you can export the result into a wave file by choosing the Spect/Output Wave option from the Spectrogram window. The program will help you choose where to place the resulting audio.
Getting Audio Data into Music++
Music++ comes equipped with the Faure Sicilienne right out of the box. While this may be enough to acquaint you with what the program does, I hope you will want to play other pieces. This section tells you how to do this.
Music++ produces its audio output by resynthesizing prerecorded audio in such a way that the ensemble instruments follow the live soloist. It is not possible to get the program to work without this prerecorded audio. These ensemble performances may come from a variety of sources, such as Music Minus One, but in nearly all cases are not something that I can distribute freely. For this reason, you will need to buy the ensemble recordings to make the program work. In case you are wondering, I derive no benefit when you buy any of these recordings.
When you choose a piece for the first time, for instance the Beethoven Romance No. 2 for Violin and Orchestra, Music++ will not have this necessary audio recording. In this case, you will be prompted by the following form:
At this point you should obtain the named cd and insert it
into the cd drive as directed. It is essential that you use the
exact recording that is specified on the form—if you try
to use a different recording, Music++ will not allow you to proceed.
When the correct recording is used, the program will read the
audio data from the cd. You will not need the cd after this stage.
You are now ready to play the new piece with accompaniment.
Correcting Performances and Training the Program
Music++ adapts to each player of each piece by learning aspects of the player’s interpretation and habits. There are two parts to this learning. The first helps the program to “hear” the player more accurately. The second examines the musical timing of past performances to help the program to predict more accurately how the music will evolve during live performance. This latter task is important for coordinating the accompaniment with the soloist. Neither of these aspects of learning requires the player to play exactly as in past performances, since this would, of course, be unrealistic.
The learning makes use of performance examples in which the locations of note onsets are marked. While the initial marking of the audio is done by the program, it will be your responsibility to make sure that the markings are correct and to fix any errors that may have occurred. While Music++ may work well without corrected performances, this process is essential if you want to get the most out of the program. In many cases you will only be required to check and “sign off” on the audio marking, or to make very minor changes. While this process may seem unfamiliar at first, with a little practice you will catch on fast.
After you play though a section of music with Music++, or recall a previous performance, the Spectrogram window will appear with a spectrogram representation of your audio. Now, using the Main/Choose Piece selection from the Spectrogram window, choose the player as chris and the piece as Mozart 4tet for oboe + strings, mvmt 3. From the Main/Choose Perf choose the only example in the directory: Mozart_k370_rondeau.125. After the program computes for a few seconds you will see an image appear in the window like this:
At this point recall that you can navigate through the file as discussed in the Reviewing a Performance section. When you do this you will hear the audio performance with clicks played at the note onsets. It would be a good idea to refresh your memory on these controls if you need to.
Now move to the note that begins at the start of the 62nd measure (marked as 62+0/1 in the status bar on the bottome of the Spectrogram window). You can do this either with the f,F keys, or by entering "62" into the Jump to Meas window. You can see by looking at the image that this note is marked as occurring later than it actually does in the audio. You can also hear this by backing up several notes before the 62nd measure (62+0/1) and playing from that location. You will hear that the click that should coincide with the beginning of the measure in fact comes late. To fix this error, move your current position to the note at 62+0/1. Now use the “l” to move the marked position to the left. Keep typing “l” (or hold down the key) until the note appears to be in the correct position. If you go too far, use the “r” to move the marked position to the right. You can verify the position by listening as well, by moving backward a note or two and playing the audio. With a little practice, you can fix any errors in a marked performance fairly quickly.
Occasionally, it may be difficult to tell if a section of very fast notes is marked correctly. In this case, it is possible to view and listen to the audio only marking the beats or measures. To do this, choose the Spect/Set Increment option from the Spectrogram window giving the following form:
Initially the program will show all note positions as indicated on the form. You could change this to show only the notes that begin on barlines by choosing “barlines.” Try this and observe how the marking of the image changes. Now when you play the audio file you will only hear clicks on “downbeats.” Similarly, you could also choose “1/4” which would highlight only the notes that begin on multiples of 1/4. This would be a rather odd choices for a piece, like the Mozart Rondeau, in 6/8 time, since it marks notes that begin on the downbeat, as well as the first and second quarter notes. 3/8 would probably make more sense in 6/8 time.
Sometimes it may be easier to first examine and correct the markings at a rather coarse level, such as the 3/8 divisions above, and then to fix the remaining notes later, viewing all note onsets, if necessary.
Once in a while a group of notes are all mismarked. When this happens, rather than fixing each individual note, find the first note that is incorrect and fix this one. Then select the Spect/Realign option from the Spectrogram form. This causes Music++ to realign its score to the audio keeping in mind the corrections you have made. Often the program can use the information you have provided to fix the errors that you have not yet corrected. It is also a good idea to Spect/Realign the markings after correcting at a coarser level such as measures or beats, then reexamining the markings viewing all of the notes.
When you have finished correcting the audio file, or just want to save the changes you have made for a later session of editing or review, hit the Save button on the Spectrogram form giving the following form:
Here you need to choose between one of the three options presented to you.
1. No. If you have not completely corrected the “match”
(the vertical line marking note boundaries), or your playing contained
significant errors or was played in a way that is definitely not
representative of what you intended, choose No. This is always
the safest choice if you are in doubt, since you can change this
decision at a later date. When you choose No, the markings are
simply saved, but will not be used to train the program in any
way. On the other hand, choosing this option will not allow Music++
to learn from your example.
2. Correct. If your performance is marked correctly
(the clicks correspond with the note onsets for all notes when
you play the performance back), and the playing does not contain
significant errors, choose Correct. You should make this choice
when the performance is accurately marked, even if you do not
feel the performance is representative of the kind of interpretation
you want to teach the program. Performance examples labeled as
Correct are used to help the program learn to “hear”
your performance better by allowing the program to observe the
actual note lengths in a particular example. Knowing the “distribution”
on length for each note is important information for generating
the match between score and audio which is the essential task
of the listening process. Thus, any performance where the marked
note lengths are more-or-less reasonable would be a good candidate
for this choice. Often places in the music that are incorrectly
marked on the first couple of performances will automatically
be fixed on subsequent performances when you correct the markings
and save as Correct.
3. Improvement. If a performance is marked correctly and is representative of the direction you want to move in from an interpretive standpoint (tempo changes and rhythmic subtleties), you should choose Improvement. A performance categorized this way will be used both to train the listening model, as in Correct, and also to train the performance model. This latter model is concerned with the timing evolution – your rhythmic interpretation. You should choose this category when the rhythmic interpretation exhibited in the performance is about as close, or closer, to your desired goal than in past performances. You should not choose this option if you think the performance is further from your eventual goal than past performances.
Music++ requires a microphone to get the sound of your playing into the computer. Many computers have microphones built in and do not require you to supply an external microphone. You can tell if you have a working microphone by watching the microphone monitor on the right of the Live Performance form during performance. When the computer is recording, the circle at the top of the monitor should turn green while the volume indicator should fluctuate with the incoming sound level. Take care to ensure that the indicator does not reach the top of the monitor since this will result in low quality audio. Try to have the indicator around ½ to ¾ of the way to the top on average. It is okay to occasionally turn the light red, though this should be relatively infrequent.
If the indicator stays fixed near the bottom of the indicator after you begin your performance, first check the Windows controls to see that the microphone is not “muted” and that the recording level is above the minimum value. If you still do not see the volume indicator fluctuate during recording, your computer probably does not have a microphone built in. For most purposes a simple and inexpensive computer microphone will work fine. However, if you wish to use the program to create nice-sounding recordings of you playing along with orchestral accompaniment, you may wish to get a higher quality microphone – even if your computer has a built in microphone.
Music++ expects to “hear” some of the accompaniments on the received microphone signal. However, if the accompaniment figures more prominently than the soloist on the microphone input, the program has a tendency to “chase its own shadow” by responding to the accompaniment it hears rather than the soloist. There are two ways to ensure that this doesn’t happen. The simplest is to use headphones to hear the accompaniment. This way the accompaniment generated by the program makes no contribution to the microphone signal. However, if you want to share your music with others, you may wish to hook up speakers to your computer so that both your sound and the accompaniment can fill the room. If you do this, be sure that the microphone and the speakers are well separated and that the distance between speakers and microphone is several times that of the distance from soloist to microphone. For instance, using a microphone that is placed directly on the instrument would be an ideal solution. Do not try to use both an onboard microphone and onboard speakers together. While this may work to some degree, you will never realize Music++’s full potential this way.
Most of the knowledge used by Music++ is learned from you performance examples. However, it is also possible to tell the program directly what to do in certain situations by setting “cues,” tempo changes, and by specifying sections where the program will not try to follow you. These situations are handled with the Custom/Set Tempo form, accessible from the Spectrogram form. Choosing this option with the Mozart Rondeau gives the following form.
When you first bring up the Set Tempo form on a piece of music, it contains the preset settings that come with the piece. The possible types of settings are
1. Cues: places where the program will wait for the
soloist to play before responding, such as after a fermata or
the beginning of a section in which the soloist and accompaniment
start together. Generally speaking, any situation in which players
would need a visual gesture to coordinate is a good candidate
for a cue. However, the accompaniment will always be a little
bit late at a cue point, since the program will not play until
it has detected to soloist’s note. So you should avoid cues
where they are not needed.
2. Tempo Settings: places where the program will play
at tempo that you specify. This is the way to control tempo in
places where the soloist does not play.
3. To Accomp and To Solo Settings: places that mark a section in which the program will not attempt to follow the soloist, as well as point at which control returns to the soloist. The tempo will change to the tempo you set at a “To Accomp” and continue to ignore the soloist until the next “To Solo” setting.
For instance, if you wished to change the presets at bar line of measure 35 in the example, click on the line beginning with 35+0/1 in the right pane of the form. The current settings will then be displayed and highlighted on the form:
You can see that the current settings direct the program to switch control to the soloist (“To Solo”) at measure 35, expecting the soloist to play at about 80 beats per minute (1 beat = 3 eigth notes). If you wished to make this a cue point and to tell the program you will begin playing at a different tempo, click on cue in the bottom left pane, and choose the beat length and beats per minute corresponding to your tempo choice. Then click on Add/Change at the bottom left of the form. Your new settings will be reflected in the right pane of the form. If you wish to make no more changes, just choose Save, causing the form to close. No change will go into effect unless you choose Save.
You may wish to add a setting at a position other than those listed in the right pane. To do this, simply type the measure number in the Meas window and choose the precise measure location from the Position window. Now choose your tempo and setting as before and hit Add/Change. Similarly, a setting can be deleted from the list by highlighting the setting and hitting the Delete key. Remember that no changes actually go into effect until you choose Save. Pending additions will be highlighted in green before the change is made final with the Save button, while pending deletions are highlighted in Red.
The Options/Mixer menu selection from the Spectrogram window brings the following form which can be used to control several performance parameters.
One of these is the mixer controls (the three bottom sliders) discussed elsewhere. For live performance, only the Volume will be relevant.
The Tuning slider at the top of the form allows you to control the pitch of the orchestral accompaniment. The initial setting is to A=440 Hz, though you shouldn’t interpret this number too literally. If the recorded orchestra actually played at a single pitch level and tuned to A=440 Hz, then this initial setting would, in fact, mean just what it says. In this case moving the slider up and down would cause the orchestra to retune to whatever level you set. In the more realistic case the orchestra plays at a variety of pitch levels and tunes to some unknown standard. In this case, moving the slider up and down moves the tuning higher and lower. You will need to find a satisfactory pitch level by listening to what is comfortable. You should not expect any consistency between pieces with regard to tuning. The Tuning slider only affects the orchestra tuning during live playing -- any previously recorded orchestral accompaniment will remain at the tuning setting used when it was recorded.
The default choice of the audio Sampling Rate is 48 KHz, meaning 48,000 samples per second are recorded while you are playing. 48 KHz is a high sampling rate by today's standards, and it is unlikely that increasing the sampling rate will produce any noticeable improvement in audio quality for a human listener. At least I can't hear any difference. The two simultaneous channels (solo and orchestra) recorded by Music++ will generate quite a bit of data when sampling at 48 KHz – about 12 MB per minute of audio. If you are only concerned with the audio quality of the program's real-time output (which is always 48 KHz), and not with the quality of the audio files that are stored on disk, you may wish to change the Sampling Rate setting to 8 KHz which leads to a much lower consumption of disk space. You will observe a decrease in the quality of the audio files saved by the program, though there will not be any degradation in Music++'s ability to understand your audio.
If you choose to listen to the audio output on headphones, rather than speakers, then you should tell the program you are doing this by choosing the "Headphones" selection from the Accompaniment Output buttons. This makes life easier for Music++ which no longer needs to factor in its audio output “shadow” to the incoming audio it receives.
I am hoping that you will help me continue to develop this program. If you find any bugs in the program, please send me email at musicplusplus@gmail.com. Describe the platform you are using and explain, as precisely as you can, the sequence of steps that led to the behavior. Also please include the file “output.txt” in the Program Files/Music++ directory. This file is rewritten every time you run the program, so please send the file before you run the program again, so that it will contain useful information.
I suppose it is also okay if you simply email to tell me how much you like the program.
Chris