Pitch & Intensity

In general, increasing the intensity of pure tones:
  a) decreases the pitch of low frequencies (<~300Hz)
  b) increases the pitch of high frequencies (>~3000Hz)
       and
  c) has no noticeable effect at middle frequencies.

In the figure to the left:
   (a) A pure tone of frequency 98Hz has
         a pitch of G₂ when quiet (ppp), and
         a pitch lower than E₂ when loud (fff).
   (b), (c), and (d) show the influence of intensity
         on the pitch for pure tones with frequencies
         392Hz, 784Hz, and 3136Hz respectively.
         (in Campbell & Greated, 1987; derived
          from Stevens & Davies, 1939).

In addition, the introduction of a high-intensity "interference" tone will change the perceived pitch of existing low-intensity tones, even if the frequency of the low-intensity tones remains unchanged.

The pitch of a low level tone will
_ rise following the introduction of a high-intensity tone at a much lower frequency and
_ drop following the introduction of a high-intensity tone at a much higher frequency.

In other words, the high intensity tone pushes the pitch of low intensity tones away from it, assuming frequency separations beyond one critical bandwidth.
(What will happen if the intense and weak tones both fall within the same critical band?)

The figure to the left  offers another, simplified illustration of the average dependence of the pitch of pure tones on intensity.
In summary:
    Increasing the intensity will
    _ lower the pitch of a low frequency tone
    _ raise the pitch of a high frequency tone
    _ have no noticeable impact on the pitch of a
       middle frequency tone

Pitch & Duration

Pitch also depends on duration. A tone must last more than a minimum amount of time (~10-60ms/3-100cycles, depending on frequency and intensity) in order to sound more than a 'click' and convey a clear sense of pitch (see below).
We will return to this during the module on Timbre.

The dependence of the pitch of periodic complex tones (i.e. of complex tones with harmonic spectra) on frequency, intensity, duration, and the introduction of intense 'interference' tones is qualitatively similar to that of pure tones but more complex, due to the variety of frequencies involved.

The pitch JND for complex tones may be larger or smaller than that for pure tones, depending on spectral context.

Pitch & Spectrum

Pitch of Complex Tones with Harmonic Spectra

Reminder: Periodic/harmonic complex tones are usually perceived as a single unit, with their multiple spectral components merging into a single tone perception rather than being perceived as a set of individual pure tones.

The pitch of periodic complex tones with harmonic spectra (i.e. spectra whose components have frequencies that are integer multiples of the frequency of the lowest component, called 'fundamental') matches (in general) the frequency of the fundamental component. This is the case regardless of whether or not the fundamental component is perceivable (i.e. even when it is too low in level and/or masked) and even if it is not physically present in the tone's spectrum at all.

Phenomenon of the missing fundamental or "virtual pitch": the pitch of a complex harmonic tone matches the frequency of its fundamental spectral component, even if this and several additional low-frequency components are missing from the tone's spectrum. 
The phenomenon of the pitch of the missing fundamental or "virtual pitch" provides evidence that pitch depends not only on frequency, intensity, and duration but also on spectral distribution. (The phenomenon challenges 'tonotopic' theories of pitch perception - see below.)

Listen to a pair of complex tones illustrating the phenomenon of the missing fundamental. 
Both have fundamental of 300Hz and up to 15 components (ramp spectrum: A_n = A₁/n for all components). The first tone has all 15 components while the second is missing the first 5. As you can hear, removing these first 5 components does not change the tone's pitch, which continues to match that of 300Hz.
 

Frequency region most significant to pitch (depending on fundamental frequency and spectral & temporal context)

• For complex tones with a fundamental of 100Hz, the pitch sensation is impacted after removing the first ~15-20 components and starts deteriorating after removing more than the first ~25 components. 
  For fundamentals of 500Hz and 800Hz, the same observations occur after removing the first ~8-10 and ~4-6  components respectively.
        Listen to this example.  It includes 13 harmonic complex tones, with fundamental of 600Hz, played in succession. 
        The first tone has all harmonic components from the 1st to the 16th and each successive tone drops one component,
        starting from the fundamental, until only the highest 3 components remain. What happens to the pitch?
    
• Experiments examining the effect of mistuning some of a harmonic tone's components on the resulting pitch have determined that the frequency region most important to pitch is between ~400 & ~1500Hz. That is, mistuning or removing components laying within this region has the most effect on pitch, while presence of frequency components within this region results in the clearest pitch sensations. 
   

Perceiving a complex signal's individual components

While harmonic complex signals are perceived as a unit, rather than a set of multiple pure tones, the individual harmonics can be heard if we draw attention to them by removing them and re-introducing them (example in Houtsma et al.,1987 - original experiments by von Helmholtz in the mid 1850s).

Tuva throat singers (peoples of Tibet and the Siberian grasslands) exploit this phenomenon to create musical passages where one singer appears to produce two pitches simultaneously: one acting as a fixed drone and one performing a sort of melody. They include:
     a) the single fundamental frequency produced, acting as a background drone, and
     b) harmonic components associated with this fundamental, selectively accentuated by the performer,
         acting as the melody line. [For more information and video/audio examples see here and here.]

Pitch of Complex Tones with Inharmonic Spectra

Inharmonic/non-periodic complex tones, whose frequency components are not integer multiples of a 'fundamental' component, do not elicit a clear, unique pitch sensation. Depending on their spectral distribution, inharmonic complex tones may
     a) elicit multiple competing pitch sensations
     b) resemble chords, or
     c) sound as noise.

As illustrated below, however, changing the spectral peaks of inharmonic complex tones without changing the frequencies of their sinusoidal components can result in a distinguishable change in pitch, matching the frequency change of the spectral peaks. This highlights the perceptual salience of changing vs. static stimuli and provides additional evidence of the dependence of pitch on spectral distribution. 

Listen to three individual major chords (C, E, & G), performed by combinations of flute, clarinet, and oboe:
Clarinet-Flute-Oboe             Flute-Clarinet-Oboe            Oboe-Clarinet-Flute

Now, listen to this "chord melody" example, consisting of a five-chord sequence, each of which is one of the three chords, above. All 3 chords are major, include exactly the same notes (C5, E5, and G5), and have inharmonic spectra (i.e. their spectral components are not integer multiples of a single fundamental, even though the spectra of the individual notes in the chords are themselves harmonic).
The frequencies of the components are identical among chords but the spectral envelopes (i.e. relative intensities of the components) are different and depend on which instrument plays what note (a flute, a clarinet, or an oboe). 

The melody you hear (C-E-G-E-C) tracks the position of the flute in each successive chord, because the flute's fundamental frequency provides the spectral peak for each chord's spectrum.  In other words, the melody you hear matches the pitch changes corresponding to the changes in the flute's fundamental frequency.
Watch the video, below.

A similar effect can also be produced through spectral shaping of noise bands. Pitch perception will track changes in the spectral peak of the noise.
 

Virtual pitch of slightly inharmonic spectra, whose fundamental or additional low-frequency components are missing:

Assume the 3-component harmonic spectrum: 800+1000+1200Hz (components 4, 5, and 6 of a 200Hz missing fundamental and a slightly inharmonic version of the same spectrum: 798+1002+1204Hz. Assume equal amplitudes for all components.

_ The virtual pitch of the harmonic spectrum will correspond to 200Hz.
 
_ The virtual pitch of the inharmonic spectrum will correspond, approximately, to the average of the three implied fundamentals: (798/4 + 1002/5 + 1204/6)/3 = (199.5+200.4+200.7)/3 = 600.6/3 = 200.2Hz.

NOTE: Minor deviations from harmonic spectra (up to ~1-2% of frequency) and the way these interact when, for example, several instruments perform together in unison, change the timbre of the combined sound rather than its pitch. They produce what is referred to as 'chorus effect': richness of ensemble sound due to slow and varying beating rates among the slightly detuned components of the complex tones involved.

Analytic & Synthetic Listening

In this example, (adapted from Smoorenburg, 1970), two complex tones, each with 2 components, are presented in succession.
(a) 800Hz + 1000Hz (b) 750Hz + 1000Hz.   When moving from (a) to (b):

• Some listeners hear the pitch going down. They follow the motion of the first component in each tone
  (800Hz 750Hz ~1 semitone drop).
  Explicit rules are employed to track the physical attributes of the two tones and determine the pitch motion.
  This is considered an example of analytic listening.
   

• Other listeners hear the pitch going up, by reconstructing the motion of the (missing) fundamental implied by the two complex tones
  (200Hz 250Hz ~4 semitone or a major third rise).
  Implicit rules are employed to synthesize a physical attribute that is implied by the rest of each tone's attributes, helping determine pitch motion. This is considered an example of synthetic listening.
  

Let's repeat the experiment, now with two complex tones, each with 5 components, presented in succession:
(a) 800Hz+1000+1200+1400+1600 (b) 750Hz+1000+1250+1500+1750.   
In this case, most listeners only hear the pitch going up by a major 3rd, perceiving the motion of the (missing) fundamental (200Hz 250Hz). (Why?)

------------------------------------------------------------------ 

In Dannenbring's (1974) demonstration:
     (masking noise bursts filling tone gaps in a steady or frequency modulated tone - from our Hearing Module)

• Listeners synthesize the sensation in a form of synthetic or holistic listening, based largely on implicit rules (rules we are not explicitly aware of).
   

• If listeners are alerted to the fact that the presented tones include gaps, they may be able to perceive the gaps by directing their attention to separate portions of the total stimulus. This is a form of analytic or 'directed' listening, based largely on explicit rules (rules we are explicitly aware of).

------------------------------------------------------------------

The McGurk effect is an auditory illusion caused by ambiguous audio/visual cues. It illustrates that perceptions are often linked to an experience-guided synthesis of information from our environment, offering another example of synthetic listening.

The McGurk effect indicates that our responses to stimuli do not necessarily match the stimuli. Rather, they reflect the best way such stimuli fit to previous experience, concurrent context, and the associated expectations.

When staring at the speaker's lips, the effect is so robust that, even when alerted to the a/v stimuli conflict, listeners/viewers are unable to listen to the a/v composite analytically. Listening with eyes closed presents no ambiguity, resulting in a different sonic perception. Watch this BBC2 clip.

PITCH THEORIES
(refer to the optional section on Basilar Membrane Encoding, Module 3)

Place (Tonotopic) Theory

Pitch relates directly to the point of stimulation on the Basilar Membrane (BM).
Different frequencies resonate at different locations on the BM (from the Hearing Module).
Frequency is encoded as pitch by the inner hair cells corresponding to the BM portion resonating for that frequency.

Watch these (simplified & exaggerated) animations of the basilar membrane motion in response to:
1000Hz,    8000Hz,    1000Hz+8000Hz,  &  a range of frequencies (.mov files).  
Read this brief outline of the place theory.

The theory was proposed by Ohm (1843), developed by Helmholtz (1862), and confirmed experimentally by von Békésy (1950s), who won the 1961 Nobel prize in medicine for his contributions to the understanding of hearing.
 

Pros

_ Applies to all frequencies.

_ Explains pathological conditions:
   diplacusis: different pitch sensations per ear for the same frequency, due to BM structural differences btw ears;
   presbycusis: pitch shift with age for the same frequency, due to BM hardening with age.

Cons

_ Cannot explain the coarseness of JND or the observed relationship of perceived pitch to intensity.

_ Unless it is modified, it cannot reliably explain two phenomena associated with the pitch of complex tones:

1. The pitch of the missing fundamental.
    
   The place theory claims that, even when missing, the fundamental frequency and other low frequency components are re-introduced as the "difference" frequencies, distortion products arising from the interaction among the existing components in the tone's harmonic spectrum (the difference frequencies between upper harmonic equals the frequencies of lower harmonics).
   The difference frequencies then excite the BM locations that correspond to the fundamental and other low-frequency components.
    
   This claim is challenged by several observations, including the pitch-shift effects (see below). These effects refer to instances where spectral modifications result in the pitch changing towards a direction that appears to be opposite to that predicted by the place theory of pitch and the "difference frequency" hypothesis.
     

2. The persistence of the missing fundamental phenomenon even when the first 8 or more components are removed.
    
   For most tones, only the first 5-8 components excite separate critical bands and can therefore elicit a unique pitch sensation. Beyond that, 2 or more components lay in the same critical band and cannot be differentiated based on a place theory of pitch. Therefore, the place theory of pitch requires energy at the BM location corresponding to the lower components for pitch sensation to register.

Temporal (Periodicity/Frequency) Theory

Periodicity/Frequency theories of pitch rely on time information, as represented on the signal itself or through its conversion to neural electrical spikes, following inner hair cell activity, to encode both frequency and intensity.

The volley theory of pitch was introduced in the late 1940s by American psychologist, E.G. Wever.
It explains pitch by combining information from a large number of inner hair cells associated with a specific place on the basilar membrane, obtained via two mechanisms:
     a) phase locking (neurons firing at a single point in the vibration cycle) and
     b) neural firing synchrony (multiple neurons firing in sync with each other).
     .
[Select the "Illustration" tab on this page for an interactive animation that describes the temporal theory.]

Whether based on the periodicity of the sound signal itself or the periodicity of the sound signal's representation in the auditory nerve (e.g. periodicity of neural spike activity), temporal models of pitch assume that the hearing mechanism extracts a signal's periodicity by performing an autocorrelation function on the signal.  Qualitatively, autocorrelation provides a measure of how well a signal matches a time-shifted version of itself, as a function of the amount of time shift (see to the right).

Pros

_ May explain why pitch perception becomes increasingly coarse past 5kHz.

_ May explain the perception of the missing fundamental.

_ Efficiently captures pitch and loudness information based on neural firing patterns and densities.

Cons

_ Cannot readily explain pathological conditions (diplacusis / presbycusis), which can be traced to
   the BM and have physiologically-based pitch perception implications.

_ Cannot readily explain the pitch shift effects.

_ Appears to break down far above 5kHz.

Hybrid Theories

Contemporary, hybrid theories of pitch incorporate aspects of both, Place and Periodicity theories (Plomp, 1964; Goldstein, 1973; Terhardt,  1974; Pierce 1990). Terhardt's theory of 'virtual pitch', introduces 'previous learning' as an additional novel component that can be developed based on either temporal or place pitch cues. It supports the development of "pitch templates" of BM disturbance patterns and neural firing patterns. These are used to support pitch decisions, including in ambiguous contexts (spectra with missing components, mistuned spectra, etc.). 

In general, it appears that the most salient/prominent components of a signal's spectrum (in terms of intensity level and frequency separation) are the most important carriers of pitch information.

In the presence of multiple complex tones, the components of each complex tone are perceptually linked together into a single percept, while each complex tone can be distinguished from the others. This appears to be due to each particular complex tone's spectral jitter (i.e. fast amplitude and frequency micro-variations that are almost synchronized among the components of each individual complex tone).

The Octave (use this virtual piano to help you with the concepts in this section)

Multidimensionality of Pitch (pitch spiral)

Pitch is multidimensional with at least three dimensions involving
a) pitch height (one dimension: frequency) and
b) pitch chroma (two dimensions, separating pitches within an
    octave and linking pitches one or more octaves apart).

Note: A variable is uni-dimensional if all its values can fit on a single straight line. E.g.:
If A, B, & C represent 3 values of a variable as points in Euclidian (i.e. regular three-dimensional) space then:
   a) if |AB|+|BC|=|AC|, the variable is uni-dimensional
   b) if |AB|+|BC|≠|AC|, the variable is multidimensional
[analogous to the linear/nonlinear distinction]

The Western chromatic scale breaks the octave down into 12 different pitch chromas. The pitch chroma dimensions represent a circularity in pitch perception.
Due to this circularity, within a single octave, the octave interval represents the largest physical distance in terms of pitch height but smallest perceptual distance in terms of pitch chroma.
 
Pitch perception therefore wraps on the octave, with scales defining sets of different pitch chromas that repeat at different pitch heights for each new octave. This is best represented by a pitch spiral (versus a pitch scale).

The perceptual circularity of pitch is explored in Shepard-tone scales/slides (after 20th century American cognitive scientist and author, Roger Shepard) that present the paradox of a continuously ascending (or descending) pitch.

Listen to two pitch spiral examples (Houtsma et al., 1987).

Shepard scales/slides are the auditory analog of the continuously ascending/descending staircases, explored conceptually by English mathematecian, Sir Roger Penrose and artistically by 20th century Dutch graphic artist, M.C. Escher (see the images, below).

                          Relativity (Escher, 1953)                                                                        Ascending-Decending (Escher, 1960) 

[OPTIONAL SECTIONS]
(also refer to the optional section on Basilar Membrane Encoding, Module 3)

This results in the neural signals of sinusoidal inputs
i) having a pulse-like shape with repetition rate equal to the input's period (i.e. they represent frequency in the time domain) and
ii) being rectified (i.e. they are constricted to the positive portion of the two-dimensional signal graph).

The process of phase locking is closely related to hearing's "temporal coding theory" of encoding frequency information:

The inner hair cells release neurotransmitters only when the basilar membrane moves in one direction, towards the scala media. They therefore only respond to the positive portions of incoming signals, with their stereocilia opening their ion channels when bending against the TM mostly at a signal’s positive peak.
 
Temporal coding assumes that, thanks to phase locking, the auditory system encodes periodic information through the firing rate of neurons. This rate corresponds to the incoming signal’s period (or some multiple of it) and, therefore frequency.

Since neurons are not fast enough to encode high frequencies, more than one neuron must be involved in the process. Each neuron fires at some of the peak portions of an incoming signal and, after adding the outputs of all neurons, the signal is represented to the brain in a manner similar to that shown at the bottom graph (left).