The Importance {or lack thereof} Of Gear Shoot-Outs
Written by Edward Vinatea
Friday, 03 September 2010
Many of you have probably participated in one or many gear shootouts throughout your audio engineering careers. Some of you may have been involved in the design process of the gear itself, have been hired as a survey participant for a research company, maybe you created your own independent research for your own purposes, or, you just downloaded samples from an audio forum and participated in the discussions regarding the differences or similarities of some audio gear.
But, exactly up to what extent do we really know that what we hear is a faithful reproduction? What sort of non-linear distortion is acceptable and even desirable for a particular gear design? The sound that is pleasant to you, that is, and most likely, the result of complex subjective processes. These are two of the many questions we should ask before we engage in making audio comparisons.
No matter what kind of gear you test, e.g. speaker monitors, AD/DA converters, pre-amps, equalizers, compressors, etc, etc, how do you judge the quality of gear and what makes a group of people collectively decide that certain sound is more desirable in the first place? What I mean is, how do we determine gear X is better than gear Y and closer to gear Z scientifically, and not by general consensus?
There are many ways of testing equipment, but I think we should talk first about the most basic of the tools, speakers. Without speakers we are deaf or unable to judge sound in the first place, correct? So, your judgment depends on what you hear. But, how do you know the speakers are any good to be used to judge the sound of another piece of gear like for example, a converter. Conversely, how do you know a room is truly flat acoustically or that your converters specs aren't already changing sound in a subtle way to test the speakers?
It seems that you depend upon the performance of these speakers {and all test elements}, but as some of you may know, speaker design has never been a perfect science either. Therefore, what you hear in your studio with speakers X, will sound different in someone else's studio with speakers Y. Factor in room acoustics, and you'll never be able to judge exactly what one hears in one place from another. You may have better results in an enclosed sound environment and a group of people, but you still have to deal with our cognitive mental processes and personal biases, so any test results are at best subjective in nature, at worst a complete waste of time. This doesn't stop any audio software or hardware equipment manufacturing company, and never will.
Knowing this, why do people bother to do gear shootouts? We are not talking about comparing the sonic qualities of a Soundcraftmen Dual 10 Band Stereo EQ with a Focusrite Blue 315 {well, they are not even the same kind of EQ}. But, you get my point, I hope.
You hear the arguments and the bickering on some forums but, how on earth do they know what they hear?
Bear in mind that while speakers with a response that can be specified as 20Hz-20kHz +/- 3dB can significantly vary. A speaker rated as 20Hz - 25kHz +/- 3dB will play lower bass and higher treble sounds than a speaker that measures 40Hz - 20kHz +/- 3dB. IMHO, that doesn't mean that one sounds more pleasant than the other {different yes} but at the very least you know their frequency range, and also if you look at frequency charts, their amplitude variations.
So, it appears that despite of the many advances in technology over the past 20 or so years, frequency response measurement to judge the performance of speaker monitors is still an imperfect science. The same speaker measured by two different labs will most likely render different response graphs.
And again, add the room acoustics and you know that what you hear in your studio is probably nothing more than an illusion that is only reinforced by the marketing departments of gear manufacturers that sold you the product.
On top of everything else, we tend to bring our own personal sense of audio aesthetics onto the table and that basically makes us, up to a certain degree, sort of biased listeners. IMO, gear shootouts appear to be no more accurate than the environments in which these audio tests are conducted. If they never are {accurate} then what's the point? I believe that our human condition also renders everyone physically inept to the task of judging sound, period; what I hear is not exactly the way you hear it. I am not referring to deciding if something sounds too bright or smooth either, that's all subjective stuff. I'm referring to minute degrees of changes across the audio spectrum. Now, neither I can prove any of this nor I know of any research study that can support my theory, but, the shape, direction and size {outer, middle and inner} of our ears affect the way we hear sound waves to the extend that people will have different perceptions of any given sound regardless of one's age, gender or experience level.
Thus, when someone is trying to review anything {including a music recording} they are unavoidably engaging their memory and cognitive function with one's unique physical features, i.e., ears with different sizes between eardrums and oval windows, which again, are unique to each one of us. Some of the people who work in design and all things audio were or still are musicians who may even have a permanent damaged hearing, and that's also scary.
They are certainly not the freshest, most accurate human ears to make the most important and critical design decisions {though they do have years of experience}, and that's probably still the reason why their companies have their individual research-marketing departments and group surveys before they make something available to the ever growing audio markets. Some of these people don't even realize that one ear may not be working as good as the other one. But forget it, I am not going to be talking about hearing problems or the issue of 'phase distortion' here :).
However, they do have memory and experience, of course, but that's not going to help you, the consumer, the artist, the producer or the newbie engineer at all. They can choose what they want to listen to, but they can't ever choose what they hear.
I have told my friends {and they all laughed} that there should be more blind people involved in the audio and music industries. Their accentuated audio sensory makes them better judges on all things audio. Cats and dogs could truly judge sound in ways we have never dreamed of, if they could only communicate what they can hear with us :). After all, short of a healthy set of ears and good cognitive functions, we are some of the poorest judges on audio in the planet. Even this can be considered silly by some, but thank God we have a brain that can - more often than not - compensate for all of our audio inadequacies and deficiencies.
Having said all this, it seems to me that no one is going to pay attention to any of what I pointed out and you will continue to read more about some gear shootout on some forum, or some magazine. Think about the thousands of times you've heard someone passionately talking about a recording system, a sound reinforcement unit, a pre-amp, a microphone, a set of speakers, etc. Discussing the subtleties of that gear with dozens of other users, some of who also owned that gear or did in the past {because they moved to something 'better'}, when in reality no one knows, other than by some frequency chart, the true nature of this elusive good sound.
There are absolutely legitimate ways to improve non-linearity issues upon anything {rooms, mics, speakers, etc.}, of course, but IMHO, the FACT still remains and it doesn't start with neither the room nor your speakers. It does look like it starts within some cognitive areas of your brain and how healthy your ears are and came 'configured'. Thus, there is a universe of non-linear audio distortion possibilities for us to conclude that we truly don't know what we hear.
In conclusion, and I guess all I am really saying is, if you find another gear shootout in some forum thread, don't waste your time; just skip it and always learn from what you actually hear or determine from your own listening expectations, otherwise, take whatever others say with a grain of salt because asking someone to describe what something sounds like is like telling a blind person to guess what I look like.
