Let me just note, with no attribution, some of the comments I received from those doing the test:
"Like everyone else, I didn't find it easy - I think there are differences between the files but they sound pretty marginal."
"None of the differences I heard was due to what I would consider as a fault (such as harmonic, phase or dynamic distortion). I couldn't point to one and say it was "better" or "worse" than another - they merely sounded different."
"Our overall result was that there was so little difference that we could not always agree with each other."
"I could hardly detect ANY real difference between extracts, certainly none that would spoil my enjoyment. It's so marginal as to be virtually a random choice."
I think the net result of this test - and this is what I had expected - is that you all realize that compressed music does not sound "bad" - it may sound different - and that your ears and your hifi equipment may not be capable of making a difference anyway. Naturally, if you knew which tracks were which, your subjective mind would have said, "Oh, of course this sounds much better because...", but not knowing made you just listen with your own points of reference. For me, when I first did such a test, I realized that I could rip music at 160 kbps (I use AAC format), and save space, while still having sufficient quality. I'm 49 years old, with some hearing degradation (like most of you, I can't hear those really high notes any more), and the music sounds just fine to me.
So thanks for playing, and on to the comments, which I think will be the most interesting part of this post. :-)
OK, my bad, I thought I was placing the image after the colon in that first graf, and there are line break problems in one location, but I don't see any way to edit the post... But you get the point. Click the image to get a larger version for more readability.
ReplyDeleteOh, one more thing (damn, if I could edit this post it would be easier); the very high bit rates are the lossless rips.
ReplyDeleteOK, never mind, I found how to edit it. Now I can't delete these extraneous comments...
ReplyDeleteThanks for setting this up, Kirk, it's been a chastening experience for me :( I was galled to see that in one instance the sample I thought was 'best' was actually the worst.
ReplyDeleteDan, that's what I've found when I've done blind tests. I think the reason is the following: you have subconscious expectations as to how an instrument/group of instruments/voice _should_ sound. And when you hear something that approximates those expectations, it "clicks" with you. Also, it depends a lot in which one you hear first; you set that up as a mental benchmark, and compare the others to it. It's a tricky process, and if you all are interested, we could do it again with just two samples: one uncompressed, and one compressed (but, say, 192 kbps, so it's at the level that most people would not be able to distinguish). That might be easier, and more interesting, since you'd just be doing an AB test, and be able to focus more on a limited set of differences.
ReplyDeleteMy thanks also, Kirk. I have been heartened to discover that, in spite of the recent damage to my hearing, it can still compensate very effectively for deficiencies in the audio information presented to it!
ReplyDeleteRegarding the results, would it be possible to present a bit more detail? For example, some sort of "performance score" against age, or how those with known deficiencies compared with those with presumed good hearing?
Regarding further tests, I'd be interested to find how we got on with MP3, which is, when all's said and done, by far the most common downloading format.
Also, how about a similar test, but extending the range of compressions right down? In a fully blind test we should then ALL be able to perceive some compression artefacts somewhere within the test set. In addition, we could each establish a ratio below which the sound becomes intolerably corrupted, and a ratio above which we can no longer detect any difference.
Maybe these further tests should be based on short samples of (say) a minute's duration. This would reduce the sizes of the downloads, and help to reduce "memory" problems when making comparisons - even a minute is quite a long stretch of signal to hold reliably in your head while you listen to the next version!
Paul, I was hoping that individual testers would post their results here. I don't want to do it, for fear of embarassing some of them... Then you could compare not only results, but also, perhaps, age.
ReplyDeleteWe could do a test with MP3 files instead of what I used here, AAC. The results would be similar, but MP3s might lose out at lower bit rates (say 128 and lower).
As for other tests, you can do this yourself. Take any track from a CD and rip it with a music managemnent program like iTunes. Then change the settings to go down through the available bit rates, and try on your own. (Tip: Create a playlist where the Bit Rate column is not visible. Put all the tracks there, then click the Random button a few times to mix them up. Listen and rate.)
And, yes, if we do another, simpler test, I'd limit it to one-minute samples.
I'll go first, shall I? My score was 13 out of 30, which sounds pretty dismal :( Oh, for statistical purposes I'm 49.
ReplyDeleteI'm not sure I'll participate in any further tests, though; I think Kirk has made his point very effectively with this one.
13 out of 30 is pretty close to random. ;-)
ReplyDeleteFWIW, I'm 49 too. It's obvious that the older you are, the less well you hear. But after all, it's about the music, right, not so much the sound...
Derailed breakdown:
ReplyDeleteSample 1: zilch
Sample 2: nada
Sample 3; 3 (I correctly identified the lossless rip)
Sample 4: 5 (was that the harpsichord?)
Sample 5: 3 (got the lowest bit rate right)
Sample 6: 2 (sussed out the best and worst)
Random? Thanks, Kirk, that make me feel a whole lot better...
I didn't do the test in the sense that I gave guesses (and that's what they would have been) on each file. Instead, I wrote Kirk an e-mail titled "Surrender". Here's the body of it:
ReplyDeleteI give up with your test... I can't tell differences that I couldn't also, just as well--or even more likely--be imagining. I've not even managed to hear those yet in the first batch (on CD, but listened to at night, without taking notes or noting any particular differences.) And just for the sake of returning some impression, I've imagined to like the following files the best on the second batch: [4]B, [5]A, [6]D.
Dazzled and confused, but with best regards all the same...
----------------------------------------------
That actually looks pretty good, now, that am checking against the list... but truth is that I still feel it was pretty random guessing. I wouldn't have bet $5 at 1 to 5, and I still wouldn't, were the test to be repeated.
Three of us sat down to listen to the first three sets of sound samples, Myself, Chris Thomas and David Dyer. We were using to top of the range equipment you can see here http://www.musicweb-international.com/MusicWeb_House/Listening_Studio.htm
ReplyDeleteWe felt that this equipment is so discriminating that we would be able to clearly notice any differences.
Our overall result was that there was so little difference that we could not always agree with each other. We were usually pretty sue of the 96kbs sample [Ha! Ha! - see results below] but found it difficult to distinguish between the others. Of course our ears are different. Chris is in his 40s, I am in my 60s and David in his 70s. Sometimes, particularly with piano, we felt we could tell which was the 'best recording' but actually preferred one of the others. We were concerned though that none of those three recordings (samples 1,2 and 3) were state of the art recordings in the first place and really for this type of discrimination they ought to have been.
Because it was so difficult to tell the differences we were constantly flipping backwards and forwards comparing one with another. We had cards on a table marked for each sample and were sliding them up an down to obtain the ranking. We found we rapidly got tired, frustrated and fed up with the process and this must have affected our final results. Nevertheless, we found the whole idea of great interest and worthwhile.
The results were, with 1 being the best as requested
1A 5 Now Known to be 320 Kbs
1B 1= Now Known to be 128 kbs
1C 1= Now Known to be 96 KBs
1D 4 Now Known to be 538Kbs
1E 1= Now Known to be 160kbs
so we actually chose the 96kbs as one of the best samples!!!
2A 1 Now Known to be 128kbs
2B 4= Now Known to be 321 kbs
2C 2= Now Known to be 464 kbs
2D 4= Now Known to be 96 kbs
2E 2= Now Known to be 160 kbs
We might have drawn some satisfaction from ranking the 96kbs bottom if we had not said the 321kbs was identical
3A 3= Now Known to be 96kbs
3B 4 Now Known to be 320
3C 3= Now Known to be 128kbs
3D 5 Now Known to be 160kbs
3E 3= Now Known to be 604kbs
A startling result ranking the 96kbs, 128kbs and 604kbs as being equally good!!
Overall conclusion? All are virtually as good as each other and it is not worth paying for higher quality downloads
Chris is early 40s, Len late 60s and David early 70s. Perhaps Chris should have done the test on his own.
We never played more than a few second of each track switching rapidly between them and ranking and resampling. We spent a long time on it and were surprised at the lack of differences. There were subtle nuances but could not agree wether that made them good or bad.
Some very good points from Len.
ReplyDeleteRegarding:
"All are virtually as good as each other and it is not worth paying for higher quality downloads."
The limit I'd set is, for MP3, 160 kbps or even 192 kpbs, and for AAC 128 kbps. As I've said, I rip in AAC at 160, just because I'd rather have that extra bit (even though I can't hear the difference). If you have a choice between, say, iTunes, at 256 kbps AAC and eMusic, at around 192 kbps AAC for most tracks, I'd say the iTunes tracks will probably always sound better, but eMusic is cheaper, so it's a tough call.
The key (for me) is this:
"There were subtle nuances but could not agree wether that made them good or bad."
Because, basically, we can't even agree on what a good recording is. (Ok, in some cases we can...) Which is why it was probably better to do this test with recordings of various quality than to try and find the best recordings. Sometimes, bad recordings will sound bad more easily when compressed (at very low bit rates).
An any case, Len's comments are interesting, and, I guess, indicative of everyone's experience. (With the exception of having top-of-the-line material; but that, too, may not make much of a difference, eh? :-)
Not much point posting my results either, because most are way off - in fact sample 1 is identical to Len's in terms of 'worst' and 'best', with the 'middle' indistinguishable. I rip and 320, like others, for that'peace of mind' that you've got higher quality, though I've just done myself another blind test with some 160s I did some time ago and - guess what - can't tell a difference. So yes, I'm now going to rip everything at 160, knowing full well I'll get equally good sound and save lots of hard drive space.
ReplyDeleteCall me a stick-in-the-mud but in spite of this test I still can't get excited about downloads. I prefer a disc and booklet to a recording stored on a hard drive, even though I accept downloads are probably where the future lies.
ReplyDeleteYes Dan but I can tell from the drop-off in sales through MusicWeb of both Lyrita and Avie since they became available as downloads that many people are quite happy to dispense with the extras if it will save money. If your main listening is through an iPod or iTunes purchasers would be quite used to that.
ReplyDeleteYes, Len, I take your point about the convenience of downloads but I seldom listen to music on my iRiver these days. When I do it's always music ripped from my CD collection. I do also wonder whether some labels realise they are on to a good thing with downloads, as they don't have to manufacture and market a physical product anymore; they simply store the music on a server and wait for the customers to come to them. And, perhaps, why bother to achieve the highest technical standards when us poor punters can't tell the difference anyway?
ReplyDeleteIt seems to me that the history of recorded music has been all about striving for ever greater fidelity to the source material. The Blumleins, Culshaws and countless others who worked so hard to improve sound quality would surely be disheartened at the way recorded music is being downsized and repackaged to suit commuters listening to compressed music on a noisy Tube train.
I wonder how DG's CD sales are doing now that they have a download shop? And what the situation is at BIS and Chandos, now that their discs are downloadable?
I must be a bit of a Luddite, but the thought of storing music on a hard drive (and, presumably, making backups, as these drives will fail eventually) just doesn't appeal to me.
Incidentally, are you certain the fall-off in demand for Lyrita and Avie issues is due to their downloadability rather than the general economic slowdown? If indeed it is all about downloads do you see MusicWeb reviews becoming exclusively download-driven in the very near future?
Even though it's late - and snowing hard here in Kent - I'm going to make myself a cup of tea, peruse the shelves and sit down to listen to a shiny silver disc.
Dan, I guess there's just two types of people (as always): those who are attached to something, and can't let go, and those who accept change. I don't mean to be in any way derisive of your attitude, but the technology is changing. What a pleasure to be able to listen to Mahler's 3rd without changing discs; what fun to be able to listen to a random playlist of my thousand-odd lieder by Fischer-Dieskau, instead of only what's on that silver disc. What a wonderful thing to be able to download a disc in 3 minutes, rather than wait 3 days or 3 weeks to get a physical disc (I live in a very rural area). What a pleasure to have the selection that downloading offers, the reissues that are now available because physical stock costs are not an issue, and even the live concerts sold only by download.
ReplyDeleteThere's much to be said for both, and you can certainly cling to your archaic "compact discs", but this is progress for many of us.
I have no problem with change - I bought my first CDs before the players were even available - and I didn't 'cling' to my 'archaic' LPs either. I readily accept downloads make a lot of sense for someone in your situation but I don't wish to be tied to my computer when listening to music (nor do I feel the urge to listen to music on the go). I also don't mind waiting a day or two for the postman and changing the discs on a Mahler symphony isn't too arduous either. And then there's the small matter of SACDs, which cannot be played on a computer. Yes, I know it's just another 'archaic' technology, so I'd be better off ripping it as an mp3. :)
ReplyDeleteDan, I'm not at all tied to my computer. I put the music on my computer, then sync it to an iPod, which I then use as a source for my stereo.
ReplyDeleteAs for waiting for the postman, I do that sometimes (especially for the low-price box sets we're seeing these days), but for single albums it's about half the cost to buy by download.
As for SACDs, yea, I don't have a player, and that doesn't matter any more anway... :-)
I'm pretty much in the middle on this one. I still cherish a 'serious' listening session on the main stereo. This means the 'ritual' we all enjoy - selecting the disc, pressing play, reading the booklet, playing the whole thing. Sitting at the computer a lot inevitably means iTunes, which I love, but I have to confess it becomes more background, and I do tend to flick thrugh what's becoming a huge library of 'tunes'. I don't tend to listen through as a 'concert', and not having the booklet notes, libretto etc. can be a pain. That's why I've not gone down the Squeezebox/ server route, as I like my discs. But I do like the iPod for sheer convenience, and increasingly find myself putting review discs straight on so I can have easy access at all times. Room for both easily, as far as I'm concerned.
ReplyDeleteBTW, using the iPod as a source for a stereo is not the only solution. You can also stream music to Apple's AirPort Express, which has an audio out (optical, in fact), which connects to your stereo. Or, you can use an Apple TV, which is a sort of "living-room iPod" to play music through your stereo via a very easy-to-navigate interface. (I'm sure there are other non-Apple solutions for some of these; the Squeezebox is one example.)
ReplyDeletePersonally, I only read booklets once or twice, so that quickly becomes moot. And the sheer number of CDs I have (thousands, probably like most of you), makes sorting them difficult. I've managed to put about ten copy-paper boxes (you know, the ones that hold five reams of paper) in the basement with CDs that I don't need to keep in my office (which is where I store my CDs). I'd like to fill another few, because things are building up. In fact, I'd like to move all of them, but I've got these CD shelf units that would then be empty, and what would I do? :-) But, in fact, I haven't listened to a physical CD in years.
This said, there is some nostalgia in doing that. I'm considering getting a new stereo for my office (which is where I do a lot of listening, both during the day and in the evening), and I would definitely get a CD player if I do. The idea of listening to a "program", on discs with works by multiple composers, is something I don't do often via my iPod.
To me, the most interesting thing about Len's feedback was the formidable assault it made on the test samples - equipment that most of us can only dream about, and a panel of judges making detailed A/B comparisons. Yet, in spite of all this, their conclusions were about as inconclusive as mine!
ReplyDeleteThe second comment Kirk quoted ("None of the differences . . .") was from me. To this I will add that I'm 61 (going on 62), and just emerging, though not altogether successfully, from a year in which my left ear was under threat from a cholesteatoma.
My right ear's response is about normal for my age, extending to about 6 KHz. Over this range, the response is 15 dB (+/- 3 dB) below the audiological benchmark. The left ear zig-zags all over the place, from 0 to 50 (fifty) bB below the right one. This has largely been compensated by a hearing aid, the downsides being that the aid's specified T.H.D is 1.0% (that's not exactly hi-fi, but good for a hearing aid).
Given that little lot, I was quite "pleased" that I could at least detect some differences, even if they were so slight as to defy any attempt at ranking them.
It strikes me that our aversion to "mp3" maybe is a hangover from the days when "mp3" was young and still fairly grotty and our ears were young and more capable of detecting the grot. Now, although "mp3" has improved immensely and our ears have deteriorated, those old opinions linger on stubbornly - until they are confounded by present realities.
I know that my early exposure to "mp3" was not a pleasant experience; it wasn't a question of subtle differences, but of a strangulated travesty of the original sound. That's why, with the advent of MD, I found ATRAC so impressive, and now these AACs are reflecting that ATRAC experience. It's a thought, anyway.
I agree with EVERYBODY about CD vs. downloading! Ever since I spun my first record, nearly 50 years ago, and felt my heart sink at hearing each new scratch or scuff, I've dreamt of a reproduction medium free of all moving parts. Yet, I never dreamt of the possibility of losing the tactile thrill or "putting on a record".
At rock bottom, I'm not really right bothered where I get the music from, or in what form, not when I now have so many options as to what to do with it once I've got it. I just hope that suppliers will find ways of preserving the pictures and texts that so enhanced the pleasure of owning CDs and especially LPs. The only real problems are (a) this damnable DRM business - it's both intrusive and ill-considered, and (b) as yet, not everybody has access to unlimited or unmetered downloading or, for that matter, broadband!
Finally, the point made by Dan ("The Blumleins, Culshaws and countless others . . .") is a very telling one. As long as there are people capable of appreciating it, our ultimate aim should remain to strive for the greatest fidelity - and make all recordings available in this form. Compression should never be more than an option, for those concerned with economy, and even then only until improvements in distribution and storage media make it both irrelevant and unnecessary.
Paul
ReplyDeleteA very sane and reasonable post, thank you :) I'm making enquiries and Googling download topics in an effort to understand the various options available. I do so reluctantly, but accept that's the way the industry will probably go.
Sorry to hear about your ear, though. Glad to hear you are recovering.
bws
I’ve been away for a week and so am tardy in following up Paul’s previous point about more analysis.
ReplyDeleteI also feel that it would be a good idea to present some aggregated data and I hope Kirk will agree to this. We have taken part in an experiment and, whilst we all now have our own data and all seem to agree that it was hard to detect differences between the files, that is a subjective conclusion. I would still feel it is worthwhile looking at the combined and relatively “objective” evidence of us as a group with some specific questions in mind.
Len presented some raw data for files 1-3 but the only attempted analysis I have so far seen was by Dan and I wasn’t entirely sure that I understood it. My guess is that he compared the rankings and found 13 exact matches out of 30. A more revealing approach to such an ordered scale would be as follows:
For each sample which was actually ranked highest (i.e. 1, the six lossless files) add up your six rankings (which will each be from 1-5). Then repeat this for the second rank (i.e. the 320 kbps files) and so on.
A “perfect” score would look like this:
1 – 6
2 – 12
3 – 18
4 – 24
5 – 30
Complete lack of discrimination would be 18 for each rank and no doubt the results for everyone will lie somewhere in between. Exactly where they lie and whether there is any plausible trend will give us an idea of whether we can discriminate between files and, if so, which ones.
Note that, the sum must be 90 if you have done this correctly. If you tied some samples as equal then the ranks would be apportioned accordingly. So, in Len’s data for sample 1, the three tied first places would each get a rank of 2 (i.e. 1+2+3 divided by 3), and so on.
Analysed as above, my own data look like this:
1 – 14
2 – 20
3 – 19
4 – 15
5 – 22
It is quite interesting that I identified the lossless file as “1” for samples 1, 2 & 6 but for sample 4 – the harpsichord which was supposed to be the easiest (!) I had it as “5” and the 96kbps file as “1”. This, and the fact that Kirk’s scale is in the opposite order to the way I have used such scales in medical research (5 is always good and 1 bad), make me wonder if I had a brainstorm and this is a data error. I don’t expect to be able to convince anyone of that and I haven’t listened again because it wouldn’t be blind. The reason I mention this is to illustrate the difference just switching those two data points would make – the 1st rank score would become 10 and 5th rank would become 26. If those data were true I think it would be reasonable to say that I probably had some ability to discriminate the best and worst files with reasonable success but not within the middle ground. As it is, there is a very slight trend in that direction but it is not at all convincing and it might just be pure chance that I scored 14 at the top and 22 at the bottom (versus the expected 18). What I would like to know is how we did as a group, on average. I suspect that there might be some collective ability to discriminate at the top and bottom but that it would be far from consistent. I wouldn’t be too worried about “confounders” like age – they probably aren’t enough of us who are young enough for this to be important. And in terms of conditions of listening, I would simply regard this as a naturalistic experiment – i.e. we all listened the way we do.
Perhaps there are others who have experience of conducting such analyses – if so, I do realise that there are much fancier statistical approaches. I am merely trying to suggest something very basic that hopefully we can all understand.
The practical questions I have in mind are (1) do we have some ability to discriminate lossless versus top of the range compressed? and (2) is there a degree of compression at the bottom which is distinguishable from lesser degrees? I know from using the Naxos Library that there is an immediate and obvious difference between streaming at 64 and 20 kbps (the sound literally breaks up at the latter bit rate) but I would like to know whether this applies (to a lesser degree) at the higher levels at which one might purchase downloads. I think the answers to these questions would be useful generally to people who do now have choices and can buy the same recording at different bit rates (or lossless) and different prices.
Finally, and sorry for the long comment, I think we should bear in mind that there are two separate questions here – to what extent, if any, can we distinguish the files and do we care? We have so far answered the second but not the first. Whilst I hope everyone will go away and analyse their individual data as suggested above, I wouldn’t see multiple posts of the data as being very efficient and there is an agreed right to privacy. Thus I hope that Kirk will either agree to analyse all the data he has and present the findings or to let someone else do so (the individual data could and should remain anonymous). I am willing to help in doing this if he wishes although I really ought to be doing some work…..
Patrick
ReplyDeleteI'm sure you're right, my 'analysis' is probably statistically meaningless :( What it does tell me, in part at least, is that I got one set of samples 100% correct, two completely wrong and the rest a mix of the two.
It was certainly a chastening experience but I don't think we need to feel too bad about the results. Our ability to listen and respond critically to recordings is conditioned by so many other factors that make this test more of a curiosity than anything else.
As for downloads, for me they are still a source of last resort.