A better way to measure a DAC's performance

[manisandher]

I've just posted this on the dCS community forum, but thought people here might be interested in it as well...

For a long time, I’ve wondered how useful the ‘standard’ battery of tests (THD+N/SINAD, IMD, jitter, etc.) really are for measuring a DAC’s performance. They all rely on the use of single or multiple tones, which don’t reflect all the elements that make up music and are certainly not what we listen to. I understand why humans might not be used to test new pharmaceuticals, but I don’t understand why music is not used to test audio gear.

It seems to me that the gold standard would be to compare what comes out of the DAC with what goes in… using real music. And that’s exactly what a null test attempts to do.

Null tesing is fraught with difficulty, which is probably the main reason why it isn’t used much. The main problem is in aligning the two signals (what comes out with what goes in). They need to be matched in time and in amplitude. The former is best done by syncing the DAC and ADC clocks and then aligning at the sample level. The latter requires that the RMS values of the two signals be matched as closely as possible (better than 0.001dB).

There’s a massive thread on Gearspace that attempts to use null testing to rank DAC/ADC combos, with very misleading results, unfortunately. The method used there is severely broken.

There is a piece of software called DeltaWave that can be downloaded for free and is by far the best null-testing tool I’ve come across.

I’ve just compared 3 DACs using DeltaWave:

1. dCS Scarlatti
2. RME ADI-2 Pro fs r
3. Okto dac8 PRO

The RME and Okto DACs are modern DACs with state-of-the-art performance according to the standard battery of tests, and would handily beat my Scarlatti in this regard, I suspect.

But can null testing shed any further insights? Well, here are the results of my null testing (using the same classical track as used in the Gearspace test):



In absolute terms, the RME and Okto are indeed superior to the Scarlatti - they have deeper nulls (RMS differences against the source of -73.80dB and -71.02dB respectively). However…

The RMS difference does not take into account certain effects which can skew the results, e.g. the effects of different anti-imaging filters out of band. The A-weighted difference does, and here the Scarlatti is superior.

The ‘PK Metric’ is a (very clever) measure that the developer created to give a better indication of the DAC’s ‘audible and perceptual accuracy’. And again, the Scarlatti is superior.

Subjectively, the Scarlatti sounds ‘fuller’ and more laid back than the modern DACs. Null testing suggests it’s also the most perceptually accurate.

Now all I need is for dCS to send me a Vivaldi APEX to test (preferably on long-term loan) ;-)

Mani.

 

[manisandher]

As a slight aside, if you look at my RMS Difference measurements, all 3 DAC/ADC combos would handily beat the best of the best on the Gearspace thread. That the 3 DACs are so good doesn't surprise me, but that a 25 year-old ADC could perform so well blows my mind.

 

[nomore landings]

Ears do the job, imho. My DAC has 8 filter options plus choice of upsampling up to 352.8/384. Plus full MQA. XLR vs rca output. Usb vs coax vs aes EBU input? Testing it against 2 other DACs with similar options would prove somewhat laborious.

 

[manisandher]

Ears do the job, imho. My DAC has 8 filter options plus choice of upsampling up to 352.8/384. Plus full MQA. XLR vs rca output. Usb vs coax vs aes EBU input? Testing it against 2 other DACs with similar options would prove somewhat laborious.

Unfortunately, my ears are easily fooled... in the short-term, at least.

I remember years and years ago when I bought my Rotel Michi RHCD-10 (beautiful CD player BTW). It was the first player I owned that had different filters - a regular brickwall and a what I now know to be a 'slow' filter. Comparing the two by ear, I was convinced that the slow was easily the better - more detail, more shimmer, more ambience. However, over a period of a few weeks (of heavy listening) I reverted back to the brickwall. The slow filter just began to sound overly 'shiny' and a bit thin low down - impressive on first hearing, but then somewhat fatiguing over the long run.

And I had a similar experience with these three DACs. On first listening, the dCS sounds almost boring in comparison to the other two, both of which seem to grasp your attention more. They genuinely seem to have more detail. But with prolonged listening, the dCS's virtues start coming through, especially from 1kHz down.

The null testing method also helped me to select which of the dCS's 6 filters to choose. Perhaps I would have chosen this filter eventually anyway (a much more difficult task than with just 2 filters), but it wouldn't have been my first choice on immediate listening. It took me less than 30 minutes to test all 6 filters and arrive at which is 'audibly and perceptually' the most accurate .

 

[JohnnyNapalm]

I've just posted this on the dCS community forum, but thought people here might be interested in it as well...

For a long time, I’ve wondered how useful the ‘standard’ battery of tests (THD+N/SINAD, IMD, jitter, etc.) really are for measuring a DAC’s performance. They all rely on the use of single or multiple tones, which don’t reflect all the elements that make up music and are certainly not what we listen to. I understand why humans might not be used to test new pharmaceuticals, but I don’t understand why music is not used to test audio gear.

It seems to me that the gold standard would be to compare what comes out of the DAC with what goes in… using real music. And that’s exactly what a null test attempts to do.

Null tesing is fraught with difficulty, which is probably the main reason why it isn’t used much. The main problem is in aligning the two signals (what comes out with what goes in). They need to be matched in time and in amplitude. The former is best done by syncing the DAC and ADC clocks and then aligning at the sample level. The latter requires that the RMS values of the two signals be matched as closely as possible (better than 0.001dB).

There’s a massive thread on Gearspace that attempts to use null testing to rank DAC/ADC combos, with very misleading results, unfortunately. The method used there is severely broken.

There is a piece of software called DeltaWave that can be downloaded for free and is by far the best null-testing tool I’ve come across.

I’ve just compared 3 DACs using DeltaWave:

1. dCS Scarlatti
2. RME ADI-2 Pro fs r
3. Okto dac8 PRO

The RME and Okto DACs are modern DACs with state-of-the-art performance according to the standard battery of tests, and would handily beat my Scarlatti in this regard, I suspect.

But can null testing shed any further insights? Well, here are the results of my null testing (using the same classical track as used in the Gearspace test):



In absolute terms, the RME and Okto are indeed superior to the Scarlatti - they have deeper nulls (RMS differences against the source of -73.80dB and -71.02dB respectively). However…

The RMS difference does not take into account certain effects which can skew the results, e.g. the effects of different anti-imaging filters out of band. The A-weighted difference does, and here the Scarlatti is superior.

The ‘PK Metric’ is a (very clever) measure that the developer created to give a better indication of the DAC’s ‘audible and perceptual accuracy’. And again, the Scarlatti is superior.

Subjectively, the Scarlatti sounds ‘fuller’ and more laid back than the modern DACs. Null testing suggests it’s also the most perceptually accurate.

Now all I need is for dCS to send me a Vivaldi APEX to test (preferably on long-term loan) ;-)

Mani.

I am very interested in these results as I am looking closely at the RME Pro as a DAC for my PC. Please could you give me your opinion on this unit as compared to the others? Would you purchase again? Thanks in advance.

 

[rabski]

I agree the idea of comparing what goes in with what comes out has some validity, but I can't see how it's practical with a DAC. What goes in is digital and what comes out is analogue, so the two cannot be directly compared.

Also, the comparison is problematic. Null testing is inherently subject to the measurement limits and tolerances of the entire 'chain' of measurement. The idea is that as you are looking only for differences, then many effects are cancelled out. Unfortunately, it is not quite that simple, as everything has a degree of error. In many cases the differences that we are actually very good at hearing may well be lower than the measurement tolerances.

There is no silver bullet, which is why most of us rely on a combination of things to assess stuff. Human hearing is incredibly acute and extremely sensitive to certain things, but then audio memory is extremely poor.

 

[manisandher]

I am very interested in these results as I am looking closely at the RME Pro as a DAC for my PC. Please could you give me your opinion on this unit as compared to the others? Would you purchase again? Thanks in advance.

I can only comment on the RME Pro black edition. I think it is excellent and would heartily recommend it even if used only as a DAC.

 

[manisandher]

I agree the idea of comparing what goes in with what comes out has some validity, but I can't see how it's practical with a DAC. What goes in is digital and what comes out is analogue, so the two cannot be directly compared.

Also, the comparison is problematic. Null testing is inherently subject to the measurement limits and tolerances of the entire 'chain' of measurement. The idea is that as you are looking only for differences, then many effects are cancelled out. Unfortunately, it is not quite that simple, as everything has a degree of error. In many cases the differences that we are actually very good at hearing may well be lower than the measurement tolerances.

There is no silver bullet, which is why most of us rely on a combination of things to assess stuff. Human hearing is incredibly acute and extremely sensitive to certain things, but then audio memory is extremely poor.

Yes, it's definitely challenging. Firstly, you need a good chain, with the ADC being the key here. You need to be able to sync the DAC and ADC clocks. And finally, you need to be able to align the digital files really, really well.

My null tests are proving totally repeatable and consistent, which gives me confidence in the results I'm getting.

I agree that our ears may be able to hear below measurement tolerances. For example, to my ears the 6 filters in my DAC sound very different to each other, and yet my null measurements show only small differences... but differences nevertheless.

Yes, no silver bullet. But another tool in our arsenal.

 

[JohnnyNapalm]

I can only comment on the RME Pro black edition. I think it is excellent and would heartily recommend it even if used only as a DAC.

Thanks for you endorsement. I am seriously considering getting one just after Christmas if finances allow.

 

[Hornucopia]

Also the stimulants consumed during the listening?

I had to throw away my little herb stalk.....

name that tune?

 

[Paul68]

Also the stimulants consumed during the listening?

I had to throw away my little herb stalk.....

name that tune?

Rebel music bob marley and the wailers.

 

[nomore landings]

Unfortunately, my ears are easily fooled... in the short-term, at least.

I remember years and years ago when I bought my Rotel Michi RHCD-10 (beautiful CD player BTW). It was the first player I owned that had different filters - a regular brickwall and a what I now know to be a 'slow' filter. Comparing the two by ear, I was convinced that the slow was easily the better - more detail, more shimmer, more ambience. However, over a period of a few weeks (of heavy listening) I reverted back to the brickwall. The slow filter just began to sound overly 'shiny' and a bit thin low down - impressive on first hearing, but then somewhat fatiguing over the long run.

And I had a similar experience with these three DACs. On first listening, the dCS sounds almost boring in comparison to the other two, both of which seem to grasp your attention more. They genuinely seem to have more detail. But with prolonged listening, the dCS's virtues start coming through, especially from 1kHz down.

The null testing method also helped me to select which of the dCS's 6 filters to choose. Perhaps I would have chosen this filter eventually anyway (a much more difficult task than with just 2 filters), but it wouldn't have been my first choice on immediate listening. It took me less than 30 minutes to test all 6 filters and arrive at which is 'audibly and perceptually' the most accurate .

Agree. Initially, MF recommends leaving at factory settings as it runs in and ears adjust. 100+ hours in my case. After that I hit on filter 8 and haven't changed it since apart from it is disabled by MQA input as that has it own proprietary filter. dont use DSD. To my non expert ears, upsampling does help shifting the inherent phase distortion out of the audible range. Digital glare or sheen? Since converting to digital in 1984 - never listen to the hype - that has been the biggest bugbear, now largely absent. Have been confusing myself due to the hope that the Innuos will arrive soon. Radio Paradise upsampled FLAC stream vs MQA stream. Both have merits. I bought the node solely for the MQA stream initially. the ears do need time to adjust and A/B can be confusing. Hope you can borrow an Apex dCS - a beast on a different level.

Relatively humble MF filter 8:

Oversampling bypass - The oversampling FIR filter, used for the 7 above mentioned presets, is bypassed and source data is upsampled to 352,8kHz/384kHz on the XMOS chip and sourced directly into the ES9038Q2M IIR filter. Bypassing the oversampling filter reduces the on chip PSU / ground noise and consequently reducing signal correlated clock modulation on the ESS DAC’s silicon die. When the source data is 705.6 / 768kHz, filter 8 is automatically turned on (NOTE: without up/downsampling to 352,8/384kHz), which turns off all remaining filtering, improving audio parameters and the frequency spectrum.