[Leica] riddle of measurement scales

[Erwin Puts <imxputs@knoware.nl>]

Let me first set a few things in the proper context. I published my 
test of the Tri-Elmar, long before CdI did, and I noted that the 
performance of the TE are as good asif not better than all previous 
generations of Leica lenses of comparable focal length. I also noted 
that while the TE challenges the image qulaity of the current asph35, 
2/50 and 2.8/28, the TE could not equal the performance of these 
three lenses when the highest level of imagery is required. So I am 
not trying to deny that the asph35 is better than the TE at 35mm. At 
5.6 however the differences are very small. This I already have 
reported upon  and it is nice to see that CdI supports my findings. 
What I object to is the notion that the asph35, in a contest with the 
TE, should be qualified as a loser or even as "clearly not as good 
as".
CdI uses an evaluation scale running from fair to good to very good 
and excellent. This is the same  as the PopPhoto classification of A, 
B, C etc. As both magazines use identical testing equipment  and use 
identical evaluation scales I always am puzzled why CdI should get 
the vote of confidence and PopPhoto be mistrusted. In my view when 
you say that lens A is good in the corners and lens B is very good, 
you get the same info as when someone says, lens A gets a C and lens 
B a B. Classifications are based on dividing lines, and so PopPhoto 
and CdI  will tell you that a resolution of 30 l/mm ot CdI: an MTF of 
30%) qualifies as good and a resolution of 31 l/mm (31%) as very 
good. Now this is ridiculous, No one can see the difference between 
the 30 and 31 lines or the 30 and 31% contrast transfer. But it is a 
logical consequence of their methodology and therefore it is not a 
good one. As I do not know when reading the CdI qualification if the 
difference of 'good' versus 'very good' is a small borderline 
difference or a big one (high end and low end of both classes) I am 
at a loss how to interpret the results. If you assume  a big 
difference between the notion of good and very good you might be 
tempted to conclude that one lens is a winner. If you look at the 
real data you see a small difference, that does not support the 
conclusion.
In the past the resolution figures were paramount and a lens that got 
80 lines was invariably better than one that got 75 lines. (it still 
is in vogue in some quarters).  Now the MTF is in danger of the same 
senseless  number juggling. To be precise: the TE at 40lp/mm at 12mm 
image height (that is at a point on the negative 12 mm away from the 
center) has a value of 60 and 65 (tang and sag) and the asph35 at the 
same position has 60 for tan and sag. Which lens is better and what 
of this difference do you see in practice? Can you see it and when? 
Is the 12mm position representative of the overall image quality? Is 
the 15mm position more or less important? And if there is a 
difference in value between 40 and 50% is there any chance you will 
notice it.
Well I am   a firm promoter of quantified measurements as it makes 
for more objective discourse than the impressionistic semantic 
bubbles so favored in some quarters. But I am against numerical 
fetishism, that tries to impress by citing and comparing numeric 
values without the relevant background and context.
To sum up: there is no sense at all in trying to read much practical 
difference in the small MTF numbers (tangentially 49% versus 58%) 
without being able to see the whole picture. There is also no sense 
in trying to extract big differences out of a measurement scale that 
is by definition nominal as good versus very good.
The evaluation of the optical performance of a lens is just more 
complicated than you can imagine.
And nor semantic eloquence nor numerical comparisons will be of any 
value to the practical photographer unless there is a clear reference 
and relation to the practical demands and technical proficiency of 
the user.

Erwin