[Leica] Value of test reports?

["Erwin Puts" <imxputs@knoware.nl>]

Recently we could read on this list a remark about the value of the
measurement of characteristics of a lens that are related to image quality.
In fact a reappearance of Mr. Johnston's well-known view about lens testing,
it is stated that any objective lens test (that is a test that tries to
establish numerical values on a set of parameters) can only capture those
characteristics of optical performance that are irrelevant or unimportant
for the true appreciation of a lens' performance by an artistically or
expressively trained photographer's eye. Those aspects of a lens that
delight or excite the working photographer in viewing his/her results in
print or on screen, cannot be measured or even discussed objectively.
As we are entering the domain of belief or even religion here, it is futile
to try to argue against this view. You can not discuss in any meaningful way
unless you try to follow the same set of rules or basic premises.
The more intriguing question is why do some persons believe that objectivity
in lens testing is irrelevant or counterproductive. The obvious fact that
all manufacturers use MTF tests and all other kinds of measurements to
create and produce the lenses with characteristics that some only wish to
discuss in personalised statements is a logical contradiction. But so be it.
Why negate the value of objectivity in lens testing and evaluation? One very
obvious reason is a commercial one. Quite recently I was emailed by a
customer in an USA store who asked me this: the salesperson had for sale two
Summilux lenses 1.4/35, one the aspherical and one the ASPH. The aspherical
was twice as expensive as the ASPH, because the salesperson stated that the
first (aspherical) version was much better optically than the current (ASPH)
version. Now this is nonsense and that I told the buyer, who went for the
ASPH version for half the price. If the salesperson had presented the buyer
with objective test reports he would never have made this statement  and so
could not justify the difference in price. Yes, yes, the aspherical is a
collectors item and because of scarcity may demand a higher price, but that
is not what the salesperson told the customer who was obviously not
interested in a collectible.
Second reason why objective lens reports are not popular is the loss of fun
factor. If we believe whatever report the discussion is closed. It is
established that lens A is better than lens B. Period. So buy lens A if you
need best quality and start taking pictures. No fun at all? But if we
believe that a test can not give conclusive evidence we are in for a never
ending discussion, which is enjoyable in itself. Then we can point out that
PopPhoto notes that the 1.4/35 asph has best wide open performance of all
lenses tested, that Modern however remarked that stopped down the asperical
is better, that Viewfinder in an article did not find significant
diferrences, but noted more coma in the far corners, that CdI gave 5 stars,
but that a friend who is a professsioal photographer swears by the ASPH, but
that a noted NatGeo-rapher had sold his as he was not content with the bo-ke
and so on. Of course I am fantasising here, but the message is clear and
recognisable.
The discussion on this list re the quality and merits of the Minolta and
Leitz designs is a proof.  I am not going to jump into this discussion, I
already overstretched, regrettably, my backbench postion by commenting on
Dan's presentation of 4 comparative pictures.
There has been a reference to a site which presents the results of several
magazines of the same lenses. While it is helpful to note that test results
stray widely, it does not answer the fundamental question: if we want to get
reliable info based on measured results, which one to trust.
There are so many stories here that are not true that I do not know where to
start:
The notion that you should need  a statistically representative sample to
make meaningful statements, is not realistic: first: a representative sample
would comprise at least 20 items. Which magazine can afford this? And what
manufacturer can give 20 lenses per magazine. As there are about 200
magazines in the world who need fair treatment, so the factory  would have
to deliver 4000 lenses. Assume the Leica  1.4/35 aspherical which has been
produced 2000 times. The full production is not enough to deliver the sample
to all magazines. And would magazines be happy with 20 lenses. Not all all!
It takes me a few months to test one item!! And is it necessary? No, QC
nowadays secures minimum standards. Is it true that a magazine gets
specially prepared versions of a lens? Most unlikely. The magazines I work
for get off the shelf boxes. My Leica test lenses are taken from the shelf
by myself. Is it true that a magazine keeps testing a series of lenses till
they find one that meets their standards? Nonsense. Try to work for a
magazine and you will find out that this is impossible. You have a deadline:
get a lens in week one, test it in week two, find in week three it is not
OK, ask a new one (often if it is a new lens, only one is available!!!) and
you get one three weeks later, you test it etc. Deadline passed. No review
needed anymore as all other magazines have reports on the lens!
Every magazine has its own procedure of testing and style of reporting. YOU
CANNOT COMPARE THEM!!! Unless you know intimately and in great technical
detail what they do and how they work.
Magazines do not tell you or in such terminology that you do not understand
what exactly they are doing. Take Photodo. MTF tests are fine. The crucial
question at what distance they set the focal plane, when testing the lens is
never answered. I asked them several times to specify this simple fact. They
refuse. Without such a knowledge the results are most misleading. If you do
not know about the basics of optical shop testing and the magazines are as
evasive as the Russians about the sinking of the Kursk, you are in the
desert. Compare this behaviour with the one at Zeiss or Leica where the
people explain to the most minute detail what they evaluate, why they do it,
what the results are, what interpretations they use, where the grey areas
are, what the margins are and I must say I believe the manufacturers data
more than the results in the magazines.

Erwin