Re: [Leica] Value of test reports?

[Stephen Gandy <Stephen@CameraQuest.com>]

Erwin,

To put it another way,  it seems you are more or less satisfied with the
accuracy of lens tests.  My biggest gripe with them is not Mike Johnston's
viewpoint, but that Published Lens Tests are not accurate enough to give precise
dependable information about the lens about to be bought, or already in their
camera bag, for some of the reason's discussed below.  It seems another
variation of "Is the glass half full, or half empty?" question.

For a high degree of accuracy, I believe the end user must establish their own
testing criteria to be sure what their OWN lenses will do, and not worry about
any one else's lens tests.    Your lens tests are of great use to you , since
they show what YOUR lenses will do, assuming some of them are actually your
own.  The question is how useful this information can accurately describe other
lenses of their types which you did not test.

Erwin Puts wrote:

> Recently we could read on this list a remark about the value of the
> measurement of characteristics of a lens that are related to image quality.
> In fact a reappearance of Mr. Johnston's well-known view about lens testing,
> it is stated that any objective lens test (that is a test that tries to
> establish numerical values on a set of parameters) can only capture those
> characteristics of optical performance that are irrelevant or unimportant
> for the true appreciation of a lens' performance by an artistically or
> expressively trained photographer's eye.

> Those aspects of a lens that
> delight or excite the working photographer in viewing his/her results in
> print or on screen, cannot be measured or even discussed objectively.

not sure about this one.  science of full of endless stories about "things that
can't be done" which later became common practice.

>
> As we are entering the domain of belief or even religion here, it is futile
> to try to argue against this view. You can not discuss in any meaningful way
> unless you try to follow the same set of rules or basic premises.
> The more intriguing question is why do some persons believe that objectivity
> in lens testing is irrelevant or counterproductive.

in my own case, the answer is lack of accuracy

> The obvious fact that
> all manufacturers use MTF tests and all other kinds of measurements to
> create and produce the lenses with characteristics that some only wish to
> discuss in personalised statements is a logical contradiction. But so be it.
> Why negate the value of objectivity in lens testing and evaluation? One very
> obvious reason is a commercial one. Quite recently I was emailed by a
> customer in an USA store who asked me this: the salesperson had for sale two
> Summilux lenses 1.4/35, one the aspherical and one the ASPH. The aspherical
> was twice as expensive as the ASPH, because the salesperson stated that the
> first (aspherical) version was much better optically than the current (ASPH)
> version. Now this is nonsense and that I told the buyer, who went for the
> ASPH version for half the price. If the salesperson had presented the buyer
> with objective test reports he would never have made this statement  and so
> could not justify the difference in price. Yes, yes, the aspherical is a
> collectors item and because of scarcity may demand a higher price, but that
> is not what the salesperson told the customer who was obviously not
> interested in a collectible.
> Second reason why objective lens reports are not popular is the loss of fun
> factor. If we believe whatever report the discussion is closed. It is
> established that lens A is better than lens B. Period. So buy lens A if you
> need best quality and start taking pictures. No fun at all? But if we
> believe that a test can not give conclusive evidence we are in for a never
> ending discussion, which is enjoyable in itself. Then we can point out that
> PopPhoto notes that the 1.4/35 asph has best wide open performance of all
> lenses tested, that Modern however remarked that stopped down the asperical
> is better, that Viewfinder in an article did not find significant
> diferrences, but noted more coma in the far corners, that CdI gave 5 stars,
> but that a friend who is a professsioal photographer swears by the ASPH, but
> that a noted NatGeo-rapher had sold his as he was not content with the bo-ke
> and so on. Of course I am fantasising here, but the message is clear and
> recognisable.
> The discussion on this list re the quality and merits of the Minolta and
> Leitz designs is a proof.  I am not going to jump into this discussion, I
> already overstretched, regrettably, my backbench postion by commenting on
> Dan's presentation of 4 comparative pictures.
> There has been a reference to a site which presents the results of several
> magazines of the same lenses.

> While it is helpful to note that test results
> stray widely, it does not answer the fundamental question: if we want to get
> reliable info based on measured results, which one to trust.

With professional staffs during their utmost to give accurate reports to their
readers,  the problem lies in believing its sufficient to test only one lens.
Quality control varies right off the assembly line, in a constant battle of cost
vs quality, plus outright mistakes and training errors.    If there were indeed
no variation, there would be no need for Magazine Lens Tests, as all the
consumer would have to do is read the manufacturers' claims of optical
performance.

While different test sources may use different procedures, their ranking of
lenses differently (for example the 16 Zeiss vs the 15 Heliar) supports the
likelihood that  lens samples indeed do vary, and so deliver conflicting and
confusing reports to the consumer.   A larger test base would build confidence
in the accuracy of the lens tests.

> There are so many stories here that are not true that I do not know where to
> start:
> The notion that you should need  a statistically representative sample to
> make meaningful statements, is not realistic: first: a representative sample
> would comprise at least 20 items. Which magazine can afford this?

Whether its 1, 5, 10 or 20 lenses tested,  anyone presenting their data as
trustworthy owes it to their audience to define the limitations of their test
procedures.  It may indeed not be practical to test more than one lens, but
without a larger test sample,  the test results are suspect, a violation of one
of the most basic rules of scientific procedure.

Stephen Gandy