Lens Reviews—Useful or not?

Back in the good ol’ days if you wanted to buy a lens or set of lenses you’d go to your nearest professional photography store with camera in hand.  Let’s say you were interested in a 35mm lens. The owner of the shop (the owner was almost always present in those days) would set out a variety of 35mm lenses on the counter top, perhaps some used ones as well.

It was considered good form to buy a roll of film from the store. Then you’d try out each lens, maybe outside in the sun, inside in low light, shoot people, shoot wide open, stopped down, etc. If the shop had a film processing service, you’d let them process and make some small prints. You’d come back the next day, or whenever the film was ready, look at the prints, discuss pros and cons of the various lenses with the proprietor, and probably buy one or more lenses. You probably bought your camera from the same store, and you’d buy all your film there, your seamless paper, umbrellas, bags and cases, and so on.

If you live in New York City, presumably you can hop onto the subway and do the same thing at B&H, although instead of shooting film you’d shoot onto a card and look at the results immediately. Some other cities in the U.S. may still have full service, well -stocked professional photography shops, but I don’t personally know of any.

Today most of us are stuck with reading online reviews and ordering lenses from B&H. In the rare event that B&H is out of stock on the lens we want, we might go to Adorama as a second choice. Both stores have good return policies, so it’s safe to buy a lens and if you don’t like it, send it back. Shopping in this manner takes more time and you might get stuck with more than one shipping cost, but what’s the alternative?

So we depend on lens reviews to get our information about lenses. There are two kinds of reviews: the DPReview type highly measurebated, pixel-peeping reviews with charts and graphs and more information than you really want to know. These reviews are accurate, well written and definitely worth reading. However, in some cases you can come away with an impression that a lens isn’t as good as it really is. For example, when they do 100% blowups and talk about chromatic aberration and you read the numbers and think, “Oh s#@^, it has CA.”

In order for that information to be meaningful you really have to compare it with other equivalent lenses that they’ve reviewed. In many cases I think you’ll find that the CA of a particular lens may be irrelevant for filmmaking because it’s no worse than other lenses. The only way to find out for yourself is to buy a lens and check it out under real world shooting conditions.

I’m not knocking DPReview here. I think they’re pretty good. I’m just saying that even their thorough reviews aren’t going to tell you what you really need to know. Also, most lens reviews are done by and for still photographers. But today many filmmakers use still camera lenses on digital cinema cameras. You might see a review that criticizes a lens because it doesn’t have auto focus, yet mention nothing about the focus throw.

The second type of review is the one written by an individual user.  On B&H these are under “Customer Reviews.”  You can google the lens you’re interested in, for example, “Rokinon 35mm cine lens review” and find hundreds of posts about the lens.  You might want to try that and read a dozen or so of the reviews for that lens. I’ll bet you’ll come away with a very positive feeling about the lens. That’s what I did.

In fact, the reviews for the Rokinon cine lenses were so positive that I ordered the 24mm, 35mm and 85mm from B&H. When they arrived in three days I went into the studio and set up a soft LED on a rack of products. The products were white plastic jugs with blue and red graphics, in front of the green screen. The meter read F4, so I did the shot (at T4 on the lens since it’s measured in T-stops), pulled the card and checked it out on my editing monitor.

It was great. The color was perfect, the contrast excellent, sharp as any of my pricier lenses. I was thrilled with my purchase and was getting ready to open up the 24 and 85 packages. But suddenly I realized that I had shot at F4. All lenses look good at F4. Between F4 and F5.6 is the sweet spot for every lens I’ve ever used.  But the main reason I wanted the Rokinons was because of their speed.  I do a lot of my shooting between F1.4 and F2.8.

I went back to the studio and did some additional shots between T1.5 and T2.8. In the camera’s screen, things didn’t look so good. When I checked the footage on the editing monitor the contrast was way down, the color wasn’t as brilliant, and everything got softer. At wide open I considered the shot unusable. At 2-2.8 it got a little sharper but was still soft and the color and contrast wasn’t so good. Also, when I compared the lens to my Nikkor 35, I found that it was probably around 40-42mm—defnitely longer than 35mm.

When I called B&H to request a return authorization, I got the usual topnotch customer service, had the RA in a few minutes, had a couple of Zeiss lenses ordered to be shipped after my return was credited. I decided that my life is too short to shoot with cheap lenses that have problems I have to work around.

So…were all the reviews I read bogus? No. Not really.

In some of the reviews, I would read things like, “…the lens really shines at T4…” A few people said it was a bit soft wide open, but the vast majority of the reviews didn’t complain about anything, though a few did say it looked the best at T4, the implication being that it’s probably softer wide open.

I read one review complaining about the contrast and color. I never saw a post showing the lens unusably soft wide open. Until I returned them: I happened to be on EOSHD the day after the return and ran across a post from a person complaining about softness. And yet in the same thread another person said his 35 was perfectly sharp at wide apertures.

Then I heard from a friend who does a lot more high end work than I do and who talks to lots of people at big rental houses. He said he’d heard that there are big variations in different copies of that lens. In other words, it’s the luck of the draw. You might get a good sharp one or you might get a soft one.

I don’t mean to knock the Rokinons. Lots of people love the lenses. Matthew Duclos gave the 35 a very favorable review. I guess he got a good one; I got a bad one. A friend of mine here got a set for his C300, and when I saw them I was favorably impressed. However, we were in a coffee house at night, doing some hand held shots. I never checked out his lenses in detail as I did mine. I think he got a good set or he would have complained by now. It could be that the 24 and 85 lenses in my order would have been fine, but I didn’t want to spend time sending the 35 back, checking out another copy, maybe sending that one back, etc.

So here’s the thing about those customer reviews: some people get a good copy and love the lens. Others like me get a bad copy and don’t like it. And I believe that still others probably checked out the lens in available light and that light happened to give them an F4 exposure or better so they think the lens is awesome.

My totally unscientific conclusion about the Rokinon 35 is that there must be variations in quality control.  I’ve been through that before with a Sigma f2.8 24-70. I got one that was too soft at 24mm at f2.8, so I sent it back. The difference in the customer reviews on that lens was that there were lots of reviews talking about softness at wide open apertures and talking about variations in different copies. I recall that one person said he sent back three different lenses before he got a good one.

In the case of the Sigma lens, I knew what to expect but decided to take a risk. The lens was soft so I sent it back to B&H. With the Rokinons, I wasn’t expecting the softness because of the overwhelming quantity of good reviews.

I think (and this is just my opinion) there is a tendency for people to get caught up in the hype. The availability of cheap cine style lenses, for example, is such a nice thought that people may be overlooking the downsides to the lenses.  One professional filmmaker I know has a more critical opinion. He thinks that many people who buy the lower cost lenses have probably never shot with quality lenses on anything other than DSLRs, so they aren’t as critical when it comes to color and contrast, and they may not go to the trouble to really check out a lens in detail. I think that may be a bit cynical, but it is a fact that people seem to be very positive about anything involving cameras and lenses that they’ve spent money on. They want everybody to buy the same thing they bought. I see this more with cameras than lenses. I don’t think they consciously post deceitful comments, but they might overlook things a more objective person would see.

Basically, it’s a let-the-buyer-beware world out there. Reviews and comments on some websites may be accurate, while others may be misleading or even totally misinformed.

What does all this mean about the usefulness of lens reviews? I think it means that you should read the reviews and learn from them, but don’t put all your trust in them. Most importantly, don’t go to eBay and buy lenses from a place where you might have trouble returning them.  Also, check out non-review sites, such as this one, EOSHD, and other sites made up of filmmakers. Filmmakers generally want their lenses to look good wide open, while still photographers shooting with “full frame” cameras may not be as concerned with that. Most importantly, buy from a reputable shop like B&H where you can return what you’ve bought if you don’t like it.