If you are at all nerdy about video camera tech, you’ll have heard of the Zacuto great camera shootouts. The first episode of the 2012 edition was just released.

Previous years focussed on relatively scientific tests. Each camera was pointed at test charts and a few staged scenes, with the same lighting and (where possible) the same lens for each camera. They picked and lit the scenes to be torture tests for the cameras. How far could they see into the shadows without disintegrating into noise? How hot a highlight could they hold, and how ugly was the burnout when it came?

I learned some useful things from those, confirming what I’d learned about how to juggle highlights and lowlights on the cameras I’d used (Panasonic AF100 and Canon dSLRs) and showing me that pretty much all the cameras in a given price bracket had similar issues.

This year’s shootout sounded much more interesting. They were going to take a scene, pre-lit, but allow the camera operators some time to fiddle around with the lighting as they liked, and similar time to grade the footage to their requirements afterwards. There were some limits on what lights they could meddle with (I think the idea for part one was to simulate shooting on location with sunlight outside, which you couldn’t fiddle with).

The range of cameras they tested was very wide, ranging from an iPhone through dSLRs (including the hacked GH2 which many people are very impressed with) through to full-on digital cine cameras including the big boys Sony F65, Arri Alexa and RED Epic. Again they used the same extremely expensive lens on all the cameras they could.

Problem 1: We are watching this on very highly compressed web video which will kill most of the detail and a lot of the subtlety. But hey, if you buy our videos you’re also looking at it after (not quite so heavy) web compression. So it’s kinda fair.

Problem 2: All the shots look pretty rubbish to me 🙁 Whether because of the above web compression or what, I thought most of them looked really muddy, and the “outside” bit looked comically bad, like a very cheap soap opera where they couldn’t afford an external shot.

Problem 3: The shots look more alike than different, overall. So the differences are fairly subtle, which is not the sort of thing you want to be using web video to look at in the first place. But now we discover that there’s a new issue because they were allowed to change the lighting. I am now looking at someone’s artistic preference for lighting too, and it is a different preference for each camera.

I happened to like shot B best on first pass. But a second look at it tells me that it has a ton more fill than some of the others (you can tell because the girl in black is casting a shadow onto the front door, which is absent in most of the rest of the shots). So I liked the shot with the most fill on the girl in black, and where I thought the face of the girl at the end looked nicest. I photograph girls, of course that’s what I was looking at.

Suppose we discover in due course that this was the iPhone (I don’t think it is, you can figure out from depth of field, since it has a tiny sensor compared with the rest) or a hacked GH2. Does that mean that the hacked GH2 is “better” than an Epic or Alexa? Clearly not. Does it mean that it is “more to my taste”? I don’t even think it does that. It is just that the way that camera operator lit it, in conjunction with the camera she/he was using, was more to my taste for the bits of the scene that were important to me: the pretty girls who drew my eye.

And here we hit the biggest problem with the test. Although it was trying to do something interesting and more “real world” it falls down, because the lighting available to everyone was the same. This is an unrealistic as having all the cameras shoot the same scene under the same lighting.

I suggest that if you have access to 5K HMI lights and multiple kino-flos, with all the light modifiers you might need and a $50,000+ lens… you would only shoot with an iPhone or a 7D out of a sense of perversity, maybe as a b-cam to get a really cool angle, or for the feeling that limitation is the friend of art.

If the budget can stand trucks worth of lighting it can handle hiring a higher spec camera for the day. And why wouldn’t you, because that higher spec camera will have the right inputs to get good sound, the right outputs to have a big monitor, shoot much less compressed footage so you can grade more easily, budget to hire a focus puller to nail the focus every time, and so on.

On paper I thought it sounded like a cool idea. Much more meaningful to be allowed to tweak the scene to make your camera shine.

Unfortunately by going part of the way towards making the test a more “real” one for the cameras, it just emphasises how unrealistic the rest of the test setup is.

A more realistic test yet would be to give a kit bag of appropriate price to each shooting crew to work with, and a page or two of script to bring to life. The GH2 and 7D guys should probably have had a bag with three red-heads, a softbox and a few CT gels. One low-power HMI at an absolute push. The iPhone guy shouldn’t have had any kit at all beyond a few dollars to buy a phone holder. The Alexa, F65 and Epic guys should have the whole lighting truck and more.

Even then, to fairly compare the results you could get with each set-up, you’d need the same director with the same aim in mind trying to wring it out of the poor crew. And that would just tell you which camera and cinematographer and crew were best at bringing that look to life for that director.

So although the exercise is bound to stimulate much discussion on the internets, and it is always informative to see detailed documentaries on how people are working and how the finished results appear, I think “Camera Shootout” is a misnomer this year.

Nothing beats using a camera in anger to shoot your own vision. I think that’s going to be the lesson from this year’s Zacuto exercise: if you make the test more realistic, you dilute it to the point where all it tells you is that you need to get your hands on a camera and try it for yourself.

It’s still a fun thing to watch, but next year- why not have a detailed and in-depth documentary about cinematography and the making of a few different films, at different budget levels, and forget the “shootout” element? There’s a dearth of good, detailed, technical documentaries about cinematography.