If megapixels don't tell the full story and the market is awash with technical jargon, what can you rely on to sort the best from the rest? The answer is a comparative test that looks at only a camera's output and ease of use, without relying on brand reputation or a tester's prejudice that could influence the results.
On this page:
- How do we choose which cameras we test?
- How do we score cameras?
- How do we assess camera video quality?
- How do we test camera functions?
- How do we assess ease of use?
- Who are our expert testers?
How do we choose which cameras we test?
Cameras don't vary in any meaningful way from country to country, which makes an international test possible. However, we only test products that are purchased from retail outlets. Then all cameras are put through the basic tests, but the more sophisticated cameras, such as SLRs go through extra testing.
How do we score cameras?
The overall score is made up of
- Image quality 50%
- Ease of use 30%
- Movie quality 10%
- Monitor/viewfinder 10%
- Flash 0% (we test for flash quality and show you the results but it doesn't contribute to the overall score as some cameras do not have a flash)
Getting started
Before we begin testing a camera, we make an exhaustive list of its features and specifications. Manufacturers' specifications often leave out protruding parts or batteries, so we measure most things to try to give you information that relates to actual use.
All computer-related tests are performed on an up-to-date PC with the latest version of its operating system. Five high-quality professional monitors are profiled and calibrated and both the monitors and video cards are adjusted to sRGB colour space. To compare the test results to previous batches, samples from the last batch are re-tested and used as an anchor for scoring.
Subjective tests are evaluated by five people each with their own calibrated monitor to provide the best possible viewing conditions. At no time do they know which camera has produced the image they are judging.
How do we assess picture quality?
Picture quality is evaluated after the photos are transferred to the PC. For all basic tests the cameras are set to full automatic function, including exposure, white balance and focus function, with the maximum resolution and image quality setting in JPEG, unless otherwise stated. Test charts are adjusted to fill the monitor display of the camera. If not stated otherwise, we use a tripod.
Normal pictures/viewing tests
Normal pictures are assessed using the following scenarios:
Portrait in a living room without flash:the light level is at 140 lux and the scene photographed is two metres from the camera. We score overall picture quality, skin tone, colour accuracy, exposure accuracy, sharpness and image noise.
Portrait in a living room with flash: the light level is at 140 lux and the scene photographed is two metres from the camera. We score overall picture quality, skin tone, colour accuracy, exposure accuracy, sharpness and image noise.
Pictures in bright light (with trees, lawn, buildings and sky): lots of contrast, natural colours and a light level of 8000 lux. We score overall picture quality, gradation of colours, colour accuracy, exposure accuracy, sharpness and image noise.
Assessing performance in well lit, and low light, environments.
Indoor image of various objects in low light: the light level is set to 30 lux and objects include skin tones, objects with high contrast as well as artificial and saturated colours. We score overall picture quality, skin tones, gradation of colours, colour accuracy, exposure accuracy, sharpness and image noise.
Face detection: an image is taken of two people left and right of the image centre, in front of a bright white screen with some high contrast objects. The image is taken as close as possible to normal viewing angle (47°). The two people are at a distance of one and three metres and the rear screen is at a distance of five metres. We score sharpness and the exposure of the two faces, with preference for the person closest to the camera.
Very low light performance: the light level is set to 11 lux with a similar scene to the pictures in bright light. We score overall picture quality, sharpness, blur, and image noise. In addition, sophisticated cameras are also set to manual and ISO 3200. Exposure, focus and white balance are done manually and images are taken in both JPEG and RAW.
Image stabilisation: we mount the camera on an electro-dynamic shaker and apply recorded human movement in different directions and at different strengths while taking a number of images. This is done with a light level of 11 lux and with the cameras flash turned off. Where possible, the test is repeated with the lens set to a viewing angle of 12°, which puts the target at around four metres from the camera and the light level is adjusted to 1000 lux.
Image stabilisation test.
Macro image: we measure the smallest sharp area which the camera can take in macro mode. We also measure the distance between test chart and lens front. The test is performed using automatic focus.
Macro measurement set-up.
Technical measurements with manual settings
The following tests are measured using manual settings where possible. Sophisticated cameras are appraised using both their JPEG and RAW output.
Resolution: measured at maximum and minimum optical zoom setting at ISO 100 (sophisticated cameras take pictures at ISO 800, ISO 1600 and ISO 3200 and in both JPEG and RAW formats where possible). The test chart is evenly illuminated by two light sources with D65 light characteristic (daylight).
Distortion: for both vertical and horizontal planes with the lens set to minimum and maximum focal length.
Vignetting: measured at maximum aperture (smallest F-number) and at an aperture of 5.6 (in both JPEG and RAW format where possible) at ISO 200. The lens is set to minimum and maximum focal length.
Veiling glare: measured at maximum and minimum aperture and at normal focal length with ISO 200 setting and using manual white balance. This measures the amount of stray light in a lens caused by reflections of light between the surface of the lens element and the inside of the barrel of the lens (ISO 9358:1994).
A lens that performs poorly will often show up as producing images with a washed-out look over most of the photo. This is often not as noticeable as other types of glare which may leave a distinctive light spot on an image. Veiling glare can result in a reduction in the contrast of the image as well as the colour saturation.
Reflections: a series of images are taken at maximum aperture (minimum F-number) and minimum and maximum focal length of a dark screen with a bright light source right and left of the maximum horizontal image angle. We do not use a lens hood, even if one is delivered.
Resolution target test.
How do we assess camera video quality?
Monitors, projectors and video cards are adjusted to sRGB colour space. All measurements and recordings are made using automatic exposure and automatic white balance settings.
Two one-minute video recordings are made of a scene featuring people with colourful clothes, other objects with fine details and realistic artificial plants at different distances, both at rest and in motion. The automatic focus system and white balance is switched on, if possible. The camera is mounted on a tripod, and about a third of the take is recorded while panning and another third while zooming. Lighting for the tests is under the following lighting conditions at the targets:
- Daylight conditions, 3500 lux (D65)
- Low light, 11 lux.
These recordings are replayed on profiled and calibrated, high-quality monitors or projectors with high resolution (FullHD with 1920 x 1080 pixels and at least 25 (24) frames per second). The test panel evaluates colour fidelity, observing the actual objects (colourful objects and the like) used in making the recordings with appropriate illumination. The recorded images are rated according to the following criteria:
- Sharpness (how faithfully structures are reproduced)
- Colour fidelity (colour neutrality and colour noise)
- Exposure and contrast gradation (also in highlights and in dark areas)
- Overall image quality
- Image and focusing irregularities.
Audio quality
All measurements and recordings are made using the camera's automatic gain control settings. An audio stereo sample from CD (classical music with high dynamic range and both male and female voices), is played over two high-quality loudspeakers in a room.
First, a reference recording from these samples is produced on a digital recorder with a high-quality stereo microphone at a distance of two metres. The sound pressure level at the location of the microphone is at an average of 65 dB. Two audio recordings are made with each camera tested under the same conditions. Recording is done using the built-in microphone in automatic mode and we create possible noise by using the zoom.
We then listen to the recordings using high-quality headphones and rate the overall audio quality of the sample recordings, at the original volume. Problems with motor, zoom and button noises are noted.
How do we test camera functions?
Flash
We check to see if the flash is 'intelligent', that is, the power of the flash automatically adapts to the distance from the subject. We also check flash quality by taking a picture of a test chart in a dark room. The camera is adjusted to fit to the full test chart in the viewfinder/monitor of the camera using the zoom lens. Pictures are taken at one, three and five metres.
The resulting images are appraised for the amount of light put out by the flash, evenness of its illumination and its ability to cope with varying distances. If necessary, we also note reflections/shadows from the camera's body.
To test flash quality over a long distance we take pictures in a dark room at the camera's widest angle setting. There are distance marks (small grey cards) at different positions within the room. Their brightness is evaluated for the flatness of the distribution of the flash light across the scene.
Monitor and viewfinder
A panel evaluates the quality of the monitor and viewfinder for readability under different light conditions (sunshine, low light indoor). Adjustments for users wearing glasses (diopter correction) is evaluated as well. In both cases we look for picture quality in terms of colour performance, brightness, contrast, resolution and the speed of the refresh rate (important for precise pictures when zooming or taking photos in panning situations).
Viewfinder: we score usability in bright sunlight conditions and in poor light conditions, the quality of resolution and refresh rate at panning scenes and the size of the image in the viewfinder. For sophisticated cameras, we measure the difference of the size of the image in the viewfinder related to the width stored in the image.
Assessing monitor usability in a bright environment.
Monitor: we evaluate the monitor's loss of readability (contrast, brightness and colour) when viewed from +/-30° to left and right and +/-20° to above and below directly in front of the monitor. This test is performed indoors in good light with no light from behind, and in poor light conditions where there is bright diffused light behind the viewer.
We also evaluate the quality of the monitor in playback mode using a test image in JPEG format with fine details and high contrast. Each monitor's performance is evaluated for resolution, sharpness, contrast and brightness of the display as well as some other errors such as a visible cell structure of the LCD panel.
Focusing
Manual focusing is assessed in low light conditions (11 lux) using objects with high and low contrast. If possible we use assistive technologies such as live view or software magnification.
Autofocus sensitivity involves taking an image in automatic mode and at normal focal length (image angle ≈ 47°) of a test chart at a distance of one metre illuminated by photo lamps. There's a pattern in the centre of the screen to allow the camera to focus automatically. The power of the photo lamps is reduced, which in turn reduces the contrast of the pattern on the test chart. We measure the luminance of the test chart in cd/m² when the auto focus doesn't reliably find the focus point any more.
How do we assess ease of use?
Shutter delay
All the camera's functions are set to automatic and the lens is focused at infinity. The camera is then aimed at a timing display at eight metres and triggered with a sensor attached to the shutter release button of the camera. The test is then repeated at one metre. It's carried out in both modes for cameras with a choice of phase-detection or contrast-detection autofocus.
Device ready for operation
The device is switched off for at least a minute then switched on and we measure the time it takes before the first still frame is taken. The process is repeated in video mode.
Speed in use
We record the time (in seconds) it takes to carry out five cycles of the following, with the camera set to maximum image quality:
- Take image.
- Switch to replay mode.
- Zoom to certain magnification.
- Delete image.
There's a subjective score for how difficult it is to manage the controls while carrying out this test.
Picture taking speed (single shooting mode)
We measure the time it takes to take a picture, from pressing the shutter release until the camera indicates it's ready to take the next picture. The camera is at its highest quality setting with the flash off.
Rig for testing shutter delay.
Picture sequence speed (continuous shooting mode)
Using a consistent illumination (1000 lux) with the camera set at medium focal length, a subject at three metres and with the flash off, we measure the number of pictures/sec or the shortest possible time interval between two pictures and the number of pictures till the buffer is full.
Speed at low luminance level (sophisticated cameras only)
We measure autofocus speed at a luminance of approximately 10 cd/m². All functions are set to automatic and single lens reflex (SLR) cameras use phase detection mode. The test is repeated for single lens reflex cameras in live view mode (contrast detection).
Manuals
The printed documentation and the manuals on CD-ROM or in-camera supplied are evaluated for the completeness of documentation with detailed explanations and examples for using the camera as well as appearance, paper quality, readability of text and sketches
We also consider how easy the structure and content are to understand, and if technical expressions are explained for 'non experts,' extra content such as safety hints for electrical danger or damage by static discharge, helpful hints in case of malfunction and tips on how to care for rechargeable batteries.
The quality of explanations for the use of digital cameras in general and the use of the camera itself also factor in. The quality of the description for connecting to a PC and file transfer do too. We also make sure the table of contents is clear and that the correct manual is supplied.
Inserting and removing memory card
The insertion and the removal of the memory cards is evaluated for how easy it is to grip when inserted, accessibility and ease of use of the eject button, clear indication of which way/side the memory card should be inserted and how easy it is to open the cover of the card compartment.
Release button
Is the release button easy to handle? Does it need much force to depress? (If it's too stiff there's a danger of moving the camera while releasing.) Is there a clear indication when the shutter releases?
Autofocus
We give a rating for the quality of the feedback of the toggle point when using the focus lock by half-pressing the shutter release.
Manual focusing
How easy is it to use the focus adjustment in manual mode? Are there easily readable scales or other information aids for determining the object distance?
Additional testing.
Zooming
We give a rating for the adjustment of the optical zoom control, speed, responsiveness, number of steps and precision.
Battery change and control of the remaining power
We check how easy it is to change the battery and what indications there are when the battery is getting low.
Lens change
We check for precision, mechanical play and friction when changing larger and smaller lenses.
Video recording
We rate how the device fits into the hand and its comfort for taking video, delay after pressing the record button and feedback during video take. We also look at control of the zoom, the record button and other functions such as focusing.
Video playback
We rate access of playback, pause and stop function, possibility of fast forward and rewind and video clip selection (including sorting, searching and filtering).
Who are our expert testers?
All the tests and procedures described in this article are carried out or overseen by professional testers in specialist camera testing laboratories in Europe.
The test method is decided on by representatives of consumer organisations from around the world who have many years' experience and specialised expertise in camera testing. Many have formal qualifications, some sit on international standards committees that relate to photography (e.g. ISO/TC42 or Standards Australia MS-065), and they all share a deep interest in photography.
Stock images: Getty, unless otherwise stated.
