Scientists at the University of Sussex found telephone conversations while driving can cause the driver to visually imagine what they are talking about.
This uses a part of the brain normally used to watch the road, the study said.
It is illegal in the UK to ride a motorcycle or drive using hand-held phones or similar devices.
Drivers can get an automatic fixed penalty notice if caught using one. They will get three penalty points on their licence and a fine of 100 pounds, the BBC reported.
The law currently says drivers can use hands-free phones, satellite navigators and two-way radios, but if the police think the driver is distracted and not in control of the vehicle, they could get penalised.
The study involved 20 male and 40 female volunteers who took part in video tests while sitting in a car seat behind a steering wheel.
One group of volunteers were allowed to "drive" undistracted while another two heard a male voice from a loudspeaker 3ft away.
Those who were distracted by the voice engaging them in conversation took just under a second longer to respond to events, such as a pedestrian stepping off the pavement, an oncoming car on the wrong side of the road or an unexpected vehicle parked at a junction.
The study showed that asking a simple question - such as, "where did you leave the blue file?"- during phone conversations could mean a driver concentrates on an area four times smaller than normal, because their brain is imagining the room where they left the file, instead of checking for hazards in front of them.
Dr Graham Hole, senior lecturer in psychology at the University of Sussex, said the research laid bare the "popular misconception that using a mobile phone while driving is safe as long as the driver uses a hands-free phone".
"The problem is enforceability - it's very difficult for the police to tell if someone's using a hands-free phone," he said.
"But on balance, I think the law should be changed to get the right message across and make it absolutely clear that any use of a mobile phone while driving is hazardous."