With just a few minutes of audio samples, attackers can replay your voice convincingly enough to trick people as well as top digital security systems, researchers said.
The consequences, from impersonating you with your friends to dipping into your bank account, are terrifying, they said.
Using only tools already on smartphones, a team of engineers led by University at Buffalo in the US is creating an app to stop voice hacking.
A prototype proved highly accurate in stopping machine-based voice impersonation attacks, researchers said.
"Every aspect of your life is now on your phone," said Kui Ren, director of the Ubiquitous Security and Privacy Research Laboratory (UbiSeC) at UB.
"That is your security hub. It is really critical now. Hackers are out there, more than you can imagine. There is a whole underground grey market to sell your password and your personal information," said Ren.
The best way to protect your cellphone, he said, is to use several security methods.
Voice recognition could become a more common security tool because more Internet-connected devices are being developed that do not have keypads.
"With the Internet of Things, what is a security interface? It is not like the phone. There is often no touch screen or keypad so voice authentication may be useful," Ren said.
Attacks can synthesise your voice, but these are detectable by existing algorithms. A human can imitate your voice, but again, existing technology can detect this.
A third method is replaying someone's actual voice, and here is where Ren's invention comes in. Any replay must be broadcast on a speaker, and speakers have magnetic fields.
Ren's system uses the magnetometer in a phone, which is there for the phone's compass, to detect a magnetic field.
In addition, the system uses the phone's trajectory mapping algorithm to measure the distance between the speaker and the phone.
It requires a phone user to be close to the phone when speaking to guarantee that anyone using a replay of a voice over a mechanical speaker is close enough that the magnetic field can be detected.
The system requires that the phone be moving - swung in front of the mouth - when the voice recognition is being used. When a replayed voice is moved, the magnetic field changes and the phone can detect this.