Shares of VirnetX plunged as much as 59.8 percent after the decision by the U.S. Federal Circuit Court of Appeals in Washington. The company and its lawyer did not immediately respond to requests for comment.
Apple had appealed a November 2012 jury finding that it infringed VirnetX's patents for virtual private network, or VPN, technology through the FaceTime feature on its iPhone, iPod Touch and iPad products, as well as on its Mac computers.
While agreeing that some patents were infringed, the appeals court said the verdict was tainted by incorrect jury instructions from the trial judge on how to calculate damages, and that the error was not harmless.
It also agreed with Apple that testimony from a VirnetX expert over how to determine potential royalties should have been excluded, saying it did not reflect the extent to which the patented features were a factor in product sales.
"The law requires patentees to apportion the royalty down to a reasonable estimate of the value of its claimed technology, or else establish that its patented technology drove demand for the entire product," Chief Judge Sharon Prost wrote for a two-judge panel. A third judge resigned before the decision was issued.
The appeals court returned the case to the U.S. District Court in Tyler, Texas for further proceedings.
VirnetX is based in Zephyr Cove, Nevada. It was assigned the four patents at issue by Science Applications International Corp in 2006, court papers show.
In May 2010, VirnetX won a $200 million settlement from Microsoft Corp over the VPN technology.
Apple, based in Cupertino, California, did not immediately respond to a request for comment.
In early afternoon trading, VirnetX shares were down $7.32, or 49.1 percent, at $7.60 on the American Stock Exchange, after earlier falling to $6.00. Apple shares were down $1.40, or 1.4 percent, at $100.23 on the Nasdaq.
The case is Apple Inc v VirnetX Inc et al, U.S. Federal Circuit Court of Appeals, No. 2013-1489.
© Thomson Reuters 2014