The National Research Council said in a congressionally-mandated report that Washington should use "stepping stones" to achieve its goal of a manned flight to Mars.
This could involve exploring an asteroid, building a moon outpost or building more international cooperation with countries like China.
"To continue on the present course... is to invite failure, disillusionment and the loss of the longstanding international perception that human spaceflight is something the United States does best," said the NRC's 286-page report.
Nasa welcomed the report's findings, saying it was consistent with the agency's Mars plan approved by Congress and President Barack Obama's administration.
It promised to "thoroughly review the report and all of its recommendations" but insisted that it was worthwhile to set a goal of walking on Mars to set the bar high for other, parallel projects.
"The horizon goal for human space exploration is Mars. All long-range space programs, by all potential partners, for human space exploration converge on this goal," it said in a statement.
"A sustainable program of human deep space exploration must have an ultimate, 'horizon' goal that provides a long-term focus that is less likely to be disrupted by major technological failures and accidents along the way and the vagaries of the political process and economic scene."
The US space agency's older Opportunity rover has been in operation for more than 10 years.
But advancing human exploration into the outer reaches of space will require decades of work, hundreds of billions of dollars of funding and "significant risk to human life," according to the NRC report.
US-China space cooperation?
That, the report said, makes it impossible for the United States to go to Mars within the current US space budget.
Instead, it called for increased cooperation with other nations, including with space rival China, as well as funding from the private sector and other sources.
Current federal law bars Nasa from participating in bilateral programs with China, which the National Research Council warned "reduces substantially the potential international capability that might be pooled to reach Mars."
"Given the rapid development of China's capabilities in space, it is in the best interests of the United States to be open to future international partnerships."
The report's authors said that returning to the moon would foster better international cooperation given the interest about the destination in other countries, and such a mission would help develop technology to land and eventually live on Mars.
The Obama administration is opposed to another moon landing, saying such a mission would be too costly. It wants instead to focus on capturing an asteroid and placing it into the Moon's orbit for future exploration.
The NRC highlighted three potential pathways to Mars, two of which include a return to the moon. The third is along the lines of the Obama administration's asteroid mission.
"It's probably the frankest assessment that there is no public demand for space exploration, that we really don't have a goal clearly stated and that the program that is being carried out won't get us anywhere," said expert John Logsdon.
However, the former director of George Washington University's Space Policy Institute said: "I don't think the report will change anything."