The UN atomic agency said that "intensive efforts" by its team visiting Iran on the way forward failed, and that they were denied access to a key military site.
"Intensive efforts were made to reach agreement on a document facilitating the clarification of unresolved issues in connection with Iran's nuclear programme," the International Atomic Energy Agency said in a statement.
"Unfortunately, agreement was not reached on this document."
The team requested access both during this visit and during a first trip in late January to the Parchin military site where it suspects suspicious nuclear activities are carried out, but Iran "did not grant permission," it said.
"It is disappointing that Iran did not accept our request to visit Parchin during the first or second meetings," IAEA Director General Yukiya Amano said in the statement.
"We engaged in a constructive spirit, but no agreement was reached."
The high-ranking IAEA team led by Herman Nackaerts, the Vienna-based agency's chief inspector, was due back in Vienna later on Wednesday.
The visit was the second in less than a month and was aimed, the IAEA had said, at clarifying all "outstanding substantive issues" surrounding Tehran's nuclear programme, in particular what it called "possible military dimensions."
Iran's envoy to the IAEA, Ali Asghar Soltanieh, was quoted by the Iranian news agency ISNA as saying the talks had been intensive and covered "cooperation and mutual understanding between Iran and the IAEA."
"These negotiations will continue in the future," Soltanieh said.
The trip was also seen as an important precursor to a possible resumption of talks between Iran and the P5+1 powers, the United States, China, Russia, France, Britain and Germany, which broke down in Turkey 13 months ago.
A watershed report from the IAEA in November said that Iran had carried out activities in a number of areas "relevant to producing" a nuclear weapon. Iran denies it is seeking to develop nuclear weapons and said the report was based on forgeries.
Since its publication, the United States and the European Union have ramped up sanctions on Iran's oil sector, and speculation has grown that Iran's arch rival Israel might launch military attacks this year.