Overturning a 2013 ruling, the judges did not, however, halt the programme but urged Congress to take action.
The NSA has collected data about numbers called and times, but not the content of conversations. It also allegedly spied on European firms.
Among individuals targeted was German Chancellor Angela Merkel.
Germany has "drastically reduced" internet surveillance for the NSA, reports say, after the US agency failed to provide "clear justification" for each search.
Some senators, however, want things to remain the same. They've pushed for an extension of a provision, Section 215 of the USA Patriot Act that allows the NSA to collect the data.
Section 215 expires in June, and an extension would allow the NSA to carry on with its work.
The latest verdict, by the 2nd US Circuit Court of Appeals in New York, came after New York District Judge William Pauley had dismissed a legal challenge by the American Civil Liberties Union (ACLU) which argued that the way the NSA tracked million of calls contravened the US constitution.
The measures - repeatedly approved in secret by a national security court since 2006 - are set to expire on 1 June.
But the ACLU's deputy legal director Jameel Jaffer said: "The appeals court's careful ruling should end any debate about whether the NSA's phone-records program is lawful."
• NSA phone surveillance first revealed in June 2013 by Edward Snowden
• Federal judge in Washington rules in December 2013 that mass collection may be unconstitutional
• A week later, a New York district judge says it is legal
• House of Representatives passes bill in May 2014 to end NSA bulk collection
• A few days later, President Barack Obama tells Congress to pass a bill ending the practice