The Authors Guild had sued Google in federal court in Manhattan 2005, claiming the Mountain View, Calif.-based company was not making "fair use" of copyright material by offering snippets of works in its online library.
Google already has scanned more than 20 million books for the project. The guild was seeking $750 for each copyrighted book that was copied.
In his ruling, Circuit Judge Denny Chin found that Google's project would not "supersede or supplant" books because it's not meant to be used for reading them.
"Google's use of copyrighted works is highly transformative," the judge wrote. "Google Books digitizes and transforms expressive text into a comprehensive word index that helps readers, scholars, researchers and others find books. ... Words in books are being used in a way they have not been used before."
Google hailed the ruling as a major victory.
"This has been a long road and we are absolutely delighted with today's judgment," the company said in a statement. "As we have long said Google Books is in compliance with copyright law and acts like a card catalog for the digital age giving users the ability to find books to buy or borrow."
There was no immediate reaction from the Authors Guild.