He served as the company's CEO from August of that year through April 2011, when he became executive chairman of the board of directors at Google.
In 2011, Schmidt became chair of Google and then continued in the role when Alphabet was formed to become a parent company to Google.
The company plans to appoint a new chairman to the board.
No reason was provided for the change.
"Sergey [Brin], Sundar [Pichai] and I all believe that the time is right in Alphabet's evolution for this transition", said Schmidt in a statement.
"Since 2001, Eric has provided us with business and engineering expertise and a clear vision about the future of technology", said Larry Page, CEO of Alphabet in a statement.
Schmidt joined Google way back in 2001, just three years after it was founded, and played a major role in growing it into the world's most powerful internet company. He worked with company cofounders Larry Page and Sergey Brin to grow Google into the publicly-traded superpower it is today.
"The Alphabet structure is working well, and Google is thriving".
John Hennessy, a member of Alphabet's board since 2004, said: "Over 17 years, Eric has been tremendously effective and tireless in guiding our board, particularly as we restructured from Google to Alphabet".
Before bringing discipline of sort to Google, Schmidt served at Sun, Novell, Bell Labs, and Xerox PARC, among other organizations.
"He did a lot of that important public-facing work on behalf of the company", John Battelle, author of 2005 book The Search, a book about Google, told The New York Times. A Forbes real-time tracker on Thursday pegged his net worth at $13.8 billion, with most of that held in the form of Alphabet shares. In 2006, he established the Schmidt Family Foundation that makes investments in startups and fellowships in the sciences, for example.