She was outplayed in the tiebreaker and in the third set.
The seven-time Grand Slam victor, in her 77th major, struggled against a player who had never before beaten her to go down 6-3, 7-5 and deprive the tournament of one of its biggest names.
It means for the first time since 1997 there will be no Williams sister in the second round at Melbourne Park after Serena pulled out following the birth of her daughter.
The Chinese player kept her nerve as she inched pass Stephens in the tie-breaker.
Since beating Madison Keys in the U.S. Open final, Stephens has lost eight matches.
Stephens has had her share of bad form since she won the US Open in NY in September.
"So I think I was ready today...."
The break at Rod Laver came during a marathon eighth game between Venus Williams and Belinda Bencic. She clinched it when Williams, who had slipped on the far side of the court, couldn't get to a ball in the open corner.
Bencic warmed up for the Australian Open by combining with Roger Federer to win the Hopman Cup for Switzerland. Previous year she was 57th.
The 20-year-old French Open champion Jelena Ostapenko enjoyed a 6-1, 6-4 win over 37-year-old Francesca Schiavone. The age gap between 20-year-old Ostapenko and 37-year-old Schiavone, the 2010 French Open victor, was the biggest of any in the first round at Melbourne Park.
In the men's draw, US Open finalist from a year ago, Kevin Anderson was beaten by Britain's Kyle Edmund in five sets 6-7, 6-3, 3-6, 6-3, 6-4. But Chinese battler Zhang would not lie down and gradually got herself on top, winning the second set on a tiebreak for the loss of only two points.
The seventh-seeded Ostapenko is playing her third Australian Open - she advanced to the third round at Melbourne Park previous year. She won the Kremlin Cup in Russian Federation in October, the WTA Elite Trophy in China in November and the Auckland Open in New Zealand earlier this month taking her world ranking up to No. 12.
While Venus, the two-time finalist, was stunned by Swiss Belinda Bencic, 3-6, 5-7, Stephens, the world number 13, lost to China's Shuai Zhang, 6-2, 6-7, 2-6.
The 29-year-old German ended the year with tournament wins at Moscow and Zhuhai, then began 2018 with a win over Caroline Wozniacki in the final of the ASB Classic at Auckland, New Zealand.