The rare meeting began shortly after 11 a.m. and ended at 1:46 p.m., an official from the presidential office Cheong Wa Dae said.
On Wednesday, the North announced that Kim Yo Jong, the sister of dictator Kim Jong-un, would attend the games, joining the country's nominal head of state, Kim Yong Nam. Another focal point is whether Kim Yo Jong has any message for Moon from her brother.
But South Korea's presidential Blue House said Pence had a meeting scheduled with USA athletes and had only planned to stay briefly to greet other officials.
The North Korean delegates are set to return home Sunday.
Tensions between the two soared past year as Pyongyang tested missiles capable of reaching the US mainland and its most powerful nuclear device to date, while Kim Jong Un and US President Donald Trump traded personal insults and threats of war.
Accepting North Korea's demand to transport more than 100 members of the art troupe by sea, South Korea treated the Mangyongbong-92 ferry as an exemption to the maritime sanctions it imposed on the North, a controversial move amid concerns that the North is trying to use the Olympics to poke holes in global sanctions.
The North's presence in the South has dominated Olympic headlines, with the winter sports festival triggering a flurry of cross-border trips culminating in the diplomatic delegation.
"We will fully cooperate with global efforts regarding sanctions against North Korea based on our stance for a denuclearised Korean peninsula and a peaceful solution (to the North Korea nuclear issue)", the statement said. The previous meetings came after rounds of worldwide talks aimed at eliminating the North's nuclear program, which eventually failed.
Abe also said he would convey to Moon Tokyo's position on a 2015 bilateral agreement on "comfort women" - many Korean - who were forced to work in Japanese military brothels during World War Two, while urging "future-oriented" two-way ties.
Analysts say the North's decision to send her to the Olympics shows eagerness to break out from diplomatic isolation by improving relations with the South, which it could use as a bridge for approaching the United States.
Abe discussed North Korea's nuclear and missile programs and brought up the issue of Japanese nationals who were abducted by the North decades ago when he shook hands and briefly spoke with Kim Yong Nam during the dinner hosted by Moon in Pyeongchang, the Kyodo News service quoted the Japanese government as saying.
USA college student Otto Warmbier was captured in North Korea in 2016, authorities said for stealing a sign praising leader Kim Jong Un.
In stark contrast, two kinds of kimchi - the fermented cabbage that features in every Korean meal - were on the menu for lunch, a Blue House official told AFP, one mild Northern style version and a spicier Southern recipe.
The North has sent almost 500 people to the Pyeongchang Games, including officials, athletes, artists and also a 230-member state-trained cheering group after the Koreas agreed to a series of conciliatory gestures for the games.