The defence secretary, who commanded United States troops as a Marine general in southern Afghanistan during the war with the Taliban in 2001, said that getting the militants on board for reconciliation may be a bridge too far.
The Taliban is likely to miss an Afghan peace conference at which participants are set to call for direct talks between the militant group and the government of President Ashraf Ghani, the foreign ministry in Uzbekistan said on Monday.
Mattis said the goal was to take advantage of Taliban fracturing to peel off insurgents exhausted of fighting and create a process that could incorporate some Taliban leaders.
The group said last month that it was open to reaching a political settlement and negotiating, but has not responded to President Ghani's peace offer so far.
But the Taliban has so far ruled out direct talks with the Western-backed government, which they say is illegitimate.
The United States a year ago stepped up its military assistance to Afghanistan, notably through a sharp increase in air strikes, with the aim of breaking a stalemate with the insurgents and forcing them to the negotiating table.
The Afghan government and the Taliban held peace talks in 2015, but they broke down nearly immediately.
Asked whether the United States would be willing to talk directly with the Taliban, Mattis reiterated the US position that the talks should be led by Kabul.
U.S. defence secretary Jim Mattis believes victory in Afghanistan is still possible, though not necessarily on the battlefield but in facilitating a Taliban reconciliation with the country's government.
U.S. President Donald Trump in August announced an increase in the number of U.S. troops in the country to push back the resurgent Taliban.
"We do look toward a victory in Afghanistan", he said. However, he added that it would not be a military victory but "the victory will be a political reconciliation" with the Taliban, which has achieved a stalemate in recent years and shown little interest in conceding to the Kabul government.
However, Taliban fighters control large parts of the country, the Kabul government itself is deeply divided and thousands of Afghan soldiers and civilians are killed every year.
According to a report by the UN Office for Coordination of Humanitarian Affairs, the people were displaced between January 1 and March 11.