Almost a million people in Yemen have been hit by a cholera outbreak, and starvation caused by warring parties blocking food supplies threatens much of the country.
Speaking with reporters on a military aircraft en route to Washington, Mattis said his death could either push the conflict towards United Nations peace negotiations or make it an "even more vicious war".
Saleh and the Houthis used to be allies in the struggle against the government headed by President Abd-Rabbu Hadi. The Arab League's general secretariat condemned the Iran-aligned Houthi movement which killed Saleh as a "terrorist organisation" and demanded that the worldwide community view it as such.
"People are reporting that there are many arrests and they are trying to shoot military men and (Saleh party) members".
Dujarric, the U.N. spokesman, told reporters that U.N. special envoy for Yemen Ismail Ould Cheikh Ahmed told a closed meeting of the U.N. Security Council that the killing of Saleh and others was "an adverse development" that will "constitute a considerable change to the political dynamics in Yemen".
"(But) one thing I think I can say with a lot of concern and probably likelihood is that the situation for the innocent people there, the humanitarian side, is most likely to (get) worse in the short term", Mattis said.
The rebels killed Saleh on Monday, days after he broke off his alliance with them against a Saudi-led coalition. "We have no conditions", Halima Jahaf said.
Former Yemeni President Ali Saleh, who was killed by the Houthi rebels, was buried on Wednesday without great solemnities, media reported.
It's not clear how many civilians are among the dead. The Saudi-led coalition battling the rebels had thrown its support behind Saleh just hours before his death, as the longtime strongman's alliance with the rebels unraveled.
Steve Purbrick, MSF's coordinator in Hajja, described the attack as a "deliberate disregard for medical facilities" that endangered the lives of patients and medical staff, and compromised "the care MSF can provide in the midst of intense fighting".
Much is likely to depend on the future allegiances of Saleh loyalists who previously helped the Houthi group, which hails from the Zaidi branch of Shi'ite Islam that ruled a thousand-year kingdom in northern Yemen until 1962.
It also shatters hopes by Yemen's Saudi-backed government that Saleh's recent split with the Iranian-backed rebels, known as Houthis, would have weakened them and given the government and the Saudi coalition backing a chance for a turning point in the stalemated war that has brought humanitarian disaster.
Saleh's body, which had appeared in a video by the militias with a gaping head wound, was taken to a rebel-controlled military hospital.
The Houthis and Saleh's forces began fighting each other in Sanaa last week.