Francesco "Frank" Cali, reputed boss of the Gambino organized crime family, was fatally shot on Wednesday night at his Staten Island home.
Cali was rushed to Staten Island University Hospital North and declared dead, the sources said.
Cali, 53, was shot multiple times outside his home shortly after 9 p.m.
No arrests were immediately made and police were scouring for video and looking for witnesses to the shooting, police said.
After his release, the feds tried to bar him from meeting with members of the Sicilian Mafia, and demanded that he avoid contact with his uncle, Giovanni Gambino - the nephew of late boss Carlo Gambino, for whom the family was named - except at weddings or holiday celebrations approved in advance.
According to the Associated Press, court documents referred to Cali as the underboss of the Gambino crime family.
The Gambino crime boss was assassinated outside Sparks Steakhouse in Manhattan in 1985.
In January 1997, an Federal Bureau of Investigation agent reportedly notified Italian police of Cali's distinctive standing as a member of crime families on both sides of the ocean, after he was "combined" into the Gambino family.
John Gotti, who organised the hit, then took power at the helm of the Gambino family.
Cali himself was jailed for 16 months for his connection to an extortion scheme surrounding an attempt to build a NASCAR track on Staten Island.
Rudolph W. Giuliani, who at the time was the United States Attorney for the Southern District of NY, condemned the killings and vowed to find those responsible.
The Sicily native, who is married to John Gambino's niece, served on the family's ruling panel for several years before being promoted to acting boss in 2015, replacing the previous leader Domenico Cefalu, also known as "Greaseball".
The Genovese, Gambino, Lucchese, Colombo and Bonanno mafia families are believed to have controlled organised crime in NY for decades.
Unlike other mobsters, Cali had few run-ins with the law.