NY media reported that 53-year-old Cali was shot multiple times outside his vehicle on Staten Island on Wednesday evening, local time, by an unidentified man who then fled the scene in a blue auto.
Medics arrived at the scene and transported Cali to Staten Island University North, where he was pronounced dead.
The New York Post reported that police sources said the shooter hit Cali with multiple bullets and a 911 caller said they heard six or seven shots.
Cali's only mob-related criminal conviction came a decade ago, when he pleaded guilty in an extortion scheme involving a failed attempt to build a Nascar track on Staten Island. Gotti's brother Gene Gotti spent 29 years in prison for heroin distribution, but he got out in September 2018, leading to speculation as to what that could mean for the future of the organization.
Cali was an associate in the Gambino family, according to court documents, when he was indicted in 2008 with more than two dozen other Gambino members for a range of alleged crimes.
Cali was believed to have been elevated to the position of acting boss in 2015, taking over from Domenico "Greaseball" Cefalu.
The Gambino family is one of New York's five historic Italian Mafia families - along with the Genovese, Lucchese, Colombo and Bonanno families.
Otherwise known as "Franky Boy", Cali was considered the de facto godfather of the Gambino crime family, having served on its ruling panel for several years.
Staten Island's affluent Todt Hill neighbourhood is renowned for its crime connections.
His killing, after a spate of hits on lower-level gangsters, reignites memories of vicious turf wars as mafiosi fought for supremacy in NY. "The word that I'm hearing is this: That there's old school tension and friction". His son Salvatore had been shot three months earlier outside the family home, but survived.