New Mountain Capital has scrapped a deal with one of its former top executives, ending a dramatic and ambitious bid by the veteran dealmaker to acquire five portfolio companies on his way out of the firm.