The exes dated from 2004 to 2006, with their relationship beginning when Duff was 16 and Madden was 25.