A federal judge has denied a petition to block the deportation of aging grandparents, despite concerns raised by court.