Malaysia has begun repatriating about 1,500 Myanmar nationals despite UN and rights groups' warnings, saying the program is voluntary.