The first after-hours session on South Korea’s main stock exchange saw erratic and retail-dominated trading, in a sign of the challenges ahead as it seeks to attract more global investors.