The area might be loved by wealthier residents and retirees - but it's lined with beggars, say locals