Woking director of football Jody Brown says there were "growing concerns" across the club before the National League side parted ways with former England striker Jermain Defoe as manager.