Around five millions workers could lose out under the plans, according to a report by the Office for Budget Responsibility