Jamie Oliver has been living a quiet life away from the spotlight in a sprawling manor house reportedly worth £6 million