Restart Brexit talks because UK is ‘less safe and less secure’, Tory group tells Boris Johnson