Our Army of Invisible Helpers

If all the energy we currently derive from fossil fuels alone (i.e. not including nuclear or renewables) was suddenly unavailable, and we had at our disposal a few handy nearby planets populated with well-fed healthy adult human males that we could capture as our ‘energy slaves’, how many of them would we need to allow us to enjoy the same lifestyle currently being provided by fossil fuels?