This has been tough to resolve for a while, and we have been working on a couple things to help out here. What @Jean_Pierre_Traets mentioned is the only current way to remove people, but you’re right, it doesn’t quite do the whole job. We’re working on this so stay tuned for an update.
We’ve also already launched this which should help with the people that are available for you to choose from when adding them to a doc…