Coda is certainly suited for this project. That said, I understand it can get complicated in a hurry.
In order to build a doc/app like this, you have to understand how Coda works, or be willing to learn how it works. Based on what you are asking, I kind of doubt you are there yet.
Just start with a simple table and add more tables and features as you go. You are asking what formula to use, but that’s a tough one to answer if we can’t see what you have done so far. Dates are really just numbers and you can calculate with them by adding and subtracting numbers or other dates. But dates are not the easiest type of data to deal with, you have to consider working days, weekends, holidays and perhaps a few more exceptions.
For the record: a Coda table is a the same as a Coda calendar - it is just a different way of looking at the data. I have many tables that show as a table on one page and show as a calendar on another page. You can even use the calendar as a place to input your data, although Coda is not always the best tool for that.
Consider also wether every member is going to enter their own vacations (because that is what you say), and also their own sick days? It makes a difference, because you would use user() to identify the person if everyone does their own thing, but if that’s not always the case, you would have a lookup from a team-member table.
If you need help building this, you are looking at a commercial job, because this goes beyond a support question.
Last: start with a dummy doc with just a few (dummy) member names. Try a couple of things and when you get stuck, you can share your doc with the community - usually someone will help you really quick.
If you want someone to build this for you, there are quite a few people in this community that can do this for you (myself included).
And one piece of advice: always check if a commercial solution is available that already does what you need. Building it in Coda gives you a lot of design and feature freedom, but if you can’t do it yourself you really have to consider if you need a custom doc/app.