The only way is to reduce the number of rows that participate in recalculation. Do you tend to edit your past Time Entries? If not, then why recalculate the .Sum() on all entries every time?
Create a column on Tasks table for “Previously tracked time”. Store current Total Time into it.
Update the Stop Timer button to modify current row: set Previously tracked time to the current value of Total time.
Update the Total Time formula to:
show Previously Tracked Time when the timer is paused
show Previously Tracked Time + Last Row’s Duration when the timer is running
This way you can keep old Time Entries for historic purposes (or get rid of them) and use only the most recent entry to recalculate time.
You could take it further by not reading from Time Entries at all, but this should suffice.
The actual table is used by the team.
And the Time Entries table is used for analysis, filtered by person in charge, time period, etc.
However, the calculations don’t have to be done every time. It is enough to be able to run the analysis when it is needed.
For example, when a button is pressed, once an hour, etc.