Working with a small team has its own challenges. One has to wear multiple hats. While it makes you a multitasker and you end up learning many things, On the positive side, focus gets hit.
The best way of handling this situation is to delegate some of the responsibilities to other members. I understand it is a challenge and for a control freak. If you have hired best folks, you should believe in them.
Delegating will foster positivity and bring everyone closer.