Although I believe the manga to be superior, I highly recommend watching S1 and S2 and then continuing the manga.
(Also, Kaguya-sama is one of the rare cases where anime holds it's ground against the manga, so anything's fine)
The voice acting, music, production value, animation all really gelled well together and made the manga experience after the anime much, much better

