Your Name hit me more, I was more invested in the characters and the story. However, you do need to pay attention as there is things that happen where if you miss them it feels like there is plot holes or something missing.
A Silent Voice was another that hit me a lot, it was easily relatable (I am not deaf but I understood the position Nishimiya was in). I also got invested in the characters to which I think were better than just a love story gone wrong, there was proper character building and huge visible differences in the characters from the start and the end.
In the end, both hit me hard but for me the love story wins (Your Name). I am not sure if it was the timing of when I watched it as the character development in A Silent Voice is something you can't ignore.