Worried that I'm neglecting relationships...

So... I've tried to play Stardew Valley hundreds of times, but could never get past 5 hours at most. I get bored, then lose interest. I never understood why it was such a great game. Then it happened, I'm 53 hours in and I am so hooked! I'm addicted to it, it's the first thing I think of when I wake up in the morning! lol!

Currently I'm in Summer of year 2 and other than quests for them, I've never really spent time building relationships with the villagers. Is there such a thing as too late in building a relationship with them? I really like ant to focus on my farm and adventuring for maybe the first 5 years, is that ok?