Is going to Versailles worth it?

My husband and I are going to france and italy at the beginning of May. We are starting our trip in Paris for 4-5 days, then on to Colmar for a few days before going to Milan. I was planning on including a day for versailles, but between all the gardens in Paris itself as well as the Opera House and the Napoleon Apartments at the Louvre, I feel like I will still see lots of opulence.. We want to make sure we have enough down time to just wander and relax at cafes.. So trying to see what could be cut out.