Either you’re at the party or you’re not.
If you’re at the party, you’re either happy to be there, or you’re looking for an opportunity to leave. You’re either ambivalent about the people who are not at the party, or you think less of them for not being there, whether they were invited or not.
Nobody who enjoys the party is envious of the people who left early, or who never showed up in the first place. The people who enjoy the party either think everybody else should attend, or they want to lock the doors to keep outsiders out.
If you’re not at the party, it’s because you left or you haven’t been there yet. If you left, you’re either at peace with your decision to leave the party, or you’re resentful for having wasted any time at the party in the first place.
Few people take joy in leaving the party. You just leave and carry on with your life. If you want to return to the party, you return.
Enjoy the party, or enjoy not going to the party. Everything else is inconsequential.