Too be fair, I highly doubt they are doing this maliciously or with malintent. It seems to me like an unfortunate miscommunication. They probably like you guys a lot and you've been really friendly and fun with them so they feel like you want them to come with you and in turn they want to come out with you!

My advice would be just to communicate with them your honest feelings. Keep it positive and upbeat but do let them know!