children of the significant other

This always troubles me as I begin to write a post. Sometimes they don't seem relevant at all and I don't want them to sound too pretentious!
Anyway, the last few days have been somewhat...I want to say awkward or strained but I'm not sure if that's what I mean. The bf's children came to visit and I always find this challenging. My children are young adults and although still at home they are self sufficient (for the most part) and therefore I have been able to return to pre-child freedom. His children are young and to be quite frank, irritating! I know they are kids but I can't help but feel annoyed that they are around. They seem rude and shouty, they bicker and are loud.
However, this could just be how I see them...I don't want young children in my life...this is an obvious dilemma in my relationship and it's going to be the cause of much grief, I realise that. I have finally forged myself a career and I now want to take advantage of the stable salary and thus I want to do things: weekends away, holidays, adult stuff basically but, I feel I will not be able to these things with him while we have to take into account them! His little holiday will be taken up with them and it will be assumed that I will be part of this and I want to say no! I know, I know this is incredibly selfish but I can't help how I feel...
Things will have to change but I'm not sure in which way...