The way he worded it, "it's not easy to meet up with me," implied that there were forces other than personal predelection at play in the scenario.
He should have said "Fuck em, I'll see when and if I damn well please," because that is what he is really saying here.