I felt like this guy was an elitist and fairly delusional.
This was suspicion for sarcasm (it sounds highly smug).
Basically, he's saying that no one could possibly be on the same page as him, unless they shared his background. And, of course, even though very few people share this...