There is a well-known saying in Sanskrit:

If we cannot get the company of people who are good, pure, deeply spiritual and wise, during the period of our sadhana, we need not seek fools, i.e.  worldly-minded people, and be in their company.  Their impure immoral vibrations affect us in our present state, though we may not know it and  think nothing has happened.

When some young disciples of Sri Ramakrishna complained to Swami Vivekananda at the Baranagore Math that they had not yet succeeded in realizing God so they had better go back to their families and live like householders, Swamiji replied: 

"If I cannot find Rama, should I go to Syama (i.e. a woman) for that reason? If I cannot realize God, does not mean that I should go back to the world? No, Never!"

This is the attitude everyone has to maintain. But ordinarily, people want somebody's company even if it is bad. They do not want to be alone. That is the whole trouble.

One sure sign of spiritual progress is that the devotee wants to hear and speak only of the Divine, and of spiritual matters. If we find that a devotee is fond of worldly company and worldly talk, there is something wrong with his devotion, and his sincerity is questionable. An external thing can attract me only when there is an inner consent or hankering for it in me. Birds of the same feather flock together because they have the same temperament. Those who are truly spiritual cannot relish the talk and company of worldly-minded people. Worldly-minded people are unwise though they may be very clever and intellectually developed; and the spiritual aspirant should see that he does not pass his time in the company of such fools. This is very very essential.  I know why I am telling this to some of you again and again.