using this logic, one can declare practically all religions as elite religions until modern era.
If you don't believe me than I'll suggest you to pick any piece of literature from either north or south which is contemporary to supposed Buddhism dominant era. You will find that there are only one or two characters who are affiliated to Buddhism. Not only this there was no imaginary Hindu-Buddhist conflict. Monks seem to be well respected even in their declining era.
I've told you earlier that Buddhists were not independent like other ascetic sects of India.