The study of psychology and the notion of brain chemistry was a stimulus for such study. Thanks to Freud and others, whose ideas gained currency, some people came to believe that human happiness and potential was there, in the brain, if only we could find a way to release it. It was soon noticed that whilst LSD remained the favoured chemical of choice, nevertheless, there were other plants and substances with similar, if weaker, effects than LSD. Mescaline, derived from the Peyote cactus, had been used by Native Americans. Most "psychedelic" drugs are characterised not so much by outright hallucinations (which can happen, nevertheless), but a heightened perception of reality. Colours become far brighter, sounds more vivid, and sensations, too. It was these effects, rather than hallucinations proper (seeing things that aren't there) which interested some people, who were also immersed in Eastern, particularly Hindi and Buddhist teachings. So, experimenting with such substances also had a spiritual element.
Sci-Fi writer Phillip K. Dick had a reputation for heavy LSD usage, but in fact, he took LSD just once, and it had a profound, terrifying effect that influenced his writing for a long time. He saw a terrifying creature, which he became convinced was the devil. He never took LSD again. LSD and many hallucingens are merciless exploiters of any character weaknesses, neuroses or fears.
But their prime motive was to try to "unlock" the secrets of the human mind, so this was primarily a mix of psychology, chemistry, spiritual belief and good old fashioned curiosity.