It's a good argument, if we remain in the realm of the hunters-gatherers. So the extinction of the megafauna wasn't connected with farming activities but with hunting. It wasn't the human impact on the environment because of proto-agricultural activities to affect the megafauna, but the instinctive behavior of the human hunters.
Also, there was the human impact on the environment due to changed fire regime (setting fires at times and places where natural causes like lightning might not have brought them). And when agriculture did arise, one major effect was supporting increased densities of hunters. Like cow and horse who did survive arrival of hunter-gatherers but were hunted to extinction by increased densities of farmers in 17th and 19th century respectively.