"But when they buy then the price of the stock goes up you loose"

That would be the case except people don't.  And what of those who average down during the dips?  They don't seem to be impacting the SP as you indicate.  Most argue about missing the big spike up which almost never happens.  Look back and notice how long those downturns lasted compared to the up times...... painfully long, and plenty long enough to buy in over time if one was concerned about affecting the stock price.  Only a select few really make money as they are the ones that truly understand the market and play it without the attachments that most people can't shake.