Twice.

It has to be a significant effect. So you test it against the placebo and measure how likely it would be that the medicine could be having the observed effect by chance (based on the performance of the placebo). If the probability is less than 10% or sometimes 5% or 1%, then it is deemed effective.

The response to the medicine has to be stronger than the response to placebo by an amount that is statistically significant. There are mathematical formulas (student's t-test amongst others) that you use to determine whether a result is significant - it asks whether the result is due to chance variation or to a consistent difference between the response of the two groups. Exactly how much of a difference that will be depends on what you are measuring - it could be a small difference, or a big one. A medication that only makes a small difference in outcome can still be significant if it is consistently a better result than placebo. When it comes to medication for a really serious condition like cancer, even a small improvement can be very important. But ideally a medication would give you much better outcomes. In trials, a medication will be studied repeatedly against placebo. There is an initial phase, called a pilot, that is small and you would do this to establish that the drug might be better than placebo. This must be reinforced in larger studies, and then studied for safety and for smallest effective dose. The more subjects are enrolled in the study, and the more repetitions of the study, the more statistically confident you can be that the result is a consistent improvement over placebo.