06 · Learning
Run tests that teach you something
Choose one decision, one hypothesis, and one primary measure before changing the page.
Start with the decision
“Try a blue icon” is an action. “Clarify whether the product or the emotional outcome should lead the page” is a decision. Tests are valuable when their result changes what you do.
Write the hypothesis first
Record the audience, proposed change, expected behavior, primary measure, guardrails, dates, and stopping rule before the experiment begins. Choose one primary question; bundled changes obscure the lesson.
Use the platform’s current tools
Apple provides product page optimization and custom product pages. Google Play provides store listing experiments and custom store listings. Their eligibility, traffic behavior, and reporting can change; confirm the current console documentation when planning a live test.
Apple supports up to 70 custom product pages, including keyword assignment and—on iOS 18 or later—optional deep links into the promised experience. Google Play supports up to 50 custom store listings that can target countries, unique URLs, Google Ads campaigns, search keywords, or user state where eligible. These pages personalize a promise; they are not randomized experiments by themselves.
For Google Play store listing experiments, you can run one default graphics experiment or up to five localized experiments at a time, with no more than two variants in one experiment. Reporting can include first-time installers and retained installers; preregistration experiments use different measures. Google automatically stops an experiment that runs for six months. Confirm those constraints in Play Console before launch because eligibility and terminology can change.
Protect against false confidence
Seasonality, paid traffic, featuring, product releases, price changes, and measurement delays can all shape a result. Keep a contemporaneous log. “Inconclusive” is a useful result when the alternative is invented certainty.
Carry the learning forward
The experiment is not done until the result, context, decision, and next question are recorded. Reuse the insight across copy, creative, onboarding, and product only as far as the evidence supports.