Switching web store providers is a decision worth measuring, and most measurements of it are wrong in the provider's favour: the new side is tested on a good week, conversion is reported without fees, and one whale in one group makes the result. Here's a design that survives those, and the arithmetic to run it.
The setup: your backend decides, both deliver
Both providers plug in the same way: the game asks your backend to open a store, the provider delivers paid orders to your backend by webhook, and your backend grants. So the switch, and the test, live in one place: the line in your backend that decides which store a player gets. The migration guide shows the mapping from an existing integration to Appside and how to run both during a switch; the test is that setup with a deliberate split and a fixed end date.
The split: by spender tier, then random
Simple randomisation fails on spend: in most games, under 5% of players produce most of the revenue, so a plain 50/50 split puts a different number of whales on each side by chance, and that chance decides the test.
Split per tier instead. For each of non-spenders, low, medium and high spenders (your backend already knows the tier; it's the same attribute you'd sign into a player token for targeting), assign players 50/50 to provider A or B, and keep the assignment for the whole test. Report results per tier and in total.
Hold out a third group of players who get no web store change at all, if you can afford it. It tells you whether anything you saw was the provider or the week.
The duration: two weeks minimum, no changes
Two full weeks is the floor, because the second purchase takes time: saved payment details, the habit of looking, the weekly offer cycle. Three or four weeks if your live-ops calendar is monthly.
During the test, change nothing: not prices, not offers, not the store design, not the group sizes. If you must change something, change it on both sides at the same hour and note it in the log.
The decision metric: net revenue per eligible player
| Metric | What it rewards | Use it as |
|---|---|---|
| Checkout conversion | Cheap packs, pre-filled cards | Diagnostic |
| Gross revenue | Lucky whales | Diagnostic |
| Order success rate | Payment reliability | Health check |
| Refund rate | Clear offers, reliable delivery | Health check |
| Net revenue per eligible player | Money that reaches you | Decision |
Net revenue per eligible player = (gross web revenue − provider fee − refunds) ÷ players assigned to that side, per tier and in total. "Eligible" means assigned, not "visited": a provider whose store players don't open has a problem the conversion rate hides.
The arithmetic, with a published fee
For an order of $9.99 on Appside: fee 3.9% = $0.39, net $9.60 before refunds. There's no setup fee and no subscription, so that's the whole side of the model. The other provider's side needs its contract: percentage, fixed per-transaction amounts, minimums, and what happens to the fee on refunds. If the contract isn't in writing, the model isn't finished.
The five ways the test lies
- Different weeks. Side A on launch week, side B after. Run both at the same time; that's what the split is for.
- Conversion without fees. A provider can win conversion and lose net revenue. Decide on net.
- One whale. Tier-split, and look at the result with the top player removed from each side. If the winner changes, the test isn't done.
- The switch tax. Players on the new side start with no saved cards and no habit. Report week 1 and week 2 separately; if week 2 closes the gap, extend.
- Lost tokens. If the old provider's player identity doesn't map cleanly onto the new one, some players on side B can't buy at all. Fix identity before the test; the passwordless design removes that failure mode.
Reading the result
Decide when the net-revenue difference per tier points the same way as the total and holds in week 2. If it's within a few percent, the provider isn't the lever; price, offers and traffic are, and the cheaper provider wins on fee alone. If one side wins by more than its fee advantage, you've learned something about checkout quality, and it's worth keeping the loser's numbers on file for the next renewal conversation.
Appcharge's own guidance on this (January 18, 2026) agrees on the bones: split by spend tier, at least two weeks, net income per eligible user as the decision metric, and about 80% confidence before deciding. The additions above are the holdout, the published-fee arithmetic, the top-player sensitivity check, and the point that both sides of the test should be measured by your backend, from the webhooks, rather than by either provider's dashboard.
Create your store free and run it as side B. No card needed to start; 3.9% per successful payment.