During the optimization of waterfall, developers often need to optimize the tiered floor price, test
the tiering effect of a new platform, add or remove tiering. These operations will inevitably cause
the overall waterfall data, which in turn affects the monetization benefits. Based on this
situation, it is extremely significant to find an alternative with low cost, low risk, and high
efficiency. The A / B Test can distribute the traffic into proportions. The divided traffic can be
set with different monetization schemes, which can minimize the risks caused by changes in
waterfall. Let's take a closer look at the method and principle of A/B Test.
What is A / B Test?
The premise of A / B Test is to first create a hypothesis about a factor, and then divide the
traffic users into two groups, A and B, and set different mediation settings for each group. By
analyzing the data indicators, we can find out which strategy is more conducive to achieve the
expectation, and eventually switch all traffic to the solution that meets the goal.
Generally speaking, A / B Test is often divided into
Group A (Control Group) and
Group B
(Test Group). By comparing the results of the monetization data of the two groups, it helps
developers to identify which monetization strategy is better.
What are the preparations before the A / B Test starts?
1. Understand the test objectives of A / B Test
First, analyze the effectiveness of the data of the existing monetization strategy, find the
variables that are expected to achieve better results, and form a test group. It is important to
choose a reasonable control group and test group variables for testing. The unique variable method
is the most intuitive and easiest way to analyze the factors that affect the results. Once the test
goals are determined, the creation of the A / B Test can begin.
Secondly, the contrast dimension of the A / B scheme needs to be clarified before the test. So which
data dimensions can be improved? Using AdTiming's foreground, you can find that there are the
following dimensions to test:
● Ad Network dimension
You can compare the monetization efficiency of the two sets of waterfalls by adding layers or
deleting layers.
● Segment traffic dimension
You can compare different groups of monetization for different advertising frequencies, user access
times, device models, and versions. It is also possible to estimate eCPM based on the user's
multiple attributes for audience granularity.
● Dimensions of sales methods
It includes Header Bidding, automatic waterfall, and manual waterfall. Header Bidding requests
multiple Programmatic Buyers in parallel to perform real-time bidding for each Ad Request, without
wasting every opportunity for high Ad Request prices.
2. Test samples and test cycle planning
Traffic division refers to randomly dividing the current population into a proportion, one as the
control group and the other as the test group. Of course, the specific ratio can be customized
during the test. For example, due to the test group's uncertainty in the data, for the sake of
insurance, the traffic division ratio between the control group and the test group can be 70% and
30%, 80% and 20%, etc. Thus, even the data fluctuates greatly, and it will not have a great impact
on the overall revenue.
The test period refers to the entire time period after the time the test group is created from the
time when the test group is turned on to the time the test group is closed. The length of the test
cycle depends on the tested sample size and often reflects the stability of the data volume.
After the optimal test plan is obtained, the strategy of gradually releasing the traffic can be
adopted to maximize the monetization profit.
3. Planning and preparation of Group B Mediation
After the test objectives of Group B are determined, relevant preparations need to be made in
advance for the variables to be tested. Since Group A and Group B cannot use the same instance,
first we need to create the placements of the relevant platform of Group B and complete the setting
of the floor price, then create a new instance and fill in the corresponding platform placement ID,
and finally A / B Test Type B to complete.
What are the usual considerations for creating group B?
1. Factors that determine the rationality of waterfall-level data
A.
Prioritization of eCPM and waterfall levels should match. The level of eCPM plays a
crucial role in revenue, and it is also an important indicator for developers. In monetization
optimization, different Ad Networks have different eCPM floor price algorithms, which means that
even if the floor price is set from high to low according to the priority of the waterfall level,
the actual eCPM may not be the same. So we need to adjust the order of the waterfall level according
to the actual eCPM level when the amount of data reaches a certain accumulation.
B.
The fill rate of different waterfall levels should be reasonable. There are typical data
problems in monetized data:
Typical problem No.1
The fill rate of the first layer is too high, even reaching more than 80%, which means that the
request level can basically be displayed by the first layer, that is, the bottom price of the first
layer is set too low and the revenue cannot be maximized.
Solution: We use Group B test, add one or more layers as the first level on the basis of the
current level, and increase the reserve price to ensure the maximum profit.
After the group A and group B add layers and adjust the prices of other levels, the waterfall level
is as follows:
Typical problem No.2:
The fill rate of the first layer is too low, and the floor price is set unreasonably, which means
that most of the display data is displayed at the bottom level, which has a huge impact on revenue.
Solution: We use Group B tests to lower all the floor price settings for all levels except
the bottom level to ensure a certain fill rate and impression volume to maximize revenue.
After the group A and group B add layers and adjust the prices of other levels, the waterfall level
is as follows:
C. The waterfall level is too finely divided. If the magnitude of the entire placements is not very
large, extremely fine division will cause the impression volume of each layer to be very low, and
the algorithm of each Ad Network has a certain relationship with the impression, and it will also
affect the eCPM as a whole, thereby affecting profits in the end.
D. The eCPM gap between adjacent levels should be reasonable. This is also an aspect that is
relatively easy to be ignored in the monetization. In addition to observing the fill rate and eCPM,
you also need to pay attention to the eCPM value before each level. If the spacing is too large, you
can increase the middle level to increase revenue.
2. The actual effect of the new ADN platform placement
At present, AdTiming supports 12 Ad Networks. In the early stage, the number of platforms is often
limited due to some objective factors. For newly added platforms, the performance can be preset
based on historical data collected in advance. Different platforms have different performances in
different categories, countries, and advertising situations. Preset test groups based on historical
experience data can make better use of them while reduce unnecessary time wastage. The only variable
in group B compared to group A is to increase the level of the new Ad Networks, and test by traffic
division to get the optimal return.
3. Test variables must be unique
The uniqueness of variable selection is very important when performing A / B Test. When analyzing
the final data, you can clearly understand the influencing factors of data changes. Therefore, when
we carry out plan B, we need to choose the only variable factor with plan A, and at the same time we
test waterfall B plan, then plan A should not be changed during period B, otherwise it will affect
the judgment of the effect of variable A and B.
4. Test confidence time and data magnitude
How to more accurately determine which of the A and B solutions is optimal? This requires sufficient
time and data accumulation. In time, it usually takes more than 3 days to complete the test in order
to rule out accidental effects due to seasonality, the platform itself, and other reasons; in data,
it usually needs to reach 5k or more for reference. The above data is for reference only. The
specific confidence time and data may vary depending on the application.
How is AB Test implemented in the system?
On the AdTiming platform, you can implement AB Test in a more concise way. The specific steps are as
follows:
Step 1:Navigate to Mediation to AB test and click "Create A / B Test"
Step 2: Define your AB Test
Step 3: Configuration needs to be completed before starting A / B Test
First, navigate directly to the Ad Network Setup-B TEST Group to set it up, and configure the
relevant parameters of the integrated Ad Network in Operation.
Second, after completing the Ad Network, start the related configuration of Mediation
Setting.
Third, start formal testing.
Step 4: Analyze the report and terminate the test
Conclusion
At this point, you have a certain understanding of the relevant content of A / B Test. It is a
low-risk test function in improving the realization of profit, which is very useful to developers.
