Clarity
Is the experience (or individual feature) easily understood by advertisers? May include a specific segment or a broader sample.
Measurement
Two data types
To comprehensively evaluate success, both qualitative and quantitative data play crucial roles. For example, “profitability” can be measured precisely from incoming key metrics, while “Clarity” is compiled of tracking and qualitative immersion studies.
Quantitative metrics provide objective, numerical insights. In advertising these might include ad’s performance metrics (such as impressions, CTR and conversions, among many others). Quantitative data allows us to track trends, identify patterns, and make data-driven decisions. For instance, in marketplace advertising, measuring the return on ad spend (ROAS) or the cost per acquisition (CPA) quantifies the effectiveness of ad campaigns.
On the other hand, qualitative metrics delve into the subjective aspects of user experience. These include user feedback, usability testing results, and customer satisfaction surveys. Qualitative data helps us understand the “why” behind quantitative trends. For marketplace advertising, qualitative insights might reveal pain points in the user journey, uncover user preferences, or highlight areas for improvement.
The need for both types of measurement is critical. Relying solely on quantitative data risks overlooking nuances and seller sentiments. Conversely, relying solely on qualitative insights may lack the rigor needed for strategic decision-making. Only by combining both, product managers and designers gain a holistic view of product performance, enabling them to iterate, optimize, and create impactful experiences in the dynamic landscape of marketplace advertising.
Equilibrium of data streams
Each area of measurement will come with its own ratio of qualitative and quantitative insights. For eBay Ads, qualitative data would come from 3 streams:
- Live immersion (advertiser-side walkthrough)
- Observation (buyer-side interaction with ads placements)
- Validation research sessions with target (or broad) advertiser segments.
The quantitative component was much more granular across areas of focus.
5 quantitative areas for measurement
Quantitative
Qualitative
- # 1st time visits across key seller segments.
- On-page tracking (completion & drop-off).
- L1 & L2 ads performance data across segments.
- Actions ratio (New campaign vs mid-flight management).
- Repeat visits at key intervals across segments.
Pertinence
Does the functionality serve actual advertiser objectives and mental modes?
Quantitative
Qualitative
- Segment-specific adoption of ads products across core seller toolset.
- Ratio of campaign creation rates across points of ingress vs Ads dashboard.
- Adoption rates of Ads within core inventory management and listing flows.
Familiarity
Is the experience aligned with advertiser expectations and industry practices?
Quantitative
Qualitative
A combination of product engagement metrics:
- Product Engagement Score (PES).
- 30-day engagement rate.
- Time on task (creation and management).
- Retention rate (per seller segment).
- Churn rate (per seller segment).
Profitability
Has advertising proven profitable for sellers and has driven sales?
Quantitative
Qualitative
Value
Do sellers perceive advertising an essential tool in the seller tool kit?
Quantitative
Qualitative
- #Of Positive actions taken based on performance data.
- Adoption decline rates (visits, time spent on task).
- #Of positive actions taken after initial success.
- CTR of task-level guidance links.