From a one off project to an ongoing, multi-team experimentation program

This task required a different approach from our usual process at Conversion, which involves drawing our strongest insights from experiment data and doubling down on early wins to drive higher impact experiment roadmaps. Nonetheless, we approached this challenge with enthusiasm, and quickly became fascinated with the opportunity to understand and advance CRO in the news media.

We started off with an in-depth research and insights plan, which combined multiple data sources: historical experiment results, interviews, usability findings, analytics reviews, heat maps, and external research. This gave us our data-driven start point for understanding and prioritizing the most important motivations, barriers, areas, and audiences of The Times.

We collated this data through multiple ideation sessions, generating bespoke hypotheses and themes of experimentation most relevant to the Times’ audience and value proposition. These robust insights drove a bounty of experiment ideas. By working closely with our in-house experts on UX design to generate concepts and iteratively refined implementations, we were quickly able to populate the requested roadmap with high-impact, data-driven experiments.

Just under 6 weeks after starting the project, we delivered the first batch of 22 experiments, followed one week later by a second session where the other 21 experiments were presented. The teams attending were highly engaged and provided valuable context and additional information which helped us select and prioritize the experiment concepts which would be ready for swift execution.

In parallel to our work delivering the experiments, we had several presentations and training sessions with The Times’ wider team in which we covered a broad curriculum on how to run strategic CRO experimentation programs. We built on their existing experience with experimentation by teaching them the most scalable and reliable approaches that we’ve cultivated during our 10+ years running advanced CRO programs.

A wrap up session was organised in week 8 in which we presented finalised concepts, alongside a ‘how to’ playbook for effective experimentation. From here, The Times began running our recommended experiments, and the ROI potential quickly became clear following a series of early successes.

Running a cross-team experimentation program

A combination of limited resources dedicated to experimentation at The Times acquisition and the strength of our early collaboration led to a ‘one-off’ project being extended to a complete CRO program.

In the first 6 months of the program, we increased our trial sign-up KPI by an impressive 28%. We saw substantial impacts from simple experiments focused on removing friction from user journeys, increasing relevance, and improving the presentation of The Times’ offering. We also introduced a basic form of personalization for returning users, which turned out to be our biggest winner, opening up potential for wider personalization in the future.

The acquisition team at The Times have done an excellent job of voicing the program’s success and championing experimentation within the business. The growing interest and trust in experimentation has allowed us to run bolder, business-critical experiments to inform key strategic decisions about product and pricing.

Ultimately, this excellent win-rate, wealth of accumulated understanding, and maturation of their culture of experimentation has since led to The Times extending the  Conversion acquisition program for another year.

Moreover, this success yielded the opportunity to enter into a new collaboration with the ‘engagement’ team at The Times. This team has a completely different set of KPIs, focussed on increasing content consumption and active days on The Times website among existing customers. This has given us the opportunity to cultivate an ‘end-to-end’ understanding of their customers, from converting prospective customers to free trials through the customer lifecycle of engagement and renewal.

“The team at Conversion really understand our business needs and challenges. Their client service is exceptional, but most importantly they deliver results.”

Victoria Mead Head of Subscriptions