Convince me: Using machine learning to make search results relevant and persuasive

When searching for information on complex topics like climate change or COVID-19 vaccinations on the internet, receiving the most relevant results to a query and factual information is critical. But is this type of information also the most persuasive? Is misinformation more convincing? 

To explore what makes information persuasive to people conducting web searches, Toronto Metropolitan University (TMU) researchers developed machine learning algorithms to retrieve the most convincing online content. Identifying the characteristics of persuasive content would allow web search engines to find relevant and factual results that are also more compelling than misinformation. This project was led by computer engineering graduate student Sara Salamat and co-supervised by electrical, computer and biomedical engineering professor Ebrahim Bagheri and Ted Rogers School of Information Technology Management professor Morteza Zihayat. 

“Understanding the degree of convincingness of content is especially important when we are searching for information on complex or controversial topics,” said professor Zihayat. “By taking convincingness into account, we can find search results that are not only factually accurate and relevant but also presented in a way that we find convincing and persuasive.” 

For this project, the team compiled a dataset from the “Change My View” Reddit forum, where users submit posts outlining their opinions on a particular topic. Other users can then comment on the posts, offering counterarguments or alternative viewpoints to engage in a thoughtful exchange of ideas to change others’ opinions. This dataset included 153,755 comments, 7,937 posts and 46,419 users published on the forum over 15 months. 

The researchers successfully trained a neural ranking model, a machine learning algorithm that ranks results, to highlight the entries considered the most convincing by the Reddit forum participants. 

Relevance matters more than style

They were surprised to find that relevance to the query was more important than writing style when it came to making content persuasive, finding a high positive correlation between the most convincing content and the most relevant in the written materials examined for this study.

“We didn’t expect to see that,” said Salamat, a sentiment echoed by both professors Bagheri and Zihayat. Instead, they hypothesized that writing style characteristics or argumentation techniques would significantly impact how convincing a person found web content. 

“That's why the title of our paper is ‘Don't Raise Your Voice, Improve Your Argument,’” said professor Bagheri, who holds the Canada Research Chair in Social Information Retrieval and leads the Laboratory for Systems, Software and Semantics. The results of this research were recently presented at the 45th European Conference on Information Retrieval (ECIR), the premier conference on that topic.

According to the researchers, convincingness, relevance and factuality are the three most important dimensions of web searching. While modern search engines have become skilled at returning relevant information, factuality and convincingness are also needed to improve information retrieval. Professor Bagheri noted that future research efforts would include tackling the difficult third pillar of web searching: factuality. 

Understanding the degree of convincingness of content is especially important when we are searching for information on complex or controversial topics.

Learn more about the Laboratory for Systems, Software and Semantics (opens in new window) . 

This research is supported by the Natural Sciences and Engineering Research Council of Canada and the Canada Research Chair program.