Google research article reveals research gap

A recent Google research paper on Answer to long form questions illustrates how difficult it is to answer questions that require longer and more nuanced answers. While the researchers were able to improve the state of the art of this type of question answer, they also admitted that their results needed significant improvement.

I read this research paper last month when it was published and wanted to share it because it focuses on addressing a gap in the research that is not discussed at all.

I hope you find it as fascinating as I am!

What Search Engines Get

This research is centered on Long answer to open domain questions, an area in which natural language processing continues to improve.

What search engines are good at is called Factoid Open-domain Question Answering or simply Open-domain Question Answering.

Open Domain Question Answering is a task in which an algorithm responds with an answer to a question in natural language.

What color is the sky? The sky is blue.


Continue reading below

Long answer to questions (LFQA)

The research paper indicates that long-form question answer (LFQA) is important but a challenge and that progress in achieving this type of question answer is not as advanced as open-domain question answers.

According to the research paper:

“The answer to long open domain questions (LFQA) is a fundamental challenge in natural language processing (NLP) that involves retrieving documents relevant to a given question and using them to generate an elaborate answer the length of a paragraph.

While there have been remarkable recent advances in answering factoid open domain (QA) questions, where a short sentence or one entity is enough to answer a question, much less work has been done in the area of ​​open domain (QA). answer to long questions.

LFQA is nevertheless an important task, in particular because it provides a test bed to measure the factuality of generative text models. But, are the current benchmarks and assessment measures really suited to progress on SQFT? “

Answering questions from search engines

The response to questions by search engines usually consists of a researcher asking a question and the search engine returning a relatively short piece of information.


Continue reading below

Questions like “What is the phone number for XYZ store?Is an example of a typical question that search engines are good at answering, especially because the answer is objective and not subjective.

Long questions are more difficult to answer because the questions require answers in the form of paragraphs and not short texts.

Facebook is also working on answering long-form questions and has come up with some interesting solutions like using a question-and-answer subroutine called Explain Like I’m 5 (a dataset called ELI5). Facebook also admits that there is still work to be done. (Presentation of the answer to long-term questions)

Examples of long form questions

Once you’ve read these detailed sample questions, you’ll have a better understanding of how we’ve been trained by search engines to ask a limited set of queries. It may even seem shocking how almost infantile our questions are compared to the long form questions.

Google’s research document offers these examples of long-form questions:

  • What is happening in these tall towers belonging to the big banks?
  • What is fire, in detail? How can light and heat come from something we can’t really touch?
  • Why do Great Britain and the rest of the English Empire always bow to the monarchs? What is the queen really for?

Facebook offers these examples of long form questions:

  • Why are some restaurants better than others if they serve essentially the same food?
  • What are the differences between bodies of water like lakes, rivers and seas?
  • Why do we feel jet lag more when we travel east?

Are researchers trained to ask short, fact-based questions?

Google (and Bing) have a hard time answering these types of long questions. This can impact their ability to bring up content that provides complex answers to complex questions.

Maybe people don’t ask these questions because they have been trained not to because of the wrong answers. But if search engines were able to answer these kinds of questions, people would start asking them.

It’s a whole world of questions and answers that are missing from our research experience.

If I shorten the sentence “Why are some restaurants better than others if they serve essentially the same food?” at “Why are some restaurants better than others?“Google and Bing still fail to provide an adequate response.


Continue reading below

The first Google search result for this question is from the blog (unsecured HTTP) of a Canadian Indian.

Google quotes this section of the Indian restaurant in the SERP:

“People pay for the overall experience and not just for the food and that’s why some restaurants charge a lot more than others. Restaurant patrons expect prices to reflect the type of food, the level of service, and the overall atmosphere of the restaurant.

What if the person had Popeye’s Fried Chicken vs.KFC in mind when they asked this question?

There is a certain amount of subjectivity that can creep into the answers to these kinds of questions which require a long and coherent answer.

I can’t help but think that there is a better answer somewhere. But Google and Bing are unable to show this type of content.

Google uses signals to identify high quality content

In a How Search Works explainer that Google released in September 2020, Google admits that it does not use the content itself to identify whether it is trustworthy or trustworthy.


Continue reading below

Google explains that it uses signals in a blog post titled “How Google Provides Information You Can Trust in Search. “

“… when it comes to high quality and trustworthy information … we often cannot tell from words or pictures alone if something is exaggerated, incorrect, low quality or otherwise useless.

Instead, search engines largely understand the quality of content through what are commonly referred to as “signals.” You can think of them as clues to the characteristics of a page that correspond to what humans might interpret as high quality or reliable.

For example, the number of quality pages that link to a particular page is a signal that a page can be a reliable source of information on a topic.

Unfortunately, this part of Google’s algorithm is unable to provide a correct answer to these kinds of long questions.

And this is an interesting and important fact to understand because it helps to realize the limits of search technology today.


Continue reading below

What about the classification of passages?

Passing ranking involves ranking long web pages that contain the short responses for normal short requests requiring an objective response.

Martin Splitt used the example of finding a relevant answer about tomatoes in a web page that mainly deals with gardening in general.

Passing ranking can’t solve the tough questions that Google currently can’t answer.

Google and Bing generally fail to respond to LFQA-type queries as this is an area that search engines still need to improve.

Obstacles to progress

The research paper itself acknowledges this shortcoming in the title:

Obstacles to progress in answering detailed questions

The research paper concludes by stating that its approach to solving this task is “at peak performance”, but there are still issues to be addressed and more research to be done.

This is how the article concludes:

“We present an ‘augmented recovery’ generation system that achieves peak performance on the ELI5 long-form question answer dataset. However, further analysis reveals several issues not only with our model, but also with the ELI5 dataset and evaluation metrics. We hope that the community will strive to resolve these issues so that we can climb the right hills and make meaningful progress. “


Continue reading below

Questions and speculations

It is not possible to provide a definitive answer, but one has to consider whether there are any web pages that lack traffic, as both Google and Bing are not able to bring up their detailed content in response to detailed questions. .

In addition, some publishers mistakenly overwrite their articles in order to be authoritative. Is it possible that these publishers are wiping out search traffic from queries that require shorter responses because search engines cannot provide nuanced responses available in longer documents?

There is no way to know these answers for sure.

But one thing this research paper clearly shows is that the answer to a long question is a gap in search engines today.


Google AI blog post
Progress and challenges of the long-term response to open-ended questions

PDF version of the research paper
Obstacles to progress in answering detailed questions

Facebook web page About SQFT
Presentation of the answer to long-term questions

Comments are closed.