From keyword-based search to AI-powered search

Site search helps site visitors discover the topics and products they are looking for. Today, companies strive to improve the customer experience – that’s why all content providers and e-commerce websites possess keyword search capability.

Recently, the expectations for a precise and robust search have increased, as site searches need greater precision than a web search engine. When a user tries to find a specific document (e.g., e-learning lesson), s/he has an exact document in mind. That’s why the company's search engines must retrieve exact file matches to be effective. If, after a few tries, the search engine doesn’t find the document the visitor needs - s/he may leave your site.

Nowadays – company searches are still based on keyword search, and visitors often miss what they need. They are expecting more meaningful and connoting results, and also instant and personalized search results. This can only be achieved with an AI-powered search.

The history of keyword-based search

Keyword search is a feature that allows website visitors to search its content for a given topic (keyword). The brief history of the classic keyword search is given as:

• In 1990 – Alan Emtage (McGill university) created Archie, the first search engine.
• In 1993 - six Stanford students created Excite – a new search engine that was sorting the results based on keywords found within the content, and backend optimization.
• In 1994 – Yahoo, Infoseek, Lycos and Alta Vista were launched.
• In 1997 – AskJeeves was launched, later becoming Ask.com.
• In 1999 – Google.com was introduced.

All of these search engines function by matching the keywords from the user queries – with the content. Keyword search is done by a persistent content indexing (crawler) to identify the new topics and re-confirm the previous topics. In late ‘90-s Larry Page and Sergey Brin (Google) introduced their famous ranking algorithm that sorted out the search results based on relevancy.

The birth of AI-powered Search 

The need to produce more relevant results ignited the usage of AI-based tools that can analyze real-time factors such as browsing history, the intent commonly associated with the words used, and high-performing content from the previous searches.

It all started with NLP (Natural Language Processing) in the ’50s (Alan Turing), that aims to understand the intent behind a search and return what the user wants. For example, in the sentences "I owe nothing to banks" and "On the banks of the river Nile", the word "banks" has two completely different meanings. To recognize the actual meaning of a word, we need to understand the context in which it is used. Thus, machine learning has become a key tool in NLP, helping to solve these problems with machines.

Aside from the text-based search - today the speech and visual searches are making trends in searches on mobile, thus opening new possibilities for AI-powered search. Gartner predicts that 70% of customer interactions will be speech-based and assistants will soon dominate the market. With advances in machine learning and computer vision, computers will also understand the images, and can perform visual searches. The ability of computers to understand natural (spoken) language, will also empower content searches. For example, users can ask their device “What is Sia’s newest song?” and when the result is displayed the command “play it” will play the song.

AI-powered Search explained

AI-powered semantic search with NLP and machine learning means that the search can work independently and produce even more relevant results. The search engines tend to discover meaningful content and always are based on some sort of NLP algorithm. NLP is based on machine learning, which means that the algorithm analyzes data (text) flow into the system and continuously finds patterns and connections in it, performing tasks that would take a human operator weeks or months.

The three essential steps in NLP include:

• Pre-processing of the text (removing special characters, tokenization, removing stop words, stemming, lemmatization).
• Text Parsing (with recognition and tagging).
• Text Mining (modeling that includes: analysis, classification, similarity calculation).

With NLP, computers are able to detect language patterns and identify relationships between words to understand what people truly want. Virtual assistants (VAs) such as Google Assistant, Cortana, Alexa and Apple Siri are mostly NLP-based systems.

On the other side – the task of automatic speech recognition (ASR) is to obtain an appropriate text record based on the audio input of a speech unit (words or sentences). In this way, the speech is practically converted to text, that is, "recognizing" what a particular speaker has said. Speech recognition and transcribing is based on machine learning algorithms – Hidden Markov model, Naïve Bayes and Neural networks. The utterances are split into phonemes – and the model is heavily pre-trained and tested.

Industries benefiting from AI-powered search

As search engines better understand the meanings of queries and people discover the benefit of AI-powered search engines, some industries can highly improve their position on the market. 

• Retailers (furniture, home appliances): AI can assist the customers to navigate the online store in an efficient manner. Retailers can provide personalized user journeys - based on their likes, brands, price limits etc. Virtual shopping experiences can be implemented that will display the searched item in different colors, or in different conditions (e.g. sofa bed in different virtual environments). Also - AI can display relevant recommendations that are in context to their purchase (or online search).
• ‍Fashion and Cosmetics industry: Many of the fashion retailers have already invested in AI, and AliBaba invested $15bn in R&D labs in an attempt to become the AI leader. Visual recognition is most often used on the detail pages of online stores, ensuring that customers will always find the right product. Another AI application for fashion retailers is inventory management. Fashion retailers have significant capital hampered in their inventory - artificial intelligence can be used to help them increase stock turnover by triggering the “need” to sell older stock as soon as possible.‍
• Streaming Platforms (music, films): Online streaming platforms also apply AI for personalized recommendations - suggesting appropriate audio and video titles. Providers can filter the content based on the age of the video, search terms, and user's browsing history. If they apply AI-based video indexing - the video content will become more visible, and they can gain valuable insights to improve their service using advanced AI technology.‍
• Online Schools / Course creators: EdTech technology has taken its grip recently. In this pandemic era – many schools, universities and companies have moved their teaching and training curricula almost entirely online. Online course creators should not only present the learning materials in different forms – they can provide benefits via site search options over video training materials, wikis, and knowledge bases. Students and attendees will highly benefit if they have the possibility to perform AI-powered content searches.‍
• News and Media Websites: The media websites are thriving today – from video news, to how-to tutorials – their content is ever-growing and changing. The visitors and video watchers can lose their path in a content-rich environment. AI-powered search can help them to find the sub-topic they need, and also to recommend similar relevant topics. Thus – they will remain regular visitors of the website.‍
• E-commerce portals: Online buyers usually know what they want to buy, but sometimes have problems locating the desired product in the entire website. You don’t want them to get lost and navigate to another portal, due to the search issues. Therefore, it is evident that (especially in busy shopping seasons), an AI-powered search can enable your visitors a smooth online shopping experience.
  

Revenue growth due to the AI-powered search

By enabling the site visitors to search, find, and buy with ease, you will get increased revenues. Since AI can be applied in different search use cases and scenarios, the benefits of AI-powered search for businesses are obvious:

• AI-powered search can provide product recommendations and personalized (customer-centric) experience on the site.
• The overall customer satisfaction will increase and the site visitors will smoothly convert into customers. 
• Possibility to analyze search data: by using the site search on your website - customers are expressing their intents in their own words. Then you can perform site search analytics and plan for strategic moves.
• When a website grows significantly, finding an exact item may cause problems to the new and previous visitors. But with the right site search, the size of the pile won’t be a concern for visitors.
• AI-powered search will definitely bring the site visitors what they need, and this can help you to achieve better customer satisfaction, higher conversion rates, and increased revenues. 

In general - an efficient search engine should display results that are personalized to individual users and their profiles / desires. At Omnisearch, we offer an effective AI-powered site search tool of all types of content – audio, video and text (including PDFs). Omnisearch’s focus is on educational and media portals. Users around the world benefit greatly from our tools that convert audio and video files to text, and perform AI-powered content searches, mapping results to exact seconds where they occur.

Another Omnisearch feature is that it performs site search analysis that helps uncover the visitors search habits on your website. The habits include which keywords and phrases have been searched for, and also - acceptance of search suggestions, thus providing invaluable insights into users' behavior.

Here you can check out Omnisearch in action, performing the whole site search over text, PDFs, audio and video content. To kick-off the implementation of your own site search, you can start by contacting us for a quick demo.