University of Edinburgh Tests AI-Driven Search with Drupal AI

The University of Edinburgh has conducted a series of experiments testing AI-enhanced search capabilities on its websites, exploring whether artificial intelligence can improve content retrieval for users. Led by Emma Horrell, User Experience (UX) Manager at the university, the study compared three search approaches: traditional keyword-based search, AI-boosted search using vector databases, and chatbot-driven AI Assistant search.

The research, conducted in collaboration with FreelyGive Ltd and Drupal AI specialists James Abrahams, Marcus Johansson, and Charlie Hull, revealed that AI-powered search tools offer advantages in precision and interactivity but require careful configuration to handle university-specific terminology effectively.

AI-Boosted Search vs. AI Assistant Search

The team used a case-study School website to simulate real-world queries. The AI-boosted search, which employs vector databases for semantic matching, performed well with precise queries but struggled when handling terms like “courses” versus “degrees.” Meanwhile, the AI Assistant chatbot provided more interactive responses, dynamically adapting to user questions, but also needed fine-tuning to improve contextual accuracy.

For example, when a query about “history courses” was tested, none of the three search methods returned the ideal result. AI-boosted search interpreted "courses" too broadly, returning links to individual course pages rather than a consolidated list of degree programs. However, after adjusting the AI Assistant’s prompt to recognize different user personas—such as prospective students—it successfully generated a structured response with relevant links.

Fine-Tuning AI for University Search Needs

The study highlighted that search effectiveness improves significantly when AI is trained with structured metadata and contextual instructions. The team experimented with AI Automators to index content more effectively, allowing AI-powered searches to return more relevant results.

Additionally, small refinements to AI prompts—such as instructing the chatbot to recognize university-specific jargon—improved search accuracy. One adjustment helped the AI Assistant distinguish between "Library," "Student Research Room," and "Study Space," leading to better search responses when users asked about borrowing books or library hours.

AI Search Requires Continuous Refinement

Horrell emphasized that AI-driven search is not a one-size-fits-all solution and requires iterative adjustments. 

“Providing AI with structured data and refining prompts based on user needs significantly enhances search accuracy,” 

she noted.

The findings underscore the need for a user-focused design approach when integrating AI into university search systems. AI can improve search experiences, but success depends on semantic understanding, query refinement, and continuous UX research.

The full study and its insights into AI-enhanced search on university websites are detailed in Emma Horrell’s blog post on the University of Edinburgh’s UX Service blog.

Read the full report here: Can AI help or hinder search?