❌

Normal view

There are new articles available, click to refresh the page.
Before yesterdayMain stream

Using a survey to understand digital design tool usage across the University (as part of the Figma design component library project)

2 November 2026 at 08:24

This year, as part of the Figma component library project, we researched how colleagues across the University use digital design tools in their work. This post explores how we approached the research, from designing and distributing the survey through to analysing the responses and identifying emerging themes. 

In a previous blog post, intern Kaile Liu introduced the wider Figma project, and its aims and objectives: 

What I am working on as a Digital Developer Intern (Figma) 

Request access to the project brief for more details:

Figma project brief

We needed to understand how digital design work happens across the University 

To begin with, we needed to understand how staff across the University design, collaborate and deliver digital work. 

There are a wide range of tools, workflows and practices currently in use. Understanding these existing ways of working would help us explore where Figma fits within the wider digital landscape, rather than looking at Figma in isolation. It also supports the work we are doing to improve our Figma component design library.

To hear from colleagues across a wide range of roles and disciplines, we chose a survey as the main method for this research.

Five questions framed our research 

As well as understanding what tools people use, we wanted to find out how those tools support their work and where there may be opportunities to improve existing ways of working.  

Our research was underpinned by five key questions: 

  • What tools are people using day-to-day? 
  • How do those tools support their work? 
  • Where do people experience challenges or inefficiencies? 
  • How do teams collaborate when creating digital content and services? 
  • Where are there opportunities to provide better support, guidance and training? 

These questions helped us build a broader picture of digital working practices across the University. Additionally, working this way allowed us to look beyond any individual or singular tool.

Hearing from more than just traditional design roles was important 

Design work happens across many different roles at the University, including roles not traditionally associated with design or that do not have ‘design’ in the job title. For example, a communications officer creating digital content, a developer sketching a wireframe, a UX practitioner co-creating a prototype or a learning technologist building online resources may all be making design decisions as part of their work. 

This meant looking beyond traditional design roles and hearing from colleagues who use digital design tools in different ways.  

Initial conversations helped us shape the research 

Before designing the survey, we carried out an initial set of user interviews with staff from across the University. We also drew on existing University communities and communities of practice to understand the different roles involved in digital design work. These conversations gave us an early view of how colleagues use and collaborate through digital tools. They helped us identify the groups we wanted to hear from and shape the questions we wanted to ask in the survey.

Adapting our survey to broaden our understanding of ways of working

Based on these interviews, we recognised that a single set of questions would not capture the differences between roles particularly well. Instead, we adapted the survey so that respondents would answer shared questions and then be filtered to questions relevant to their role and ways of working.

This, in essence, would give us a more consistent set of responses while still allowing us to explore some of the differences between roles.

Role categories allowed us to tailor the survey 

We created a filter with seven options: 

  • Marketing and Communications 
  • UX and Human-centred Design 
  • Graphic and UI Design 
  • Developer 
  • Project or product manager 
  • Research 
  • Other (for participants who did not feel represented by the other groups) 

Participants selected the options that best reflected their work, determining which role-specific questions they were shown. They also provided their job title.

The survey combined common and role-specific questions 

The survey consisted of three main sections. The questions included a mixture of multiple-choice and open-ended responses. This allowed us to identify broader patterns while also giving colleagues opportunities to explain their experiences in more detail. 

The opening section included five common questions about participants’ roles, the types of digital design work they do and the tools they use, including tools for collaboration, diagramming and presentations. 

The middle section included around four to six role-specific questions, depending on the participant’s selected group. These explored aspects of digital working relevant to each group. For example, marketing and communications colleagues were asked about the tools they use when designing marketing materials, while developers were asked about the tools and approaches they use during the design phase of projects. 

The closing section included 15 common questions focused on colleagues’ experiences and expectations of digital design tools, including Figma. We asked which tools colleagues would be likely to use, whether they experienced challenges with their current tools and how their work might be affected if they lost access to Figma. 

We wanted the survey to reach as many colleagues as possible

We distributed the survey primarily through existing Teams channels and University communities, which helped us reach a wide range of roles. These included groups focused on marketing and communications, human-centred design and digital development, as well as distribution via heads of IT. 

The survey received 55 responses  

We heard from 55 staff across a range of roles. Based on the categories participants selected when completing the survey, these were as follows: 

  • Marketing and Communications: 20 
  • Developers: 10 
  • UX and Human-Centred Design: 6 
  • Graphic and UI Design: 3 
  • Project or product manager: 2 
  • Other: 14 

Although seven categories were available in the survey, no participants selected ‘Research’ as their primary category, so it is not shown in the breakdown. 

The ‘Other’ category included colleagues working in areas such as product and project management, learning technology and academic support. 

We refined the groups when analysing the responses 

The survey categories gave us a useful starting point, but participants did not always recognise themselves within the roles specified (even when their job title directly reflected one of the options). For example, a participant selected themselves as ‘Other’, but their job title indicated that their work was within graphic design.

In order to evaluate the responses effectively, we revisited participants’ selected categories alongside their job titles and descriptions of their work.

This helped us six create groups that gave us a more useful basis for comparing responses:

  • Developers and technical specialists 
  • UX practitioners 
  • UI and graphic designers 
  • Marketing and communications staff 
  • Product and project managers 
  • Learning technology and academic support staff 

Some of these groups needed to be created on the back of this, as we had not identified those specific roles. For example, Learning Technologist was not an option for participants to select, but emerged as a useful category based on the number of respondents who had this as their job title. 

After this assessment, we were in a better place to begin analysis, as these groups gave us a more useful basis for comparing responses while still reflecting the range of roles represented.

Analysing the survey results required two approaches 

The survey produced both quantitative and qualitative responses, and we used different approaches depending on the type of question and response. 

For the role-specific questions, we used FigJam (Figma’s whiteboard for online collaboration) to review and compare responses across the different groups. This included both quantitative and qualitative responses, allowing us to look at the detail within each role as well as compare responses across the wider dataset. We used Power BI, a data visualisation platform, to explore the common quantitative questions, which gave us a consistent set of responses across all participants. 

We used FigJam to explore the role-specific responses 

We created a large collaborative FigJam board to organise the responses. FigJam supported discussion within the project team, allowing us to work in tandem and analyse the data as a group (in person and asynchronously).

We read through responses from each group, compared them across different roles, and identified recurring patterns and themes. FigJam also allowed us to consider the quantitative responses alongside the more detailed qualitative answers, giving us a fuller picture of ways of working. 

Screenshot of three frames on FigJam full of stickies. There is a blue section, titled Developers, an orange section titled Marketing and Communications, and a green section titled Graphic visual and UI design.

Section of our analysis on the FigJam board

Power BI helped us explore the common quantitative responses 

Power BI allowed us to visualise responses across the opening and closing questions. We wanted to do this as they were common questions asked of all participants. Power BI allowed us to bring our data to life by presenting the data in a more visual way, highlighting the common patterns and themes more easily.

Screenshot of Power BI visualisations. There are seven individual bar graphs of different colours, alongside a larger bar graph with these seven combined. The large bar graph has a right skewed narrow tail. There are also two sets of pie charts, each consistening of seven smaller charts and one larger combined. The data itself is not visible.

Section of our Power BI visualisations for the project.

Grouping responses helped us identify recurring themes 

The open-ended questions in the role specific sections had more detailed responses. This required us to compare responses across the different role groups and group similar comments together to identify recurring themes. 

As we worked through the responses, we also noted areas where a follow-up conversation with a participant could provide more context.

We also captured key quotations that illustrated what participants were telling us in their own words. These will be useful for when we playback findings to colleagues and complete a written report. 

The research will inform the next stage of the project 

The research highlighted both the diversity of digital working practices across the University and common themes across different roles. 

The next stage will bring these findings together into a report. It will highlight where Figma currently supports existing workflows and where its integration will provide more consistent, unified delivery of digital practices across the University.

Testing with users informed the Institute for Academic Development’s content migration

28 October 2026 at 16:27

The User Experience Service worked with colleagues from the Institute for Academic Development to test their content with users. Results from the testing supported content decisions in migration work that these colleagues carried out later in the summer.

This is part two of a two-part series. Part one is available here:

How we helped the Institute for Academic Development to establish user needs for their website

We had established some user needs but wanted to learn more

In two online sessions, we had worked with Institute for Academic Development (IAD) staff to establish a shared understanding of what users needed from their service.

We now wanted to find out about the experience of students who use the IAD website. Specifically, we were interested in two audience groups:

  • Postgraduate research (PGR) students
  • Students from all stages who might need academic skills support

Along with tests on the existing site, we also wanted to try out some new layout ideas and learn about what types of content these audiences would find most useful.

An in-person event gave us an opportunity to hear from PGR students

We knew that presenting information about workshops would be a priority for the new sites and we wanted to know the best way to present this. To inform the design, we needed to speak to people who might use the website to look for these workshops.

Fortunately, the perfect opportunity was just coming up. On 17 June, a large group of research students would be attending PhD Horizons, an all-day careers event at McEwan Hall. We planned two simple research tasks that we could carry out on the day. Emma Horrell and I attended the event with UX team interns Hannah Watson and Shlok Gupta. This was a great opportunity for Hannah and Shlok to gain hands-on experience of in-person UX research.

Usability testing helped us to learn about the existing site

The first task was a quickfire usability test asking PhD students to use the IAD website. For the testing, we handed PhD students an iPad and presented them with two scenarios:

  • Imagine your viva is coming up. How would you find a workshop to help you prepare for this?
  • You’re going to a conference and want to get better at presenting. What can you find on this site that could help you to do this?

Paper prototyping gave us insight into alternative designs

For the second task, we presented four paper prototypes of potential homepages for postgraduate researchers. PhD students compared the prototypes and talked about which areas of the site they would find most useful. This gave us an indication of how closely these homepages aligned with their needs, and it also provided insight into how the language used in the prototypes might be understood.

Clipboards and sticky dots on a table.

We used paper prototypes of homepages to test ideas with students.

A piece of A4 paper with a rough mockup of a webpage. Page is titled Support for postgraduate researchers. There are 4 boxes with titles Explore our workshops and courses, Plan your PhD journey, Browse our writing support courses, and Not sure what training you need? Launch the Development Needs Analysis tool. There are sticky dots on the page.

Students used sticky dots to indicate which areas of the paper prototype would be most useful to them.

What we learned from the usability testing and paper prototyping

In the end, we heard from about 20 PhD students who told us how they might search for workshops on the IAD website. From these conversations, we learned how these students responded to items presented on the live site and what they thought of some potential alternative ideas.

In the usability testing activity, we learned which ways of presenting workshops were perceived as being more useful than others. For example, most participants did not use the embedded PDF brochure, despite it being prominently located on the PGR landing page. Instead, we saw users gravitate to webpages showing workshops grouped by theme, or as an A to Z list.

Screenshot of the Postgraduate Researchers page on the Institute for Academic Development website. An embedded brochure is at the top of the page.

Users did not interact with the embedded PDF brochure.

Observations like this gave us a steer on which categories and methods of presenting workshops would be most useful to PGR students in future designs of this content.

In the homepage paper prototype activity, one of our mock-ups included an option to complete a Development Needs Analysis. This is a form that new PGR students can complete to assess their learning needs. But when we presented the text ‘Development Needs Analysis’ in our prototype homepages, most students could not guess what this was. That seemed like something that could be addressed in the new website.

We shared the findings with colleagues tasked with developing the new site for postgraduate researchers.

We ran usability tests on the academic skills content

Alongside this work, we were helping IAD staff to develop content for a new academic skills website. We agreed that a short round of usability testing on the existing IAD site would help us to get a better understanding of what it’s like to use it. We wrote a script for a 15-minute test and recruited four students working on summer internships at the University. A week later, we were on a Teams call with those students, watching them use the site.

The tasks we set in the usability tests were as follows:

  • You’re an undergraduate student at the University of Edinburgh. You’ve just started a new course. You’ve got your reading list, and you’re feeling overwhelmed by it. Your student advisor has said to book a one-to-one meeting with the Institute for Academic Development (IAD). You land on the IAD website homepage. Can you find out how to book a one-to-one?
  • You’re a postgraduate taught student. You’ve seen a poster advertising dissertation workshops. You arrive at the IAD homepage. Can you find a workshop that will help you with writing your dissertation?
  • You’ve just completed your first year of your History degree. Your exams didn’t go so well and you have a resit coming up in August. Your friend said there’s some advice on preparing for exams on the IAD site. Starting from the homepage, can you show me what advice you can find?

What we learned from testing the academic skills content

None of the participants clicked on the Study Hub resources. The Study Hub contains valuable content that could help students with the problems we presented in the test. But none of the participants followed links to the Study Hub, despite these links being present on several of the pages that participants used.

The Exam Bootcamp page did not lead participants to other exam content. When looking for support for exams, participants scanned the homepage for any mention of exams. This led them a page about Exam Bootcamp, a resource available on the Learn VLE. Participants seemed pleased to find this but didn’t find any of the other available resources or workshops about exams.

Participants did not seem phased by the amount of dissertation workshops on offer. When tasked with finding a workshop to help with dissertations, participants reached a page that listed 17 different workshops on the topic of dissertations. We had anticipated that that might be a problem. But none of the participants seemed worried about how they would choose a workshop to attend. They scanned through the list and picked those that looked like they might be relevant.

The new sites are now live

The teams working on the new sites took the findings from the strategy sessions and the user research and fed this into the development of four new sites. The new sites went live in July, using layout, wording and information architecture choices that reflect some of the insights gained from the content review work.

For example, the Postgraduate Researcher and Career Development site puts PGR workshops in a prominent location on the homepage. Users can browse all workshops, or view them grouped by theme. Next to that, a card shows development support options for PGR students at different stages of their studies (‘Starting out’, ‘Keeping on track’ and ‘Final stage’), echoing the variation between needs at different stages that we identified in the content review.

Postgraduate Researcher and Career Development

Screenshot of the Postgraduate Researcher and Career Development homepage. There are cards on the homepage reading Explore our PGR workshops, Find development for every stage, and Not sure what training you need?

In a similar way, the new Academic Skills homepage leads with the priority content we identified in the review. Study Hub is given the most prominent location at the top left of the screen on desktop, and at the top of the screen on mobile. Cards for workshops and one-to-one meetings show the support services that are available to students. Elsewhere on the page, students can find support for topics such as writing, making notes, literature reviews and exam preparation.

Academic Skills

Screenshot of the Academic Skills homepage. Cards on the homepage are titled Visit Study Hub, Browse our workshops, and Book a one-to-one.

Reflections on this work

This work was a great example of how you can take an evidence-informed approach to content strategy when you’re working to a tight deadline. It also showed how you can use a website migration as an opportunity to review the content you have and think about how it supports your service. By drawing on existing knowledge about users, and testing ideas and prototypes with them, the teams were able to make decisions about their new websites with user needs in mind.

If you’re reviewing your content

If you’re thinking about reworking what you have on your website, get in touch. The User Experience Service can help you to evaluate what you have, develop your content strategy and bring user research into the process.

Contact the User Experience Service

How we helped the Institute for Academic Development to establish user needs for their website

28 October 2026 at 16:26

This summer, the User Experience Service worked with colleagues from the Institute for Academic Development to review their content ahead of a website restructure. In this post, we look at how we developed a shared understanding of user needs to help guide this work.

Supporting the migration of IAD content

At the end of the academic year 2025 to 2026, the services performed by the Institute for Academic Development (IAD) moved to different areas of the University. As part of this change, the IAD website was being shut down.

Content from the IAD website was moving to four new sites:

In June 2026, there were 900 published pages on the IAD website. Time was tight, with the content needing to have moved to its new location by August.

Members of the User Experience Service (Emma Horrell, myself, and interns Hannah Watson and Shlok Gupta) worked with colleagues from the IAD to prepare for this move. In this post, I’ll outline how we worked together to establish who uses the content and what user needs it supports.

We ran two online sessions to identify service users and their tasks

In a content review, we typically start by working out who’s using the site and what they’re trying to do when they arrive at it. It’s also useful to know what kind of events trigger people to visit the site and what information is likely to be most valuable to them at that point.

To answer these questions, Emma Horrell and I ran two online sessions with colleagues who were familiar with the existing content and who would be involved in creating the new sites. We wanted to draw together existing knowledge about users of the IAD’s services so we could establish what these users need from the IAD website. Each online session brought together a small group of specialist staff. We met over Teams and used Miro to gather the information.

Session 1: Postgraduate research students

In the first session, we focused on content for postgraduate research (PGR) students. Through discussion, it became clear that this audience group has shifting needs as they complete their programme of study.

We grouped the student needs on Miro and three stages became evident:

  1. They become a PGR student
  2. They get started with their PGR programme
  3. Later in the programme
Screenshot of Miro showing three circles with sticky notes in each one. Text on the sticky notes is too small to read.

We collected user needs and grouped them into three stages.

New PGR students have distinct needs

We learned that new PGR students might hear about the University’s academic development services in Welcome Week or in a conversation with their supervisor.

At this early stage in their programme, their tasks might include:

  • find workshops that are relevant to me
  • sign up for newsletters that tell me about future workshops
  • learn what I can expect from my upcoming programme of study
  • connect with other PGR students at the University of Edinburgh

PGR student needs change as they progress in their studies

Later in their PhD, these same students will likely have different needs. For example, they might be looking for a particular workshop that will help them to:

  • write a dissertation
  • put together a literature review
  • prepare for a viva

They might also be thinking about what comes after their PhD and what they need to do when they take the next steps in their career.

Some tasks apply at all stages of a PGR student journey

At all stages, PGR students could have these needs:

  • find out which services support academic skills development
  • find out what these services do
  • engage with these services in the event of questions
  • find out what’s on (workshops, support sessions)
  • get updates about upcoming workshops (for example through a newsletter)
  • access policy and standards information

At times, PGR students may also need to know about other University services that can support them. For example, Edinburgh Research Office and Edinburgh Innovations can provide support with funding applications. The Student Counselling Service can support with health and wellbeing issues.

Presenting available workshops was the priority

Our discussions made it clear that presenting information about workshops was going to be the top priority on the new site for PGR students. On the IAD website, workshops were presented in a variety of ways:

  • in an A to Z list
  • listed by theme
  • on a page called “At a glance – workshops this month”
  • in an embedded PDF brochure

Session 2: Users of academic skills content

Alongside this work, other IAD staff were preparing content for the new academic skills site. This would have a different purpose to the site for PGR students, and it would be pitched at a wider audience of students at all stages of study.

An online session helped us to understand user needs

We ran a second session with a small group of IAD colleagues to learn about the existing academic skills content. Specifically, we wanted to know:

  • who uses academic skills content
  • how they arrive at it
  • what they need from it

As in the first session on postgraduate research students, we gathered together known user needs and looked for common themes.

Screenshot of sticky notes on a Miro board. Groups of sticky notes are titled Writing, Workshops, AI, How to get in touch, Exams, Wellbeing, Finding literature and Planning and organising time.

We used Miro to collect the needs of students looking for academic skills support.

Through these discussions, we developed some scenarios to illustrate how a student might arrive at the site. For example:

An undergraduate student gets a poor grade for an assignment. They speak to their student adviser about how to improve their essay-writing skills. The adviser directs them to the IAD. The student goes on the IAD website to look for resources or workshops about essay-writing.

By thinking about user tasks in this way, we developed our understanding of what it might be like to be a student scanning the homepage, looking for content that will help them to complete the task they have in mind.

We also established that the priority for the new academic skills website would be content that supports the following:

  • Booking on a one-to-one study skills consultation session.
  • Finding a study skills workshop.
  • Finding online resources to support study skills such as writing and presenting.

Understanding student users and their priority content like this gave us a foundation for what we needed to do to assess the site and consider alternative designs.

The sessions uncovered a need for testing

Through these two sessions, we collected existing understanding of who was using the IAD website and what they needed from it.

This led us to the next stage of this work. We wanted to know what it was like for these users when they interacted with the site. Could they find what they were looking for? What might be the best way of organising this content on the new sites?

To answer these questions, we carried out two sets of user research:

  1. To support the work developing content for PGR students, we joined a Careers Service event at McEwan Hall to hear directly from those students, seeing how they responded to existing pages and some prototyped alternatives.
  2. To support the work developing academic skills content, we ran a short series of usability tests to observe students using the existing IAD website.

Read more about this work

This is part one of a two-part series. In the next post, I’ll explain how we used paper prototypes and usability testing to learn how users interacted with various site layouts.

Testing with users informed the Institute for Academic Development’s content migration

Architecting a new website to support dementia research at the University

23 October 2026 at 12:32

Alzheimer’s and dementia research is a flagship strength of the University.  Websites of different research units play a key role in supporting this work. The UX Service worked with the Alzheimer Scotland Dementia Research Centre to design and launch Dementia.ed, a new home for dementia research at the University.

The University of Edinburgh is internationally recognised for research on dementia and Alzheimer’s, with different research activities occurring across groups, labs, centres in and across the various Colleges. The team at the Alzheimer Scotland Dementia Research Centre had identified a need to join up different research activities, to ensure connections could be made across different research initiatives by facilitating an interdisciplinary University network.

A website was the proposed way to meet this need, with the intention that this site would provide visibility of the different research areas, to enable academic researchers to exchange knowledge and expertise within the wider dementia and ageing research community. The site was also anticipated to have a secondary, non-academic audience: people with a general interest in learning about and supporting dementia research.

With the go-ahead to build an EdWeb2 site confirmed, Angel Garmpi from the ASDRC contacted the UX Service for help to shape the new site.

Before creating a site, we thought through a content strategy

In the seminal textbook on content strategy, ‘Content strategy for the web’ by Kristina Halvorsen and Melissa Rach, the authors introduce content strategy as a way to avoid being tormented by a burden of excess content to be managed. Starting a site from scratch is a golden opportunity to make decisions upfront to prevent content spiralling out of control, leading to website bloat and content debt further down the line.

The framework in the book outlines four elements of content strategy:

  1. Substance – what content does the site need and why?
  2. Workflow – how does the content get created and maintained?
  3. Governance – who owns the site and is responsible for its content?
  4. Structure – how should the content be organised so it can be found?

These elements served as a useful way to tease out key considerations from the start, setting a path for good decision-making and planning for the longer-term, past the website launch.

Substance: Content for dementia researchers and for people interested in dementia research

By considering questions about what the site was for, and who would visit the site and why, we were able to establish the site substance.

The ASDRC team had identified five main purposes for the site:

  1. To facilitate a network where researchers could come to collaborate with others on research work
  2. To showcase and bring together the different groups, centres and parties involved with and supporting dementia research
  3. To be a source of information where people interested in dementia research could come to read about latest insights
  4. To serve as a volunteer hub to capture details of people wishing to participate in dementia research (or make tissue donations)
  5. To act as a payment portal where supporters of dementia research could make financial donations.

Using these purposes as a starting point, we defined primary and secondary site audiences and their associated tasks to flesh out detail of the site substance.

Site audience 1 (primary): Academic researchers working on dementia and ageing

Needs and tasks:

  • Meeting other researchers and groups working in related fields
  • Establishing peers to collaborate with and exchange knowledge
  • Getting support and help with grant and funding applications

Content to support:

  • Contact details of the different groups
  • An entry point to join the network
  • Ways/touchpoints to stay informed such as events

Site audience 2 (secondary): People interested in tracking dementia research developments

Needs and tasks:

  • Finding out about dementia research in relation to a specific condition of interest
  • Staying up-to-date with advances in specific areas of dementia and/or ageing research
  • Identifying relevant academic papers and publications to form the basis of journalistic content

Content to support:

  • Publications and updates about dementia research
  • News articles about topical research areas

Site audience 3 (secondary): People wishing to take part in dementia research

Needs and tasks:

  • Registering to donate brain tissue
  • Volunteering to participate in academic research studies

Content to support:

  • A place to register interest in becoming a research participant

Site audience 4 (secondary): People wishing to make monetary donations

Needs and tasks:

  • Donating money to contribute to the progress of dementia or ageing research

Content to support:

  • A portal where financial donations can be made

Workflow: content is curated and published by the ASDRC

Considering the site substance led us to think about where this content would come from – the sources and the subject matter experts behind it. All too often, websites are built and filled with content for a launch, but without mapping out the content sources and understanding the underlying creation processes, launch-day content is at risk of quickly going stale because there is no plan in place to maintain it.

To fulfil its primary purpose of facilitating dementia research networks, the bulk of the content for the Dementia.ed site was planned to come from the different University-wide dementia research groups and initiatives. This content would be captured as links to the different sources, to ensure it was current. Events from the different groups needed to be highlighted as network-forming opportunities, so these were planned to be pulled together in a single location on the Dementia.ed site by the ASDRC team. It was also felt that the site should provide a round-up of news from different areas, which again would be curated by the ASDRC team.  Finally, the site would include static content such as: practical information about the ASDRC and the dementia research network, contact details, information about how to join the network and details for how to contribute, either donations or by participating in a study.

Taken together, this meant the initial content-related work requirement from the ASDRC team would be: to create the static content for the site in the different sections and to establish the correct links for the different research groups. Going forward, there would be a maintenance required to check the static content and the links remained correct, and to produce events and news content, which all  seemed doable given resource in the ASDRC team.

Governance: ASDRC team owns the site and manages the content

Considerations of how the new site would be governed were related to the workflow processes. A prerequisite for obtaining an EdWeb2 site is the need for a named site owner to take overall responsibility for the site, and the same ASDRC team member responsible for creating, curating and maintaining the site content had been assigned the role of site owner. This kept things straightforward in terms of decision-making about the site’s content and overall direction, although there was a need for the site owner to forward-plan a content calendar for maintenance, curation and creation tasks, and to assign responsibility to other ASDRC team members as a back-up to avoid a single point of dependency.

Structure: content arranged under common user tasks

Starting the design of a new website from a blank canvas can be difficult as there are lots of questions to consider at once. What should the menu contain? What should be on the top of the homepage? How should the content link together?

Another core textbook useful in site building is ‘Information Architecture’ by Peter Morville and Jose Arango. In this book, the authors encourage thinking about how to group or categorise content (in an organisation scheme) as well as how to establish relationships between those groups or categories (in the organisation structure).

Typically, websites adopt ambiguous organisation schemes – with content arranged by topic, subject, audiences and/or tasks to encourage browsing and discovery. This approach was appropriate for the Dementia.ed site to enable it to support the different audience groups and their associated needs and areas of interest. We considered a task

For the organisation structure – a non-hierarchical flat arrangement made most sense. Given the site was anticipated to have a relatively small number of pages, and would have clear calls-to-action for the different audiences, it didn’t seem necessary to plan a heavily hierarchical structure. For the content in the different research group sites, a hub-and-spoke model could be adopted, with Dementia.ed acting as the central hub, linking off to the different sources.

Before building, we needed to sense check the proposed site map with users

With ideas for the site mapped out, it was tempting to just get building in EdWeb2. Before creating the new site, it was wise to test assumptions with some testing.

Usability testing – where audiences are given a task to perform online and are observed as they do it – always provides an excellent source of insight into how audiences naturally interact with a digital artefact like a website, how they make sense of the menu labels and use on-page content elements to find their way to complete their tasks.

When it comes to new sites, however, it can be difficult to know what to test with users. If you mock-up a prototype, which parts should you include and which can you omit? Asking participants to make sense of menu labels that lead to empty pages can feel unnatural, and setting participants tasks can fall flat if there’s no real-life site content where they can find the information they need to complete tasks authentically. All that said, it’s impractical to build out an entire site just for testing purposes, as it’s likely the testing will reveal ways it needs to change which can end up adding more work.

We used another research network site as a proxy to test with

Thinking through the options with the ASDRC team, we came up with a plan. The ASDRC team had started looking at other research network sites for inspiration and ideas about how to build Dementia.ed, so we decided to adopt one of these sites for the basis of a test.

We chose the Edinburgh Mental Health Network (EMHN) as a testing site. This site included top level menu items: People, Research, News, Events, Blog and Contact. It had a total of 13 main pages with additional pages within the News, Events and Blog sections.

Access the Edinburgh Mental Health Network website

Since the site supported researchers in mental health as well as audiences with a more general interest in the topic, we were able to shape a usability testing protocol to use the site to assess how two audience types sought to complete representative tasks:

Audience 1: Academic researcher wanting to join the research network

Representative tasks:

  1. Find out what the network is, and its goals and values
  2. Find out how to become part of the network and what this would entail
  3. Find out about upcoming activities
  4. Find research opportunities in the network
  5. Find out how to develop your research project and involve people with lived experience
  6. Find content you can use in support of a funding application

Audience 2: Non-academic with a personal interest in the research area

Representative tasks:

  1. Find out what research is being done across the network
  2. Find out how the network is managed
  3. Find out how to keep up with what’s happening in the network
  4. Find out how you can contribute as a research participant for studies
  5. Find out how to donate money to support the research

Some of the tasks mapped clearly to specific sections of the EMHN site, whereas others were more ambiguous – they could have been completed in multiple sections. In all cases, it was useful to see how participants approached the tasks, so we could use their expectations to design Dementia.ed in the most audience-focused way possible.

We learned which menu labels worked, and which didn’t

The results of the testing gave us a clear steer on which elements of the Edinburgh Mental Health Network site supported the different user groups completing their tasks and therefore would be a good fit to adopt in the new Dementia.ed site. The tests also showed areas where the approach of the EMHN site was lacking – and therefore where different structuring and/or labelling was more appropriate.

The Events and News sections worked well

To complete tasks about finding out about activities and keeping up with what was happening, participants naturally gravitated towards the Events and News sections of the EMHN site, where their expectations of finding events to sign up to and articles to read were met.

The EMHN site was lacking a specific place for information about the network

When working on tasks geared at finding out what the network was, what its goals and values were, and detail about how the network was run, participants looked for an ‘About’ section when they used the EHMN site. When they realised this didn’t exist, they tried multiple other locations including the homepage and the Research section. This suggested an ‘About us’ section would be useful to include in the Dementia.ed site.

The People section didn’t resonate as the way to become part of the network

The EMHN site presented information for academic researchers to join the network in a ‘People’ section, which also contained information about the management of the network as well as researcher support resources. In the tests, participants struggled to find this content when tasked with finding out how to join the network, suggesting that a more explicit ‘How to join’ section would be more helpful to support this key task.

The Research section was too broad to usefully support specific tasks

Test participants gravitated towards the ‘Research’ section of the EHMN site for various tasks: to find research opportunities, to find help with research project development, to find content to support funding applications, to find out how to be a research participant and to find out how to financially support research activities.

On arrival in this section, when they realised it only contained content detailing different research activities, they went elsewhere. This suggested the need to separate out content about research activities from content supporting specific research-related tasks and to label it accordingly. In other words, instead of having a large, general ‘Research’ section, opt for smaller, more specific sections such as: ‘Research areas’, ‘Participate in research’ and ‘Donate’.

The new site has a flat structure, with content arranged by topic and tasks

Keeping site of the content strategy work, and using the results of testing the proxy site to make key decisions about structure, the new Dementia.ed site was built and launched.

The site contains six main sections, adopting a mix of topics and tasks to signpost the different sorts of content. The ‘About’ section contains background information about the dementia research network, with a dedicated page bringing together links to the different research centres and groups in a hub-and-spoke approach. A ‘What’s On’ section combines pages to Events and News content. The other four sections are labelled in task-focused ways, to ensure relevant audiences can quickly get to the content they need to complete their tasks: Join the network, Get involved, Contact us and Donate.

To ensure audiences reach content in the most direct way possible, the content in each section is either available as a top-level page, or a sub-page, avoiding long pages and deep hierarchies which may confuse those visiting the site.

Access the Dementia.ed website

If you need help architecting your website, get in touch

From start to finish, the process to design and launch the Dementia.ed website took a few months, including a few meetings between the UX Service and the ASDRC team, some short usability tests and an overall site review.

The ASDRC team provided the vision of what they wanted to achieve, and subject matter knowledge of their audience base and their associated tasks. With this information, the UX team could apply expertise to translate goals into a user-centred website, which is manageable for the ASDRC team to maintain.

We’re happy to offer help, advice and support to other University teams. If we can help, please contact us through the UX Service website.

Contact the UX Service

Using scenario-based research to understand how colleagues find inclusive language guidance

Earlier this year, the User Experience (UX) Service researched how colleagues find and use the University’s Inclusive Language Guide. This post explores why we used scenario-based research, what it revealed about colleagues’ behaviour and how the findings are improving the guide. 

The Inclusive Language Guide is only useful if people can find it 

The Inclusive Language Guide sits within the University’s Editorial Style Guide and provides practical guidance when writing about disability, race and ethnicity, and sex, sexuality and gender. The guide was developed in 2022 through a co-design process involving colleagues from across the University.

Inclusive Language Guide 

Having useful guidance is only part of the challenge. Colleagues also need to know it exists and be able to find it when they need it. 

More recently, we reviewed the Inclusive Language Guide to understand how discoverable it was for colleagues creating digital content. Emma Horrell has written about how this work prompted improvements to the guide’s visibility and signposting across the University. 

You don’t know what you don’t know: Improving the way we position inclusive language at the University

Why we used scenario-based research 

We wanted to understand more than whether colleagues were aware of the Inclusive Language Guide. Awareness alone does not tell us what people do when they need support, so we used realistic scenarios to observe how colleagues responded to situations where they might need guidance. 

We focused our research questions on four areas: 

  • How colleagues approached decisions about inclusive language. 
  • Where colleagues looked for support and whether they could find relevant guidance. 
  • Colleagues’ awareness of the Inclusive Language Guide and what they expected it to help with. 
  • Any gaps, ambiguities or areas of confusion. 

These questions shaped the structure of the interview and activities we designed. 

Researching colleagues approach to inclusive design decisions 

We combined semi-structured interviews with scenario-based activities to understand both colleagues’ existing experiences and how they approached real content decisions. 

We spoke to nine colleagues who create digital content 

We carried out nine interviews with colleagues in different roles across the University. Participants had a range of experiences creating digital content, including two student interns who brought perspectives of those newer to the University’s digital landscape. 

We began by exploring participants’ existing experiences 

Before looking at the scenarios or the guide itself, we asked participants about the types of content they create and whether they had ever needed to check terminology or wording before publishing. For example, we asked whether they had been in a position where they needed to consider how to word something to make sure the language was not excluding anyone. 

This helped us understand the approaches colleagues already used and the sources they relied on when dealing with these topics. 

Realistic scenarios showed how colleagues look for guidance 

We introduced a series of realistic scenarios based on everyday content tasks. 

Rather than asking whether participants knew about the guide, the scenarios allowed us to observe how they approached these situations in practice.

Example scenarios included: 

  • writing promotional content about a Pride initiative and wanting to ensure the language used was fair and respectful 
  • writing about someone who had experienced trauma and wanting to avoid misrepresenting them 
  • checking the preferred terminology for the chair of a company board before publishing a news article 

In posing the scenarios, we gained insight into where participants would usually look for this type of guidance, how confident they felt making language decisions and what they understood by the term ‘inclusive language’.

We adapted our scenarios as we learned more 

One of the original scenarios asked participants how they would write about someone elected to lead a company board. Rather than focusing on terminology, several participants described how they would research the individual or organisation before publishing. 

While this reflected realistic editorial practice, it did not help us answer the research question we were exploring. 

For later sessions, we refined the scenario so that it focused specifically on checking the preferred terminology for the role before publication. This encouraged participants to think about language choices and produced richer insights into where they expected to find guidance. 

What we learned: Four themes around finding and using inclusive language guidance

Across interviews, there were four themes that emerged consistently. 

Colleagues valued inclusive language even when they had not heard of the guide 

Across the interviews, participants consistently demonstrated that they valued using respectful and inclusive language. They were already thinking carefully about inclusive language, but they did not always know where to access University specific guidance.  

Many described that they would check terminology with colleagues or seek advice from specialist teams when writing about unfamiliar topics. They wanted confidence that they were using appropriate language. 

Awareness of the importance of inclusive language was much higher than awareness of the Inclusive Language Guide itself. 

Colleagues expected to find guidance through different routes 

The scenario-based activities helped us understand where colleagues naturally looked for support. 

Participants frequently described turning to colleagues, managers, Equality, Diversity and Inclusion (EDI) resources, staff networks and external websites. A few participants did eventually land on the guide, but they had not recalled it or looked there first. 

Several participants also relied on previous examples when writing content. Others spoke about balancing official guidance with lived experience and perspectives from the communities being represented. 

This highlighted an important consideration. Colleagues expected inclusive language guidance to be connected to the places they already look for support, linked from elsewhere rather than existing solely within the Editorial Style Guide.  

Improving signposting therefore remains just as important as the guide itself. 

Colleagues understood the phrase ‘inclusive language’ in different ways 

Participants interpreted the phrase ‘inclusive language’ in different ways. 

Some described it as “using the correct terminology”, while others framed it more broadly as “not leaving anyone out”. Those with accessibility backgrounds often drew direct connections between inclusive language and accessibility. 

Despite these differences, the term ‘inclusive’ was recognisable, with several participants using it unprompted when working through the scenarios.

Colleagues looked for practical guidance first 

After exploring how participants would approach different situations, we asked them to navigate the Inclusive Language Guide. This helped us understand whether the structure matched their expectations when looking for support. 

Several participants were unsure where the practical guidance began. Some mistook the introductory blog post for the guide itself, while others expected the examples and practical advice to appear much earlier on the page. 

Although participants valued the context and principles behind the guidance, they prioritised practical information they could apply immediately. The examples within the guide were consistently highlighted as the most useful content. 

The research continues to inform improvements to the Inclusive Language Guide 

Our findings have allowed us to identify some key areas for improvements to the Inclusive Language Guide.

Making practical guidance easier to find 

One area of focus is the guide’s landing page. We are reviewing its structure so that practical guidance is easier to find while retaining the contextual information that participants found valuable. 

The findings highlighted that the current structure does not always support the way colleagues use the guide. Participants wanted to move quickly to practical guidance while still being able to access the wider context when needed.

We are also reviewing page titles, navigation labels and the ordering of content to make it clearer where the guide begins and how it relates to the wider Editorial Style Guide. We’re considering how to better support both first-time and repeat use, making it easier for people to quickly find practical guidance while still providing the context and principles that many participants found valuable.

We’ll also consider how best to surface practical examples and support the different ways colleagues use the guide, whether they are reading it for the first time or returning to quickly check terminology and writing conventions.

Improving connections with related guidance 

The research also prompted us to consider how the guide connects with related guidance across the University’s web estate. 

Participants frequently expected to find inclusive language guidance within EDI resources or other specialist guidance. Improving discoverability means not only improving the guide itself, but also making sure it is connected to the places colleagues naturally go when seeking support.

As outlined in Emma’s blog, since completing the research, we have worked with colleagues responsible for complementary guidance across the University’s web estate to improve signposting between services.

This work will help ensure colleagues can find appropriate support regardless of where they begin their journey. 

Scenario-based research helped us understand behaviours 

The research reinforced an important principle in content design: guidance needs to be easy to find as well as useful.  

By combining colleagues’ existing experiences with scenario-based activities, we gained a better understanding of how people approach inclusive language decisions. These insights have already informed improvements to the Inclusive Language Guide and will continue to shape how we approach future content research.

QuillMark: The Technical Details Behind My Drupal Style-Guide Assistant

I previously introduced QuillMark, a tool I built to help web publishers check content against the University’s editorial style guide. In this post, I explain the technical design and methodology behind the prototype.

Introduction

My name is Shlok, and this summer I am working as an AI and UX Innovation Intern in the Information Services Group (ISG). During my internship, I have been developing QuillMark, a prototype tool designed to help web publishers check their content against the style guide. This post outlines the system design, methodology and AI pipeline behind the prototype, including how deterministic checks, language models and a MapReduce approach work together to improve reliability while keeping publishers in control.

 

QuillMark highlights violations in your existing text rather than generating an entirely new version. This:

  1. avoids the risk of AI making unnecessary changes to unrelated parts of the page, which would require a manual audit each time
  2. allows publishers to clearly see each violation and accept or reject suggested changes
  3. reduces output tokens, lowering both hallucination risk and cost

 

Problem: The style guide contains a large number of rules, around 60–70. If all rules are sent to the LLM at once, the model may not check the text against every rule reliably.

 

Research: https://openreview.net/pdf?id=R6q67CDBCH

 

Solution → MapReduce (Divide and Conquer: Map → Collapse → Reduce)

 

System Design and Methodology

 

Pipeline:

  1. Deterministic checks to flag blatant violations
  2. Fine-tuned small model flags likely obvious violations
  3. Large model audits the final page deeply for complex deeper violations

 

The deterministic checks are used to flag “obvious” violations that are directly detectable using regex or code. For example, this could include the use of a forbidden word, or a spelling convention such as using “benefited” instead of “benefitted”. These issues can be detected using a direct search of the page and do not require AI.

 

The second stage uses a smaller fine-tuned model. This model would be fine-tuned on the Style Guide rules and training examples. It would again be used to detect relatively “obvious” violations, but ones that are not simple enough to be detected reliably through deterministic checks. These issues are not highly complex, so they should still be visible to a smaller model.

 

After the publisher fixes the deterministic violations and the violations found by the fine-tuned small model, there are likely to be remaining deeper and more complex issues that both previous approaches did not pick up. The final large model can then focus mainly on these more complex issues. This means it spends less attention on obvious violations and more attention on issues that are easier to miss.

 

This improves the process in two ways.

 

First, it allows the final large model to focus on deeper issues. If the whole page was passed directly to the large model from the beginning, it would likely flag many obvious violations first and might miss some subtler issues. By fixing the most obvious issues earlier, the large model can focus more on smaller, judgement-based, or easily missed problems.

 

Second, it may reduce cost compared with running the large model multiple times. One alternative would be to run the large model once, fix the obvious issues + some deeper ones it finds, and then run the large model again to find the remaining deeper issues. In the first run, the large model may spend much of its output on obvious violations. In the second run, after those obvious issues are fixed, it may be able to look more deeply. The proposed approach tries to achieve a similar effect more cheaply by using deterministic checks and a smaller fine-tuned model before the final large-model review.

 

This pipeline is shown in Figure 1.

 

Style Guide Violation Detection Pipeline
Figure 1: Style Guide Violation Detection Pipeline

 

Using MapReduce to reduce instruction-following degradation

Based on personal experience with large language models, as well as existing research such as the Curse of Instructions, we cannot rely on a large language model to follow a very large number of rules at once. In this case, the style guide contains around 60–70 rules. If all of these rules are passed to the model in a single request, the model may not check the page against every rule properly. It may follow some rules, ignore others, or miss violations because there are too many instructions to apply at the same time.

 

To solve this, I propose using a MapReduce approach, which is a divide-and-conquer technique: Map → Collapse → Reduce.

 

The rules would be broken into smaller chunks, with each chunk containing a fixed number of rules. Each chunk would then be sent as a separate request, asking the model to check the page only against that smaller set of rules. This allows the model to focus on fewer rules at a time and check them with more attention.

 

After all the chunks have been processed, the responses would be gathered together and combined into one report. This is the collapse stage. The combined findings would then be passed through another large language model request in the reduce stage. This final request would clean up the output by removing duplicate findings, resolving overlapping issues between chunks, and filtering out possible false positives or hallucinated violations.

 

This approach makes the checking process more reliable because it avoids asking the model to apply all 60–70 rules at once. Instead, each group of rules is checked more carefully, and the final report is produced by combining and cleaning the results. This helps ensure that all rules are checked with more equal rigor, rather than relying on the model to remember and apply every rule in one large prompt.

 

The MapReduce approach is shown in Figure 2.

 

MapReduce Approach for Style-Guide Rule Checking
Figure 2: MapReduce Approach for Style-Guide Rule Checking

 

Update: Large-Model Cleanup from the Reduce stage was excluded from the prototype as the probability of duplicates occurring is low and the cost of such a cleanup is relatively high with minimal benefit.

 

Read more about the technical system plan: https://blogs.ed.ac.uk/website-communications/building-quillmark-testing-a-drupal-style-guide-assistant-with-web-publishers/

Watch a video demo: https://media.ed.ac.uk/media/t/1_yozad8ie

 

Future enhancements:

  • Strengthen Deterministic Regex-matching with more training data (web pages)
  • Audit deterministic findings using a cheap, local AI

Building QuillMark: testing a Drupal style-guide assistant with web publishers

I introduce QuillMark, a Drupal prototype designed to help web publishers check content against the University’s editorial style guide. I reflect on how I developed and tested the tool with publishers, focusing on what their feedback revealed about usability and how it will shape the next version.

Introduction

My name is Shlok, and this summer I am working as an AI and UX Innovation Intern in the Information Services Group (ISG). During my internship, I have been developing QuillMark, a prototype tool designed to help web publishers check their content against the style guide. This blog post explains the problem I was trying to solve, how I built and tested the prototype, what I learned from publishers and what I plan to do next.

 

The problem

Our style guide contains more than 60 rules covering areas such as spelling, punctuation, tone, formatting and terminology. Previous research showed that web publishers found it difficult to remember and apply every rule while writing and editing content.

 

This is understandable. Publishers are often working under time pressure, and checking a page manually against dozens of rules adds a significant cognitive burden. Even experienced publishers can miss small details, particularly when they are concentrating on the meaning and accuracy of the content.

 

I began exploring whether a tool could make this process easier. The aim was not to replace publishers’ judgement or rewrite their content automatically. Instead, the tool would identify possible style-guide violations, explain the relevant rule and allow the publisher to decide whether to apply or dismiss each suggestion.

 

This became QuillMark.

 

Read more about the Style Guide

 

Why I built the prototype in Drupal

Drupal is where publishers already create and edit web content, so it made sense to build the prototype directly into that environment rather than create a separate tool.

Before starting development, I attended Drupal in a Day to understand the platform, its content-editing interface and how a custom tool could fit into an existing publishing workflow.

 

Building QuillMark within Drupal meant publishers could check their content without copying it into another system. It also allowed me to test the tool in a realistic environment, using the same types of fields, buttons and interactions that publishers encounter in their day-to-day work.

 

Read more about Drupal in a Day

 

Planning the checking process

Before writing the prototype, I planned the system in detail.

 

One of the main design decisions was that QuillMark should highlight individual issues in the existing content instead of generating a completely rewritten version. Rewriting an entire page could introduce unnecessary changes and would require the publisher to audit every sentence. Showing individual findings makes it clearer what the tool has identified and keeps the publisher in control. This also prioritises the HITL (Human-in-the-Loop) framework, which I intend to use in my AI-based projects to ensure that people remain involved in reviewing decisions made by AI.

 

The system design proposed three types of checking:

  1. Deterministic checks for clear violations that can be identified using code or regular expressions, such as prohibited words or spelling conventions.
  2. A smaller AI model for relatively straightforward issues that require more context than a simple text search.
  3. A larger language model for more complex, judgement-based rules involving areas such as tone, structure or plain language.

 

The plan also used a MapReduce-style approach. Rather than asking a language model to apply all 60–70 style-guide rules in one prompt, the rules would be divided into smaller groups. Each prompt could then concentrate on a limited number of rules before the results were combined. This was intended to reduce the risk of the model overlooking instructions because it had been given too many at once.

 

The original design included an additional AI step to remove duplicate findings and resolve overlaps. I left this out of the prototype because duplicates were expected to be relatively uncommon, while the extra model request would increase cost and complexity.

 

Read more about the technical system plan: https://blogs.ed.ac.uk/website-communications/quillmark-the-technical-details-behind-my-drupal-style-guide-assistant/

Coding the prototype

I developed the prototype iteratively, beginning with a small number of style-guide rules and expanding the checks once the basic workflow was working. For development, I used Codex. I gave it my plan and asked it to design the architecture. After reviewing it, I asked it to proceed with the implementation.

 

The first stage used deterministic checks for issues that could be found reliably without AI. These checks searched the content for recognisable patterns and returned the location of the issue, the relevant style-guide rule and, where appropriate, a suggested correction.

 

The second stage used a large language model to identify issues that depended more heavily on context. I divided the rules into smaller prompt groups so that the model could focus on a manageable set of instructions during each request.

 

The prompts asked the model to return structured findings rather than a rewritten page. Each finding needed to include enough information for QuillMark to:

  • identify the relevant content;
  • explain the problem;
  • show the related style-guide rule; and
  • suggest a possible correction.

 

Publishers still had to approve or dismiss each finding. This human-in-the-loop approach was intentional. Style rules can depend on context, and an automated suggestion will not always be appropriate. QuillMark was designed as a decision-support tool, not an automatic editor.

 

Watch a video demo: https://media.ed.ac.uk/media/t/1_yozad8ie

 

Designing the usability test

Once the first iteration was complete, I adapted an existing usability-testing template to create a test for QuillMark.

 

I conducted three sessions with web publishers. Participants were asked to work through a series of tasks covering the main parts of the prototype, including running the checks, understanding the two stages, locating an issue in the editor, reviewing a rule and applying or reverting a suggested fix.

 

I observed how participants used the interface, where they hesitated and whether the information on screen matched their expectations. I also asked how they might use the tool as part of their normal publishing process.

 

The purpose was not only to find technical bugs. I wanted to understand whether the concept itself was useful and whether the interface communicated how the tool was intended to work.

 

What publishers told me

The publishers thought the tool would be useful as a final check before publishing. They said they would still use their own judgement and would not automatically accept every suggestion.

 

Most of the feedback was about the usability of the tool. Some parts were unclear, such as the difference between the two stages and some of the button labels. There was also too much information on the screen, and some issues were repeated. The publishers also wanted to see what a suggested fix would change before applying it.

 

What I learned

The testing suggested that the core idea is useful, but the user experience needs to become simpler.

 

Publishers do not necessarily need to understand which checks use regular expressions and which use an AI model. They need to know what issue has been found, why it matters, what the proposed change is and what action they can take.

 

The sessions also reinforced the importance of designing for scanning. More explanation does not always create more clarity. In a publishing workflow, concise labels, clear states and well-grouped findings may be more valuable than displaying every piece of supporting information at once.

 

Next steps

My next step is to refactor and review the prototype code before making changes based on the usability feedback. This will include reviewing the AI-generated code to check whether each section is needed, whether the same result could be achieved more simply, and whether any parts repeat code that already exists. I will compare the generated code with the rest of the codebase so that it follows the same structure and does not add extra functions or files without a clear reason.

 

Where the code is longer than needed, I will simplify it by removing repeated checks, combining similar sections, and reusing existing functions. I will also keep individual files to a manageable size, generally no more than 1,000 to 3,000 lines depending on the file. Larger files will be split where this makes the code easier to follow, and hardcoded rules will be moved into the database where possible.

 

The next iteration will focus on:

  • clarifying or simplifying the two-stage workflow;
  • reducing repeated and overwhelming findings;
  • improving button labels, status messages and colour distinctions;
  • previewing changes before they are applied; and
  • making the interface more concise and easier to scan.

 

What I am working on as a Digital Developer Intern (Figma)

By: kliu3
9 July 2026 at 10:29

In March, I started my internship as a Digital Developer within the Website and Communications team. My internship focuses on exploring Figma, the industry-leading digital design tool, and the user interface (UI) component library that the University has been developing within it.  

First impressions as an intern 

On my first day – somewhere between onboarding, learning a whole new set of acronyms, and unexpectedly getting free pizza – I began to get a sense of what the next few weeks might look like. I also met with colleagues I’d be working with closely on the project: Sonia Virdi (Human Centred Specialist in Web Strategy and Governance) and Mel Batcharj (Content Designer in UX and Digital Consultancy).

Looking back, that first week was a bit of a whirlwind. From getting to grips with the University’s structure, to navigating room bookings in the Edinburgh Futures Institute, to settling into the Forrest Hill office (and quickly becoming a fan of the huge monitors), it took some time to find my rhythm. 

Attending my first stand-up and adjusting to a new way of working was all part of that process. At the same time, I began exploration and research, which quickly became central to my day-to-day work. What stood out most was how quickly the experience shifted from unfamiliar to genuinely engaging, especially as I explored the problem space in more depth. 

The platforms and tools behind this project 

Several platforms and tools underpin this work, helping us design, document and maintain digital components while ensuring consistency between design and development. 

EdGEL (the University’s coded component pattern library)

EdGEL is the University’s coded component pattern library. For those unfamiliar, a pattern library contains all the digital assets that are required when building and creating digital products and services. EdGEL’s underlying design framework is Bootstrap, which provides the starting point when developing coded components.

Bootstrap

EdGEL is widely used across websites and web applications. It acts as a central source of truth for how interfaces should be branded, structured and implemented at the University. It was developed in 2015 when the first Drupal installation of the University’s central web platform, EdWeb, was launched. 

EdGEL website 

Documenting EdGEL components (Pattern Lab and Storybook)

The assets contained within EdGEL are currently documented within Pattern Lab, which is being replaced by the tool Storybook. Both tools provide a practical way for developers to explore the available EdGEL components and understand how and when they should be used.

Storybook implementation of EdGEL pattern library 

Figma Design 

Figma Design is an industry-standard tool used to create user interface designs (UI), prototypes and support teams through its collaborative features. It’s widely adopted by organisations (from startups to large enterprises ) because it brings design and development workflows closer together, reducing inefficiencies in how digital products are created and iterated on. 

More recently, Figma has introduced AI-powered features for prototyping and layout generation, making it an exciting tool in the current digital software landscape. 

What is Figma? (Figma article)

Building a University UI component library in Figma

In 2021, as part of the University of Edinburgh’s design system project, Figma was adopted to develop a library of digital UI components.  

Blog on the University’s design system project 

The goal was to make the University’s branded EdGEL components more accessible to a wider range of colleagues, align the two libraries and streamline digital development workflows. Additionally, the project’s aim was to simplify the process of making design changes and improvements to the EdGEL components, ensuring they are fit for purpose across a broad range of University environments. The work to create the Figma UI component resource would support a variety of roles involved in designing and developing digital platforms. 

 

Screenshot of the University's Component Library showing various components, such as Buttons & Links, Card Blocks, Event cards

The University of Edinburgh’s User Interface Library of components.

 

Why Figma was chosen 

Figma was selected for several reasons: 

  • Its functionality outweighed any other digital tool in respect to component development. 
  • It helped bring design and development practices closer together, creating a more efficient workflow. 
  • It enabled component and resource sharing across the University, reducing duplication of effort. 
  • It supported more consistent use of the University’s branded components across digital products and services. 
  • The strong user community provided a rich resource for learning best practices and obtaining support. 

My internship has two key aims

The two aims of my internship both focus on Figma, however, they approach the challenge from different angles. The first is about understanding how staff across the University currently design and deliver digital work, and where Figma fits within that landscape. The second is about improving the University’s existing Figma component library to better support its users. 

Understanding how Figma fits in at the University

The primary aim of my internship is to learn more about how Figma supports digital development processes across the University. This involves assessing Figma alongside the other tools teams use every day and understanding where it could add value.

We focused on understanding how people work

As a second-year computer science student, I was keen to move away from a purely engineering perspective and instead immerse myself in the user side – exploring not just Figma as a tool, but the people using it.  

Understanding how staff design, collaborate, and deliver digital work is central to this. Importantly, we’re not just interested in Figma itself, but in the wider workflows and tools that people rely on.

To guide this work, we focused on a few key questions: 

  • What tools are people using day-to-day? 
  • How do these tools support their work? 
  • What works well and what does not? 
  • How does or could Figma improve their workflow process? 

A survey helped us reach staff across the University

To help us understand how people work, we created a survey to reach a large proportion of internal staff across a range of roles. These included: 

  • marketing and communications staff producing high volumes of digital content 
  • visual designers working on platforms and services 
  • developers building and maintaining applications and systems 
  • user experience (UX) professionals carrying out human-centred research and design 
  • academic staff designing and prototyping their own tools 
  • others (such as project managers working with product teams) 

A key decision we made from the start of my work was not to focus solely on “designers.” Through discussions, we recognised that design isn’t confined to traditional visual design roles. People across the University make design decisions about layout, content, structure, and user experience, often using whatever tools are easiest to access within the University. 

Working with This Is Milk to enhance and improve the existing Figma UI library

Our work is being supported by the design agency This Is Milk, who are helping us make better use of Figma’s newer features while improving the University’s UI component library.

This is Milk 

This will involve: 

  • making the library easier to access and integrates into users’ workflows
  • improving synchronisation between the EdGEL codebase and the Figma UI library to reduce maintenance effort 
  • developing training resources to help colleagues get started with Figma 

What I’ve learned so far

Developing my own understanding of design systems and Figma

Alongside our research, I’ve been developing my understanding of both design systems and Figma. 

The internship has given me the opportunity to build on concepts I was already familiar with, like components and variant properties, while exploring newer ideas, including design tokens and more scalable system structures.

Adopting a user-first mindset

So far, the experience has shown me that designing digital systems isn’t just about creating components or choosing the “best” tools. It’s about understanding people, workflows, and the small details that make something either frictionless or frustrating to use.

Importantly, the internship has pushed me to think more from a user-first perspective, grounding decisions in usability, accessibility, and real user needs rather than focusing only on the tehcnical or visual aspects of a solution.

Looking ahead

As my internship continues, I’m particularly looking forward to getting more hands-on with improving the design system and component library, as well as continuing to engage with staff to better understand how these tools can support their work. 

Myself, Sonia and Mel will continue to blog about the project as it progresses. Our next post will explore the survey research in more detail, including how we carried it out, who took part and our initial findings.

How to run a usability test

In our May Content Improvement Club session, we focused on how to run a usability test. We ran through the basics of putting a script together, watched a clip from a test and had a go at prioritising some issues.

Two problems for people working with content

We started this session by outlining two problems for people working with content.

We don’t see people using our websites

The first problem is that we don’t typically see people using the things we create. We build a website based on our best assumptions of what our users need and how we think they will interact with it. Then at some point in the future, someone uses the site. They might find it easy to use. They might find it difficult. But we don’t know, because we don’t get to see.

It’s hard to self-assess our own website

The second problem is that it’s hard to self-assess a website that we’re already familiar with. When we look at the site, we bring our understanding of the structure and the context it sits in. We know what the acronyms mean. We know where to find the contact form. We know where the links go.

This isn’t necessarily the case for a user coming to the site for the first time.

Here’s Steve Krug, author of Rocket Surgery Made Easy:

“If you’re building something, you’re not going to be able to see where it’s going to confuse people. It’s not going to confuse you. You know too much about it.”​

Steve Krug interviewed on the Brave UX podcast

This is sometimes known as the ‘curse of knowledge’. Erika Hall, author of Just Enough Research, explains:

Whenever we learn things, we forget what it’s like not to know those things. […] The more you know, the more you expect other people to know. And if you become a real expert in a topic, forget it.

Erika Hall writing in the Mule Design Studio newsletter: Don’t chicken out about talking to people

Usability testing is a way of addressing these problems. It gives us a chance to see what it’s like for someone visiting our site for the first time and it counteracts the curse of knowledge. That gives us valuable insight into where a design is working and where it isn’t.

A quick definition of usability

Before we go any further, a quick word about what we mean by ‘usability’.

In short, usability refers to how easy, effective and satisfying something is to use.​

The cover of Don Norman’s book ‘The Design of Everyday Things’ features a coffeepot with a spout and handle on the same side. This would score poorly in a usability test as it wouldn’t be easy, effective or satisfying to use.

The cover of the Design of Everyday Things, featuring a coffeepot with a spout on the same side as the handle.

One of French artist Jacques Carelman’s impossible objects, as featured on the cover of the Design of Everyday Things.

ISO definition

There’s also an international standard (ISO 9241-11) definition of usability:​

“The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.”​

ISO definition of usability

This is a more technical definition, but it’s worth bearing in mind. It emphasises the fact that we need to think about specified users and their goals. That will come up when we get to writing our script.

A simple usability test

The simplest usability test you can do takes about 5 minutes:

  1. Grab a colleague / friend / family member who doesn’t know your site​.
  2. Sit them at a computer and open the homepage of your site.
  3. Set them a task. For example: “Imagine you’re a student and you want to get a replacement student card”.​
  4. Sit quietly and watch as they try to complete the task.

This doesn’t take much effort, and sometimes you uncover a usability issue or two.

In Content Improvement Club, we looked at how to run a more extensive usability test. But we wanted attendees to know from the outset that usability testing doesn’t have to be a big complicated process. It can still be beneficial even if you do it on a small scale.

There are three roles in every test

In a usability test, there are three roles:

  • A moderator, who reads the script, sets tasks for the participant and asks questions​.
  • A participant​, who completes tasks on a website​, thinking out loud​ as they do so.
  • An observer​, who watches and takes notes​.

There can be any number of observers, who watch the test live or on a recording.

We watched an example of a usability test

The best way to understand how a usability test works is to watch a recording of one.

Here’s Steve Krug demonstrating how he runs a test:

Usability Test Demo by Steve Krug (YouTube video, 24 minutes)

In Content Improvement Club, we watched a short clip of someone completing a task on a library website at a UK university.

This was the task:

You’re working on a presentation with four other people from your course. You need to find a study room in the library for the four of you this weekend. Can you find out if a suitable study room is available at the library?

We noted down the issues that we saw, and then we shared them on a Microsoft Whiteboard. Even though it was a single task, there were a lot of issues, which is fairly typical. That’s why it’s often helpful to follow your observations with a prioritisation exercise.

We prioritised the issues

In the session, we demonstrated how you can prioritise issues using a matrix like this:

A matrix showing Easy to fix and Hard to fix on the vertical axis, and minor issue for users and major issue for users on the horizontal axis. Blank sticky notes are in some quadrants.

To place things on the matrix, you make a quick assessment of how significant the issue is. Then you assess how easy it would be to fix. The issues you typically want to focus on first are those in the top right quadrant: major issues that are easy to fix.

It can be helpful to have a set of questions to establish whether an issue is major or minor.

These are adapted from Dave Travis’s work on Red Route usability testing:

  • Does the problem occur in a task that is highly important to you or your users?
  • Is the problem difficult for users to overcome?​
  • Did multiple participants experience the same problem?

Red route usability: The key user journeys with your web site (Dave Travis / archive.org)

Using questions like these give you a shared set of criteria when assessing the significance of usability issues in a group.

But we’re getting ahead of ourselves. How did we get to this point?

How to write a script

Before you can run a series of tests, you need a script.

Use a template

We shared our usability testing script template, which draws on the work of Steve Krug:

Research brief and usability testing script (DOCX, 51KB)

The template includes:

  • a brief, where you articulate the goals of the testing
  • a lead-in script, which you read to participants before testing begins. This covers what the session will involve and sets expectations. ​
  • a set of tasks, presented in a table with two columns. One column contains the task, and another the expected path to complete the task.​

Come up with tasks

We start by identifying tasks and then develop them by adding a scenario.

We practised this in the session. Attendees suggested tasks that someone might need to carry out on a university library website. For example:

  • Check opening times
  • Find a book on your reading list

Develop tasks by adding scenarios

Next, attendees developed these tasks by adding a scenario.

For example, “Check opening times” becomes:

  • You’re a student and you want to check the opening times of the library during the winter break. How would you go about doing this?

“Find a book on your reading list” becomes:

  • You’re a new student and you want to find the book “Campbell’s Biology”, which is on the reading list for your course. Can you show me how you would find out if the library has a copy of this book?

Tips for writing good tasks and scenarios

We shared some tips for turning tasks into questions​:

  • Aim for 5 to 10 scenarios per session.​
  • Keep tasks focused: one goal per task.​
  • Frame tasks as realistic scenarios, not instructions.​
  • Add light context to make it realistic.​

We recommend running a pilot usability test before going ahead with a round of testing. This helps you see how the full usability test flows from start to finish. ​It also gives you a chance to refine the script and ​ensures the session runs smoothly for participants on the day.​

Logistics

Ahead of the Content Improvement Club session, we asked attendees if they wanted us to cover any topics in particular. Most of these came under the wider topic of logistics.

When should you test in a design process?​

Test earlier rather than later. Testing earlier in a design process is ideal because it’s easier to make changes based on what you learn. If you’re creating something new, this usually involves creating rough drafts or prototypes, and then later, a high-fidelity version.

Changing a prototype is easy because no one is very attached to it. But when a design is a later stage of development, it tends to be more painful to learn that something isn’t working. By this point, you and your colleagues have usually invested a significant amount of time in an idea. If the fix involves changing something fundamental to the product, you might have to undo work that’s already been done.​

How long does a testing session take?

It varies, but we usually run testing sessions of about 30 minutes.

How many tasks do you set?

In 30 minutes, we can usually fit in between 5 and 10 tasks.

How many participants do you need?

Three to five participants is a good target. Jakob Nielsen argued that five participants is enough for one round of tests. This is because when you run the same tests with multiple people, you tend to see the same usability issues reoccurring. It’s a case of diminishing returns: by test number six, you aren’t usually learning anything new.

Why you only need to test with five users (Nielsen Norman Group)​

In Rocket Surgery Made Easy, Steve Krug recommends three participants per round of testing. This way you can run more rounds of tests – for example, by testing an initial idea and then a later iteration of the same design.

How do you recruit participants?

Mailing lists, Teams channels and surveys are great places to put call outs. We advise that you avoid revealing the testing topic or content in advance.​

Aim for participants who reflect real users where possible. This can prove difficult, so don’t let it stop you if you can’t find participants who don’t match the profile of your users.

How do you take notes?

Obviously, you can scribble notes anywhere you like. When we’re running a series of tests, we often take notes in a spreadsheet. This allows us to quickly spot which tasks caused problems for multiple participants.

Usability test results spreadsheet (XSLX, 36KB)

How can we make testing more accessible and inclusive?

Accessibility and inclusivity should be considered from the very start of the testing process. One important step is asking participants early on whether they use any assistive technology, so you can understand their setup and make any necessary arrangements. Our own introduction questions and templates include this for that reason. In the template, this is an introductory question, but if you’re recruiting via a survey, you could mention this there.​

​Having a diverse participant group will usually lead to more useful and representative findings. For example, if you’re testing a student-facing service, including both undergraduate and postgraduate students can help capture a wider range of experiences and needs. Again, this can sometimes be a challenge, but if possible something to aim for.​

Ideally, usability testing should include participants who regularly use assistive technologies. This provides valuable insight into accessibility barriers that might otherwise be missed. However, this can sometimes be challenging due to recruitment costs, specialist panels, or limited budgets.

Some resources that can help:

Acting on the findings​

So you’ve run a round of testing. What happens next?

Involve senior colleagues in playbacks

In 2015, Caroline Jarrett wrote about a phenomenon she and Steve Krug had noticed when working on websites. They would run a set of tests on a website and uncover some usability problems. But then six months later, the problems were still there: no one had gone in and fixed them. To investigate why this was happening, they ran a survey of UX professionals.

Caroline Jarrett and Steve Krug’s analysis of why usability problems go unfixed

Out of 131 responses, the most common reason was that findings from usability testing conflicted with a decision maker’s opinion.

One way to address this is to get decision makers in the room (or on the Teams call) when you watch a highlights reel of test recordings. There really is no substitute for seeing user behaviour first hand. Reading about it in a report doesn’t carry the same weight.

Use the momentum created by testing

Testing creates a shared motivation to fix things​. Use this to your advantage. If you can, take action to fix usability problems while people’s memories are fresh​.

Make small changes first

Sometimes usability testing highlights small problems that are easily fixed. A link that doesn’t go where someone expects it to. An item missing from an A-Z.

Sort these things out first. Small wins like this give you the motivation to sort out the knottier problems.

If you want to find out more

This was a quick introduction to running your own tests. If you want to learn more,  Steve Krug’s introductory guide is a great place to start:

Rocket Surgery Made Easy by Steve Krug (listing on DiscoverEd)

We post about case studies of usability testing at the University on this blog:

The Prospective Student Web Team do the same:

Acknowledgements

Various points made in this blog post are taken from this 2015 post by Neil Allison:

Making usability testing agile

How to hear about future sessions

We promote these sessions via our mailing list. If you’re interested, please sign up:

Join the UX and Content Design mailing list (University login required)

Suggest a topic for a future session

We picked usability testing following a suggestion from the community. We’re keen to continue covering topics that colleagues across the University would find useful. It would be really helpful if you could let us know any ideas you have using this form:

Suggest a topic for Content Improvement Club (University login required)

Other training that we offer

More training is listed on the User Experience Service website:

Training | User Experience Service

Do students actually watch videos on websites?

It is difficult to think of a university that does not treat video as a core part of its content strategy. Across admissions, communications, and academic departments, the assumption has quietly taken hold that if something can be filmed, an open day, a student or alumni testimonial, it probably should be. The result is that video has become the default format for a huge amount of content that might once have been written.

What that assumption rarely accounts for is the environmental cost sitting behind every piece of video content. Streaming and hosting video is energy intensive, and the carbon footprint of digital content at scale is a growing concern across the sector. For universities with sustainability commitments, that adds another layer of responsibility to decisions about when and why video is the right choice.

But when we sat down to observe how students actually engage with video content, the findings were more nuanced than that assumption would suggest. Rather than confirming that video is universally valued, the research pointed that engagement depends heavily on context, framing, and the individual’s immediate needs, and that for a significant portion of users, video is simply not the preferred way to receive information.

We ran a small qualitative study with three student participants, observing their natural behaviour as they encountered video content embedded in university communications, including emails and webpages. It is worth noting that the videos in question were informational videos hosted on university websites, part of a central university service, and not lecture recordings, online courses, or other educational media.

Most students do not want to watch videos

This was the clearest finding from the research. Two of the three participants showed low or no interest in video content. One stated outright that she actively avoids videos, citing a short attention span and a strong preference for reading. Another would only engage if the video was directly and immediately relevant to her situation, and even then her decision rested heavily on the title and how long the video appeared to be.

Only one participant engaged readily with video, and notably, he was at a specific point of relevance: he was actively working on job applications, and the content matched his immediate need. This matters because it suggests that video engagement is not simply a question of format preference, it is a question of timing and context. The same video, shown at the wrong moment, might not get watched at all.

Even though 15 minutes isn’t long, I would like to know what I spent it learning. You decide in the first few seconds if you want to continue watching.

– Participant, Graduating student

The title is the first and most important decision point

For participants who were willing to consider watching, the title was the single biggest factor in whether they clicked. Titles that named a problem the viewer already had, and implied a clear answer, consistently outperformed broader or more generic framing.

A title like “Why wasn’t my application successful?” performed well because it is direct, addresses something the viewer is likely already worried about, and signals that the video will answer a specific question. By contrast, titles like “Top Tips” or broad sector titles were seen as vague, leaving participants uncertain about what they would actually gain from watching. One participant put it plainly: she did not know what she would get out of a broader title, and that uncertainty was enough to stop her from starting.

What this tells us is that if a title does not tell someone what they will learn and why it matters to them, most students will not give the video a chance to do so itself. The window to make that case is a matter of seconds.

Attention spans are shrinking among students

Multiple participants mentioned short attention spans without being prompted. None were willing to watch a video over an hour long, and one participant was explicit that long videos were simply not something she would engage with at all. The upper threshold for something being considered watchable appeared to sit somewhere under 15 minutes, and even then only if the content felt directly relevant.

This is not simply a case for keeping videos short. One participant suggested that chapter-style signposting at the start of a video would significantly change her willingness to engage. If she could see upfront what would be covered, and at what point in the video, she would feel more in control of her time. Knowing that the section she needed was four minutes in would make her far more likely to watch than facing an unlabelled block of content with no way to navigate it.

When written content already exists, the video is less likely to be watched

When participants knew or suspected that the same information existed in text form elsewhere on the page, or as an automated transcript on Media Hopper, they were noticeably less interested in watching a video covering the same ground.

Videos should not be used as a parallel format for content that already works well in writing. It should be reserved for things where the medium genuinely earns its place, such as demonstrations, walkthroughs, or anything where showing something is meaningfully better than describing it. Using video to duplicate written content risks producing material that most users will simply scroll past. It is also worth noting that every video hosted and streamed carries an environmental cost. According to the Carbon Trust, streaming one hour of video generates approximately 55g of CO2 equivalent.

Can audio do the job?

Video should not be the default. Given the low and conditional engagement observed in the research, it should be reserved for content that genuinely benefits from the medium, such as demonstrations and walkthroughs, rather than anything that could be communicated equally well in text. Where written content already exists in the newsletter, avoid duplicating it in video form. Participants are less likely to watch if they can read it instead, and every unnecessary video adds to the environmental footprint of the platform.

Where audio adds value but visuals are not needed, consider using the MP4 audio format instead. Audio files are significantly smaller, require less energy to stream, and can carry the same content at a fraction of the environmental cost.

Get the title right

If the title does not tell a student what they will learn and why it matters to them, most will not click. Use direct, problem-focused titles that address something the viewer is likely already thinking about. “Why wasn’t my application successful?” works because it names a specific concern and implies an answer. “Top Tips” does not, because it could mean anything. Where possible, address the viewer directly in the title. It signals relevance immediately, and relevance is what drives the decision to watch.

Keep it short and make it navigable

The research pointed to an upper threshold of around 15 minutes for something to feel watchable, and even then only if the content felt directly relevant. But length alone is not the whole picture. One participant was clear that chapter-style signposting at the start of a video would significantly change her willingness to engage. Knowing what would be covered, and when, made her feel in control of her time in a way that an unlabelled block of content did not. Opening each video with a summary of what will be covered, and using timestamps or chapter markers throughout, gives viewers the information they need to decide whether to watch and where to start.

Turn off autoplay

Autoplay is likely to cause disengagement, particularly among users who prefer text and did not actively choose to watch. It also adds unnecessary weight to the page, causing video to load and stream regardless of whether the user has chosen to engage with it. Disabling autoplay is one of the simplest changes a publisher can make, and it has both a user experience and a sustainability benefit.

Always provide a transcript and a written summary

A text transcript alongside every video supports users who prefer reading, users with accessibility needs, and those who want to skim content before committing to watching. A short written summary below each video allows text-preferring users to get the key points without watching at all, and reduces the likelihood of users loading a video only to abandon it partway through.

Think before you film

The research does not argue against videos. It argues for using it more carefully, with a clearer sense of when it earns its place and when something simpler would serve students better. And given the environmental cost sitting behind every videos we produce and host, that question of whether it is really necessary is worth asking every time.

Stop chasing, keep researching: Why continuous contextual learning is the only way to build useful AI features

AI development keeps evolving as do the ways people seek to use AI. Traditional software development runs the risk of trying to perfect AI features people won’t use. Revisiting our previous AI research helped me tease out new opportunity spaces for AI features to help with content design tasks.

Last summer, Mostafa Ebid joined the UX team for a summer internship and built an AI assistant tool which integrated the University’s main AI provider ELM into EdWeb2, our Drupal content management system. The idea for the tool came from hearing University describe the difficulties they experienced when publishing web content. The tool included the capability to write content, design content and proofread, and was designed to work by typing prompts in a chatbot interface, right-aligned to the main part of the editorial interface. When prompted, an orchestration of AI agents were triggered to read textual content and use ELM to make suggestions for improvement based on what the publisher had asked for. Improved text was displayed in the sidebar chatbot interface for the publisher to review and consider using.

Read Mostafa’s blog post about how he developed the tool:

Integrating ELM with EdWeb – Building an AI tool for publishers

Initial tests of the tool with publishers revealed some potential, but identified the need to do further tests, specifically to understand limitations around the user interface display and to learn if the tool was useful to publishers in the context of content they were familiar with (as opposed to generic stock content that was used in the first round of tests).

Read my blog post about the initial tests of the tool:

Initial insights from UX testing our Drupal AI content assistant tool 

Mel Batcharj accessibility tested the tool, focusing on keyboard navigability, and Nick Daniels ran tests with two publishers in October last year. I recently reviewed the findings of this research to consider advancements in Drupal AI in the past 12 months, to assess whether the original premise for the tool was still valid, and to think about residual content design challenges AI could potentially help with.

Advancements in Drupal AI have resulted in improved AI features

The Drupal AI Initiative began in April 2025 with a group of participating organisations making a commitment to collaborate to build the future of AI in Drupal. As a result of the initiative, various workstreams began to shape Drupal’s infrastructure to support AI, to experiment with new innovations and to improve the UX.

Read more about this workstream:

Drupal AI Initiative project page on Drupal.org

A more accessible AI chatbot is now available as a Drupal recipe

Our original AI content assistant tool was built into a panel of the editorial interface as a custom build, which worked well when using a mouse, but which accessibility testing showed was restrictive when navigating using a keyboard. Since the tool was built, however, accelerated development in the wider Drupal AI community prompted the creation of an open-source AI chatbot freely available to apply. Adopting this chatbot was preferable as it avoided the need to maintain custom code and it was possible to use it with a keyboard only.

There’s an active Drupal issue to address the need for the AI chatbot interface to be expandable

Several of the participants who took part in Mostafa’s tests of the tool last year found it awkward to scroll through the AI chatbot output as it was presented in the narrow left-aligned interface. The same problem had been noted in the wider Drupal community, and therefore I raised an issue to have this rectified, which is being worked on as part of the AI Initiative task backlog.

We did more tests on our AI tool – this time using participants’ own content

In the first round of tests, four participants were all presented with the same piece of content (on the topic of safety procedures) and asked to use the AI tool to improve it in specific ways (such as writing it for user needs, making link text better, and so on). This approach turned out to be limited, as since participants were unfamiliar with the content, they were unable to assess whether the outputs of the AI tool were an improvement on the original content or not.

In a subsequent round of tests we therefore adopted a looser approach – asking participants to supply a piece of content they were already working on, and then asking them to use the AI tool to improve it to suit their needs. The results from these tests were more indicative of how useful publishers found the tool to make content improvements.

Results of these tests highlighted how the AI tool needed to change

Since we placed participants in a situation where they were using AI on content they knew well and could critique and appraise authentically, the results of these second-round tests gave clearer indications of what worked with the existing tool, what didn’t and where AI could be best applied to help with content design tasks.

Participants didn’t notice the content options in the AI tool, or the help text

The tool contained three different content options: Design Content, Write Content and Proofread, presented in a dropdown menu in its interface.

Initially, participants didn’t notice these options and used the default Write Content option.  When they later experimented with the Proofread option they found no discernible difference between these options in terms of outputs, which led them to believe that a simpler version with a single conversational interaction option would be preferable.

The tool defaulted to reading the content on the page, and working on this when prompted. Participants were initially unclear that this is how it worked, and they didn’t notice the help text to enable or disable this mechanism presented in the tool interface. Taken together, this feedback suggested that a simpler version of the AI chatbot, such as the one from the Drupal recipe, would be easier for publishers to use.

Close-up screenshot showing detail on the AI assistant tool chatbox

Close-up screenshot showing the content options and the help text on the AI tool

The AI tool had some value as a writing partner to suggest restructures to textual content

Responding to the task to experiment with the AI tool to improve their content, the participants quickly got used to how the tool worked, and recognised its use as a writing partner to prompt about their content and receive suggestions for improvement in a conversational way.

Prompts they used to improve a page of content in the body text field included:

  • ‘Rephrase copy to condense, highlight key messages and make it accessible to pet owners looking to join practice’
  • ‘Proofread copy so that it appeals to pet owners non clinicians’
  • ‘Turn this page into web ready content. It needs to be concise, easy to scan, readable to a wide range of audiences’

These prompts resulted in edited versions of the page content, typically including structural elements like headings, bullet points and calls to action, delivered in the chatbot interface.

Screenshot showing the output from the AI tool before and after a prompt to rephrase the content for a specific audience (before on the left, after on the right)

Side-by-side screenshots showing the prompt entered in AI tool to improve content for an audience (on the left) and after (on the right), with the output response to the prompt.

The tool lacked capacity to tweak or iterate on previous versions of content – which participants wanted

Once they had reviewed the tool’s initial outputs, both participants entered conversational turns with the tool, asking it to perform successive tasks on the content it had previously produced, to edit it further, in line with their specific requirements and rules.

Prompts they used to tweak initial AI outputs included:

  • ‘Remove adjectives and exclamation marks’
  • ‘Remove brackets’
  • ‘Remove any unnecessary words, fix typing errors, suggest improvements for SEO’

With every new prompt in the conversation, the tool produced a fresh output, meaning the publisher was left to review a succession of different versions of the edited content, presented in the chatbot interface, without any indication of what had been changed. Participants found it difficult to review edits, as they would usually do when working on a piece of content, in order to compare the ‘before’ with the ‘after’ – ultimately to assess whether the AI tool outputs were to their satisfaction.

They said they would have liked the tool to have presented the edits in a ‘tracked changes’ format that they were familiar with from word processing programmes.

Screenshots showing outputs of the tool before and after a prompt to iterate on improved content

Side-by-side screenshots showing a prompt entered to improve existing content (on the left) and (on the right) the output from this prompt – showing a new version of the content

Participants didn’t really need the tool in the interface as they tended to edit text content elsewhere

When describing their usual content process, participants said they would usually prepare their content in a word processing programme like Microsoft Word rather than edit directly in the Drupal editorial interface. There were several reasons they chose this method – a key ­­reason being the need to involve others to check (and in some cases, sign off) their content in preparation for the website. They were more familiar with referring others to check their content or proofread it when it was in the Microsoft suite, than when it was in the Drupal editorial interface.

Furthermore, Microsoft Word accommodated the addition of comments and tracked iterative changes to pieces of content which was not possible within the Drupal editorial interface. This content preparation habit suggested that while the AI tool was useful to suggest content edits, this would have been more useful before the content was in the interface, and therefore could be achieved by pasting content to be edited into a browser-based AI tool or app (such as ELM).

Within the interface, the tool only had use as a ‘final check’ mechanism, to catch any typos, errors or style misalignments before the content was ultimately published.

The tool needed to be able handle more than text as pages were typically made of multiple elements

Reviewing the test set-up and comparing it to their usual ways of working with EdWeb2, the participants said the pages they worked on would usually be made up of more than just textual content in the body text field. They would typically work on pages with multi-column layouts, making more extensive use of Drupal paragraphs or including structural elements like accordions, feature boxes and cards. They were interested to know how the tool may make appraisals or suggest improvements for those sorts of pages to help them arrange their content in appropriate ways.

We identified new opportunities for AI content publishing features

Extrapolating on the feedback from the tests, several use cases and scenarios for applying AI to content design tasks emerged, which will help inform our ongoing work to apply AI to make content design tasks easier for publishers.

AI-assisted content structuring

Describing their typical content writing workflow, participants said they found it difficult to move from a text-based editor like Microsoft Word into the Drupal editorial interface where they needed to make use of Drupal Paragraphs as well as page elements like accordions to structure the content. Potential areas for AI development could therefore include:

  • A mechanism to convert textual content into appropriate structural elements
  • A way to make suggestions for accordion labels or structure
  • A feature to ensure uniform creation of cards or feature boxes
  • A means of cross-checking style consistency of pages made of multiple elements or with a specific layout

AI- assisted content design for SEO/GEO/AEO

Having their content picked up by search engines or being machine read was something participants wanted, but they were unsure how to write, tag and structure their content to make this happen effectively. Potential areas for AI development could therefore include:

  • A way to have their content analysed for SEO effectiveness, based on signals like content quality, scannability and key word alignment
  • A mechanism to suggest content changes aligned to specifically defined SEO goals and target user engagement measures

AI assisted content reviews – against specific style conventions and contexts

As well as ensuring their content followed the rules of the University’s Editorial Style Guide, both participants mentioned other conventions that they needed to apply to their content, that existed at a more local website level. For example, one participant’s site had a rule not to use brackets or exclamation marks, or to overuse adjectives, so they would have found it helpful to have a way to cross check content against these rules before publishing.

The Drupal Context Control Center is an emerging feature designed to handle the application of context rules within a site across various scopes and use cases, and Drupal AI Content Review is a related feature, designed to appraise content against given context rules and conventions. Together, these Drupal features may be a good fit to help University web publishers make use of AI to shape their content with the uniformity they require.

Read more about the Drupal Context Control Center and its development in my recent blog post:

Think like a machine: How building a Drupal context-handling feature is providing a new lens of content design and style rules

Read about AI Content Review on Drupal.org

We’re changing the direction of the AI tool based on what we’ve learned

Last summer it seemed certain that an in-interface AI chatbot content assistant helper was what we needed to build. A few improvements to the UI to make it expandable and navigable with a keyboard and it would be ready to go. As it turned out, things had moved on, and these problems were addressed by the wider Drupal community. This meant we could go back to our research findings to re-examine how AI could be best applied to aid content design tasks, and to consider how it could best fit into existing workflows of our publishers to assist them with difficulties they faced. As we continue with internships this summer, we’re excited to re-focus and plan more research to explore some of these emergent opportunity areas. Aligning with in-progress Drupal AI developments, we’re open to learning how we can apply and adopt the work of the Drupal community to our University digital publishing context.

When good product practice tells you to stop: What we learned trying to externalise our Effective Digital Content course

Riding on the success of our internal Effective Digital Content course, we set out to expand by building an external version for the short courses platform, taking a product thinking approach. Three months on, experimenting with a proof-of-concept course has convinced to pause this work – to avoid falling into a build trap.

The success of our relaunched Effective Digital Content (EDC) course, to date completed by more than 400 University staff, prompted ambitions to reposition the course for external audiences by including it in the University’s Short Online Courses platform. With the help of colleagues from the Short Online Courses team and the Learning Technology team, we did some market research, identified target audiences, defined a product vision and goals and began using the Canvas platform to develop a proof-of-concept.

Read more about our plans to reposition EDC as an external course in my blog post from March 2026:

Repositioning Effective Digital Content as a short online course: A product approach

At the end of sprint three (of a total of seven planned sprints) we faced unknowns and unanswered questions preventing us from achieving some of the fundamentals we’d defined in the product vision and goals. Resisting a sunk-cost fallacy-motivated urge to continue building, we made a sensible decision to stop and to pause until we’re in a better-informed place to have the confidence to continue.

In this post, I reflect on the benefits of adopting product thinking, the discomfort with facing difficult questions upfront and the practicalities of learning about audiences for expansion opportunities.

Revisiting the Product Kata helped clarify feelings of uncertainty

Melissa Perri’s book ‘The Build Trap’ contains a helpful framework to guide product development which I have become familiar with through my involvement with Drupal product teams. On the strategic level, this framework helped me set out a vision for an external version of the Effective Digital Content course, establish the problem the course was aiming to solve and its audiences, and work out the learning outcomes associated with each course module.

Entering the execution phase, however, taking each course module at a time and building each one out to meet the needs of the established audiences, things took longer than planned, and felt more difficult than we’d anticipated.

Pausing for reflection, we unpicked where things were going wrong. Going back to our vision, we had wanted to repeat the success of the internal EDC course, giving learners practical experience of writing digital content for their audiences. We had been able to instill this experience in the internal course because we had been able to regularly engage with University web publishers, to fully understand their content design challenges and design practical experiences for them that specifically addressed their friction points. Without such detailed insight into external audiences, we were effectively basing our course design decisions on guesswork and assumptions, which explained why our initial efforts seemed to be missing the mark. The strategic vision was sound, but our capacity to fully explore the problems and optimise associated solutions was limited, therefore execution was flawed.

Adaptation of the Product Kata diagram from Melissa Perri's book 'The Build Trap' showing the stages: Understand the direction, (Company vision and strategic intent), Analyse the current state, (Current state of awareness), Set the next goal, (Product initiative), Choose step of product process (Problem exploration, Solution exploration and Solution optimisation)

Adaptation of the Product Kata from Melissa Perri’s book ‘The Build Trap’ showing the split between strategy creation and deployment and execution

The practical workbook is EDC’s best asset – but is hard to replicate for broader audiences

A lightbulb moment in our reflections came when considering what worked well about our internal EDC course. When staff complete this course, they are required to complete exercises in a workbook which they submit to the UX team for feedback. This tests learners’ ability to do fundamental content design tasks like structure headings, write hyperlinks and turn lengthy text into chunks to be scan-read. Reviewing more than 400 submitted workbooks, we affirmed the importance of this element of the course both for learners and for our team as owners of the course as a product. Through these workbooks,  learners are able to practice content design in the specific context of the University sites they are responsible for, and as the product team, we are able to see impact the course has had in improving content design knowledge.

Replicating the workbook concept for external audiences would require knowing about the contexts and content they were working with, in order to design exercises that required them to test those skills. Without knowledge of our audiences’ circumstances, the best we could do would be to design a generic set of exercises, devoid of the nuances needed to really engage learners in practising content design techniques.

Testing what we had built so far taught us what we didn’t know

In UX and product design there’s a saying: the best time to test is yesterday, failing that, test now. Resisting the urge to keep building EDC modules, we made the decision to take the three modules that had been built and to test them with University publishers. Feedback from the tests was positive, which was nice to hear, but didn’t really help us assess if the modules we had made were a good fit for external audiences. University staff wouldn’t take an external Effective Digital Content course as they had already taken the internal version. To assess if we’d made a good product we needed to know answers to questions such as:

  • What needs should we prioritise for our target audiences?
  • How do our audiences currently meet these needs?
  • Would univerisities with a distributed content model find an external course useful?
  • What would motivate our target audiences to take this type of course?
  • How well do existing courses meet user needs and expectations?
  • Will our proposed course be valuable to target audiences?

In another of Melissa Perri’s books ‘Product Operations: How successful companies build better products at scale’ by Melissa Perri and Denise Tilles, the authors use a case study at a financial services organisation, Fidelity, to show the relationship between user research and the product design lifecycle. Applying this relationship by positioning EDC external as the product, it was clear that our outstanding questions fell into the ‘Core UXR question’ category at the ‘Discover’ ‘Define’ and ‘Design’ stages and therefore needed to be addressed before proceeding to the ‘Develop’ and ‘Deploy’ stages.

Having worked with other teams both within and outside the University, we knew all too well the potential risks and consequences of building without adequate research – and resolved that it was better to stop building to avoid wasting effort.

Successful expansion will rely on targeting audiences and researching their needs more fully

Taking a pause in the build will allow us to take time to assess the gaps in our knowledge of our target audiences, and work out ways of learning what we don’t know. Understanding the nuanced needs of our target audiences will help us familiarise with the market for content design training in the public sector to assess the value of the expansion opportunity. Referring again to ‘Product Operations’, learning the differences between the markets associated with our expansion opportunity (the total addressable market, the serviceable addressable market and the serviceable obtainable market) will help us decide whether the proposition is viable, deliverable and desirable or not.

In the meantime, we’re using what we’ve learned to improve internal EDC and training

As a team, we really value what we learn from time spent in research, and we always endeavour to act on what we have learned and make sure research does not go to waste. In this case, going through the process of reworking three course modules to aim them at external audiences has pinpointed ways to improve the internal version of our course – in particular to make the introductory module clearer and more impactful and to be clearer on some accessibility concepts. Submitted workbooks and feedback from our regular Content Improvement Clubs also provide a constant source of learning, to identify areas publishers still struggle with that we can continue to address with subsequent tweaked versions of EDC and topics for additional publisher training sessions.

How the Effective Digital Content course led to a collaborative project to explore ways to improve the School of Informatics course materials site

In January 2026, Alex Burford, a learning technologist from the School of Informatics contacted the UX Service following completion of the new Effective Digital Content online course. Alex had really enjoyed the course and was keen to explore ways to implement the content design best practice principles it teaches within Open Course – the platform that the School of Informatics use to share their course materials.

Informatics Open Course Materials

Screenshot of the homepage of the School of Informatics Open Course materials page. There is a title 'Welcome to the Informatics Open Course Materials' heading, followed by some paragraph text and part of a table listing out the courses available to access.

Screenshot of the homepage of the School of Informatics Open Course Materials platform.

Identifying key areas for improvement

Alex had received positive feedback from students who were able to successfully locate their course materials. However, from our discussions and a review of the content on the Open Course platform, (which relies heavily on tables), we both felt there was scope to review the current layout and explore potential alternatives.

Within the tables, we also identified that a more consistent and informative approach to formatting links as well as creating more effective headings could improve the user experience.

In addition, Alex was keen to look at the best ways to support Open Course publishers with content design and embed ways to make it easier for them to prepare effective content. To this end, we talked through potential guidance and training that could be developed with a specific focus around Open Course content.

We spoke with Open Course publishers to gain their perspectives

Before progressing any further with our ideas, we were keen to establish a baseline understanding of how Open Course is currently used by those who publish content on it. This would help us to learn about their processes and approaches, identify pain points as well as areas that were working well.

Alex reached out to Informatics colleagues to see if they would be happy to speak to us about their experiences of using Open Course. In late January / early February 2026 we carried out a series of semi-structured interviews.

Format and aims of the interviews

The UX Service facilitated the interviews via Teams, observed by Alex, to learn about the experiences of Informatics colleagues who published content on Open Course. Informatics colleagues shared their screen and talked through how they achieve content publishing and formatting tasks in Open Course. This enabled us to see first-hand how they went about these tasks and hear their perspectives and feedback.

The aim was to confirm areas that were working well and also the main pain points. We were keen to learn the reasons why tables were used, what the experience was like for those editing content within them and understand more about their approach to writing headings and links.

We gained a lot of useful information from the interviews both from a content perspective, but also in relation to more technical aspects relating to the platform itself which Alex could feedback to Graham Dutton, the developer working on Open Course.

This information then provided the basis on which to form an action plan around how to make improvements and decide on the best way to provide support and/or guidance.

Tables: were they necessary, if so could they be improved?

Following our discussions with Alex and the semi-structured interview insights, we felt we had a good understanding of how tables were used and the pain points relating to editing content within them.

We first wanted to explore whether tables were needed, or if there were alternative layout options to avoid the need for tables completely. If they were needed, we could then look at how they could be improved?

Prototypes were created by the UX Service

The UX Service mocked up some prototype tables, initially in Word before also creating them in an Open Course playground site we had access to. We used the following ‘Schedule and materials’ page, where almost all information was displayed in a table.

INF2-SEPP: Schedule and Materials | Open Course Materials

Prototype one

Our first prototype involved removing tables completely and using headings and paragraph text to display the same information. What we soon realised was that this made the information hard to scan and increased the length of the page. We felt from trying this approach, there were definite benefits to having a structured column layout to help communicate key dates and materials for course participants efficiently.

A screenshot of a word document prototype taking information from the Open Courses platform and displaying it using headings and paragraph text rather than tables. The screenshot includes the page title and a short summary paragraph. Followed by the Week 1 heading which tells you the week, dates and name of the topic. Following this there are section headings for lectures, tutorials, drop-in labs and milestones, with necessary details under each as paragraph text.

Screenshot of a word document prototype displaying information from Open Course using headings and paragraph text rather than tables.

Prototype two

Our second prototype then explored how we could improve the current table layout and make it as accessible as possible. The existing page had a continuous table with six columns. All headings and links were included in the table itself which made it quite hard to navigate. Also, the number of columns meant that text was wrapping onto two lines quite often, which again impacted on readability.

We explored presenting the same information in a two-column layout, using rows with clear left-hand headings to signpost key information. We also split the continuous table into multiple tables, one for each week of the term. Column and row headers were also applied within the table editor tools. These changes we felt improved the overall look and feel of the page but also made it easier to navigate to the information you needed.

A screenshot of a word document prototype taking information from the Open Course platform and displaying it in an alternative table format. The screenshot includes the page title, a short summary paragraph. Followed by the heading 'Week 1' and then a table with two columns and six rows, one column has various headings such as 'Dates', 'Themes', 'Lectures', with the corresponding information displayed in the column next to it.

Screenshot of a word document prototype displaying information from Open Course in an alternative table format.

Stress-testing prototype two

After sharing both options with Alex, we decided to proceed with prototype two and carried out some further testing to make sure it was fit for purpose.

JAWS – Screen reader testing

Using the prototype in the our Open Course playground site, we used JAWS to test out how the content on the page would be navigated and interpreted by a screen reader.  Whilst this approach may not fully capture how all users of assistive technology navigate, interact and experience the page, it did provide useful insights around how tables in Open Course interact with screen readers. Overall, the new table layout responded well.

We found that JAWS:

  • read out the number of columns and rows in a table as expected.
  • navigated the table content in a logical order. It interpreted tabled cells moving from left to right, first reading the column headers (‘dates’, ‘themes’) before information and links within the cells.

JAWS didn’t pick up on blank cells within the table. Instead, it moved to the next table cell which had content within it. Therefore, we recommended that if this table layout was adopted that there should be no blank cells to aid clarity. Where there was no information to put within certain cells, text such as ‘no milestone’ or ‘no tutorial’ should be used.

We are also aware that screen reader users can and may wish to jump from table to table on a page. When navigating in this way, JAWS moved table content to table content without reading out the assigned ‘Week X’ heading we had inserted. Therefore, when you have multiple tables on a page, it’s hard for the user to tell which table they are on. A potential solution to aid understanding could be to incorporate the week number within the table itself, rather than using headings.

Magnification and responsiveness on mobile

When tested, the page (including tables) responded well to 400% magnification, with no loss of information. This check was carried out to ensure that the content reflows correctly for users who routinely view web pages using zoom functionality. We wanted to ensure that information did not move off screen or overlap when zoom was used.

The tables also responded well in mobile view, including when the screen was rotated. This is important to test given the increased usage of mobile devices to access digital content, to ensure all users were able to easily read / navigate the information on the page.

UX Service recommendations

Recommendation to use prototype two table layout going forward

Following our prototyping and stress-testing, we reported back to Alex that prototype two was the option we would recommend.

Recommendation to consider best formats for guidance on headings and links

From reviewing the pages and considering the insights gained from the interviews, we also felt that it would be beneficial to create guidance around:

  • writing effective headings and using correct heading levels in tables
  • how to use links and write effective link text.

Having this guidance close to hand, ideally accessible directly from the edit screen of the Open Course platform would be preferable to enable ease of access for users.

Reflections and next steps

As a team, we really enjoyed working with Alex on the project. It provided us with the opportunity to apply the content design best practice principles that we teach specifically to learning and teaching content. It was interesting to explore and test out options for using tables in a different context, outwith the central EdWeb 2 Content Management System, for student-facing content.

We are glad that the Effective Digital Content online course provided the impetus for this work. We hope that the course continues to provide the starting point for interesting conversations about how digital content can be improved as it reaches wider audiences around the University.

We look forward to continuing to work with Alex in relation to the guidance and potential implementation of the new table layout before the next semester or at an appropriate time in the future.

Think like a machine: How building a Drupal context-handling feature is providing a new lens on content design and style rules

AI tools to support content tasks are becoming more and more widespread. As part of my contributions to open-source Drupal I’ve been researching how to prepare and package content design and style rules that these tools can use effectively.

Using AI to help with content design tasks (and indeed, any other type of tasks) is now standard practice for many. As anyone who has experimented with it knows, the value the AI can provide is largely dependent on the prompt it receives, and the contextual data and information it is given to work with.

The way you provide your chosen AI with context affects the results you get

I have encountered two main methods for providing AI with context data. The first is the ‘prompt and pray’ approach, where you start with vague instructions and then engage in back-and-forth to drip-feed the AI the necessary information in the conversational turns. The second is the ‘context engineering’ approach, where you front-load information to proactive guide the AI, for example, defining its role, setting explicit goals, providing background and adding contextual constraints and rules.

Context engineering is more proactive than prompt and pray

Prompt and pray is better suited to a standard LLM interface or chat assistant (like ELM or Claude), and is the preferred choice when you’re looking to brainstorm and experiment, and are happy to take variable outputs. Context engineering relies on a more structured interface, designed to handle workflows (like Claude Skills) and is therefore the best choice if you’re looking for consistent, predictable outputs within set parameters, but it requires you to prepare your context data upfront, and for your chosen AI mechanism to have a way of handling it.

The Drupal Context Control Center is a management system for context data

Context data is a term given to the structured, site-specific knowledge – such as brand guidelines, regulatory requirements, editorial rules, and terminology – that tells an AI how you want it to work, to ensure its outputs reflect your actual preferences and standards rather than generic defaults.

In January 2025, our team was fortunate to spend a day of Drupal AI experimentation with specialist company Freely Give, in which we applied AI solutions to some of the web content management and design problems faced by the University. One of our experiments involved prototyping a basic style-guide checker into EdWeb2.

Read about this early experiment in my blog post:

An automated Editorial Style Guide? Experimenting with Drupal AI Automators

Looking back, this experiment was a very basic form of context handling, and since then, following the launch of the Drupal AI Initiative in June 2025, there have been rapid developments in thinking about AI and context. The Drupal Context Control Center (CCC) was initiated specifically for the purpose of enabling AI to handle context data and I am proud to have worked on its design, collaborating with Kristen Pol, Aidan Foster and others from the AI Initiative.

Read about the Context Control Centre on its project page on Drupal.org

To design the CCC architecture we brainstormed context application scenarios

Starting with a blank page, we started trying to come up with the main building blocks that the CC needed to have, but we found it very difficult. Drupal is phenomenally flexible which meant every function we wanted could be achieved in various ways. We quickly moved from thinking in the abstract to considering real-life scenarios when people would want to use AI to make use of context data to control what happened with their content which helped us tease out the tasks we wanted the CCC to support.

An example scenario was as follows:

Content creator wants to refresh a prospective student recruitment campaign. They ask the AI, integrated in the content management system, to help. Acting on the instructions, the AI prepares a sample campaign page which includes a slogan that doesn’t match the voice and tone and contains images from winter when the campaign is for summer. The content creator rectifies this by providing the voice and tone guidelines and updated imagery. The AI tries again, it’s better but this time there’s a word spelled differently from the style guide. The creator uploads the style guide rules. The page looks good, but the content creator wants another version to compare. The AI comes up with a version 2. The content creator wants to track interactions with the first version and swap to the second based on performance. To do this they connect their Google Analytics Acquisition reports and give the AI instructions of the thresholds to look for, to initiate the change.

We defined building blocks of the CCC and the relationships between them

Considering what would be needed to support the scenarios we came up with, we identified that the CCC needed to house several types of data and we thought about the dependencies and interactions the CCC would need to support.

1.     Data items that set out ‘the What’

First and foremost, the CCC needed to include the context items (rules and guidelines) such as:

  • Brand guidelines
  • Voice and tone rules
  • SEO keywords
  • Campaign or project briefs
  • Style guide
  • Image bank
  • Pattern library

In addition, it needed to contain subcontext items (more refined versions or child/subsets of the context items) such as:

  • New product brand guidelines
  • Seasonal campaign copy
  • Seasonal image set
  • Landing page pattern set

2.     Data that set out ‘the When and How’

As well as the various sorts of rules and guidelines, the CCC also needed to include details of the circumstances and situations in which to apply them. Collectively, these were called the context scopes and they mapped out boundaries and constraints such as:

  • Global (site-wide)
  • Site sections
  • Content types
  • Use cases (for example – writing teaser copy, working with images)

3.     Mechanisms that control how agents select context

Connecting the ‘what’ with the ‘when and how’ relied on AI agents identifying context scopes and selecting appropriate context items to apply. For this to occur successfully, the CCC needed to include some means of facilitating as well as guardrails to steer suitable choices. These included:

  • Context limits (limiters on numbers of context items and tokens)
  • Scope subscriptions (agent configurations to define opt-in or opt-out)
  • Target entities (specified content entities with context built-in)

We tried out terms in practice before deciding on labels

Labelling parts of interfaces is always tricky, particularly within Drupal where certain words have legacy meanings and, owing to its flexibility the functionality associated with terms may change over time. It was especially important to make careful label choices for the different parts of the CCC architecture, given its anticipated universality.

To inform my work on the CCC I decided to re-read ‘Design by Definition’ by Elizabeth McGuane, which book sets out and appraises a range of approaches for deciding upon terminology, labels and names for objects.

When choosing a name to fit a system, the author recommends appraising the options against three criteria:

  • Novelty – how standard or unique the name needs to be
  • Flexibility/mutability – how well the name works in different contexts
  • Memorability – how easily the name is recalled

Applying this idiom helped us decide upon several key CCC terms, including:

Context item: any piece of information fed into an AI powered mechanism (typically an agent) to help the AI’s working memory to produce responses and outputs that are accurate and tailored to the users’ requests.

Context source: origin of any piece of information fed into an AI powered mechanism (typically an agent) to help the AI’s working memory to produce responses and outputs that are accurate and tailored to the users’ requests.

A screenshot showing an interface from the beta release of the Drupal Context Control Center, showing the use cases and the context items (with inter-relationships). Arrows point out these different areas.

A screenshot showing an interface from the beta release of the Drupal Context Control Center showing the main building blocks

For the CCC to work as intended, inputs need to be machine-readable

With key parts of the CCC defined and the architecture mapped out, we started to consider how well the CCC could handle variations of context data loaded into it. This required us to think more broadly about how AI makes sense of information.

In the excellent book ‘Machine customers: The evolution has begun’ by Katya Forbes, the author outlines many use cases where people enlist AI agents to take care of tasks for them and diagnoses the underlying factors necessary for the AI to satisfactorily work on behalf of the people concerned.

In one such use case she breaks down the stages required for a human to take a purchasing decision compared to a machine.

For a human to decide on a purchase, they go through the following stages:

  1. Awareness – ‘I have a problem’
  2. Interest – ‘This might solve it’
  3. Consideration – ‘Let me evaluate my options cognitively and emotionally’
  4. Intent – ‘I’m leaning toward this choice’
  5. Purchase – ‘This feels right’

For a machine to decide, the stages are different:

  1. Query initialisation – parameters received ready to begin search
  2. Discovery – options identified that meet basic criteria
  3. Evaluation – comparing options against weighted parameters
  4. Verification – validating performance claims and reliability
  5. Selection – optimal choice identified based on data

From this comparison use case, it is clear that designing contextual data such as guidance documents to be used by AI requires a different approach to designing content to be read by humans. In particular, writing guidance information that relies on sentiment and/or subjective interpretation will be lost on AI and therefore not applied in the ways intended.

In its current form, our Editorial Style Guide is only partially AI-ready

Last year, in the early days with ELM, members of the UX Service tested ELM’s ability to apply the rules of the style guide to check content. They found it had limited success and found that in some cases, ELM applied its own rules to checking the content which couldn’t be traced to the style guide. Reviewing this experiment in light of the machine customer journey, it becomes clear that for an AI like ELM to faithfully and consistently apply style guide rules, the rules need to be written in a way that demands minimal interpretation, in other words, a deterministic way.

Read John Wilson’s blog about experimenting with ELM and the style guide:

Testing ELM’s ability to return useful results with prompts about the Editorial Style Guide

Adopting a ‘machine-first’ lens to the Editorial Style Guide to analyse content in selected sections, it is possible to pull out some of the deterministic (automatable) rules to compare with non-deterministic (requiring judgement) ones.

Analysis revealed deterministic and non-deterministic rules in the Editorial Style Guide

I completed a quick review of three sections of the style guide to assess how AI-ready they were.

  1. In the Headings and page titles section

Deterministic rules:

  • Do not skip heading levels
  • Do not use H5 or H6 headings (maximum four levels)

Non-deterministic rules:

  • Use word and language your users will be looking for and familiar with
  • Write descriptive headings – avoid vague words
  1. In the Acronyms and abbreviations section

Deterministic rules:

  • Do not put full stops or spaces between letters of an acronym
  • Do not abbreviate Professor to Prof
  • Do not use eg, ie or etc

Non-deterministic rules:

  • Spell out acronyms multiple times on long pages or pages with accordions
  • Well-known acronyms don’t need spelling out
  • Use abbreviations only when better known than the full version, or when space is limited
  1. In the Links section

Deterministic rules:

  • Do not use a URL as link text
  • Do not use ‘click here’, ‘more information’, ‘learn more’ as link text
  • Put links on a new line, not inline in a sentence

Non-deterministic rules:

  • Add details about what a link will do (open in a new tab, require a University login) where relevant
  • Avoid duplicating links where you can
  • Reserve button styling for the most important links

Going through this short exercise with our style guide established that before we can make valid use of AI-powered features like the CCC, there is preparatory groundwork required to some of our content design guidance and rules more AI-readable, and potentially to create a new machine-readable version of the Editorial Style Guide.

I’m looking forward to investigating the potential for the CCC at the University

My work with others in the Drupal community on the CCC has led to a successful release of a beta version, with a release candidate coming soon, planned alongside the regular Drupal AI module release cycle.

Building on our early AI experiments with the style guide, I am keen to lead the UX team to explore the CCC’s potential to handle and apply editorial rules within a Drupal-powered CMS like EdWeb2. Our previous research with web publishers tells us that applying the style guide consistently can be genuinely difficult, and as AI-powered content design tools like the CCC continue to emerge, there is real opportunity to put this to work on the challenges our publishers face.

Read about our previous research with publishers about using the Editorial Style Guide

An analysis of responses to our Editorial Style Guide survey by Hannah Watson

Usability testing the Editorial Style Guide site by me in my previous Content Designer role

Full automation of good content design is unlikely and undesirable but it may be possible to offload some of the more straightforward rules to an AI mechanism like the CCC, freeing publishers to focus their attention on the trickier aspects of content preparation. Taking a machine-assisted view will also provide the chance to revisit existing non-deterministic rules to assess whether some could be made more granular, broken into sub-contexts or context scopes for more precise application.

In line with our broader aim of improving the tools available to content publishers, I am excited to see what the CCC can offer for real-world content design challenges at the University.

 

❌
❌