❌

Reading view

There are new articles available, click to refresh the page.

Just do it for me? Balancing human and AI input in content improvement

Working with student interns and contributing to the Drupal community, I’ve experimented with AI in various  tools designed to support content improvement. I reflect on what I’ve learned about peoples’ preferences for engaging with AI tools and consider wider design challenges for web content to support co-browsing.

Writing good quality, user-centred content has always been a challenge, and accordingly, a suite of tools evolved to address this; Hemingway, Grammarly and many more. With their ability to learn patterns and predict word sequences, the advent of LLMs provided a new direction for content improvement. Where previous tools had used fixed rules, algorithms and formulas to flag content problems and score sentences, LLMs changed the playing field as they were able to be prompted to rewrite and create text and to adapt words and sentences not only in line with fixed rules, but also aligned to different interpretations of style.

The UX Service’s content design course, Effective Digital Content, and our ongoing Content Improvement Club meet-ups are both designed to help staff publishing digital content for the University. It’s been encouraging to see a positive uptake for both training formats, however, we realise we can do more to help staff publish the best quality content they can, and to this end, for the past few years it’s been a privilege to support a series of interns in the UX Service to look at how we can innovate and apply AI to the persistent challenge of writing good quality, user-centred content.

Thinking beyond – embracing the spirit (or grasping the nettle, depending on how you look at it) of AI innovation, it’s sensible to note that, more broadly, the browsing landscape of the web is increasingly shifting to include AI agents as well as humans, giving cause to consider what using AI to design content to be consumed by AI might look like. That’s the subject of another blog post (or several).

For now, as I begin another phase of work with a new student in the role of AI and UX Innovation intern (Viktor Sharha), it felt timely to pull together what I’ve learned so far.

Designing, creating and editing content happens in various ways

When designing any user-centred assistive features, a good place to start is the current workflow or user journey people adopt to achieve the task in question. In other words, to look at the stages content editors currently go through to make sure their content is as good as it possibly can be.

One of my earliest research tasks when I joined the University was to usability test the Editorial Style Guide, to find out how publishers used it to keep their content consistent in EdWeb. This work gave me insight into the type of workflows publishers followed to get compliant content published. Some publishers worked in Microsoft Word, using the in-built spelling checker for typos, and cross-checking against a printed-out copy of the style guide for University-specific errors before pasting content into the EdWeb body text. Others went straight to EdWeb, checking their content line-by-line with a PDF version of the guide. Others relied on colleagues giving their work the once-over before pressing publish. Some used Grammarly and Hemingway, some didn’t.

Overall, there wasn’t a dominant content-publishing workflow, with unified friction points waiting to be alleviated through a standard user-centred design process. Instead, there were varied approaches, habits and ways of working which people had adapted over time to suit their specific circumstances and to address the areas they found difficult as individuals. Tools were used at different stages of the process, and to a greater or lesser extent depending on personal preferences.

Therefore, designing content improvement tools isn’t one-size-fits-all

It follows that, since people follow different pathways to create, design and edit content, tools to assist this process need to be developed in different ways to suit varied preferences. Since 2025, interns in the UX team have prototyped our own different content improvement tools to test with publishers, and we’ve also tested open-source tools that exist already, developed within the Drupal community. From our testing and research, we found pros, cons and nuances associated with different tools. The summary below pulls together detail of the tools and how they compared.

Tool 1: Drupal Editoria11y – no AI, flags errors in the editing interface

Editoria11y is a Drupal module built by John Jameson from Princeton University that automatically checks content inside the editorial interface. It was originally designed to catch accessibility issues, but it can be configured to check content against other rules too, such as institutional style guide conventions.

What it does and how it works: Editoria11y checks content as authors type, and also on previews and published pages. It includes an indicator bar to show how many issues are on the page and has inline markers to show where each issue is, as well as a dashboard summarising site-wide errors. When an issue is opened, a pop-up modal explains the problem and suggests a fix.

What worked well: In tests, publishers were able to make sense of the inline markers and understand how these related to the indicator bar for the whole page. They liked the tracked change visualisation in the pop-up modals, and were able to work through successive modals, accepting or ignoring the suggested changes.

What needed improvement:  Some publishers found the number of markers overwhelming, and since the pop-ups covered the text, it was difficult to judge each recommendation in context. In the pop-ups, some of the rule descriptions used formal grammatical terms (such as ‘ordinal suffixes’) which were unfamiliar to some publishers meaning they were confused about the problem to be fixed. Since rules were applied mechanically, there were instances where errors were missed, or where awkward fixes were suggested (such as swapping ‘1st floor’ for ‘1 floor’ due to straight removal of the ordinal suffix) – these degraded publishers’ trust in the tool.

Read more about what we found testing Drupal Editoria11y

Embedding the rules: What we learned UX testing a style guide helper tool built with Drupal Editori11y

Tool 2: ELM-powered content design chatbot – with conversational advice

This tool was developed by student Mostafa Ebid during a 2025 summer internship. It connected ELM, the University’s AI platform, to EdWeb2, so publishers could get help with their content by conversing with the University’s suite of LLMs without leaving the Drupal editing interface.

What it does and how it works: This tool sits in a chat sidebar next to the main Drupal editorial interface and offers three modes: Write Content, Design Content and Proofread. In response to a user prompt to improve content, it activates an orchestration of agents (each with specific tasks such as checking the style guide, structuring pages, profiling audiences) which read the page content and act on it accordingly, publishing the revised content in the chatbot window.

What worked well: In tests, publishers understood how the tool could be used as a writing partner, and were impressed with the way it restructured content with headings, bullet points and calls to action and hyperlinks.

What needed improvement: When faced with the empty chatbot interface, some publishers weren’t sure where to start, and they didn’t notice the three modes to work on content. Some were also unsure of the sources the chatbot was using for its revised content and skeptical that the revised content contained hallucinogenic content. They would have appreciated tracked changes to be able to pinpoint what had changed and found the long thin sidebar chat interface awkward to review the updated text. The chatbot was also limited to reviewing body text content, whereas publishers needed it to review content in other page components used in Drupal.

Read more about what we found developing and testing the ELM-powered content design chatbot:

Initial insights from UX testing our Drupal AI content assistant tool

Tool 3: QuillMark v1 and v2 – some AI, clusters errors with correction suggestions

QuillMark was an AI-enhanced tool which worked in Drupal to check content against the University’s Editorial Style Guide. Shlok Gupta built and tested two iterations of Quillmark during his 2026 summer placement as AI and UX Innovation Intern.

What it does and how it works: QuillMark sits in a side panel of the Drupal editorial interface and scans content on the page in two stages. The first stage uses code-based pattern matching (without AI) to catch straightforward errors. The second stage brings in AI to find deeper issues (such as passive instead of active voice) and to suggest fixes. Findings from both stages are displayed in the side panel – each finding describes the problem, links to the relevant rule in the style guide and offers an ‘Apply fix’ button (which can be reverted). Findings are grouped by category, and this panel also includes an indicator of the environmental impact and cost of the scan.

What worked well: In tests, publishers understood that the tool was there to help them correct content on the page, they liked the environmental and cost indicator, the suggested fixes and the links to the style guide reassured them that the tool was applying University-specific rules.

What needed improvement: Publishers reported feeling overwhelmed by the number of findings on a single page and they wanted a way to deal with related findings and fixes in bulk, to avoid going through them one-by-one. Related to this, they wanted

Read more about what we found developing and testing the QuillMark tool:

QuillMark, iterated. How user research helped us evolve an AI-enhanced Drupal style guide assistant

Tool 4: Drupal AI Content Review – full AI, with a plan provided to improve content

As part of the Drupal AI Initiative, I’ve contributed to shaping the self-explanatory Drupal AI Content Review module. This module plugs into a Drupal site, references content rules and good practice from context items in Drupal Context Control Center (within the same site) and uses the configured AI provider to review site content

What it does and how it works: The feature reads the page content and checks it against criteria set by the site owner, such as tone and voice guidelines, inclusive language rules and target readability levels. These rules are stored in the Drupal Context Control Center, along with examples of good and bad content. The review gives the page a score for each criterion and recommends fixes to improve it. Accepted recommendations become an improvement plan, which is applied in the editorial interface. Re-reviewing the page shows the updated scores, with the option to keep making improvements. Over time, the rules can be refined by adding more examples of good and bad content to the Context Control Center.

Read more about Drupal AI Content Review on Drupal.org

Read more about the Drupal Context Control Center on Drupal.org

View a demonstration of AI Content Review in a mocked-up university website as part of the Drupal AI demo on the Drupal Forge platform

What worked well:  In tests of this module within the AI demo, people liked seeing the red, amber and green indicators against page content for an easy visual highlighting areas to be worked on, and they liked being able to select a checklist of recommended improvements to apply to improve their content.

What needs improvement: The screens explaining the recommended improvements were difficult to scan read, and it was also difficult to compare the original text and the text with the recommended improvements side-by-side (despite colour shading to denote which was which). The stage where the recommendations were turned into an improvement plan felt like an unnecessary extra step, it would have been better to have implemented the recommendations directly. Also, it was presumed that implementing the plan would have corrected all errors resulting in a score of 100%, without the need for further work and re-review. Finally, it wasn’t immediately clear how to use the Drupal Context Control Center and the AI Content Review together.

When designing AI-enhanced tools, consider three AI design maxims

Taking on board variations in blending human effort and automated operations, there are three approaches to designing AI features and AI-assisted workflows worth considering. These are defined below, with ideas for how they may apply to content improvement.

1.      Human-in-the-loop

In this set-up, humans typically share the task-load with AI – actively working alongside an automated operator to supervise or influence actions, and to approve AI decisions before they happen (ensuring AI outputs are not released without the green-light from a person).

Human-in-the-loop tends to be adopted when:

  • there is a need for human judgement to get something correct
  • the consequences of an error could be high-profile or damaging
  • trust in an automated tool is still being built, and the boundaries of its capabilities are still being realised
  • the goal is for the system to facilitate shared learning between the person and the automation

For content improvement, human-in-the-loop is appropriate in these scenarios:

  • ensuring correctness of important factual data – for example, prices, dates, contact details, terms and conditions
  • checking the tone of sentences in line with inclusive language guidance
  • verifying accuracy of high-stakes published content such as legal information, policy wording or written instructional content
  • rewriting complex topics in plainer language
  • writing meta descriptions or page summaries to be accessed by search mechanisms
  • translating content into other languages

2.      Human-on-the-loop

These systems involve the AI or automator acting on its own, with the human reviewing and spot-checking at intervals, monitoring output quality over time, and intervening to correct failures periodically, where required.

Human-on-the-loop tends to be adopted when:

  • there is a high volume of low-risk tasks to be completed, which would be tedious for humans to do themselves
  • consequences of errors are minimal
  • there’s a need for tasks to be completed quickly, and including human-input at each task would be too time-consuming
  • consistency is required across tasks, and there’s a risk of human error introducing variability

For content improvement, human-on-the-loop is appropriate in these scenarios:

  • fixing straightforward style rule violations such as correcting spelling errors
  • categorising and tagging text content (for example, to facilitate its discovery in search)
  • scanning content for readability and accessibility, to flag or report areas to be addressed

3.      Human-out-of-the-loop

In this arrangement, the human sets up the AI to act autonomously, and leaves it to proceed, with no routine review.

Human-out-of-the-loop tends to be adopted when:

  • there are very low-risk, behind-the-scenes tasks to be completed where errors have little or no consequence and can be fully reversed

For content improvement, human-out-of-the-loop is appropriate in these scenarios:

  • autocorrecting typing errors
  • detecting content duplication
  • auto-optimising media content to defined parameters

Choice, pacing and transparency are constant design requirements

Testing four different tools has showed that options always need to be available, whatever balance of human and AI the tool includes. People approach AI differently based on preferences and past experiences. Some love nothing more than a good grappling of a sticky sentence, some just want an agent to do the work for them, and many sit somewhere in between, happy to have some AI help as long as they have the final say.

Alongside humans in, on and out of the loop, it’s important to remember the fourth option: human only, where people can choose to do their content unaided, using the methods they relied on before AI came along. AI shouldn’t be forced on anyone, so all tools need to come with a ‘no thanks to AI’ option.

AI is a fast-evolving and non-deterministic technology, so at any point in time it may work differently to the way it did before.  Responding to challenge from a design perspective, it’s important to factor in pacing. People need to be taken through how a tool works, and the options available. They need time and space to experiment and try it out, so they can learn for themselves how it can work for them. Pacing not only matters in the way tools are introduced, but also in the way their outputs are presented, to avoid the overwhelm that comes when AI does the heavy lifting all at once, and so people know what changed and how, and they have a way to accept, decline or edit what it produced.

If people don’t trust an AI tool, they won’t use it. Earning trust requires transparency, explainability, relatability and reliability – so AI tools need to be upfront about what they can and can’t do, with clear, intuitive interfaces and labels. Reliability will always be hard to promise given AI is probabilistic, so instead of accuracy guarantees which may fall short, a better approach is to design for experimentation, so people can find their feet and make their own judgements.

Designing for humans and AI together – metadata, markdown and MCPs

Looking at the broader digital landscape, the challenge widens from designing tools that help humans make content to designing websites with content ready for both humans and agents to use.

In co-browsing, people are increasingly arriving at a website with an agent alongside them, handing off some tasks while keeping control of others and wanting to see what their agent is doing. That means firstly deciding which tasks belong to humans and which one to AI, and secondly designing for each. Humans need intuitive interfaces. Agents need data they can read and act on without ambiguity:

  • metadata, such as schema markup that tells an agent what a page is about, who it’s for and how it connects to other content
  • markdown versions of pages and llms.txt files, which give agents clean, structured text without layout and navigation getting in the way
  • MCPs (Model Context Protocol servers), which let sites expose what they can do as well as what they say, from simple actions declared directly in HTML to more complex ones written in JavaScript – in preparation for use by the emerging WebMCP browser standard.

Labels matter for humans and agents alike, because words are becoming a currency that machines read as well as people. Alignment on terminology facilitates the creation of a single source of truth, to prevent the human experience from drifting from the agentic experience. On another level, humans need a way to consistently communicate with AI – to give AI the rules and instruct it and to train it with new insight. ‘Rosetta stone’ mechanisms like the Drupal Context Control Center will play an important part in designing for human and AI interaction and collaboration and I am excited to keep working and researching in this area to address these evolving design challenges.

 

Show me your sources! What students make of content provenance data, AI signifiers and sparkles

The line between human-made and AI-generated content is blurring fast. I added a disclosure indicator to an AI-powered search tool and tested it with students, which taught me about perceptions of AI-influenced content and what signals authenticity. I reflected on the need for a shared language around provenance and disclosure.

Estimations of the proportion of AI-generated content on the web vary and are becoming progressively difficult to measure. New legislation advocates for disclosure of AI use, but the nuances of communicating how AI has been used within a feature, and detailing the finer points of the provenance of AI-generated content both remain a challenge.

Earlier in 2026, I took part in a study to design a C2PA provenance indicator into a prototype AI searchbot. I revisited my designs to test them with students, to assess if this authenticity data made sense and to understand its impact on students’ potential trust of the AI searchbot. In this blog post, I describe what I tested and what I learned, and make broader considerations of designing with provenance data, the impact of AI legislation and what makes people trust AI features.

People’s trust in technology depends on explicit and implicit factors

The ISO 9241-210 standard for UX defines user experiences as ‘a person’s perceptions and responses that result from the use of anticipated use of a product, system or service’, which sums up the influence trust can have on the way a person experiences a technology.

I was interested to explore more about the relationships between human trust and AI, so I read a selection of papers on the topic and extrapolated four factors that commonly affect how people trust AI technologies:

  1. Transparency (surface and presumed credibility)
  2. Reliability (earned and reputed credibility, how consistently it works and how dependable it is)
  3. Explainability (how clear the purpose of the AI is and how clear it is how it works)
  4. Relatability (how well the AI supports the person’s interests, how easily they can use it and how it resonates with them)

When it comes to designing AI features people trust, and communicating how AI has been used within those features, it’s important to try and convey each of these factors in ways that make sense to the user in context. In other words, striking a delicate balance of providing sufficient trust indicators to allow people to make reasoned judgements, without going overboard and running the risk of confusion or skepticism.

The EU AI Act has set out obligations to disclose AI usage

From an ethical perspective, advising people that they’re using an AI feature has always been the right thing to do, in order to foster a sense of user trust. Legislation published in August 2026 made the requirement to disclose AI usage more explicit and provided more detail. Article 50 of the EU AI Act details Transparency Obligations for Providers and Deployers of Certain AI systems and includes recitals which cover areas including the following:

  • Designing AI systems to ensure users know they are talking to AI
  • Marking content from AI systems (text, images, videos, audio) so it is detectable as such
  • Informing people of AI systems that include emotion recognition or biometric categorisation
  • Disclosing deepfakes (audio/video/image content that has been manipulated as well as public-interest text, unless human-reviewed with editorial insight) as AI-made
  • Clarity, timing and accessibility of AI disclosures – to be provided at the first interaction or exposure and to be clear and accessible.

With this legislation now in place, it’s anticipated that a growing number of ways to disclose AI usage will emerge, making AI disclosure as mainstream as AI features.

I adopted the C2PA digital content provenance standard into an AI searchbot prototype

When it comes to icons that disclose AI usage, sparkles, neurons, and robots are all widely used, but they only signify that AI is present or available, they don’t provide any detail of how it has been used or how it has shaped content. C2PA is an established provenance standard which has been designed to convey progressive levels of content provenance data. I incorporated C2PA to disclose how AI was used to shape synthetic chatbot responses in a search feature and wrote about my process in an earlier blog post:

How can people trust AI-generated content? Designing provenance data into our prototype AI searchbot

I concluded this design work with a resolve to test my designs with users to understand the following:

  • if they cared about seeing provenance data about the AI searchbot response
  • if they grasped the concept of provenance data
  • if they understood what the C2PA Content Credentials icon signified
  • if the names for the different categories of data made sense to them

Drawing on my recent reading into trust factors, and taking into account the new EU AI disclosure guidance I also wanted to use my designs to understand more about how users make sense of AI features, and what affects their trust of them in context.

Testing revealed students’ responses to progressive levels of AI disclosure in the searchbot

In a simple test, I asked the student participants to imagine they were in a scenario where they had used the AI searchbot to find information from a website. I then showed them mock-up screens with the indicator in progressive levels of disclosure, going from the C2PA ‘CR’ logo, to ‘content credentials’, ‘About this response’, ‘Certification data’, ‘Digital source data’ and ‘Transformation processing data’. At each stage, I asked them to speak aloud to describe what they saw, what they thought it meant, and how it resonated with them.

Screenshot showing the AI searchbot interface showing all the provenance data available

Screenshot showing the AI searchbot interface with progressive levels of provenance data on display

The C2PA icon was unfamiliar but content credentials made more sense

Students’ first exposure to the provenance data was as the C2PA ‘cr’ icon which was displayed at the opening interface of the searchbot (when the searchbot was ready to receive a user query). On its own, the icon was pretty meaningless to them. One felt it could be a watermark, others thought it was a button to make the chat window bigger, others said they would Google ‘cr’ to find out what it was. The next level of disclosure was a signifier that read ‘content credentials’, which appeared once the searchbot had received a query and produced a response. Students found this indicator more meaningful – they felt it was a way to verify where the response came from, and said they would be inclined to press it to find out more.

The level 1 disclosure ‘About this response’ was too vague to be useful

When asked what they thought they would find in the section ‘About this response’, students were largely unsure. Some thought it would contain a hallucination warning, others thought it would contain sources. When they read the generic descriptor about the C2PA provenance data, which explains that the response was AI generated and had content credentials and assertions to demonstrate authencity, they felt it was too vague to be useful to them, and didn’t feel it was telling them anything new. They were unfamiliar with ‘content credentials’ as a term and by the language, they reasoned that this message was aimed at a legal team rather than someone reading the output of the searchbot.

‘Certification data’ wasn’t a term that resonated with student participants

Reading the label ‘certification data’ students had mixed ideas of what this section would contain. Some felt it would reveal intellectual property details, some thought it would show licensing agreements relating to data entered in the searchbot, and others thought it would explain how the searchbot handled data. There was a further expectation that this section would contain the sources behind the searchbot response.

When they expanded this section to see the identity data (name of the organisation certifying the content) and the associated timestamp of when the content was certified, they were unclear what was being certified (the output of the searchbot or the AI model), and how they would use this information. Some reasoned that the responses from the AI searchbot were stored in a database and that this certification data could help keep track of a bank of these responses.

‘Digital source data’ made more sense to them

Of all the data categories displayed, ‘Digital source data’ was the one students gravitated towards. This is where they assumed they would find the links to the sources used to create the AI searchbot response, and their assumptions were correct. For many students, seeing the links to the sources proved the authenticity of the AI content more strongly than any other sections of the data they had seen. As well as the source links, this section included the C2PA Actions assertion (which recorded the stages used to create the digital content (or asset) including the AI model used). Students recognised the AI model but questioned the terminology ‘Actions assertion’, thinking it would be more straightforward to just list the AI model used.

In the ‘Transformation processing data’, detail of the AI prompts was of particular interest

Preconceptions of what they would find in the ‘Transformation processing data’ section varied from a logical set of steps to show how the responses were formed from source data, to details of how data inputs would be used to train AI models, to sustainability data showing energy expenditure in the formation of the AI searchbot responses. When they saw information labelled ‘Search mechanisms’ which detailed how embeddings were created and a vector database was used in semantic matching, most students felt this was too much technical detail. Some who were familiar with the process felt this section would be more appropriately labelled as a technical stack, rather than search mechanisms. The ‘Transformation processing data’ section also contained the AI prompt, and most students responded positively to this, saying they felt it gave them a fuller picture, it helped them understand ‘the personification of the AI’ and made the process more transparent.

The effectiveness of my C2PA designs largely depended on students’ existing knowledge

Reviewing the findings together, I concluded that my designs partly succeeded in communicating details of how AI had formed the searchbot responses, but this depended on students’ perceptions and what they knew already, as well as what they wanted to know. Overall, my designs fell short when it came to instilling a sense of trust in the students, largely due to unfamiliarity with the C2PA terminology, the challenges of targeting implicit trust factors and the partic.

Students familiar with AI responded positively, but others wanted a non-AI option

The C2PA indicators focused on the provenance of the AI-generated searchbot responses, but for some students this was irrelevant as they wanted a choice to be able to not use the AI feature at all. Several students were skeptical and negatively pre-disposed to AI features based on their previous experiences and concerns about the environmental cost, and therefore upon seeing the sparkle icon realising the searchbot was AI-powered they were sought for a way to turn it off, or to make a choice about whether to use it or not. This reinforced the need to ensure AI features are a choice, and in the case of the AI searchbot example, an alternative way to achieve the same task is provided.

The C2PA terminology was off-putting to students

As they read the data in the different categories, students felt alienated by some of the C2PA language, particularly terms like ‘content credentials’, ‘provenance’ and ‘assertion’. When they were able to see contextual detail associated with these terms, most were able to fathom what the terms meant, but I concluded that including these terms in the designs and in the labels of the different data categories was giving students an extra unnecessary hurdle of jargon to overcome before they could reach an understanding of the data presented to them.

Some C2PA data was clear and useful, other data felt unnecessary

The four categories of data resonated differently with the students, and, based on their reactions, some of these data sources felt unnecessary to show data provenance.

Most students found the source links useful, and what they expected in the category labelled ‘Digital source data’. They also found the AI prompt useful, but they didn’t expect to find it in ‘Transformation processing data’. Almost all felt the notification that the response was AI-generated (in the ‘About this response’ section) was unnecessary. Other data points that students felt confused and nonplussed by included the ‘CR’ logo, the identity data, the certification data (including the timestamp), and the technical terms describing the response-generation process.

Some of the data students were expecting was missing

As they reviewed the different categories of data they were presented with, students commented on data they would have expected to see. This included: environmental impact data, information about how input data would be used to train AI models, and details of how the data was processed and held by the searchbot feature. While these data points largely fell out of the realm of content credentials, it was interesting to understand the types of data students looked for when making judgements about whether to use and trust this type of AI feature.

The C2PA designs targeted some trust factors but not others

Re-examining factors that affect trust highlighted that trust is heavily influenced by implicit factors or emotional cues which are specific to individuals. Targeting subjective trust factors such as relatability, (which depends on how well something supports a person’s interests and resonates with them) was very difficult to achieve without research to understand the nuances of the target audience. Some students were reassured by the mention of AI models involved in generating the content, but for others, this merely caused confusion.

Targeting explicit trust factors like transparency and explainability in displaying content provenance was more straightforward, but it meant striking a balance. Providing too much technical detail about the production process, although this was intended to build trust by being upfront, could end up causing confusion and therefore result in the opposite effect, as I observed in the student tests.

Conveying reliability remains a challenge when AI is involved. Since AI is probabilistic, its results are variable, and in the context of a feature like an AI searchbot, it was not possible to make assurances in the provenance data. Instead, I used the progressive disclosure levels to encourage exploration and a gradual building of understanding, to help people come to their own conclusion about whether to trust the AI-generated content. I saw this working in the student tests, as they made reliability judgements about the AI responses when they saw the source data revealed.

Provenance indicators should replace the sparkles and it’s time for a unified AI vocabulary

The C2PA standard is several years old, yet content provenance is still a relatively unfamiliar concept. With new AI legislation published and the advent of more formalised requirements to disclose AI usage and application, I predict more rhetoric and discussions around disclosure, authenticity and provenance, to evolve the language and terminology around these concepts and to make them more mainstream.

Those who consume AI-generated content want to know how it was made, and the sources it came from, and those who use AI to generate or shape the content they publish have an obligation to disclose how AI was used, to ensure transparency from both perspectives. When designing AI features, consistent labelling will play an growing part in conveying meaning. As people become more familiar with AI, they will rely on these labels more and more to make judgements and decisions about AI features and AI-influenced content.

This small study has re-emphasised the importance of choosing terms and wording labels carefully, to strike a balance between technical and non-technical and to provide just enough detail to resonate with audiences but not so much that it confuses them. It highlights the need for a unified vocabulary for AI, to effectively communicate about its usage and application. As AI usage progresses, I expect provenance indicators and associated design patterns to continue progressing, and I’m keen to keep a close eye, to keep iterating on ways to display provenance data. I look forward to seeing sparkles replaced by more meaningful signifiers of AI usage and disclosure.

QuillMark, iterated. How user research helped us evolve an AI-enhanced Drupal style guide assistant

AI and UX Innovation Intern Shlok Gupta and I worked together to build and test Quillmark, an AI-powered Drupal content design tool that tackles the persistent problem of keeping content consistent with the University’s style guide. This post covers how we iterated from version 1 to 2.

Leading a UX Service, it’s critical to stay informed about the latest technologies, as these dictate user expectations and perceptions. I recognised the potential for AI to transform publisher experiences of using Drupal when Drupal AI experts Freely Give came to work with our team in January 2025. Since then, I’ve been committed to experimenting with AI application in content management, both through my involvement with the Drupal AI initiative and by supporting a succession of interns in my UX team at the University.

As a technology, AI is becoming increasingly mainstream. That said, its rapid rate of evolution and flux means adopting it consistently remains a challenge, as does designing AI features that are centred around user needs and expectations. AI features can be built and released in a matter of days, but can just as quickly become superseded and obsolete as technologies and capabilities change, and user expectations and perceptions of AI features can vary widely depending on individual experiences, trust factors and preferences. Continuous research is necessary when developing any user-centred software, but it’s particularly important when designing user-centred AI features, which I wrote about in a related blog post:

Stop chasing, keep researching: Why continuous contextual learning is the only way to build useful AI features

In the spirit of continuous research, the students who occupy the AI and UX Innovation Intern role in my UX team spend as much time testing as they do building. Shlok Gupta joined the team from May – August 2026, in which time he prototyped and tested two versions of an AI-enhanced style guide checker tool in Drupal. Shlok described his process to build and test Quillmark v1 in two blog posts, and also produced a video to show it in action:

Building QuillMark: testing a Drupal style-guide assistant with web publishers

Quillmark: the technical details behind my Drupal style-guide assistant

View a video of Quillmark v1 on MediaHopper

In this post, I continue the story, to explain how version 2 was built based on feedback from version 1, and what we learned testing version 2 with publishers.

QuillMark v1 had a non-AI stage and an AI stage and the ability to flag and fix errors

Shlok designed QuillMark v1 to have two stages as a way of making most economical use of AI time and tokens. Stage 1 used basic pattern-matching and code-based rules to catch straightforward style-guide errors without AI, and stage 2 then brought in AI to find and recommend fixes for deeper issues. This meant AI resource wasn’t spent identifying and resolving simpler problems that didn’t need it. The screenshots indicate how v1 worked.

Screenshot showing Quillmark v1 (in the right-hand panel of a Drupal site) ready to review content

Screenshot showing Quillmark v1 (in the right-hand panel of a Drupal site) ready to review content

From right-left, four screenshots showing detail of the QuillMark guide in the right-hand panel before a scan, the panel expanded to show the guide, results of a stage 1 scan and detail of selected stage 1 findings

From left to right, detail of QuillMark in the right-hand panel – before a scan, expanded to show the guide, results of a stage 1 scan and detail of selected stage 1 findings

Three screenshots of QuillMark, showing the panel ahead of running the stage 2 scan, combined results of the stage 1 and stage 2 scans and detail of selected stage 1 and stage 2 findings alongside each other

From left to right: the panel ahead of running the stage 2 scan, combined results of the stage 1 and stage 2 scans and detail of some stage 1 and stage 2 findings alongside each other

QuillMark screenshot showing how a finding from a stage 2 scan mapped to a sentence in the passive voice highlighted in the editorial interface

QuillMark screenshot showing how a finding from a stage 2 scan mapped to a sentence in the passive voice highlighted in the editorial interface

QuillMark screenshot showing the highlighted sentence changed to active voice - which resulted from pressing the ‘Apply fix’ button on the finding

QuillMark screenshot showing the highlighted sentence changed to active voice – which resulted from pressing the ‘Apply fix’ button on the finding

Tests revealed the need to simplify QuillMark v1

In the tests of QuillMark v1, participants were asked to read a guide about how the tool worked, use it to scan some pre-prepared content and then review the resulting findings and say how they would act upon them. The results indicated several areas for improvement.

All participants understood that QuillMark v1 was a tool to help them correct content errors. Despite being directed to read the guide upfront, however, some participants were unclear of the difference between stages 1 and 2.  Having run stage 1, participants were uncertain what stage 2 of the tool would do. Some identified the AI detail in the stage 2 labels and therefore realised that stage 2, unlike stage 1 used AI, but others thought stage 2 was the ‘repair’ or ‘fix’ stage to follow stage 1 as the diagnostic stage.

As they reviewed the different categories of finding in the right-hand panel, most participants said they felt the amount of information was overwhelming, and difficult to scan read. They especially found it difficult to interpret abstracted recommendations to fix errors, preferring to receive suggested fixes through the ‘Apply fix’ option. As a way to reduce the overwhelm, some participants said they would be inclined to go and fix several of the errors manually first, and then rescan, so that the tool only presented them with the errors they missed themselves.

Other observations from the test participants fell into two categories, lack of understanding of how Quillmark worked, and feedback on how they felt it could work better for them.

Participants were confused about how to use QuillMark and how it worked

Several aspects of participants’ behaviours indicated they weren’t sure how QuillMark worked. These were as follows:

  • There was some confusion over how to start QuillMark running. Participants did not immediately realise that the Stage 1 and Stage 2 buttons would start the tool. They looked for a button that said ‘Run’ or similar. When they worked out how to activate the different stages, they were unclear about what was happening in each.
  • Several participants raised questions about the cost and environmental impact of running the AI stage of the tool, saying this would affect whether they would choose to use it.
  • There was a lack of clarity from participants about whether QuillMark was using generic grammatical rules to flag errors and make recommendations or if it was using the specific University of Edinburgh style guide rules
  • Several participants did not initially recognise that QuillMark could suggest fixes as well as flagging errors.

Participants suggested ways to change QuillMark, to help it support publishing practices better

When they had got used to working with QuillMark, several participants suggested ways they felt it could work better. There were as follows:

  • A request to be able to apply bulk fixes to recurring errors, rather than addressing them one-by-one (for example, removing full stops in repeated headings or switching sets of times from the 24 hour clock to the 12 hour clock)
  • An ask from several participants to be able to preview fixes offered by the ‘Apply fix’ buttons before accepting them to change the text in the editorial interface (however, they soon realised they could switch between the ‘Apply fix’ and ‘Revert fix’ buttons to achieve this).

QuillMark v2 worked in the same way but the interface was simpler

Taking on board the feedback from the QuillMark v1 tests, Shlok made a version 2 which included several changes:

  • A single ‘Scan’ button to start both stages of the tool working, replacing the need for publishers to run stage 1 and stage 2 scans in sequence
  • Progress indicators that appeared once ‘Scan’ had been pressed, to communicate what was happening, and to affirm that the editorial style guide was being checked
  • Inclusion of an ‘Advanced settings’ panel giving users the option to decide whether to include deterministic error scanning and fixing and the use of AI to suggest replacements
  • A panel that estimated monetary cost and environmental impact of each scan resulting in the findings

View a short video demonstration of QuillMark v2 on Media Hopper

The screenshots indicate how v2 worked and show the differences between v1 and v2

QuillMark screenshot showing the tool in the right-hand panel ahead of a scan being run on a page of content titled ‘Research projects’

QuillMark screenshot showing the tool in the right-hand panel ahead of a scan being run on a page of content titled ‘Research projects’

Series of QuillMark screenshots, from left to right, showing the tool with the guide displayed, and the successive stages once ‘Scan’ has been pressed – reviewing, preparing, checking editorial style rules and then the notification that QuillMark has completed the scan.

QuillMark screenshots, from left to right, showing the tool with the guide displayed, and the successive stages once ‘Scan’ has been pressed – reviewing, preparing, checking editorial style rules and then the notification that QuillMark has completed the scan (showing that AI suggestions are included in the ‘Advanced settings’ pane

Tests showed v2 was better, but still had room for improvement

The testing protocol for QuillMark v2 remained the same as for v1, however, this time participants were asked to use the tool on three separate pages of University website content, to give them more practice of engaging with the tool to learn how it worked.

Participants found it easier to understand how QuillMark v2 worked

With the new blue ‘Scan’ button, participants were more easily able to get started with running QuillMark. Since it was running an AI and a non-AI stage QuillMark v2 took longer to produce findings than QuillMark v1, but the newly added progress indicators kept participants informed about what was happening.

The indicator with the data about environmental and monetary costs was noted by all participants, and some said this was welcome information to help them make responsible choices about AI use.

Participants were still directed to the guide, which some wanted to refer back to once they had got started with the tool, to remind them of how it worked.

There were fewer questions about the source of the rules powering Quillmark v2 than for v1, and several participants verified that the tool was using rules from the University editorial style guide by pressing the ‘Link to rule’ button on individual findings (which connected to the relevant page or section of the style guide).

Some still found the interface and display overwhelming

Although QuillMark’s interface had been simplified by the removal of stage 1 and 2 indicators, participants still commented it contained a lot of textual information and was difficult to scan. Some were unclear what the different labels in the findings meant (Errors vs Warnings vs Notes), and several found it easy to miss some of the tool’s functionality simply because there was too much to read through.

Participants were still uncertain about the ‘Apply fix’ button

As they worked through the individual findings, participants were able to recognise the need for corrections that had been highlighted, but some did not appreciate that QuillMark v2 offered a fix as well as flagging the error.

As they used QuillMark v2 to scan a page about research projects, the tool flagged an instance of poorly worded link text (‘Project Website’). The ‘Apply fix’ button contained a suggested improvement (‘Visit the Hillforts Atlas website’) but several participants did not select this, instead they manually addressed the error themselves, by checking the link destination and then writing appropriate link text. Those that did notice that ‘Apply fix’ had made a suggested improvement were happy to accept it and apply it.

Screenshot showing a QuillMark v2 finding flagging poorly written link text

Screenshot showing a QuillMark v2 finding flagging poorly written link text

Screenshot showing QuillMark v2 a fix to improve poorly written link text applied in the editorial interface

Screenshot showing QuillMark v2 a fix to improve poorly written link text applied in the editorial interface

They still wanted repeated findings to be consolidated, to be addressed together

Some categories of findings contained recurring instances of the same issue. Participants wanted these to be grouped or summarised instead of being listed individually, as they wanted to address all the findings on a particular issue together in one action (either dismissing them or applying suggested fixes), rather than going through them one-by-one. This problem was particularly prevalent on pages where the content followed a structural pattern, for example a series of headings all written in non-sentence case (with misuse of capital letters) or a labelling convention applied consistently to rows of a table (for example numerals in sequence).

Screenshot showing individual instances of an error flagged as separate findings in the QuillMark v2 interface

Screenshot showing individual instances of an error flagged as separate findings in the QuillMark v2 interface

There was concern over saving progress when working through lots of findings

In the tests, one of the pages scanned by Quillmark v2 generated 166 findings to be addressed. As participants worked through those in a particular category (for example ‘Links and calls to action’) they felt uncomfortable that they were unable to save as they went, seeking some reassurance that their work was being logged sequentially. They looked for a ‘Save’ button associated with each individual finding, and some wrongly assumed the ‘Dismiss’ button saved the change they had made (whereas in actual fact, this button removed the finding from the list to be addressed).

QuillMark was unable to reconcile multiple errors on the same content

In some instances, a piece of content contained multiple style guide errors, so it appeared in separate findings in separate categories. An example was ‘Prof Smith’s website’ used as a piece of link text. This error appeared in the ‘Spelling, wording and abbreviation’ category, flagged for the incorrect use of ‘Prof.’, but also in the ‘Links and calls to action’ category, flagged for unclear link text.

When participants encountered an error in one finding, they corrected it there, but then when the same piece of content was also flagged for another error in another category, Quillmark v2 had already noted that the content had been changed, and had no way of layering on another, different correction. In these instances, a warning flag disabled the ‘Go to field’ preventing participants from going to the incorrect content, therefore forcing them to try and find the location of the incorrect content themselves, which they found disorientating and tricky, especially in long pages of content.

Screenshot showing the blocker message that displayed when there were multiple errors on a single field of content

Screenshot showing the blocker message that displayed when there were multiple errors on a single field of content

Participants wanted QuillMark to fix heading levels and reposition inline links

Errors relating to content structure, such as incorrect heading hierarchies and inline links were flagged by Quillmark, but the tool was unable to suggest an automatic fix to apply. Several participants said they would have liked for an autocorrect functionality for these errors, since they found them difficult and fiddly to manage within the EdWeb2 editorial interface.

The next iteration of the tool will focus on one category of style rules: Links

Since the last round of tests on QuillMark v2, Shlok completed the internship, and was succeeded by Viktor Sharha in the role of AI and UX Innovation Intern, to continue the work on the Drupal style guide checker tool. Taking all the findings from the tests of QuillMark v1 and v2 together, I concluded that the most pressing problem to address was that of overwhelm.

When a page contains many errors or style guide violations, there is a dual design problem: Firstly how to detail each error, the root cause and the suggested fix, so that publishers understood what was wrong and how it can be corrected, and secondly to do this concisely, taking up the minimal space possible, to avoid publishers having a long scroll through multiple errors with the associated cognitive load.

Going forward, it made sense to focus on the first problem of presentation of individual findings first, and to do so at a site level (recognising that publishers may not do their checking on a page-by-page basis), before scaling this up to design for multiple errors. To maintain a tight remit, and to focus on the style guide violations which are highest priority to fix, we have decided to specifically look at links, to consider how the tool can spot instances of poor link text, how it can describe ways to improve these, and the options for applying automated fixes.

❌