“Copilot works brilliantly when you already know what you want it to help you with.”

Translated from “Copilot er helt glimrende når du allerede kan det du vil at den skal hjelpe deg med” – Interesting find by the NTNU project, but in my ears this has tripped over another one of the elephants in the room with respect to this report: Tools work better when you know how to use them. Training users on how to use a product (any product) is going to produce results more in-line with the tool’s intended use. (Note I did not say better; Sometimes innovation comes from not following guidelines!) For readers who might be familiar with The Expanse series by James S. A. Corey, there are references to Monkeys and Microwaves, and I think this excerpt captures the sentiment well (emphasis mine):

“Holden was starting to feel like they were all monkeys playing with a microwave. Push a button, a light comes on inside, so it’s a light. Push a different button and stick your hand inside, it burns you, so it’s a weapon. Learn to open and close the door, it’s a place to hide things. Never grasping what it actually did, and maybe not even having the framework necessary to figure it out. No monkey ever reheated a frozen burrito. So here the monkeys were, poking the shiny box and making guesses about what it did.”

Research from Erik Brynjolfsson (Stanford & NBER), Danielle Li (MIT & NBER) and Lindsey Raymond (MIT) which predates NTNUs report and has a nice summary from MIT Sloan has a much different take from NTNUs: Access to AI-based conversational assistants increases productivity and is important for novice and low-skilled workers.

In another study, Experimental Evidence on the Productivity Effects of Generative Artificial Intelligence by Shakked Noy (MIT) and Whitney Zhang (MIT), published in the journal Science, where the productivity of incentivized workers was measured, findings indicated that all workers benefited to a degree. This study was on the heels of ChatGPT3.5 where it was not ubiquitously mentioned in common media.

Among the highlights from the study that piqued my interest as confirmation of what I’ve seen in my professional life:

• Participants enjoyed their tasks more with ChatGPT
• Participants showed an increase in optimism regarding AI in terms of their profession
• 80% of users opted to use ChatGPT on a subsequent task
• Time-to-completion was shorter for tasks
• Quality was higher
• Participants who scored lower on a first task benefitted more from using ChatGPT on a subsequent task
• Access to ChatGPT reduced inequalities among workers, particularly those of lower ability

This study specifically noted “ChatGPT used in this experiment cannot, by its nature, access or supply context-specific knowledge and is not a reliable source of precise factual information.” This is in sharp contrast to Microsoft Copilot for M365 which is grounded in an organizations M365 data, thus providing this missing context noted here. In order to take advantage of the added organizational data grounding from Copilot, we need to inform users of the capabilities and use.  

When we deprive users of training they will then use what is intuitive to them; They have only superficial mastery of a product’s capabilities.  It may be argued that this has to do with “simplicity of use” drawing the conclusion that this product is not, in fact, easy to use. We disagree.  Communicating in natural language is not necessarily an easy task, and training is helpful.  Training is useful for all our business tools.  As an example: When one uses Excel for the first time, a VLOOKUP might not be the most intuitive feature but it certainly could make a task easier (You can even ask Copilot how to use VLOOKUP!). Discovering the fill-handle in Excel only after extensive copy and paste actions would be frustrating–it’s helpful to know the potential in advance.  Copilot for Microsoft 365 is no different. 

Our findings:  The context of the tool itself is something that should also be conveyed to users prior to release. This is part of a well-worked implementation plan which is, among other things, part of the recommended process from Microsoft. Users should be introduced to the tool’s potential after the necessary data security and governance steps have been taken within the organization.