Findings that missed the mark: NTNU and Copilot

The NTNU Copilot report findings and recommendations is something of a conundrum. It has many important high-level questions we should be asking as either society or an organization. However, these have very little to do with Copilot for Microsoft 365 as a product, which the report title implies is the subject. It is not until page 3 that we have the acknowledgement stating this is not research nor this is not a scientific analysis, but rather a snapshot of an experience. We think it is worth emphasizing two points here: This report is not scientific research, and this that report does have significant value when taken in context. It’s just maybe not the value you were looking for when you picked it up.

The purpose of the report

We’ll begin by saying that there is a long-standing friction between the more and the less technical people working in the IT space. We are not interested in engaging in this friction, as both the highly-technical and the highly-informed from other areas aren’t enemies and should, in fact, work together more for the good of the whole. We are not looking to tell anyone to “stay in their lane” or belittle the significant amount of work done here. Nor do we want to underemphasize the passion and dedication to a field of study it takes to undertake a large project like this. But, we will go as far as to say that when it comes to lanes, we’re in both and we’re going to weigh in on this.

This report is not a Copilot guide. This report is some good advice wrapped in the perspective of users but absent the technical and leadership perspectives. We cannot ignore that a report issued from Norway’s University of Science and Technology carries with it the implicit impression that this is going to be something the report simply isn’t.

The (false) impressions from the report

We see these already in the Media. We see this from NTNU’s own employees tweeting about monitoring of employees, and we see this in customer questions:

“Yeah but NTNU says Copilot is problematic…”

Much greater care should have been given to how this report was both organized and issued to avoid giving this false impression. As we mentioned before, the authors have plainly stated on page 3 that this isn’t a research study with findings–but what should this look like to a lay person? Do opinion pieces typically come with surveys or does this start then to look like research? If it looks like a duck…

We immediately started looking for the missing literature review and methods sections. There are certain assumptions most lay persons make when an organization like this issues a document sharing experience with a product that this was a study of some sort. There is an officiality here that has the potential to harm; The product is misrepresented in areas, nearly all technical issues are omitted and the lay reader is left with the belief that this magic Copilot box is hard to use and dangerous.

Copilot is just a technology with prerequisites and requirements like any other. What a large-scale Copilot implementation requires first and foremost is that you as an organization have adhered to the well-known principles of data governance, identity management and access management. It requires that you have mastery over what it is that you ‘do’. That is to say, if you have consistently de-prioritized completing projects’ final clean-up stages, elected to migrate ‘everything’ with the last system upgrade or move to the cloud, or you just like to ‘save it for later, just in case‘ in undesignated or inappropriate locations, then your organization needs to make serious changes. There are laws, regulations and principles from which you should be making informed decisions on what data to store, who has access to what, what to discard and when, and what to archive.

We’re going to give our comments and suggestions for the whole document with this as a starting point for our English-only speakers. We hope to round off with what we think would be a good set of guidance for most organizations wondering where to start. Stay tuned.