可用性的总结性评价应该怎么做——第一部分

RA小编辑

本文转载自Open Regulatory

The IEC 62366 (Usability) Summary: All You Need To Know (It's Not Much)Dr. Oliver Eidel

Oh man, where to start! While probably not much changed for you, as you were happily reading along our Blueprint article and came across this one as simply the next one in your queue of semi-interesting-rather-boring regulatory articles to read, a lot has actually changed for me in the meantime. If you check the dates, you might notice that many of the original articles, mainly on the IEC 62304, were written by me in 2020, when I was mostly-unemployed, figuring out what to do with my life and braindumping my regulatory knowledge onto this weird “OpenRegulatory” website. Since then, this “OpenRegulatory” website has become a legimitate company (at least that’s what I tell myself every night when I go to sleep), with lots of consulting clients, an eQMS software with many happy users, and real team members who just returned with me from our yearly company retreat in Croatia.

Okay, none of this probably interests you as you arrived on this page actually looking for IEC 62366 information. But still, this is me picking up my writing where I left off 3 years ago. Damn..

Back to less interesting stuff. Settle down in your comfortable office chair, grab a mug of coffee so you don’t fall asleep, and prepare to learn one thing or another about the IEC 62366, the standard on usability for medical devices.

The Only IEC 62366 Summary You Ever Need To Read

If I had to summarize the IEC 62366 for startups developing medical software, I would say you only need to know two important things: First, don’t pay consultants, you don’t need them, and second, 95% of the work is doing a usability test with at least 5 (better, 15) users. That’s it. Those are the important parts. The rest is noise.

Let’s dig into those now.

The Important Parts Of The IEC 62366Doing a Usability Test

The IEC 62366 calls this performing a “summative evaluation” (standards seem to be terrible at choosing easy-to-understand words for humans) of your usability. In practice, that means that you have to assemble a group of people who resemble the actual, future users of your product, and subject those to tasks, while observing them whether they achieve those tasks.

As an example, if you’re developing software for radiologists, you’ll have to find some radiologists who are willing to take part in your usability tests. That might be hard, because radiologists are often busy staring at black-and-white screens in dark rooms, earning a lot of money.

As another example, if you’re developing an app for people suffering from depression, it might simply suffice to test your app with a broad range of people, regardless of whether they’re suffering from depression. Why? Because depression arguably wouldn’t influence the way they understand and interact with your app (then again, it depends on the details, as always, I suppose).

And finally, if you’re developing an app to treat radiologists suffering from depression, well, all hope is lost. That’s really a small subgroup of people. I’m kidding, of course. Then again, maybe not. Depression is a serious disease and many doctors (radiologists also count as doctors) suffer from it.

Okay, where were we.. ah yes, my point simply was that you need a group of people which resembles your future users.

Now, you get that group of people and come up with tasks which those people should do. You observe whether they’re successful. That’s your summative evaluation. Read more in my post on Summative Usability Evaluation For The IEC 62366.

Not Hiring Consultants

The next important part for IEC 62366 compliance is not hiring consultants. Yes! As you read at the beginning of the article, OpenRegulatory now is a company (and a consulting company, too), but, at heart, I’m still very skeptical of consultants. Especially usability consultants. Don’t hire them. Don’t even hire us for usability consulting. There’s no point. Some caveats apply, see further below.

Again, 95% of the work for IEC 62366 compliance is performing a summative usability evaluation. And that again requires you to find an suitable group of future users, coming up with a list of tasks for them, telling them those tasks and observing them, while writing everything down. You think you can do that? I hope so, because, comparing it with all other tasks you have to do in a Healthcare startup (founding it, building a product, wrestling with German bureaucracy), this is a rather easy task. Believe me.

So, where to usability consultants actually come in? I’ve met a few usability consultants in the past, and I’d say that a surprisingly high amount of them (≈20%) are actually not shady (contrast that with the rest of the regulatory consulting industry).

When should you hire them? I can think of a few reasons:

• You’re a huge enterprise company which doesn’t get much done anyway due to internal politics and chaos, and you want to outsource all of the usability stuff. Usability consultants are happy to take your money.
• Performing a usability test for it would be very complex. Think of something like a ventilator (in the context of anesthesiology) - you’d need to set up a mock operating theatre and maybe even staff it with the appropriate people to recreate a real-world scenario in which it’s used. Damn. My gut feeling is that this “complex test set-up” problem applies more to hardware devices than to software, but I’m not sure. Still, it’s a real problem. You get the idea.
  

Those are the legitimate reasons I can think of. Maybe you know more. Let me know.

The Less Important Parts Of The IEC 62366

So we covered the most important parts for your IEC 62366 compliance so far. Let’s look into the less-important parts next.

What The Hell To Test In The Usability Test?

I hope you’re having a good laugh right now, because I certainly had one. A very friendly usability consulting lady (she was not shady!) and I recently had a call, and we both joked about auditors actually only checking the structure of your documentation while pretty much ignoring its content. While that might not be 100% true (only 98%), it’s still what I often see in audits.

Specifically, auditors will check whether you did a summative evaluation (the usability test I mentioned above) and whether you’ve created all the right documents. They might not check whether the content of those tests actually made any sense at all from a usability point of view.

This is very similar to sitting down in a wheelchair, searching meticulously for where the CE label is applied on it (wheelchairs are medical devices), finding the CE label, then concluding that the device is safe and subsequently driving the wheelchair off a cliff. It’s mixing up adherence to standards with actual safety. Those two things might overlap, but they often do not. In our audit experience regarding usability documentation, it seems to be that mainly adherence to standards is checked.

By the way, I’ve been told that the FDA is more rigorous there. This goes along with their theme of being more focused on content in medical device documentation, a concept which I appreciate. Still, the EU experience however seems to be to mainly check adherence to standards.

This was a bit of a rant. So what the hell do you actually test in your usability tests? Consultants tend to make this sound very complicated, but it’s just two things: Your hazard-related use scenarios, and your main use scenarios. Wait, what? Let’s look at those.

What would the main use scenarios for a wheelchair (I’m still chuckling) be? Let’s see:

• A user can comfortably sit down in the wheelchair. (hazard-related)
• A user can drive the wheelchair around to wherever they want.
• A user can brake safely. (hazard-related)
  

As all my examples, this might be simplified. Then again, I can’t think of many additional use scenarios (maybe another person pushing the wheelchair etc.). Still, I hope you get the idea. This is a list of use scenarios which the wheelchair must fulfil so that, well, it actually fulfils its purpose (or rather, intended use, in regulatory slang).

You probably noticed that I marked the first and third one as “(hazard-related)”. Those are use scenarios which are related to hazards - in the words of the standard, hazard-related use scenarios. Again, simplified for our example, but let’s just assume that those truly have associated hazards - for the first one, that could be that the wheelchair rolls back while the patient wants to sit on it (that may or may not have happened to me in the hospital when I did my internship there). For the third one, it could be that the patient crashes into something if the braking doesn’t work properly.

Let’s get back to the standard. The IEC 62366 only tells you to do a summative evaluation of the hazard-related use scenarios. Not of all use scenarios. So why did we list all? That’s because the ISO 13485 requires you to do a validation of whether your product fulfils all your user needs.

So, as you might notice now, different standards have different names for things which are essentially the same. The simple solution here is to realize that what you’re doing for IEC 62366 compliance also fulfils the validation requirements of the ISO 13485, and the simple “hack” is to expand your list of hazard-related use scenarios to also include, well, all use scenarios. Just as I did above for the wheelchair. So we add the second use scenario (a user can drive it around) and also test it during our summative evaluation.

Another thing is the so-called formative evaluation.

Formative Evaluation: Whatever You Want

When small kids (and, if you’re in Berlin, grown-ups) go to a costume party, their parents (or, if you’re in Berlin, their friends) tell them “you can be whatever you want!”. So you’d have kids (and, if you’re in Berlin, grown-ups) dress up as a fridge or so.

The same goes for formative usability evaluation. It can be whatever you want. I won’t even go into what “formative evaluation” actually means, because you’ll figure it out yourself, because it can be whatever you want it to be.

Some examples of formative usability evaluation:

• Showing a prototype of your product to someone, walking them through it and asking them for their opinion.
• Letting a UI/UX person review wireframes / mock-ups / drafts of your product and getting their opinion on what can be improved.
• Doing a user test on a prototype (yes, essentially the same procedure as the summative evaluation above).
  

So you get the idea. It’s something you do with your product while it’s in development, with the goal of getting some sort of information on what to improve.

I like to rant about regulations a lot, but I actually really like this one. Because, well, it really makes sense - involving your customers during development is a huge aspect of the whole agile software development methodology (at least its good parts), and that’s very useful. And the standard doesn’t even pretend to know what method might make sense here - they just leave it up to you. I think that’s really cool.

Want to have a Google Meet call to walk a doctor through your prototype to get some feedback? Just write down what you did and what the doctor said, and that’s a valid formative evaluation.

You showed your prototype to a potential customer at a conference and they gave you some feedback? Also a valid formative evaluation. Just write it down.

Very cool.

I guess the somewhat tragic aspect of this is that “minimal compliance for audits” is incredibly easy to achieve and doesn’t fulfil its actual goal - you’d just need at least one formal usability evaluation in your virtual drawer somewhere, and that might only be some crappy notes from a Google Meet call with a doctor. Even worse, I’ve actually never seen this being checked during audits.

Then again, would checking it lead to better products? That moves us into this whole philosophical discussion of whether more rules and more audits lead to better products. The tragic reality is that we simply don’t know as we don’t have any data on that. The safety (not to be confused with actual greatness) of medical devices seems to have improved, so regulators seem to conclude that their regulation is working. It’s not based on data though. So it’s like alchemy right now.

Anyway, I actually like the aspects regarding formal evaluations. You decide for yourself!

Read The IEC 62366!

The IEC 62366 is actually a rather short standard and is surprisingly easy to read - probably the easiest of all the medical device standards. Again, I tend to be critical of many regulatory requirements, but this one is actually a rather useful one and that’s something which is worth respecting.

Obviously, you should read all standards, but if you’ve been procrastinating that, the IEC 62366 is a good place to start. Check out our article on how to get cheap access to them via the friendly Estonian website.

On a slighty different note: You want to get your medical software certified under MDR but don't know where to start? No worries! That's why we built the Wizard. It's a self-guided video course which helps you create your documentation yourself. No prior knowledge required. You should check it out.

Or, if you're looking for the most awesome (in our opinion) eQMS software to manage your documentation, look no further. We've built Formwork, and it even has a free version!

If you're looking for human help, did you know that we also provide some limited consulting? It's limited because we are not many people. We guide startups from start to finish in their medical device compliance.