[Vp-integration-subgroup] Update: Draft Guidance: Assessing Credibility of Computational Modeling and Simulation

[Jacob Barhak]

Greetings group members,

The purpose of this email is to start a discussion on the FDA draft:
Assessing the Credibility of Computational Modeling and Simulation in
Medical Device Submissions.

I went back to the paper draft and looked at it. The draft does a good job
on focusing on procedures and steps to assess credibility. Especially
important is the work flw suggested. From those perspectives it's a good
document that may help others in the future.

One basic idea that we also discussed in the paper was the "question of
interest" - Early on John Rice introduced this concept into our paper and
others supported it later. However, even in our paper there were different
opinions on how to approach credibility and reuse models defined for
different questions in mind. The design process may start from a different
question than the one that we wish to satisfy. Since this document is a
draft from a regulatory body, it is less about the design process and more
about compliance. However, our discussion in the paper may be still
relevant since it introduced the concept of comparison among models.

I looked through the draft and could not see an element of relativity , or
in other words, a reference to compare to. And this concept may be
important in the future since the modeling field evolves quickly,
especially with regards to machine learning techniques that evolve rapidly.
A model that seems credible today and passes some tests may not be very
good in a few years. Also one model may not perform as good as a
combination of models. Yet this model, although outdated may outperform
human decision - therefore I believe there should be 2 elements considered
by the FDA that include a reference / relative performance.

1. Group of models - how does a model perform compared to existing models
with regards to satisfying the question of interest. Models will eventually
become outdated and although  models may be sufficient to answer a
question, within time some models may not be as good as others. And this
time period may be relatively short. So the regulator has to consider
outdated models and not only set a threshold for compliance. So to keep
compliance over time a model has to outperform existing models.  This idea
can be extended to allow models to cooperate in an ensemble, yet the first
step is benchmark models taking into account other models as a relative
reference.

2. Relative to human practice - the model also has to be compared with
current human standards without  the model. A device with a model may not
be accurate, yet human accuracy may be even worse without that model to
aid. Moreover, humans may not agree with the same evidence produced. To
give an example, please look at this paper: https://arxiv.org/abs/1806.02121
. In this paper the authors test their model against humans and show that
humans test to agree with their model more than they agree with each other.
So the FDA should keep in mind that humans that are certified may not
perform well and the lower threshold for compliance should take this into
consideration.

I am writing those ideas in hope this will start a discussion where we can
formulate and combine answers by group members and submit them together.
Otherwise I will submit a version of this text myself.

I hope there will be interest in providing this feedback.

                  Jacob



On Thu, Jan 27, 2022 at 9:37 PM Jacob Barhak <jacob.barhak at gmail.com> wrote:

> Greetings reproducibility and integration groups,
>
> John Rice has sent this message. about the FDA document:
>
> https://www.fda.gov/regulatory-information/search-fda-guidance-documents/assessing-credibility-computational-modeling-and-simulation-medical-device-submissions
>
> And indeed this is relevant to our work - in fact this was my suggestion
> for the next activity.
>
> FDA has released a request for comments for a document drafted. We
> actually referred to this document in our paper.
>
> I was planning to start a discussion on this paper so we can
> submit responses, either individually or as a group or sub-groups after we
> finish with the paper.
>
> Yet since the paper is almost done and time is running short, it is
> perhaps a good idea to discuss this now.
>
> Looking forward to your feedback.
>
>                Jacob
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.simtk.org/pipermail/vp-integration-subgroup/attachments/20220130/624c36ec/attachment.html>