[Vp-reproduce-subgroup] Update: Draft Guidance: Assessing Credibility of Computational Modeling and Simulation

[Jacob Barhak]

Greetings group members,

Hopefully this thread caught your attention and you plan to submit feedback
to the FDA.
Here is the FDA link again:
https://www.fda.gov/regulatory-information/search-fda-guidance-documents/assessing-credibility-computational-modeling-and-simulation-medical-device-submissions

I do hope we hold a discussion to generate more impactful ideas.

Regardless, I plan to submit a response based on the text I sent and our
paper in mid March.

I do hope many of you will read the document and then their thoughts to
this mailing list as well.

         Jacob



On Sun, Jan 30, 2022, 11:51 Jacob Barhak <jacob.barhak at gmail.com> wrote:

> Greetings group members,
>
> The purpose of this email is to start a discussion on the FDA draft:
> Assessing the Credibility of Computational Modeling and Simulation in
> Medical Device Submissions.
>
> I went back to the paper draft and looked at it. The draft does a good job
> on focusing on procedures and steps to assess credibility. Especially
> important is the work flw suggested. From those perspectives it's a good
> document that may help others in the future.
>
> One basic idea that we also discussed in the paper was the "question of
> interest" - Early on John Rice introduced this concept into our paper and
> others supported it later. However, even in our paper there were different
> opinions on how to approach credibility and reuse models defined for
> different questions in mind. The design process may start from a different
> question than the one that we wish to satisfy. Since this document is a
> draft from a regulatory body, it is less about the design process and more
> about compliance. However, our discussion in the paper may be still
> relevant since it introduced the concept of comparison among models.
>
> I looked through the draft and could not see an element of relativity , or
> in other words, a reference to compare to. And this concept may be
> important in the future since the modeling field evolves quickly,
> especially with regards to machine learning techniques that evolve rapidly.
> A model that seems credible today and passes some tests may not be very
> good in a few years. Also one model may not perform as good as a
> combination of models. Yet this model, although outdated may outperform
> human decision - therefore I believe there should be 2 elements considered
> by the FDA that include a reference / relative performance.
>
> 1. Group of models - how does a model perform compared to existing models
> with regards to satisfying the question of interest. Models will eventually
> become outdated and although  models may be sufficient to answer a
> question, within time some models may not be as good as others. And this
> time period may be relatively short. So the regulator has to consider
> outdated models and not only set a threshold for compliance. So to keep
> compliance over time a model has to outperform existing models.  This idea
> can be extended to allow models to cooperate in an ensemble, yet the first
> step is benchmark models taking into account other models as a relative
> reference.
>
> 2. Relative to human practice - the model also has to be compared with
> current human standards without  the model. A device with a model may not
> be accurate, yet human accuracy may be even worse without that model to
> aid. Moreover, humans may not agree with the same evidence produced. To
> give an example, please look at this paper:
> https://arxiv.org/abs/1806.02121  . In this paper the authors test their
> model against humans and show that humans test to agree with their model
> more than they agree with each other. So the FDA should keep in mind that
> humans that are certified may not perform well and the lower threshold for
> compliance should take this into consideration.
>
> I am writing those ideas in hope this will start a discussion where we can
> formulate and combine answers by group members and submit them together.
> Otherwise I will submit a version of this text myself.
>
> I hope there will be interest in providing this feedback.
>
>                   Jacob
>
>
>
> On Thu, Jan 27, 2022 at 9:37 PM Jacob Barhak <jacob.barhak at gmail.com>
> wrote:
>
>> Greetings reproducibility and integration groups,
>>
>> John Rice has sent this message. about the FDA document:
>>
>> https://www.fda.gov/regulatory-information/search-fda-guidance-documents/assessing-credibility-computational-modeling-and-simulation-medical-device-submissions
>>
>> And indeed this is relevant to our work - in fact this was my suggestion
>> for the next activity.
>>
>> FDA has released a request for comments for a document drafted. We
>> actually referred to this document in our paper.
>>
>> I was planning to start a discussion on this paper so we can
>> submit responses, either individually or as a group or sub-groups after we
>> finish with the paper.
>>
>> Yet since the paper is almost done and time is running short, it is
>> perhaps a good idea to discuss this now.
>>
>> Looking forward to your feedback.
>>
>>                Jacob
>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.simtk.org/pipermail/vp-reproduce-subgroup/attachments/20220206/02707967/attachment.html>