[Vp-reproduce-subgroup] Update: Draft Guidance: Assessing Credibility of Computational Modeling and Simulation

[Jacob Barhak]

Greetings Group members.

This is a reminder that the submission deadline time for comments to the
FDA document is on 3/24.

https://www.fda.gov/regulatory-information/search-fda-guidance-documents/assessing-credibility-computational-modeling-and-simulation-medical-device-submissions

Unless there is an interest by the group members to submit any text in a
group, I will submit the text I send below next weekend and mention our new
published paper.

Hopefully others will choose to submit comments.

               Jacob



On Sun, Feb 6, 2022 at 5:36 PM Jacob Barhak <jacob.barhak at gmail.com> wrote:

> Greetings group members,
>
> Hopefully this thread caught your attention and you plan to submit
> feedback to the FDA.
> Here is the FDA link again:
>
> https://www.fda.gov/regulatory-information/search-fda-guidance-documents/assessing-credibility-computational-modeling-and-simulation-medical-device-submissions
>
> I do hope we hold a discussion to generate more impactful ideas.
>
> Regardless, I plan to submit a response based on the text I sent and our
> paper in mid March.
>
> I do hope many of you will read the document and then their thoughts to
> this mailing list as well.
>
>          Jacob
>
>
>
> On Sun, Jan 30, 2022, 11:51 Jacob Barhak <jacob.barhak at gmail.com> wrote:
>
>> Greetings group members,
>>
>> The purpose of this email is to start a discussion on the FDA draft:
>> Assessing the Credibility of Computational Modeling and Simulation in
>> Medical Device Submissions.
>>
>> I went back to the paper draft and looked at it. The draft does a good
>> job on focusing on procedures and steps to assess credibility. Especially
>> important is the work flw suggested. From those perspectives it's a good
>> document that may help others in the future.
>>
>> One basic idea that we also discussed in the paper was the "question of
>> interest" - Early on John Rice introduced this concept into our paper and
>> others supported it later. However, even in our paper there were different
>> opinions on how to approach credibility and reuse models defined for
>> different questions in mind. The design process may start from a different
>> question than the one that we wish to satisfy. Since this document is a
>> draft from a regulatory body, it is less about the design process and more
>> about compliance. However, our discussion in the paper may be still
>> relevant since it introduced the concept of comparison among models.
>>
>> I looked through the draft and could not see an element of relativity ,
>> or in other words, a reference to compare to. And this concept may be
>> important in the future since the modeling field evolves quickly,
>> especially with regards to machine learning techniques that evolve rapidly.
>> A model that seems credible today and passes some tests may not be very
>> good in a few years. Also one model may not perform as good as a
>> combination of models. Yet this model, although outdated may outperform
>> human decision - therefore I believe there should be 2 elements considered
>> by the FDA that include a reference / relative performance.
>>
>> 1. Group of models - how does a model perform compared to existing models
>> with regards to satisfying the question of interest. Models will eventually
>> become outdated and although  models may be sufficient to answer a
>> question, within time some models may not be as good as others. And this
>> time period may be relatively short. So the regulator has to consider
>> outdated models and not only set a threshold for compliance. So to keep
>> compliance over time a model has to outperform existing models.  This idea
>> can be extended to allow models to cooperate in an ensemble, yet the first
>> step is benchmark models taking into account other models as a relative
>> reference.
>>
>> 2. Relative to human practice - the model also has to be compared with
>> current human standards without  the model. A device with a model may not
>> be accurate, yet human accuracy may be even worse without that model to
>> aid. Moreover, humans may not agree with the same evidence produced. To
>> give an example, please look at this paper:
>> https://arxiv.org/abs/1806.02121  . In this paper the authors test their
>> model against humans and show that humans test to agree with their model
>> more than they agree with each other. So the FDA should keep in mind that
>> humans that are certified may not perform well and the lower threshold for
>> compliance should take this into consideration.
>>
>> I am writing those ideas in hope this will start a discussion where we
>> can formulate and combine answers by group members and submit them
>> together. Otherwise I will submit a version of this text myself.
>>
>> I hope there will be interest in providing this feedback.
>>
>>                   Jacob
>>
>>
>>
>> On Thu, Jan 27, 2022 at 9:37 PM Jacob Barhak <jacob.barhak at gmail.com>
>> wrote:
>>
>>> Greetings reproducibility and integration groups,
>>>
>>> John Rice has sent this message. about the FDA document:
>>>
>>> https://www.fda.gov/regulatory-information/search-fda-guidance-documents/assessing-credibility-computational-modeling-and-simulation-medical-device-submissions
>>>
>>> And indeed this is relevant to our work - in fact this was my suggestion
>>> for the next activity.
>>>
>>> FDA has released a request for comments for a document drafted. We
>>> actually referred to this document in our paper.
>>>
>>> I was planning to start a discussion on this paper so we can
>>> submit responses, either individually or as a group or sub-groups after we
>>> finish with the paper.
>>>
>>> Yet since the paper is almost done and time is running short, it is
>>> perhaps a good idea to discuss this now.
>>>
>>> Looking forward to your feedback.
>>>
>>>                Jacob
>>>
>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.simtk.org/pipermail/vp-reproduce-subgroup/attachments/20220227/f91887ca/attachment.html>