[Vp-reproduce-subgroup] Some key points to remember Re: Update: Draft Guidance: Assessing Credibility of Computational Modeling and Simulation

[John Rice]

1. This draft as did the current Guidelines came from the medical device side of FDA.

2. Although this is a U.S. FDA draft, the FDA and European Commission are working very closely together and will probably have very similar guidelines for model credibility on the medical DEVICE regulation processes.

3. Both FDA and EC are also working on the Pharmacology side. But it has very different issues.

     A. Note the parenthetical remark in the introduction to this draft regarding its applicability to PHYSICS BASED and more specifically FIRST PRINCIPLES based physics models, and specifically not data based models.   However, within a couple paragraphs of that delimiter, they talk about the use of the device model with human models to asses how the device effects the human.   ????  

Our life science MSM / Mechanistic models, as far as I know, lack life science “First Principles”.   Not at all a new discussions in our community and something frequently heard from life science modelers in a context of bio modeling being way more complex then physics modeling.  Point here is how to address credibility of a first principle physics model of a device/material that gets used to see how it works in or effects a model of a, much less many different  humans.  

   B.  Are there things that can be done now to significantly improve confidence in life science models?  It’s the question that crosses “state of the art” for testing insilco devices in insilco humans and the future of insilco testing or even study of effects of pharma products.

As usual I have lots of questions but few suggested solutions.

John 



 

On Jan 30, 2022, at 11:51, Jacob Barhak <jacob.barhak at gmail.com> wrote:


Greetings group members,

The purpose of this email is to start a discussion on the FDA draft: Assessing the Credibility of Computational Modeling and Simulation in Medical Device Submissions.

I went back to the paper draft and looked at it. The draft does a good job on focusing on procedures and steps to assess credibility. Especially important is the work flw suggested. From those perspectives it's a good document that may help others in the future.

One basic idea that we also discussed in the paper was the "question of interest" - Early on John Rice introduced this concept into our paper and others supported it later. However, even in our paper there were different opinions on how to approach credibility and reuse models defined for different questions in mind. The design process may start from a different question than the one that we wish to satisfy. Since this document is a draft from a regulatory body, it is less about the design process and more about compliance. However, our discussion in the paper may be still relevant since it introduced the concept of comparison among models. 

I looked through the draft and could not see an element of relativity , or in other words, a reference to compare to. And this concept may be important in the future since the modeling field evolves quickly, especially with regards to machine learning techniques that evolve rapidly. A model that seems credible today and passes some tests may not be very good in a few years. Also one model may not perform as good as a combination of models. Yet this model, although outdated may outperform human decision - therefore I believe there should be 2 elements considered by the FDA that include a reference / relative performance. 

1. Group of models - how does a model perform compared to existing models with regards to satisfying the question of interest. Models will eventually become outdated and although  models may be sufficient to answer a question, within time some models may not be as good as others. And this time period may be relatively short. So the regulator has to consider outdated models and not only set a threshold for compliance. So to keep compliance over time a model has to outperform existing models.  This idea can be extended to allow models to cooperate in an ensemble, yet the first step is benchmark models taking into account other models as a relative reference.

2. Relative to human practice - the model also has to be compared with current human standards without  the model. A device with a model may not be accurate, yet human accuracy may be even worse without that model to aid. Moreover, humans may not agree with the same evidence produced. To give an example, please look at this paper: https://arxiv.org/abs/1806.02121  . In this paper the authors test their model against humans and show that humans test to agree with their model more than they agree with each other. So the FDA should keep in mind that humans that are certified may not perform well and the lower threshold for compliance should take this into consideration. 

I am writing those ideas in hope this will start a discussion where we can formulate and combine answers by group members and submit them together. Otherwise I will submit a version of this text myself. 

I hope there will be interest in providing this feedback.

                  Jacob



On Thu, Jan 27, 2022 at 9:37 PM Jacob Barhak <jacob.barhak at gmail.com> wrote:
> Greetings reproducibility and integration groups,
> 
> John Rice has sent this message. about the FDA document:
> https://www.fda.gov/regulatory-information/search-fda-guidance-documents/assessing-credibility-computational-modeling-and-simulation-medical-device-submissions
> 
> And indeed this is relevant to our work - in fact this was my suggestion for the next activity. 
> 
> FDA has released a request for comments for a document drafted. We actually referred to this document in our paper. 
> 
> I was planning to start a discussion on this paper so we can submit responses, either individually or as a group or sub-groups after we finish with the paper.
> 
> Yet since the paper is almost done and time is running short, it is perhaps a good idea to discuss this now.
> 
> Looking forward to your feedback.
> 
>                Jacob
> 
> 
_______________________________________________
Vp-reproduce-subgroup mailing list
Vp-reproduce-subgroup at lists.simtk.org
https://lists.simtk.org/mailman/listinfo/vp-reproduce-subgroup
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.simtk.org/pipermail/vp-reproduce-subgroup/attachments/20220130/c8db9e50/attachment-0001.html>