[Vp-integration-subgroup] [Vp-reproduce-subgroup] Paper review and revision check list
Alexander Kulesza
alexander.kulesza at novadiscovery.com
Tue Jan 4 03:45:13 PST 2022
Dear all,
I have thought about the additional Figure requested by the reviewer. Jacob
your Figure nicely shows the assignment of the difficulties we detail in
our central table with the 4 big concepts.
I feel however that the text-heavy figure is a lot o redundancy with the
rest of the elements (existing figure & table). Therefore I wanted to throw
another suggestion forward (inspired by systematic reviews narrowing down
eligible studies/data).
@all please notice that the assignment I would do is a bit different to
what is currently done in the paper. The order of sections would have to be
altered. Not sure if that is acceptable.
[image: image.png]
The table with Difficulties (hurdles) and solutions would then be
Difficulty
Potential Solutions
Reproducibility
Models are written in different languages
Common transport specifications such as SBML or CellML, and proper
documentation and annotation
Models are hard to locate
Archive web sites such as: BioModels, SimTK, IMAGWiki, and the future
modeleXchange
Lack of common platforms for executing models and simulations
Platforms such as BioSimulators, and runBioSimulations
Unit standardization
Standardization efforts, and machine learning solutions such as
ClinicalUnitMapping.com
Credibility
Models have built-in barriers to Evaluating model credibility
Better modeling practices, documentation, and tests.
Data availability and measurement definitions
Models that merge human interpretation, and newer measurement devices
Missing annotations in models
Adoption of policies such as those COMBINE suggests
Utility
Models are not consistently licensed to allow for reuse
Abandoning some old school open source licenses and promoting licenses that
release to public domain
Different scales and modeling paradigms
Standardization effort and centralization tools
Stochastic modeling difficulties
Development of tools that guarantee repeatability and standards to address
stochastic simulations
Integration
Model application and implementation barriers
Education of modelers, users, and the public
Modeling requires adaptation towards integration
Tools for composing models such as SBML-Comp, and SemGen
Following this assignment.
Please let me know what you think. I have added this suggestion in track
changes mode to the revision draft.
Best
Alexander
On Mon, 3 Jan 2022 at 07:27, Jacob Barhak <jacob.barhak at gmail.com> wrote:
> Thanks Alex,
>
> You did great by adding the new material that the reviewers actually asked
> for. This is great - I found myself adding the new FDA document to find out
> that you already added it - I just voice read it ona long drive and
> figured it fits well - it seems you were faster - great.
>
> I merged your edits and I am ok with adding your response to the reviewers
> - I only had an issue with one example paragraph that you deleted - I left
> it with a comment - please check if it is possible to keep this example
> somewhere in the text.
>
> I will assemble a letter that includes all those responses point by point
> and will try to use your text almost verbatim if possible. You really bring
> important knowledge with you.
>
> It seems most points have been answered to some level - even a figure was
> created - although I wish I was a better artist.
>
> I will try to assemble the response letter for everyone within the next
> few days. And unless there will be objections I will try to pass it by
> reviewers with a new version. If the new version is accepted, we will do a
> final approval round before publication.
>
> Many thanks to all those who contributed.
>
> Jacob
>
>
>
>
>
> On Sun, Jan 2, 2022 at 7:27 AM Alexander Kulesza <
> alexander.kulesza at novadiscovery.com> wrote:
>
>> Dear all,
>>
>> Happy new year to everyone. I hope that you are all well and safe and
>> have spent a nice holiday season.
>>
>> I have tried to address
>>
>> *Reviewer 2 *
>>
>>
>> *Moreover, when the authors mention ASME V&V40 procedure, it could be
>> useful to spend few words about the model credibility pipeline drawn for
>> predictive models in biomedicine, such as the one suggested in "Credibility
>> of in Silico Trial Technologies-A Theoretical Framing", Viceconti, M.,
>> Juarez, M.A., Curreli, C., ...Russo, G., Pappalardo, F., IEEE Journal of
>> Biomedical and Health Informatics, 2020, 24(1), pp. 4–13, 8884189.For this
>> specific aspect, the authors should talk about "specific question of
>> interest" and not "specific purpose" in the paragraph called "Credibility
>> of Models" to follow in such a way a standardised language.*
>>
>> I have worked on the followng document in "track changes" mode:
>>
>> https://docs.google.com/document/d/1IMEgmdNkx-EsnOjGuegpenSIMmKIkK00Lc8Gred3QxM/edit
>> Actually, I ended up in changing rather drastically the section. I think
>> that all ideas and all text is conserved, but I would very much like to
>> encourage you to read, comment and challenge this revision. Please tell me
>> if that goes to far. No problem to revert to an earlier version if
>> necessary.
>>
>> I suggest the following formulation in the rebuttal letter:
>>
>> When trying to better describe the "credibility pipeline" (CoU
>> definition, verification, validation, uncertainty quantification, e.g. as
>> described in Viceconti et al. 2019) or others, we noticed that the section
>> about credibility could be restructured in order to put the reader (not
>> necessarily familiar with regulatory assessment of models) in the position
>> to follow the argumentation.
>>
>> We now roughly follow the following flow of writing:
>>
>> - Every model needs a purpose and credibility is tied to that purpose as
>> well as repeatability and reproducibility
>> - Credibility is essential for models that have impact on regulated
>> areas or people's lives which is why regulatory authorities issue guidance
>> - The probably most advanced guidance ASME V&V40 suggests a risk-based
>> approach and pipeline to establish credibility of a model. It is
>> overarching and widely applicable
>> - Regulators and modelers work together (for example in frame of the
>> model informed drug development program MIDD) in order to better
>> understand, better apply and better uptake different kinds of models in
>> regulatory applications
>> - More work is needed to harmonize, stay up to date and to be more
>> inclusive
>>
>> We hope that with the rather extensive change of the wording and
>> additional passages as well as citations (see below) we could address the
>> concern of reviewer 2 (and the general remarks of reviewer 1).
>>
>> " In 2018 the American Society of Mechanical Engineers (ASME) issued an
>> important guidance ASME V&V 40 (ASME, 2018) of how to assess credibility of
>> computational models of medical devices through verification and validation
>> (V&V). The guideline is centered around the definition of the context of
>> use (CoU) of the model which is formulated based on the questions of
>> interest the model will answer. The CoU is then analyzed in terms of the
>> “model risk” - being the influence the model exerts on a decision and the
>> potential consequences these decisions might incur. Commensurate with this
>> model risk the modeler suggests establish the credibility goals, perform
>> verification validation and uncertainty quantification actions and then
>> assess the outcome of this exercise in order to allow judging the
>> acceptability of the model CoU. Key to this guidance is its overarching
>> nature which also allows adoption in other (e.g. drug development) fields
>> irrespective of the model type (Kuemmel 2020, Viceconti 2019)."
>>
>> We also feel that it is essential to underline the cross-discipline
>> viewpoint of the ASME V&V40 which is further elaborated by the cited paper
>> by formalizing the verification, validation and uncertainty quantification
>> VVUQ pipeline across model types.
>>
>> We therefore have added the following statement
>> " In the paper by (Viceconti 2019) the verification, validation and
>> uncertainty quantification (VVUQ) pipeline is streamlined to different
>> types of models. It is, perhaps, the closest to score credibility across
>> model types from mechanistic physics driven models to machine learning
>> models. However, it is still short of including very recent developments
>> such as ensemble models, although it touches upon the topic."
>>
>> We additionally strengthened our argumentation by citing additional
>> literature
>>
>> Bai, JPF, Earp, JC, Florian, J, et al. Quantitative systems
>> pharmacology: Landscape analysis of regulatory submissions to the US Food
>> and Drug Administration. CPT Pharmacometrics Syst Pharmacol. 2021; 10:
>> 1479– 1484. doi.org:10.1002/psp4.12709
>>
>> FDA (2021b). Assessing the Credibility of Computational Modeling and
>> Simulation in Medical Device Submissions. FDA
>> Available at:
>> https://www.fda.gov/regulatory-information/search-fda-guidance-documents/assessing-credibility-computational-modeling-and-simulation-medical-device-submissions
>> [Accessed January 2, 2022]
>>
>> Galluppi, G.R., Brar, S., Caro, L., Chen, Y., Frey, N., Grimm, H.P.,
>> Rudd, D.J., Li, C.-C., Magee, M., Mukherjee, A., Nagao, L., Purohit, V.S.,
>> Roy, A., Salem, A.H., Sinha, V., Suleiman, A.A., Taskar, K.S., Upreti,
>> V.V., Weber, B. and Cook, J. (2021), Industrial Perspective on the Benefits
>> Realized From the FDA’s Model-Informed Drug Development Paired Meeting
>> Pilot Program. Clin Pharmacol Ther, 110: 1172-1175. doi:10.1002/cpt.2265
>>
>> Kuemmel, C., Yang, Y., Zhang, X., Florian, J., Zhu, H., Tegenge, M.,
>> Huang, S.-M., Wang, Y., Morrison, T. and Zineh, I. (2020), Consideration
>> of a Credibility Assessment Framework in Model-Informed Drug Development:
>> Potential Application to Physiologically-Based Pharmacokinetic Modeling and
>> Simulation. CPT Pharmacometrics Syst. Pharmacol., 9: 21-28. doi.org:10
>> .1002/psp4.12479
>>
>> Musuamba, FT, Skottheim Rusten, I, Lesage, R, et al. Scientific and
>> regulatory evaluation of mechanistic in silico drug and disease models in
>> drug development: Building model credibility. CPT Pharmacometrics Syst
>> Pharmacol. 2021; 10: 804– 825. doi:10.1002/psp4.12669
>>
>> Viceconti M, Emili L, Afshari P, et al. Possible Contexts of Use for In
>> Silico Trials Methodologies: A Consensus-Based Review. IEEE J Biomed
>> Health Inform. 2021;25(10):3977-3982. doi:10.1109/JBHI.2021.3090469
>>
>>
>> All the best
>> Alexander
>>
>> On Sun, 19 Dec 2021 at 03:10, Jacob Barhak <jacob.barhak at gmail.com>
>> wrote:
>>
>>>
>>> Greetings to the paper contributors,
>>>
>>> You may have seen my other message where I posted the reviews to our
>>> paper. I am sending those again below to start a new thread discussing
>>> possible revisions.
>>>
>>> Here is a checklist for revisions required:
>>>
>>>
>>> 1. We should discuss the references that reviewer 1 raised and add
>>> them as reference
>>> 2. Reviewer 1 asked for more discussion around V&V40 - I
>>> believe Alex added the text there originally - Alex do you think you can
>>> address the request of the reviewer?
>>> 3. Reviewer 1 asked to focus on COVID-19 at the title - I am not
>>> sure we wish to limit our scope since many of our ideas are applicable far
>>> beyond COVID-19 - I leave it up for discussion in the group on how to
>>> address this request by the reviewer
>>> 4. Reviewer 1 asked for additional graphics - I am unsure how to
>>> address this beyond ur current diagram - ideas will help
>>> 5. We need to check the "Contribution to the field" section for
>>> grammar and spelling - if someone can contribute more elegant text - now is
>>> the time - I looked over it now and found no issues - yet I may have missed
>>> something - this section may have been missed since it was added last and
>>> perhaps vetted less than other sections. Also Reviewer 2 asked for grammar
>>> corrections, so it is worthwhile proof reading the paper as a whole.
>>> 6. Reviewer 2 asks for major rewrite to emphasize why the problems
>>> are there - I think we should explain that the core of the paper is the
>>> table and perhaps emphasize it in the paper beyond what the current text -
>>> we also should reply to the reviewer and explain that the paper is composed
>>> of contributions from a large group and we made an effort to include every
>>> voice in the choir - each of those voices is important and needs to be
>>> preserved - hopefully it will convince the reviewer that change we will add
>>> will be sufficient
>>> 7. Reviewer 2 asks to revise the introduction - I believe some
>>> changes are possible - yet the introduction includes contributions from
>>> many authors - at least 10 - and I fear losing something important someone
>>> contributed - if someone has an idea on how to address this reviewer
>>> without a painful transformation, please reply to this message.
>>> 8. Reviewer 2 wants additional discussion around the
>>> "reproducibility crisis" section and asks a valid question about
>>> expectations we should address - "“Computational biomedical modeling… was
>>> expected to be less affected by the reproducibility crisis.” Is this true,
>>> and why would it be so? " - My answer to the reviewer is that unlike
>>> biological processes that have random nature and experiments
>>> would not repeat if repeated, while computer software should be
>>> deterministic and it should be repeatable if designed well. Unfortunately
>>> we are not experiencing this promise - yet I believe the reviewer wants
>>> more discussion beyond this section so I am happy to discuss this.
>>> 9. Reviewer 2- writes: "Models are Hard to Locate: Are the authors
>>> suggesting that entire simulation workflows, from model construction to
>>> analysis, should be publicly available? At what point does one consider
>>> intellectual property? Do the authors advocate for such extensive
>>> publishing for all models, or only ones that are intended to be widely
>>> re-used? - those are important points we need to discuss - we need to
>>> better explain the difficulties we are having when creating models and the
>>> reviewer is absolutely correct about expanding the discussion to IP - I
>>> suggest we create a thread for this discussion and reference it - I
>>> suggest it we merged with the licensing issue that I will address later -
>>> there is a strong connection there - the reviewer was very observant and
>>> sees the bigger problem. However,we need to eventually distill our
>>> discussion to recommendations that will inline.
>>> 10. Reviewer 2 asks that we fix the unit standardization section - I
>>> believe Hana and myself were the largest contributors there - Hana - I will
>>> start a discussion on that topic in a separate email where we can publicly
>>> discuss how to fix this - others will be welcome to contribute.
>>> 11. Reviewer 2 asks that we better handle the section on Data
>>> availability and measurement definitions: in think we need to emphasize
>>> solutions and separate it from issues that may not be solvable. Ideas are
>>> welcome.
>>> 12. Reviewer 2 asks good questions with regards to licensing
>>> following our text- I personally have good answers to the reviewer and
>>> William and I had some discussion on the topic in this list I suggest we
>>> expand this discussion in a separate thread - hopefully William and perhaps
>>> others will join the discussion. This discussion should also address the IP
>>> issues raised by the reviewer for the "models are hard to locate section."
>>> 13. Reviewer 2 asks to handle the Model Application and
>>> Implementation Barriers section. We should decide what to do there, the
>>> section may need expansion since the ideas there are solid, yet the section
>>> is short so perhaps enhancing it makes better sense. I am open to
>>> suggestions.
>>>
>>>
>>> Those are the items I located and my suggestions. It seems we need
>>> attention from Alex, Hana, myself and william. However, anyone on the list
>>> is welcome to participate and suggest changes.
>>>
>>> I will start the discussion threads on specific topics. Hopefully we can
>>> get it done quickly.
>>>
>>> Jacob
>>>
>>>
>>>
>>>
>>>
>>>
>>> -
>>>
>>>
>>>
>>>
>>>
>>> ########### Original Reviews #############
>>>
>>> There are 2 reviews - both require major changes. I am copying the
>>> relevant text below. If more appear, I will let you know, yet I only got
>>> this message today although the reviews are dated a few days ago.
>>>
>>> Reviewer 1:
>>> Recommendation for the Editor: Substantial revision is required
>>>
>>> Please list your revision requests for the authors and provide your
>>> detailed comments, including highlighting limitations and strengths of the
>>> review. If you have additional comments based on Q2 and Q3 you can add them
>>> as well.
>>>
>>> Karr and co-authors made an interesting and exaustive point about the
>>> reproducibility crisis that leads to inability to reuse and integrate
>>> models, especially about COVID-19 disease.
>>> Within the manuscript some typos and missing information are present.
>>> I'd suggest the authors to revise the entire manuscript especially in
>>> terms of the state of the art, revising and updating the most relevant
>>> examples of computational models dealing with COVID-19 and in general about
>>> some semi-standardised proposals about the pipeline to follow for the
>>> verification and validation of model credibility. In particular, the
>>> authors failed to mention and cite some major results on in silico modeling
>>> about COVID-19 up to now. See for example:
>>> a."In silico trial to test COVID-19 candidate vaccines: a case study
>>> with UISS platform", Russo, G., Pennisi, M., Fichera, E., ...Viceconti, M.,
>>> Pappalardo, F., BMC Bioinformatics, 2020, 21, 527.
>>> b. Russo G, Di Salvatore V, Sgroi G, Parasiliti Palumbo GA, Reche PA,
>>> Pappalardo F. "A multi-step and multi-scale bioinformatic protocol to
>>> investigate potential SARS-CoV-2 vaccine targets" [published online ahead
>>> of print, 2021 Oct 5]. Brief Bioinform. 2021;bbab403.
>>> doi:10.1093/bib/bbab403.
>>> Moreover, when the authors mention ASME V&V40 procedure, it could be
>>> useful to spend few words about the model credibility pipeline drawn for
>>> predictive models in biomedicine, such as the one suggested in "Credibility
>>> of in Silico Trial Technologies-A Theoretical Framing", Viceconti, M.,
>>> Juarez, M.A., Curreli, C., ...Russo, G., Pappalardo, F., IEEE Journal of
>>> Biomedical and Health Informatics, 2020, 24(1), pp. 4–13, 8884189.
>>> For this specific aspect, the authors should talk about "specific
>>> question of interest" and not "specific purpose" in the paragraph called
>>> "Credibility of Models" to follow in such a way a standardised language.
>>> Moreover, the authors should refer to COVID-19 also in the title, with a
>>> specific mention about the fact that the main topic of model integration in
>>> computational biology will be discussed inside the COVID-19 context.
>>> Furthermore, for the model key concepts such as
>>> Reusability-Extensibility-Extractability-Portability the authors should
>>> described and outlined through a graphical sketch or visual representation
>>> that summarises these key point.
>>> The authors should also fix some grammar and writing typos present in
>>> the "Contribution to the field" section.
>>>
>>> a. Is the quality of the figures and tables satisfactory?
>>> - No
>>>
>>> b. Does the reference list cover the relevant literature adequately and
>>> in an unbiased manner?
>>> - No
>>>
>>> c. Does this manuscript refer only to published data? (unpublished or
>>> original data is not allowed for this article type)
>>> - Yes
>>>
>>> d. Does the review include a balanced, comprehensive, and critical view
>>> of the research area?
>>> - Yes
>>>
>>>
>>> Reviewer 2
>>>
>>> Please list your revision requests for the authors and provide your
>>> detailed comments, including highlighting limitations and strengths of the
>>> review. If you have additional comments based on Q2 and Q3 you can add them
>>> as well.
>>>
>>>
>>> The article often reads as a stream-of-consciousness account of the
>>> discussions that took place but lacks a clear thesis or recommendations. It
>>> is not clear what, if anything, the authors are advocating for. It is not
>>> always clear why the issues being discussed are problematic, or that they
>>> can be reasonably addressed. Some of the issues raised are indeed important
>>> and should be discussed, but the paper lacks focus and does not tell a
>>> cohesive story. I believe this manuscript requires a major re-write to be
>>> suitable for publication. The authors should consider narrowing the scope
>>> of the discussion and focusing on a cohesive set of recommendations or open
>>> questions. More specifically, I make a few suggestions below:
>>>
>>> Major comments
>>> 1. The introduction is long and repeats itself (e.g., “much less is
>>> known about how viral infections spread throughout the body…” is repeated
>>> verbatim). It is not clear from the introduction what the main goal of the
>>> paper is or why the “reproducibility crisis” is truly a crisis. Why is the
>>> discussion of composition and black/white box models relevant to the
>>> introduction? Further, this section is subtitled “the promise of modeling”,
>>> which does not seem to match the content.
>>> 2. The Reproducibility Crisis: “Computational biomedical modeling… was
>>> expected to be less affected by the reproducibility crisis.” Is this true,
>>> and why would it be so? One would think that more complex models would
>>> suffer more from a lack of reproducibility. It may be helpful to define
>>> what exactly the “reproducibility crisis” refers to.
>>> 3. Models are Hard to Locate: Are the authors suggesting that entire
>>> simulation workflows, from model construction to analysis, should be
>>> publicly available? At what point does one consider intellectual property?
>>> Do the authors advocate for such extensive publishing for all models, or
>>> only ones that are intended to be widely re-used?
>>> 4. Unit standardization: The conversion from PFU or TCID50 to individual
>>> virions is likely to differ across viruses – are the authors focused on
>>> COVID here? Are the authors advocating for a standard conversion factor? It
>>> is not clear what the purpose of this discussion is. As the authors
>>> mention, different scales require different units. Even at a single scale,
>>> different models may require different units for numerical reasons. It is
>>> not clear what the authors are advocating for here.
>>> 5. Data availability and measurement definitions: This section seems to
>>> outline limitations of available data, but again makes no recommendations
>>> or proposed solution to any of the issues raised. Is this the intention?
>>> Most of the issues raised here reflect limitations of experimental science
>>> or data privacy, which likely cannot be meaningfully addressed by the
>>> modeling community.
>>> 6. Models are Not Consistently Licensed…: Are the authors implying here
>>> that all modeling work should be published with no rights reserved? Is it
>>> reasonable to expect modelers to make their work freely usable by others
>>> for profit? Is it reasonable for institutions to allow this? How much does
>>> this really contribute to reproducibility and utility?
>>> 7. Model Application and Implementation Barriers: This section seems
>>> unnecessary and out of place.
>>> 8. There are grammar and punctuation errors scattered throughout; please
>>> edit carefully.
>>>
>>>
>>> a. Is the quality of the figures and tables satisfactory?
>>> - Not Applicable
>>>
>>> b. Does the reference list cover the relevant literature adequately and
>>> in an unbiased manner?
>>> - Yes
>>>
>>> c. Does this manuscript refer only to published data? (unpublished or
>>> original data is not allowed for this article type)
>>> - Yes
>>>
>>> d. Does the review include a balanced, comprehensive, and critical view
>>> of the research area?
>>> No answer given.
>>>
>>>
>>> ========
>>> _______________________________________________
>>> Vp-reproduce-subgroup mailing list
>>> Vp-reproduce-subgroup at lists.simtk.org
>>> https://lists.simtk.org/mailman/listinfo/vp-reproduce-subgroup
>>>
>>
>>
>> --
>> Alexander Kulesza
>> Team leader
>> Modeling & simulation / Biomodeling
>> alexander.kulesza at novadiscovery.com
>> +33 7 82 92 44 62
>> nova
>> DISCOVERY
>> www.novadiscovery.com
>> 1 Place Verrazzano, 69009 Lyon
>> +33 9 72 53 13 01
>>
>> *This message is intended only for the personal and confidential use of
>> the designated recipient(s) named above. If you are not the intended
>> recipient of this message you are hereby notified that any review,
>> dissemination, distribution or copying of this message is strictly
>> prohibited. Email transmission cannot be guaranteed to be secure or
>> error-free. Therefore, we do not represent that this information is
>> complete or accurate and it should not be relied upon as such. All
>> information is subject to change without notice.*
>
>
--
Alexander Kulesza
Team leader
Modeling & simulation / Biomodeling
alexander.kulesza at novadiscovery.com
+33 7 82 92 44 62
nova
DISCOVERY
www.novadiscovery.com
1 Place Verrazzano, 69009 Lyon
+33 9 72 53 13 01
--
_This message is intended only for the personal and confidential use of the
designated recipient(s) named above. If you are not the intended recipient
of this message you are hereby notified that any review, dissemination,
distribution or copying of this message is strictly prohibited. Email
transmission cannot be guaranteed to be secure or error-free. Therefore, we
do not represent that this information is complete or accurate and it
should not be relied upon as such. All information is subject to change
without notice._
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.simtk.org/pipermail/vp-integration-subgroup/attachments/20220104/75927623/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 144605 bytes
Desc: not available
URL: <https://lists.simtk.org/pipermail/vp-integration-subgroup/attachments/20220104/75927623/attachment-0001.png>
More information about the Vp-integration-subgroup
mailing list