[Vp-integration-subgroup] [Vp-reproduce-subgroup] Paper review and revision check list
James Glazier
jaglazier at gmail.com
Fri Jan 14 09:04:50 PST 2022
APPROVE
On Fri, Jan 14, 2022 at 11:02 AM Yaling Liu <yal310 at lehigh.edu> wrote:
> I also APPROVE
>
> Thanks for all the efforts!
>
> Yaling
>
> On Fri, Jan 14, 2022 at 1:38 AM Jacob Barhak <jacob.barhak at gmail.com>
> wrote:
>
>> Thanks Marcella,
>>
>> In that case, it seems we have no reviewer objection and hopefully the
>> editor will be positive as well . To speed up the process we need
>> the following::
>> 1. Sheriff - please send a high resolution image for Figure 1
>> 2. Alex - Please send a high resolution image for Figure 2
>> 3. Everyone please review the manuscript and Reply to all in this email
>> with one word - either APPROVE, DISAPPROVE, ABSTAIN
>>
>> Where:
>>
>> APPROVE means you are ok with the version we reached - please note that
>> we do have some new elements there and your approval means you understand
>> what it means - please read the paper well and understand the nuances of
>> approval.
>>
>> DISAPPROVE- means that you no longer agree with the version we reached
>> and we need to discuss the paper more and withdraw it - if you have some
>> serious objection, it is the last chance to get the paper back to the
>> drawing board
>>
>> ABSTAIN - You have no objection for the paper being published, yet you
>> want your name withdrawn
>>
>> Also, if anyone else has any objections for this paper being published,
>> please raise them now so we can discuss them.
>>
>> I call again to John Gennarri to join us as an author - he did
>> substantial work and many of his contributions are in the final version -
>> including the order in figure 2. I can trace back many of his contributions
>> and I think he should get credit for it - hopefully my last call will be
>> accepted and he will rejoin.
>>
>> William - I did not forget your second affiliation - I just could not
>> change it in the system - I will write to the editor about that when we get
>> the final acceptance notice.
>>
>> I look forward to seeing your final decisions - hopefully it will be fast
>> this time.
>>
>> Jacob
>>
>>
>> On Thu, Jan 13, 2022 at 8:50 PM Torres, Marcella <mtorres at richmond.edu>
>> wrote:
>>
>>> Sorry, I was not clear enough: no response is required for Reviewer 2,
>>> as they have endorsed the manuscript. A response to Reviewer 1 is overdue,
>>> however; see the email below.
>>>
>>> Frontiers in Systems Biology Editorial Office has sent you a message.
>>> Please click 'Reply' to send a direct response
>>>
>>> This is to remind you that your response to the reviewer 1 is overdue in
>>> the interactive review forum of your manuscript 822606 under consideration
>>> for publication in Frontiers in Systems Biology, section Multiscale
>>> Mechanistic Modeling.
>>> To access the review forum, click:
>>>
>>> http://www.frontiersin.org/Review/EnterReviewForum.aspx?activationno=5f3876ba-0de4-47c1-8dff-1c2bcb5c6b11&retab=1
>>> <http://links.email.frontiersin.org/ls/click?upn=99YMbkkQ8WNJiHLectmHSn5XnTmYe57aPeiziTpYNtyfClFbQqFlwPq2nUMZBzQ7srQvkLkPbc-2F-2BZbZ-2BczlOsMnoBdURYE7vBE6wrLrCJR4-2BCXMD-2FyylaGS-2FaxCSwEXxntm7RnNdDdmDL5k3W2WQo0ie3jVFLbWE6g0wuBJBzfQ-3DOLrv_B-2Fy9jbSVGoT8qhSwbw0AFtm6DlDsGMQJjIadnFC13-2FJyATqJzBD73VYfuDTc-2F-2BJaS4kVj6eazmIcCMUijXaF7cwmpIPfXo62oyUmieGoN2C1Y5Or-2Fqw1mDFet-2FbgSMXcJvyRgL8VJvJK-2Bw-2BULW-2FawzD0QPdwLV1hkLM4-2F9kG-2BJqcbrOIV4DgfEtFEs9z2qt9wTumRZRHdyMGfQyetT9cia1k0sDolZ7ukB5Ik6E4DTmj3yHv4voIEhFHR5lhm-2Bbs>
>>>
>>> Should you require a deadline extension for your resubmission, please do
>>> let us know by selecting "Request Extension" in the review forum, if you
>>> have not done so already. We encourage you to submit your revised
>>> manuscript with tracked changes to facilitate the review.
>>>
>>> We look forward to receiving your revised manuscript. For any issues,
>>> please immediately contact the Associate Editor as well as the Editorial
>>> Office.
>>>
>>> Best regards,
>>>
>>> Your Frontiers in Systems Biology team
>>>
>>> Frontiers in Systems Biology Editorial Office
>>> www.frontiersin.org
>>> Avenue du Tribunal Federal 34
>>> Lausanne, Switzerland | T 41(0)21 510 17 93
>>>
>>> For technical issues, please contact our IT Helpdesk -
>>> support at frontiersin.org
>>>
>>>
>>>
>>>
>>> Marcella Torres, Ph.D.
>>> Director of Mathematical Studies
>>> University of Richmond
>>> Jepson Hall Room 212
>>> 221 Richmond Way
>>> Richmond, VA 23173
>>> (804) 289-8081
>>> Pronouns: she/her
>>>
>>> ------------------------------
>>> *From:* Jacob Barhak <jacob.barhak at gmail.com>
>>> *Sent:* Thursday, January 13, 2022 8:14 PM
>>> *To:* Torres, Marcella <mtorres at richmond.edu>
>>> *Cc:* Rahuman Sheriff <sheriff at ebi.ac.uk>;
>>> vp-reproduce-subgroup at lists.simtk.org <
>>> vp-reproduce-subgroup at lists.simtk.org>;
>>> vp-integration-subgroup at lists.simtk.org <
>>> vp-integration-subgroup at lists.simtk.org>; John Gennari <gennari at uw.edu>;
>>> Faeder, James R <faeder at pitt.edu>
>>> *Subject:* Re: [Vp-reproduce-subgroup] [Vp-integration-subgroup] Paper
>>> review and revision check list
>>>
>>> *External Email:* Use caution in opening links, attachments, and buying
>>> gift cards.
>>>
>>> Thanks Marcella, Thanks Gilberto,
>>>
>>> Since the second reviewer did not respond, I think we have more time -
>>> we did respond in time however.
>>>
>>> The figures were provided by Alex and Sheriff - they seem pretty good
>>> resolution to me, yet if we want increased resolution, they will have to
>>> generate those again from the same software that generated thm originally -
>>> Basically the images have to be at least 300 dpi and export it as pdf and
>>> extract it as tiff or eps - not jpg.
>>>
>>> Section 2.2.2 in the instruction provides information about the image
>>> quality:
>>> https://www.frontiersin.org/about/author-guidelines
>>>
>>> Hopefully we can have those ready.
>>>
>>> However, I do ask that all authors be ready to approve the paper
>>> assuming the other reviewer will be positive - we worked on thai for a year
>>> and there were many discussions - before this paper gets published I want
>>> all authors to be good with the final outcome. We will do this formally
>>> like we did before - yet I want everyone to be ready to avoid any delays.
>>> Hopefully it will take us less than 3 weeks this time.
>>>
>>> Jacob
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Jan 13, 2022 at 1:05 PM Torres, Marcella <mtorres at richmond.edu>
>>> wrote:
>>>
>>> Hi again all,
>>>
>>> On the 7th I shared comments from Reviewer 1 that we are now late
>>> responding to. Is it possible to provide high resolution figures, or no
>>> (see below)?
>>>
>>> Reviewer 2 has endorsed the manuscript.
>>>
>>>
>>> Reviewer 1 comment:
>>> *Reviewer 1* | 07 Jan 2022 | 14:36
>>> #3
>>>
>>> I would like to thank the authors for having fully addressed my
>>> comments. I'm satisfied about the quality the manuscript has been reached
>>> in this latest version. Finally, I would only suggest the authors to
>>> provide an high quality resolution of both new figures. Thank you.
>>>
>>>
>>> Marcella Torres, Ph.D.
>>> Director of Mathematical Studies
>>> University of Richmond
>>> Jepson Hall Room 212
>>> 221 Richmond Way
>>> Richmond, VA 23173
>>> (804) 289-8081
>>> Pronouns: she/her
>>>
>>> ------------------------------
>>> *From:* Vp-reproduce-subgroup <
>>> vp-reproduce-subgroup-bounces at lists.simtk.org> on behalf of Rahuman
>>> Sheriff <sheriff at ebi.ac.uk>
>>> *Sent:* Monday, January 10, 2022 3:09 PM
>>> *To:* Jacob Barhak <jacob.barhak at gmail.com>
>>> *Cc:* vp-reproduce-subgroup at lists.simtk.org <
>>> vp-reproduce-subgroup at lists.simtk.org>;
>>> vp-integration-subgroup at lists.simtk.org <
>>> vp-integration-subgroup at lists.simtk.org>; John Gennari <gennari at uw.edu>;
>>> Faeder, James R <faeder at pitt.edu>
>>> *Subject:* Re: [Vp-reproduce-subgroup] [Vp-integration-subgroup] Paper
>>> review and revision check list
>>>
>>> *External Email:* Use caution in opening links, attachments, and buying
>>> gift cards.
>>>
>>> Thanks Jakob for submitting the revision and others for the support.
>>> Thanks William for editing my response.
>>>
>>> Best
>>> Sheriff
>>>
>>>
>>> On 7 Jan 2022, at 04:52, Jacob Barhak <jacob.barhak at gmail.com> wrote:
>>>
>>> Thanks Hana, Thanks William, and thanks to Gilberto, Alex, and Sheriff,
>>>
>>> With your help and some effort, the revised paper was resubmitted in
>>> time - just now.
>>>
>>> William, please keep on reminding me about your second affiliation - I
>>> kept a comment as a reminder to do this - the submission system did
>>> not allow making any changes in Author information - not sure why - if this
>>> gets accepted we should ask the editor to add your affiliation manually.
>>>
>>> I urge all other authors to look at the changes we made and our
>>> response. We will have to approve the manuscript with everyone before it
>>> gets published - hopefully we will convince the reviewers this review round.
>>>
>>> Jacob
>>>
>>> On Thu, Jan 6, 2022 at 3:51 PM Dobrovolny, Hana <h.dobrovolny at tcu.edu>
>>> wrote:
>>>
>>> I've read through the paper and fixed a few grammar mistakes.
>>>
>>> Hana
>>>
>>>
>>> *******************************************************
>>> Dr. Hana Dobrovolny
>>> Associate Professor of Biophysics
>>> Texas Christian University
>>> TCU Box 298840
>>> Fort Worth, TX 76129
>>>
>>> phone: (817) 257-6379 fax: (817) 257-7742
>>> email: h.dobrovolny at tcu.edu
>>> *******************************************************
>>>
>>>
>>> ------------------------------
>>> *From:* Vp-integration-subgroup <
>>> vp-integration-subgroup-bounces at lists.simtk.org> on behalf of Jacob
>>> Barhak <jacob.barhak at gmail.com>
>>> *Sent:* January 5, 2022 11:48 PM
>>> *To:* William Waites; James Glazier
>>> *Cc:* vp-reproduce-subgroup at lists.simtk.org;
>>> vp-integration-subgroup at lists.simtk.org; John Gennari; Faeder, James R
>>> *Subject:* Re: [Vp-integration-subgroup] [Vp-reproduce-subgroup] Paper
>>> review and revision check list
>>>
>>>
>>> *[EXTERNAL EMAIL WARNING]* DO NOT CLICK LINKS or open attachments
>>> unless you recognize the sender and know the content is safe.
>>> Greetings paper contributors,
>>>
>>> Gilberto has addressed the final point and added text in the paper and
>>> review, thus concluding our response.
>>>
>>> I went over the paper from start to finish again and cleaned it up.
>>>
>>> There are two last comments in the paper addressed to William Waites and
>>> James Glazier - if you can quickly check those, it would help.
>>>
>>> If someone else wants to read the paper once more, please do this today
>>> - otherwise I will convert the paper and our response to reviewers and
>>> submit it to the Journal portal.
>>>
>>> If anyone has any objections, now is the time to raise them so we can
>>> withdraw rather than submit.
>>>
>>> Here is the link to the revised paper:
>>>
>>> https://docs.google.com/document/d/1IMEgmdNkx-EsnOjGuegpenSIMmKIkK00Lc8Gred3QxM/edit?usp=sharing
>>> <https://urldefense.com/v3/__https://docs.google.com/document/d/1IMEgmdNkx-EsnOjGuegpenSIMmKIkK00Lc8Gred3QxM/edit?usp=sharing__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0hvK3OmJ8$>
>>>
>>> Here is the link to response for reviewers:
>>>
>>> https://docs.google.com/document/d/1bO1CDbMMlyyweYayP9rGohPIfgvIhfe14sFchtJMJ-s/edit?usp=sharing
>>> <https://urldefense.com/v3/__https://docs.google.com/document/d/1bO1CDbMMlyyweYayP9rGohPIfgvIhfe14sFchtJMJ-s/edit?usp=sharing__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0haRkeB0o$>
>>>
>>>
>>> Hopefully there will be no objections and can submit this tomorrow.
>>>
>>> Jacob
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Jan 4, 2022 at 11:30 PM Jacob Barhak <jacob.barhak at gmail.com>
>>> wrote:
>>>
>>> Hi Alex, Hi William, Hi Gilberto, and all other contributors.
>>>
>>> Alex, William and Gilberto are mentioned since we are the most active
>>> contributors.
>>>
>>> We have 2 days to finish edits.
>>>
>>> William - I attempted to add your suggestions to the paper
>>> regarding issue 11. You are free to revise further.
>>>
>>> Gilberto - I merged some of your texts, yet it seems you are the only
>>> one I saw interested in resolving issue 12 where the reviewer asks to
>>> remove the "Model Application and Implementation Barriers" section. Please
>>> suggest your corrections as soon as possible. If you cannot do this by
>>> tomorrow I will do my best to add a response based on your idea. Yet I
>>> really hope you can do this before me.
>>>
>>> Please finalize any edits you have - and if anyone has any objection to
>>> the text, please raise it within one day since after tomorrow I will start
>>> closing down all issues.
>>>
>>> I would ask for a volunteer that will go over the paper once more after
>>> its finalized and check for typos and grammar - we do not want the paper to
>>> return again due to this issue. Time is tight, so please volunteer fast.
>>>
>>> Hopefully we can get it done in time.
>>>
>>> Jacob
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Jan 4, 2022 at 7:25 AM Alexander Kulesza <
>>> alexander.kulesza at novadiscovery.com> wrote:
>>>
>>> Dear Jacob, dear all,
>>> thanks for endorsing my figure. I'll gladly add this. Please find
>>> enclosed the .png.
>>>
>>> What you say makes sense and I agree that exaggerating changes is not
>>> speeding up approval. Please still consider it as a tool to perform "major"
>>> change (ordering of sections is a major change of a paper in my opinion)
>>> without changing too much the sections as such. But I agree that this could
>>> be tried out should the reviewers insist.
>>>
>>> I therefore reverted the table change (disconnecting ist from Figure 1)
>>> and suggest the following Figure caption:
>>>
>>> "Figure 2: Sketch of difficulties persistent that impede
>>> reproducibility, credibility, utility and integration of models, especially
>>> in computational biology. An assignment of these difficulties to the
>>> four different concepts interpreting them as hurdles is attempted. We would
>>> like to point out that this sequential assessment model indicated in the
>>> graph is only one of the possibilities a modeler would assess the
>>> suitability of available models. In the rest of the paper we therefore
>>> address each difficulty separately. "
>>>
>>> With that sentence we can discuss the difficulties in any order.
>>>
>>> Best
>>> Alexander
>>>
>>> On Tue, 4 Jan 2022 at 13:24, Jacob Barhak <jacob.barhak at gmail.com>
>>> wrote:
>>>
>>> Yes Alex,
>>>
>>> Your image is much better than mine, please use it instead in the paper
>>> and send a link to the image itself in an acceptable image format so I can
>>> upload it to the submission system later on.
>>>
>>> As for changing the table and order, I am not sure. Although it may be
>>> possible. We need to think about it. Yet we really are crunched in time, so
>>> I am leaning to say no.
>>>
>>> First, because some of the difficulties span more than one category, for
>>> example unit standardization affects integration as well as
>>> reproducibility, only it appears first at reproducibility, therefore it was
>>> placed there in the figure.
>>>
>>> Second, if you look at the text there are some transition sentences
>>> among sections, that connect them, so swapping those around may just mess
>>> things. I am not sure it if will also mess up acronym definitions on the
>>> first appearance as well, so shuffling order is not something's I would do
>>> unless necessary. And it was not requested by reviewers anyway.
>>>
>>> I am also trying to minimize changes and focus only on what was
>>> requested rather than change a lot more. Otherwise we may never finish
>>> changing things since each one of the 18 of us will have a one last item
>>> they MUST have corrected. I personally did not add some recent publications
>>> I had, although I added one publication James asked to add in the previous
>>> approval round and corrected one that Sheriff mentioned. Yet I really want
>>> to avoid opening the paper to significant changes.
>>>
>>> If you start opening corrections beyond what was requested, this will
>>> drag on forever since we do have a large list of contributors and each one
>>> of us will generate a new idea. So please focus on what was asked unless
>>> absolutely necessary.
>>>
>>> So I suggest not changing the table, and changing only the figure and
>>> keeping order intact. I hope I am projecting the sense of urgency - we had
>>> the paper lying around for a couple of weeks during the holiday without
>>> much progress, I ask not to escalate things just because a deadline is
>>> looming.
>>>
>>> If you have extra time on your hands in the next couple of days, I
>>> suggest you focus on addressing the last two points in the response that we
>>> have not resolved yet. I know Gilberto wanted to fix one of them, yet you
>>> are welcome to do it. Those are things requested by the review
>>>
>>> Hopefully my arguments make sense.
>>>
>>> Jacob
>>>
>>> On Tue, Jan 4, 2022, 06:45 Alexander Kulesza <
>>> alexander.kulesza at novadiscovery.com> wrote:
>>>
>>> Dear all,
>>>
>>> I have thought about the additional Figure requested by the reviewer.
>>> Jacob your Figure nicely shows the assignment of the difficulties we detail
>>> in our central table with the 4 big concepts.
>>>
>>> I feel however that the text-heavy figure is a lot o redundancy with the
>>> rest of the elements (existing figure & table). Therefore I wanted to throw
>>> another suggestion forward (inspired by systematic reviews narrowing down
>>> eligible studies/data).
>>>
>>> @all please notice that the assignment I would do is a bit different to
>>> what is currently done in the paper. The order of sections would have to be
>>> altered. Not sure if that is acceptable.
>>> <image.png>
>>> The table with Difficulties (hurdles) and solutions would then be
>>>
>>> Difficulty
>>> Potential Solutions
>>>
>>> Reproducibility
>>> Models are written in different languages
>>> Common transport specifications such as SBML or CellML, and proper
>>> documentation and annotation
>>> Models are hard to locate
>>> Archive web sites such as: BioModels, SimTK, IMAGWiki, and the future
>>> modeleXchange
>>> Lack of common platforms for executing models and simulations
>>> Platforms such as BioSimulators, and runBioSimulations
>>> Unit standardization
>>> Standardization efforts, and machine learning solutions such as
>>> ClinicalUnitMapping.com
>>> Credibility
>>> Models have built-in barriers to Evaluating model credibility
>>> Better modeling practices, documentation, and tests.
>>> Data availability and measurement definitions
>>> Models that merge human interpretation, and newer measurement devices
>>>
>>> Missing annotations in models
>>> Adoption of policies such as those COMBINE suggests
>>> Utility
>>> Models are not consistently licensed to allow for reuse
>>> Abandoning some old school open source licenses and promoting licenses
>>> that release to public domain
>>> Different scales and modeling paradigms
>>> Standardization effort and centralization tools
>>> Stochastic modeling difficulties
>>> Development of tools that guarantee repeatability and standards to
>>> address stochastic simulations
>>> Integration
>>> Model application and implementation barriers
>>> Education of modelers, users, and the public
>>> Modeling requires adaptation towards integration
>>> Tools for composing models such as SBML-Comp, and SemGen
>>>
>>> Following this assignment.
>>>
>>> Please let me know what you think. I have added this suggestion in track
>>> changes mode to the revision draft.
>>>
>>> Best
>>> Alexander
>>>
>>>
>>> On Mon, 3 Jan 2022 at 07:27, Jacob Barhak <jacob.barhak at gmail.com>
>>> wrote:
>>>
>>> Thanks Alex,
>>>
>>> You did great by adding the new material that the reviewers actually
>>> asked for. This is great - I found myself adding the new FDA document to
>>> find out that you already added it - I just voice read it ona long drive
>>> and figured it fits well - it seems you were faster - great.
>>>
>>> I merged your edits and I am ok with adding your response to the
>>> reviewers - I only had an issue with one example paragraph that you deleted
>>> - I left it with a comment - please check if it is possible to keep this
>>> example somewhere in the text.
>>>
>>> I will assemble a letter that includes all those responses point by
>>> point and will try to use your text almost verbatim if possible. You really
>>> bring important knowledge with you.
>>>
>>> It seems most points have been answered to some level - even a figure
>>> was created - although I wish I was a better artist.
>>>
>>> I will try to assemble the response letter for everyone within the next
>>> few days. And unless there will be objections I will try to pass it by
>>> reviewers with a new version. If the new version is accepted, we will do a
>>> final approval round before publication.
>>>
>>> Many thanks to all those who contributed.
>>>
>>> Jacob
>>>
>>>
>>>
>>>
>>>
>>> On Sun, Jan 2, 2022 at 7:27 AM Alexander Kulesza <
>>> alexander.kulesza at novadiscovery.com> wrote:
>>>
>>> Dear all,
>>>
>>> Happy new year to everyone. I hope that you are all well and safe and
>>> have spent a nice holiday season.
>>>
>>> I have tried to address
>>>
>>> *Reviewer 2 *
>>>
>>>
>>> *Moreover, when the authors mention ASME V&V40 procedure, it could be
>>> useful to spend few words about the model credibility pipeline drawn for
>>> predictive models in biomedicine, such as the one suggested in "Credibility
>>> of in Silico Trial Technologies-A Theoretical Framing", Viceconti, M.,
>>> Juarez, M.A., Curreli, C., ...Russo, G., Pappalardo, F., IEEE Journal of
>>> Biomedical and Health Informatics, 2020, 24(1), pp. 4–13, 8884189. For this
>>> specific aspect, the authors should talk about "specific question of
>>> interest" and not "specific purpose" in the paragraph called "Credibility
>>> of Models" to follow in such a way a standardised language. *
>>>
>>> I have worked on the followng document in "track changes" mode:
>>>
>>> https://docs.google.com/document/d/1IMEgmdNkx-EsnOjGuegpenSIMmKIkK00Lc8Gred3QxM/edit
>>> <https://urldefense.com/v3/__https://docs.google.com/document/d/1IMEgmdNkx-EsnOjGuegpenSIMmKIkK00Lc8Gred3QxM/edit__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0hrtFQ-XI$>
>>> Actually, I ended up in changing rather drastically the section. I think
>>> that all ideas and all text is conserved, but I would very much like to
>>> encourage you to read, comment and challenge this revision. Please tell me
>>> if that goes to far. No problem to revert to an earlier version if
>>> necessary.
>>>
>>> I suggest the following formulation in the rebuttal letter:
>>>
>>> When trying to better describe the "credibility pipeline" (CoU
>>> definition, verification, validation, uncertainty quantification, e.g. as
>>> described in Viceconti et al. 2019) or others, we noticed that the section
>>> about credibility could be restructured in order to put the reader (not
>>> necessarily familiar with regulatory assessment of models) in the position
>>> to follow the argumentation.
>>>
>>> We now roughly follow the following flow of writing:
>>>
>>> - Every model needs a purpose and credibility is tied to that purpose
>>> as well as repeatability and reproducibility
>>> - Credibility is essential for models that have impact on regulated
>>> areas or people's lives which is why regulatory authorities issue guidance
>>> - The probably most advanced guidance ASME V&V40 suggests a risk-based
>>> approach and pipeline to establish credibility of a model. It is
>>> overarching and widely applicable
>>> - Regulators and modelers work together (for example in frame of the
>>> model informed drug development program MIDD) in order to better
>>> understand, better apply and better uptake different kinds of models in
>>> regulatory applications
>>> - More work is needed to harmonize, stay up to date and to be more
>>> inclusive
>>>
>>> We hope that with the rather extensive change of the wording and
>>> additional passages as well as citations (see below) we could address the
>>> concern of reviewer 2 (and the general remarks of reviewer 1).
>>>
>>> " In 2018 the American Society of Mechanical Engineers (ASME) issued an
>>> important guidance ASME V&V 40 (ASME, 2018) of how to assess credibility of
>>> computational models of medical devices through verification and validation
>>> (V&V). The guideline is centered around the definition of the context of
>>> use (CoU) of the model which is formulated based on the questions of
>>> interest the model will answer. The CoU is then analyzed in terms of the
>>> “model risk” - being the influence the model exerts on a decision and the
>>> potential consequences these decisions might incur. Commensurate with this
>>> model risk the modeler suggests establish the credibility goals, perform
>>> verification validation and uncertainty quantification actions and then
>>> assess the outcome of this exercise in order to allow judging the
>>> acceptability of the model CoU. Key to this guidance is its overarching
>>> nature which also allows adoption in other (e.g. drug development) fields
>>> irrespective of the model type (Kuemmel 2020, Viceconti 2019)."
>>>
>>> We also feel that it is essential to underline the cross-discipline
>>> viewpoint of the ASME V&V40 which is further elaborated by the cited paper
>>> by formalizing the verification, validation and uncertainty quantification
>>> VVUQ pipeline across model types.
>>>
>>> We therefore have added the following statement
>>> " In the paper by (Viceconti 2019) the verification, validation and
>>> uncertainty quantification (VVUQ) pipeline is streamlined to different
>>> types of models. It is, perhaps, the closest to score credibility across
>>> model types from mechanistic physics driven models to machine learning
>>> models. However, it is still short of including very recent developments
>>> such as ensemble models, although it touches upon the topic."
>>>
>>> We additionally strengthened our argumentation by citing additional
>>> literature
>>>
>>> Bai, JPF, Earp, JC, Florian, J, et al. Quantitative systems
>>> pharmacology: Landscape analysis of regulatory submissions to the US Food
>>> and Drug Administration. CPT Pharmacometrics Syst Pharmacol. 2021; 10:
>>> 1479– 1484. doi.org:10.1002/psp4.12709
>>>
>>> FDA (2021b). Assessing the Credibility of Computational Modeling and
>>> Simulation in Medical Device Submissions. FDA
>>> Available at:
>>> https://www.fda.gov/regulatory-information/search-fda-guidance-documents/assessing-credibility-computational-modeling-and-simulation-medical-device-submissions
>>> <https://urldefense.com/v3/__https://www.fda.gov/regulatory-information/search-fda-guidance-documents/assessing-credibility-computational-modeling-and-simulation-medical-device-submissions__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0harP7IHc$>
>>> [Accessed January 2, 2022]
>>>
>>> Galluppi, G.R., Brar, S., Caro, L., Chen, Y., Frey, N., Grimm, H.P.,
>>> Rudd, D.J., Li, C.-C., Magee, M., Mukherjee, A., Nagao, L., Purohit, V.S.,
>>> Roy, A., Salem, A.H., Sinha, V., Suleiman, A.A., Taskar, K.S., Upreti,
>>> V.V., Weber, B. and Cook, J. (2021), Industrial Perspective on the Benefits
>>> Realized From the FDA’s Model-Informed Drug Development Paired Meeting
>>> Pilot Program. Clin Pharmacol Ther, 110: 1172-1175. doi:10.1002/cpt.2265
>>>
>>> Kuemmel, C., Yang, Y., Zhang, X., Florian, J., Zhu, H., Tegenge, M.,
>>> Huang, S.-M., Wang, Y., Morrison, T. and Zineh, I. (2020), Consideration
>>> of a Credibility Assessment Framework in Model-Informed Drug Development:
>>> Potential Application to Physiologically-Based Pharmacokinetic Modeling and
>>> Simulation. CPT Pharmacometrics Syst. Pharmacol., 9: 21-28. doi.org
>>> :10.1002/psp4.12479
>>>
>>> Musuamba, FT, Skottheim Rusten, I, Lesage, R, et al. Scientific and
>>> regulatory evaluation of mechanistic in silico drug and disease models in
>>> drug development: Building model credibility. CPT Pharmacometrics Syst
>>> Pharmacol. 2021; 10: 804– 825. doi:10.1002/psp4.12669
>>>
>>> Viceconti M, Emili L, Afshari P, et al. Possible Contexts of Use for In
>>> Silico Trials Methodologies: A Consensus-Based Review. IEEE J Biomed
>>> Health Inform. 2021;25(10):3977-3982. doi:10.1109/JBHI.2021.3090469
>>>
>>>
>>> All the best
>>> Alexander
>>>
>>> On Sun, 19 Dec 2021 at 03:10, Jacob Barhak <jacob.barhak at gmail.com>
>>> wrote:
>>>
>>>
>>> Greetings to the paper contributors,
>>>
>>> You may have seen my other message where I posted the reviews to our
>>> paper. I am sending those again below to start a new thread discussing
>>> possible revisions.
>>>
>>> Here is a checklist for revisions required:
>>>
>>>
>>> 1. We should discuss the references that reviewer 1 raised and add
>>> them as reference
>>> 2. Reviewer 1 asked for more discussion around V&V40 - I
>>> believe Alex added the text there originally - Alex do you think you can
>>> address the request of the reviewer?
>>> 3. Reviewer 1 asked to focus on COVID-19 at the title - I am not
>>> sure we wish to limit our scope since many of our ideas are applicable far
>>> beyond COVID-19 - I leave it up for discussion in the group on how to
>>> address this request by the reviewer
>>> 4. Reviewer 1 asked for additional graphics - I am unsure how to
>>> address this beyond ur current diagram - ideas will help
>>> 5. We need to check the "Contribution to the field" section for
>>> grammar and spelling - if someone can contribute more elegant text - now is
>>> the time - I looked over it now and found no issues - yet I may have missed
>>> something - this section may have been missed since it was added last and
>>> perhaps vetted less than other sections. Also Reviewer 2 asked for grammar
>>> corrections, so it is worthwhile proof reading the paper as a whole.
>>> 6. Reviewer 2 asks for major rewrite to emphasize why the problems
>>> are there - I think we should explain that the core of the paper is the
>>> table and perhaps emphasize it in the paper beyond what the current text -
>>> we also should reply to the reviewer and explain that the paper is composed
>>> of contributions from a large group and we made an effort to include every
>>> voice in the choir - each of those voices is important and needs to be
>>> preserved - hopefully it will convince the reviewer that change we will add
>>> will be sufficient
>>> 7. Reviewer 2 asks to revise the introduction - I believe some
>>> changes are possible - yet the introduction includes contributions from
>>> many authors - at least 10 - and I fear losing something important someone
>>> contributed - if someone has an idea on how to address this reviewer
>>> without a painful transformation, please reply to this message.
>>> 8. Reviewer 2 wants additional discussion around the
>>> "reproducibility crisis" section and asks a valid question about
>>> expectations we should address - "“Computational biomedical modeling… was
>>> expected to be less affected by the reproducibility crisis.” Is this true,
>>> and why would it be so? " - My answer to the reviewer is that unlike
>>> biological processes that have random nature and experiments
>>> would not repeat if repeated, while computer software should be
>>> deterministic and it should be repeatable if designed well. Unfortunately
>>> we are not experiencing this promise - yet I believe the reviewer wants
>>> more discussion beyond this section so I am happy to discuss this.
>>> 9. Reviewer 2- writes: "Models are Hard to Locate: Are the authors
>>> suggesting that entire simulation workflows, from model construction to
>>> analysis, should be publicly available? At what point does one consider
>>> intellectual property? Do the authors advocate for such extensive
>>> publishing for all models, or only ones that are intended to be widely
>>> re-used? - those are important points we need to discuss - we need to
>>> better explain the difficulties we are having when creating models and the
>>> reviewer is absolutely correct about expanding the discussion to IP - I
>>> suggest we create a thread for this discussion and reference it - I
>>> suggest it we merged with the licensing issue that I will address later -
>>> there is a strong connection there - the reviewer was very observant and
>>> sees the bigger problem. However,we need to eventually distill our
>>> discussion to recommendations that will inline.
>>> 10. Reviewer 2 asks that we fix the unit standardization section - I
>>> believe Hana and myself were the largest contributors there - Hana - I will
>>> start a discussion on that topic in a separate email where we can publicly
>>> discuss how to fix this - others will be welcome to contribute.
>>> 11. Reviewer 2 asks that we better handle the section on Data
>>> availability and measurement definitions: in think we need to emphasize
>>> solutions and separate it from issues that may not be solvable. Ideas are
>>> welcome.
>>> 12. Reviewer 2 asks good questions with regards to licensing
>>> following our text- I personally have good answers to the reviewer and
>>> William and I had some discussion on the topic in this list I suggest we
>>> expand this discussion in a separate thread - hopefully William and perhaps
>>> others will join the discussion. This discussion should also address the IP
>>> issues raised by the reviewer for the "models are hard to locate section."
>>> 13. Reviewer 2 asks to handle the Model Application and
>>> Implementation Barriers section. We should decide what to do there, the
>>> section may need expansion since the ideas there are solid, yet the section
>>> is short so perhaps enhancing it makes better sense. I am open to
>>> suggestions.
>>>
>>>
>>> Those are the items I located and my suggestions. It seems we need
>>> attention from Alex, Hana, myself and william. However, anyone on the list
>>> is welcome to participate and suggest changes.
>>>
>>> I will start the discussion threads on specific topics. Hopefully we can
>>> get it done quickly.
>>>
>>> Jacob
>>>
>>>
>>>
>>>
>>>
>>>
>>> -
>>>
>>>
>>>
>>>
>>>
>>> ########### Original Reviews #############
>>>
>>> There are 2 reviews - both require major changes. I am copying the
>>> relevant text below. If more appear, I will let you know, yet I only got
>>> this message today although the reviews are dated a few days ago.
>>>
>>> Reviewer 1:
>>> Recommendation for the Editor: Substantial revision is required
>>>
>>> Please list your revision requests for the authors and provide your
>>> detailed comments, including highlighting limitations and strengths of the
>>> review. If you have additional comments based on Q2 and Q3 you can add them
>>> as well.
>>>
>>> Karr and co-authors made an interesting and exaustive point about the
>>> reproducibility crisis that leads to inability to reuse and integrate
>>> models, especially about COVID-19 disease.
>>> Within the manuscript some typos and missing information are present.
>>> I'd suggest the authors to revise the entire manuscript especially in
>>> terms of the state of the art, revising and updating the most relevant
>>> examples of computational models dealing with COVID-19 and in general about
>>> some semi-standardised proposals about the pipeline to follow for the
>>> verification and validation of model credibility. In particular, the
>>> authors failed to mention and cite some major results on in silico modeling
>>> about COVID-19 up to now. See for example:
>>> a."In silico trial to test COVID-19 candidate vaccines: a case study
>>> with UISS platform", Russo, G., Pennisi, M., Fichera, E., ...Viceconti, M.,
>>> Pappalardo, F., BMC Bioinformatics, 2020, 21, 527.
>>> b. Russo G, Di Salvatore V, Sgroi G, Parasiliti Palumbo GA, Reche PA,
>>> Pappalardo F. "A multi-step and multi-scale bioinformatic protocol to
>>> investigate potential SARS-CoV-2 vaccine targets" [published online ahead
>>> of print, 2021 Oct 5]. Brief Bioinform. 2021;bbab403.
>>> doi:10.1093/bib/bbab403.
>>> Moreover, when the authors mention ASME V&V40 procedure, it could be
>>> useful to spend few words about the model credibility pipeline drawn for
>>> predictive models in biomedicine, such as the one suggested in "Credibility
>>> of in Silico Trial Technologies-A Theoretical Framing", Viceconti, M.,
>>> Juarez, M.A., Curreli, C., ...Russo, G., Pappalardo, F., IEEE Journal of
>>> Biomedical and Health Informatics, 2020, 24(1), pp. 4–13, 8884189.
>>> For this specific aspect, the authors should talk about "specific
>>> question of interest" and not "specific purpose" in the paragraph called
>>> "Credibility of Models" to follow in such a way a standardised language.
>>> Moreover, the authors should refer to COVID-19 also in the title, with a
>>> specific mention about the fact that the main topic of model integration in
>>> computational biology will be discussed inside the COVID-19 context.
>>> Furthermore, for the model key concepts such as
>>> Reusability-Extensibility-Extractability-Portability the authors should
>>> described and outlined through a graphical sketch or visual representation
>>> that summarises these key point.
>>> The authors should also fix some grammar and writing typos present in
>>> the "Contribution to the field" section.
>>>
>>> a. Is the quality of the figures and tables satisfactory?
>>> - No
>>>
>>> b. Does the reference list cover the relevant literature adequately and
>>> in an unbiased manner?
>>> - No
>>>
>>> c. Does this manuscript refer only to published data? (unpublished or
>>> original data is not allowed for this article type)
>>> - Yes
>>>
>>> d. Does the review include a balanced, comprehensive, and critical view
>>> of the research area?
>>> - Yes
>>>
>>>
>>> Reviewer 2
>>>
>>> Please list your revision requests for the authors and provide your
>>> detailed comments, including highlighting limitations and strengths of the
>>> review. If you have additional comments based on Q2 and Q3 you can add them
>>> as well.
>>>
>>>
>>> The article often reads as a stream-of-consciousness account of the
>>> discussions that took place but lacks a clear thesis or recommendations. It
>>> is not clear what, if anything, the authors are advocating for. It is not
>>> always clear why the issues being discussed are problematic, or that they
>>> can be reasonably addressed. Some of the issues raised are indeed important
>>> and should be discussed, but the paper lacks focus and does not tell a
>>> cohesive story. I believe this manuscript requires a major re-write to be
>>> suitable for publication. The authors should consider narrowing the scope
>>> of the discussion and focusing on a cohesive set of recommendations or open
>>> questions. More specifically, I make a few suggestions below:
>>>
>>> Major comments
>>> 1. The introduction is long and repeats itself (e.g., “much less is
>>> known about how viral infections spread throughout the body…” is repeated
>>> verbatim). It is not clear from the introduction what the main goal of the
>>> paper is or why the “reproducibility crisis” is truly a crisis. Why is the
>>> discussion of composition and black/white box models relevant to the
>>> introduction? Further, this section is subtitled “the promise of modeling”,
>>> which does not seem to match the content.
>>> 2. The Reproducibility Crisis: “Computational biomedical modeling… was
>>> expected to be less affected by the reproducibility crisis.” Is this true,
>>> and why would it be so? One would think that more complex models would
>>> suffer more from a lack of reproducibility. It may be helpful to define
>>> what exactly the “reproducibility crisis” refers to.
>>> 3. Models are Hard to Locate: Are the authors suggesting that entire
>>> simulation workflows, from model construction to analysis, should be
>>> publicly available? At what point does one consider intellectual property?
>>> Do the authors advocate for such extensive publishing for all models, or
>>> only ones that are intended to be widely re-used?
>>> 4. Unit standardization: The conversion from PFU or TCID50 to individual
>>> virions is likely to differ across viruses – are the authors focused on
>>> COVID here? Are the authors advocating for a standard conversion factor? It
>>> is not clear what the purpose of this discussion is. As the authors
>>> mention, different scales require different units. Even at a single scale,
>>> different models may require different units for numerical reasons. It is
>>> not clear what the authors are advocating for here.
>>> 5. Data availability and measurement definitions: This section seems to
>>> outline limitations of available data, but again makes no recommendations
>>> or proposed solution to any of the issues raised. Is this the intention?
>>> Most of the issues raised here reflect limitations of experimental science
>>> or data privacy, which likely cannot be meaningfully addressed by the
>>> modeling community.
>>> 6. Models are Not Consistently Licensed…: Are the authors implying here
>>> that all modeling work should be published with no rights reserved? Is it
>>> reasonable to expect modelers to make their work freely usable by others
>>> for profit? Is it reasonable for institutions to allow this? How much does
>>> this really contribute to reproducibility and utility?
>>> 7. Model Application and Implementation Barriers: This section seems
>>> unnecessary and out of place.
>>> 8. There are grammar and punctuation errors scattered throughout; please
>>> edit carefully.
>>>
>>>
>>> a. Is the quality of the figures and tables satisfactory?
>>> - Not Applicable
>>>
>>> b. Does the reference list cover the relevant literature adequately and
>>> in an unbiased manner?
>>> - Yes
>>>
>>> c. Does this manuscript refer only to published data? (unpublished or
>>> original data is not allowed for this article type)
>>> - Yes
>>>
>>> d. Does the review include a balanced, comprehensive, and critical view
>>> of the research area?
>>> No answer given.
>>>
>>>
>>> ========
>>> _______________________________________________
>>> Vp-reproduce-subgroup mailing list
>>> Vp-reproduce-subgroup at lists.simtk.org
>>> https://lists.simtk.org/mailman/listinfo/vp-reproduce-subgroup
>>> <https://urldefense.com/v3/__https://lists.simtk.org/mailman/listinfo/vp-reproduce-subgroup__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0hhiovw3M$>
>>>
>>>
>>>
>>> --
>>> Alexander Kulesza
>>> Team leader
>>> Modeling & simulation / Biomodeling
>>> alexander.kulesza at novadiscovery.com
>>> +33 7 82 92 44 62
>>> nova
>>> DISCOVERY
>>> www.novadiscovery.com
>>> <https://urldefense.com/v3/__http://www.novadiscovery.com/__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0hwI_2O8g$>
>>> 1 Place Verrazzano, 69009 Lyon
>>> +33 9 72 53 13 01
>>>
>>> *This message is intended only for the personal and confidential use of
>>> the designated recipient(s) named above. If you are not the intended
>>> recipient of this message you are hereby notified that any review,
>>> dissemination, distribution or copying of this message is strictly
>>> prohibited. Email transmission cannot be guaranteed to be secure or
>>> error-free. Therefore, we do not represent that this information is
>>> complete or accurate and it should not be relied upon as such. All
>>> information is subject to change without notice.*
>>>
>>>
>>>
>>> --
>>> Alexander Kulesza
>>> Team leader
>>> Modeling & simulation / Biomodeling
>>> alexander.kulesza at novadiscovery.com
>>> +33 7 82 92 44 62
>>> nova
>>> DISCOVERY
>>> www.novadiscovery.com
>>> <https://urldefense.com/v3/__http://www.novadiscovery.com/__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0hwI_2O8g$>
>>> 1 Place Verrazzano, 69009 Lyon
>>> +33 9 72 53 13 01
>>>
>>> *This message is intended only for the personal and confidential use of
>>> the designated recipient(s) named above. If you are not the intended
>>> recipient of this message you are hereby notified that any review,
>>> dissemination, distribution or copying of this message is strictly
>>> prohibited. Email transmission cannot be guaranteed to be secure or
>>> error-free. Therefore, we do not represent that this information is
>>> complete or accurate and it should not be relied upon as such. All
>>> information is subject to change without notice.*
>>>
>>>
>>>
>>> --
>>> Alexander Kulesza
>>> Team leader
>>> Modeling & simulation / Biomodeling
>>> alexander.kulesza at novadiscovery.com
>>> +33 7 82 92 44 62
>>> nova
>>> DISCOVERY
>>> www.novadiscovery.com
>>> <https://urldefense.com/v3/__http://www.novadiscovery.com/__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0hwI_2O8g$>
>>> 1 Place Verrazzano, 69009 Lyon
>>> +33 9 72 53 13 01
>>>
>>> *This message is intended only for the personal and confidential use of
>>> the designated recipient(s) named above. If you are not the intended
>>> recipient of this message you are hereby notified that any review,
>>> dissemination, distribution or copying of this message is strictly
>>> prohibited. Email transmission cannot be guaranteed to be secure or
>>> error-free. Therefore, we do not represent that this information is
>>> complete or accurate and it should not be relied upon as such. All
>>> information is subject to change without notice.*
>>>
>>> _______________________________________________
>>> Vp-integration-subgroup mailing list
>>> Vp-integration-subgroup at lists.simtk.org
>>> https://lists.simtk.org/mailman/listinfo/vp-integration-subgroup
>>>
>>>
>>> _______________________________________________
>> Vp-integration-subgroup mailing list
>> Vp-integration-subgroup at lists.simtk.org
>> https://lists.simtk.org/mailman/listinfo/vp-integration-subgroup
>>
>
>
> --
> -- ------------------
> Yaling Liu, Professor
> Department of Bioengineering, Director of Graduate Studies
> Department of Mechanical Engineering and Mechanics
> Lehigh University
> www.lehigh.edu/~yal310
> _______________________________________________
> Vp-reproduce-subgroup mailing list
> Vp-reproduce-subgroup at lists.simtk.org
> https://lists.simtk.org/mailman/listinfo/vp-reproduce-subgroup
>
--
James A. Glazier, PhD
Professor of Intelligent Systems Engineering, Adjunct Professor of Physics
Director Biocomplexity Institute
Indiana University, Bloomington
(812) 391-2159
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.simtk.org/pipermail/vp-integration-subgroup/attachments/20220114/40c56a74/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 28718 bytes
Desc: not available
URL: <https://lists.simtk.org/pipermail/vp-integration-subgroup/attachments/20220114/40c56a74/attachment-0001.png>
More information about the Vp-integration-subgroup
mailing list