[Vp-reproduce-subgroup] [Vp-integration-subgroup] Paper review and revision check list
Yaling Liu
yal310 at lehigh.edu
Fri Jan 14 08:01:42 PST 2022
I also APPROVE
Thanks for all the efforts!
Yaling
On Fri, Jan 14, 2022 at 1:38 AM Jacob Barhak <jacob.barhak at gmail.com> wrote:
> Thanks Marcella,
>
> In that case, it seems we have no reviewer objection and hopefully the
> editor will be positive as well . To speed up the process we need
> the following::
> 1. Sheriff - please send a high resolution image for Figure 1
> 2. Alex - Please send a high resolution image for Figure 2
> 3. Everyone please review the manuscript and Reply to all in this email
> with one word - either APPROVE, DISAPPROVE, ABSTAIN
>
> Where:
>
> APPROVE means you are ok with the version we reached - please note that we
> do have some new elements there and your approval means you understand what
> it means - please read the paper well and understand the nuances of
> approval.
>
> DISAPPROVE- means that you no longer agree with the version we reached and
> we need to discuss the paper more and withdraw it - if you have some
> serious objection, it is the last chance to get the paper back to the
> drawing board
>
> ABSTAIN - You have no objection for the paper being published, yet you
> want your name withdrawn
>
> Also, if anyone else has any objections for this paper being published,
> please raise them now so we can discuss them.
>
> I call again to John Gennarri to join us as an author - he did
> substantial work and many of his contributions are in the final version -
> including the order in figure 2. I can trace back many of his contributions
> and I think he should get credit for it - hopefully my last call will be
> accepted and he will rejoin.
>
> William - I did not forget your second affiliation - I just could not
> change it in the system - I will write to the editor about that when we get
> the final acceptance notice.
>
> I look forward to seeing your final decisions - hopefully it will be fast
> this time.
>
> Jacob
>
>
> On Thu, Jan 13, 2022 at 8:50 PM Torres, Marcella <mtorres at richmond.edu>
> wrote:
>
>> Sorry, I was not clear enough: no response is required for Reviewer 2, as
>> they have endorsed the manuscript. A response to Reviewer 1 is overdue,
>> however; see the email below.
>>
>> Frontiers in Systems Biology Editorial Office has sent you a message.
>> Please click 'Reply' to send a direct response
>>
>> This is to remind you that your response to the reviewer 1 is overdue in
>> the interactive review forum of your manuscript 822606 under consideration
>> for publication in Frontiers in Systems Biology, section Multiscale
>> Mechanistic Modeling.
>> To access the review forum, click:
>>
>> http://www.frontiersin.org/Review/EnterReviewForum.aspx?activationno=5f3876ba-0de4-47c1-8dff-1c2bcb5c6b11&retab=1
>> <http://links.email.frontiersin.org/ls/click?upn=99YMbkkQ8WNJiHLectmHSn5XnTmYe57aPeiziTpYNtyfClFbQqFlwPq2nUMZBzQ7srQvkLkPbc-2F-2BZbZ-2BczlOsMnoBdURYE7vBE6wrLrCJR4-2BCXMD-2FyylaGS-2FaxCSwEXxntm7RnNdDdmDL5k3W2WQo0ie3jVFLbWE6g0wuBJBzfQ-3DOLrv_B-2Fy9jbSVGoT8qhSwbw0AFtm6DlDsGMQJjIadnFC13-2FJyATqJzBD73VYfuDTc-2F-2BJaS4kVj6eazmIcCMUijXaF7cwmpIPfXo62oyUmieGoN2C1Y5Or-2Fqw1mDFet-2FbgSMXcJvyRgL8VJvJK-2Bw-2BULW-2FawzD0QPdwLV1hkLM4-2F9kG-2BJqcbrOIV4DgfEtFEs9z2qt9wTumRZRHdyMGfQyetT9cia1k0sDolZ7ukB5Ik6E4DTmj3yHv4voIEhFHR5lhm-2Bbs>
>>
>> Should you require a deadline extension for your resubmission, please do
>> let us know by selecting "Request Extension" in the review forum, if you
>> have not done so already. We encourage you to submit your revised
>> manuscript with tracked changes to facilitate the review.
>>
>> We look forward to receiving your revised manuscript. For any issues,
>> please immediately contact the Associate Editor as well as the Editorial
>> Office.
>>
>> Best regards,
>>
>> Your Frontiers in Systems Biology team
>>
>> Frontiers in Systems Biology Editorial Office
>> www.frontiersin.org
>> Avenue du Tribunal Federal 34
>> Lausanne, Switzerland | T 41(0)21 510 17 93
>>
>> For technical issues, please contact our IT Helpdesk -
>> support at frontiersin.org
>>
>>
>>
>>
>> Marcella Torres, Ph.D.
>> Director of Mathematical Studies
>> University of Richmond
>> Jepson Hall Room 212
>> 221 Richmond Way
>> Richmond, VA 23173
>> (804) 289-8081
>> Pronouns: she/her
>>
>> ------------------------------
>> *From:* Jacob Barhak <jacob.barhak at gmail.com>
>> *Sent:* Thursday, January 13, 2022 8:14 PM
>> *To:* Torres, Marcella <mtorres at richmond.edu>
>> *Cc:* Rahuman Sheriff <sheriff at ebi.ac.uk>;
>> vp-reproduce-subgroup at lists.simtk.org <
>> vp-reproduce-subgroup at lists.simtk.org>;
>> vp-integration-subgroup at lists.simtk.org <
>> vp-integration-subgroup at lists.simtk.org>; John Gennari <gennari at uw.edu>;
>> Faeder, James R <faeder at pitt.edu>
>> *Subject:* Re: [Vp-reproduce-subgroup] [Vp-integration-subgroup] Paper
>> review and revision check list
>>
>> *External Email:* Use caution in opening links, attachments, and buying
>> gift cards.
>>
>> Thanks Marcella, Thanks Gilberto,
>>
>> Since the second reviewer did not respond, I think we have more time - we
>> did respond in time however.
>>
>> The figures were provided by Alex and Sheriff - they seem pretty good
>> resolution to me, yet if we want increased resolution, they will have to
>> generate those again from the same software that generated thm originally -
>> Basically the images have to be at least 300 dpi and export it as pdf and
>> extract it as tiff or eps - not jpg.
>>
>> Section 2.2.2 in the instruction provides information about the image
>> quality:
>> https://www.frontiersin.org/about/author-guidelines
>>
>> Hopefully we can have those ready.
>>
>> However, I do ask that all authors be ready to approve the paper assuming
>> the other reviewer will be positive - we worked on thai for a year and
>> there were many discussions - before this paper gets published I want all
>> authors to be good with the final outcome. We will do this formally like we
>> did before - yet I want everyone to be ready to avoid any delays. Hopefully
>> it will take us less than 3 weeks this time.
>>
>> Jacob
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Thu, Jan 13, 2022 at 1:05 PM Torres, Marcella <mtorres at richmond.edu>
>> wrote:
>>
>> Hi again all,
>>
>> On the 7th I shared comments from Reviewer 1 that we are now late
>> responding to. Is it possible to provide high resolution figures, or no
>> (see below)?
>>
>> Reviewer 2 has endorsed the manuscript.
>>
>>
>> Reviewer 1 comment:
>> *Reviewer 1* | 07 Jan 2022 | 14:36
>> #3
>>
>> I would like to thank the authors for having fully addressed my comments.
>> I'm satisfied about the quality the manuscript has been reached in this
>> latest version. Finally, I would only suggest the authors to provide an
>> high quality resolution of both new figures. Thank you.
>>
>>
>> Marcella Torres, Ph.D.
>> Director of Mathematical Studies
>> University of Richmond
>> Jepson Hall Room 212
>> 221 Richmond Way
>> Richmond, VA 23173
>> (804) 289-8081
>> Pronouns: she/her
>>
>> ------------------------------
>> *From:* Vp-reproduce-subgroup <
>> vp-reproduce-subgroup-bounces at lists.simtk.org> on behalf of Rahuman
>> Sheriff <sheriff at ebi.ac.uk>
>> *Sent:* Monday, January 10, 2022 3:09 PM
>> *To:* Jacob Barhak <jacob.barhak at gmail.com>
>> *Cc:* vp-reproduce-subgroup at lists.simtk.org <
>> vp-reproduce-subgroup at lists.simtk.org>;
>> vp-integration-subgroup at lists.simtk.org <
>> vp-integration-subgroup at lists.simtk.org>; John Gennari <gennari at uw.edu>;
>> Faeder, James R <faeder at pitt.edu>
>> *Subject:* Re: [Vp-reproduce-subgroup] [Vp-integration-subgroup] Paper
>> review and revision check list
>>
>> *External Email:* Use caution in opening links, attachments, and buying
>> gift cards.
>>
>> Thanks Jakob for submitting the revision and others for the support.
>> Thanks William for editing my response.
>>
>> Best
>> Sheriff
>>
>>
>> On 7 Jan 2022, at 04:52, Jacob Barhak <jacob.barhak at gmail.com> wrote:
>>
>> Thanks Hana, Thanks William, and thanks to Gilberto, Alex, and Sheriff,
>>
>> With your help and some effort, the revised paper was resubmitted in time
>> - just now.
>>
>> William, please keep on reminding me about your second affiliation - I
>> kept a comment as a reminder to do this - the submission system did
>> not allow making any changes in Author information - not sure why - if this
>> gets accepted we should ask the editor to add your affiliation manually.
>>
>> I urge all other authors to look at the changes we made and our response.
>> We will have to approve the manuscript with everyone before it gets
>> published - hopefully we will convince the reviewers this review round.
>>
>> Jacob
>>
>> On Thu, Jan 6, 2022 at 3:51 PM Dobrovolny, Hana <h.dobrovolny at tcu.edu>
>> wrote:
>>
>> I've read through the paper and fixed a few grammar mistakes.
>>
>> Hana
>>
>>
>> *******************************************************
>> Dr. Hana Dobrovolny
>> Associate Professor of Biophysics
>> Texas Christian University
>> TCU Box 298840
>> Fort Worth, TX 76129
>>
>> phone: (817) 257-6379 fax: (817) 257-7742
>> email: h.dobrovolny at tcu.edu
>> *******************************************************
>>
>>
>> ------------------------------
>> *From:* Vp-integration-subgroup <
>> vp-integration-subgroup-bounces at lists.simtk.org> on behalf of Jacob
>> Barhak <jacob.barhak at gmail.com>
>> *Sent:* January 5, 2022 11:48 PM
>> *To:* William Waites; James Glazier
>> *Cc:* vp-reproduce-subgroup at lists.simtk.org;
>> vp-integration-subgroup at lists.simtk.org; John Gennari; Faeder, James R
>> *Subject:* Re: [Vp-integration-subgroup] [Vp-reproduce-subgroup] Paper
>> review and revision check list
>>
>>
>> *[EXTERNAL EMAIL WARNING]* DO NOT CLICK LINKS or open attachments unless
>> you recognize the sender and know the content is safe.
>> Greetings paper contributors,
>>
>> Gilberto has addressed the final point and added text in the paper and
>> review, thus concluding our response.
>>
>> I went over the paper from start to finish again and cleaned it up.
>>
>> There are two last comments in the paper addressed to William Waites and
>> James Glazier - if you can quickly check those, it would help.
>>
>> If someone else wants to read the paper once more, please do this today -
>> otherwise I will convert the paper and our response to reviewers and submit
>> it to the Journal portal.
>>
>> If anyone has any objections, now is the time to raise them so we can
>> withdraw rather than submit.
>>
>> Here is the link to the revised paper:
>>
>> https://docs.google.com/document/d/1IMEgmdNkx-EsnOjGuegpenSIMmKIkK00Lc8Gred3QxM/edit?usp=sharing
>> <https://urldefense.com/v3/__https://docs.google.com/document/d/1IMEgmdNkx-EsnOjGuegpenSIMmKIkK00Lc8Gred3QxM/edit?usp=sharing__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0hvK3OmJ8$>
>>
>> Here is the link to response for reviewers:
>>
>> https://docs.google.com/document/d/1bO1CDbMMlyyweYayP9rGohPIfgvIhfe14sFchtJMJ-s/edit?usp=sharing
>> <https://urldefense.com/v3/__https://docs.google.com/document/d/1bO1CDbMMlyyweYayP9rGohPIfgvIhfe14sFchtJMJ-s/edit?usp=sharing__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0haRkeB0o$>
>>
>>
>> Hopefully there will be no objections and can submit this tomorrow.
>>
>> Jacob
>>
>>
>>
>>
>>
>>
>>
>>
>> On Tue, Jan 4, 2022 at 11:30 PM Jacob Barhak <jacob.barhak at gmail.com>
>> wrote:
>>
>> Hi Alex, Hi William, Hi Gilberto, and all other contributors.
>>
>> Alex, William and Gilberto are mentioned since we are the most active
>> contributors.
>>
>> We have 2 days to finish edits.
>>
>> William - I attempted to add your suggestions to the paper
>> regarding issue 11. You are free to revise further.
>>
>> Gilberto - I merged some of your texts, yet it seems you are the only one
>> I saw interested in resolving issue 12 where the reviewer asks to remove
>> the "Model Application and Implementation Barriers" section. Please suggest
>> your corrections as soon as possible. If you cannot do this by tomorrow I
>> will do my best to add a response based on your idea. Yet I really hope you
>> can do this before me.
>>
>> Please finalize any edits you have - and if anyone has any objection to
>> the text, please raise it within one day since after tomorrow I will start
>> closing down all issues.
>>
>> I would ask for a volunteer that will go over the paper once more after
>> its finalized and check for typos and grammar - we do not want the paper to
>> return again due to this issue. Time is tight, so please volunteer fast.
>>
>> Hopefully we can get it done in time.
>>
>> Jacob
>>
>>
>>
>>
>>
>> On Tue, Jan 4, 2022 at 7:25 AM Alexander Kulesza <
>> alexander.kulesza at novadiscovery.com> wrote:
>>
>> Dear Jacob, dear all,
>> thanks for endorsing my figure. I'll gladly add this. Please find
>> enclosed the .png.
>>
>> What you say makes sense and I agree that exaggerating changes is not
>> speeding up approval. Please still consider it as a tool to perform "major"
>> change (ordering of sections is a major change of a paper in my opinion)
>> without changing too much the sections as such. But I agree that this could
>> be tried out should the reviewers insist.
>>
>> I therefore reverted the table change (disconnecting ist from Figure 1)
>> and suggest the following Figure caption:
>>
>> "Figure 2: Sketch of difficulties persistent that impede
>> reproducibility, credibility, utility and integration of models, especially
>> in computational biology. An assignment of these difficulties to the
>> four different concepts interpreting them as hurdles is attempted. We would
>> like to point out that this sequential assessment model indicated in the
>> graph is only one of the possibilities a modeler would assess the
>> suitability of available models. In the rest of the paper we therefore
>> address each difficulty separately. "
>>
>> With that sentence we can discuss the difficulties in any order.
>>
>> Best
>> Alexander
>>
>> On Tue, 4 Jan 2022 at 13:24, Jacob Barhak <jacob.barhak at gmail.com> wrote:
>>
>> Yes Alex,
>>
>> Your image is much better than mine, please use it instead in the paper
>> and send a link to the image itself in an acceptable image format so I can
>> upload it to the submission system later on.
>>
>> As for changing the table and order, I am not sure. Although it may be
>> possible. We need to think about it. Yet we really are crunched in time, so
>> I am leaning to say no.
>>
>> First, because some of the difficulties span more than one category, for
>> example unit standardization affects integration as well as
>> reproducibility, only it appears first at reproducibility, therefore it was
>> placed there in the figure.
>>
>> Second, if you look at the text there are some transition sentences among
>> sections, that connect them, so swapping those around may just mess things.
>> I am not sure it if will also mess up acronym definitions on the first
>> appearance as well, so shuffling order is not something's I would do unless
>> necessary. And it was not requested by reviewers anyway.
>>
>> I am also trying to minimize changes and focus only on what was requested
>> rather than change a lot more. Otherwise we may never finish changing
>> things since each one of the 18 of us will have a one last item they MUST
>> have corrected. I personally did not add some recent publications I had,
>> although I added one publication James asked to add in the previous
>> approval round and corrected one that Sheriff mentioned. Yet I really want
>> to avoid opening the paper to significant changes.
>>
>> If you start opening corrections beyond what was requested, this will
>> drag on forever since we do have a large list of contributors and each one
>> of us will generate a new idea. So please focus on what was asked unless
>> absolutely necessary.
>>
>> So I suggest not changing the table, and changing only the figure and
>> keeping order intact. I hope I am projecting the sense of urgency - we had
>> the paper lying around for a couple of weeks during the holiday without
>> much progress, I ask not to escalate things just because a deadline is
>> looming.
>>
>> If you have extra time on your hands in the next couple of days, I
>> suggest you focus on addressing the last two points in the response that we
>> have not resolved yet. I know Gilberto wanted to fix one of them, yet you
>> are welcome to do it. Those are things requested by the review
>>
>> Hopefully my arguments make sense.
>>
>> Jacob
>>
>> On Tue, Jan 4, 2022, 06:45 Alexander Kulesza <
>> alexander.kulesza at novadiscovery.com> wrote:
>>
>> Dear all,
>>
>> I have thought about the additional Figure requested by the reviewer.
>> Jacob your Figure nicely shows the assignment of the difficulties we detail
>> in our central table with the 4 big concepts.
>>
>> I feel however that the text-heavy figure is a lot o redundancy with the
>> rest of the elements (existing figure & table). Therefore I wanted to throw
>> another suggestion forward (inspired by systematic reviews narrowing down
>> eligible studies/data).
>>
>> @all please notice that the assignment I would do is a bit different to
>> what is currently done in the paper. The order of sections would have to be
>> altered. Not sure if that is acceptable.
>> <image.png>
>> The table with Difficulties (hurdles) and solutions would then be
>>
>> Difficulty
>> Potential Solutions
>>
>> Reproducibility
>> Models are written in different languages
>> Common transport specifications such as SBML or CellML, and proper
>> documentation and annotation
>> Models are hard to locate
>> Archive web sites such as: BioModels, SimTK, IMAGWiki, and the future
>> modeleXchange
>> Lack of common platforms for executing models and simulations
>> Platforms such as BioSimulators, and runBioSimulations
>> Unit standardization
>> Standardization efforts, and machine learning solutions such as
>> ClinicalUnitMapping.com
>> Credibility
>> Models have built-in barriers to Evaluating model credibility
>> Better modeling practices, documentation, and tests.
>> Data availability and measurement definitions
>> Models that merge human interpretation, and newer measurement devices
>>
>> Missing annotations in models
>> Adoption of policies such as those COMBINE suggests
>> Utility
>> Models are not consistently licensed to allow for reuse
>> Abandoning some old school open source licenses and promoting licenses
>> that release to public domain
>> Different scales and modeling paradigms
>> Standardization effort and centralization tools
>> Stochastic modeling difficulties
>> Development of tools that guarantee repeatability and standards to
>> address stochastic simulations
>> Integration
>> Model application and implementation barriers
>> Education of modelers, users, and the public
>> Modeling requires adaptation towards integration
>> Tools for composing models such as SBML-Comp, and SemGen
>>
>> Following this assignment.
>>
>> Please let me know what you think. I have added this suggestion in track
>> changes mode to the revision draft.
>>
>> Best
>> Alexander
>>
>>
>> On Mon, 3 Jan 2022 at 07:27, Jacob Barhak <jacob.barhak at gmail.com> wrote:
>>
>> Thanks Alex,
>>
>> You did great by adding the new material that the reviewers actually
>> asked for. This is great - I found myself adding the new FDA document to
>> find out that you already added it - I just voice read it ona long drive
>> and figured it fits well - it seems you were faster - great.
>>
>> I merged your edits and I am ok with adding your response to the
>> reviewers - I only had an issue with one example paragraph that you deleted
>> - I left it with a comment - please check if it is possible to keep this
>> example somewhere in the text.
>>
>> I will assemble a letter that includes all those responses point by point
>> and will try to use your text almost verbatim if possible. You really bring
>> important knowledge with you.
>>
>> It seems most points have been answered to some level - even a figure was
>> created - although I wish I was a better artist.
>>
>> I will try to assemble the response letter for everyone within the next
>> few days. And unless there will be objections I will try to pass it by
>> reviewers with a new version. If the new version is accepted, we will do a
>> final approval round before publication.
>>
>> Many thanks to all those who contributed.
>>
>> Jacob
>>
>>
>>
>>
>>
>> On Sun, Jan 2, 2022 at 7:27 AM Alexander Kulesza <
>> alexander.kulesza at novadiscovery.com> wrote:
>>
>> Dear all,
>>
>> Happy new year to everyone. I hope that you are all well and safe and
>> have spent a nice holiday season.
>>
>> I have tried to address
>>
>> *Reviewer 2 *
>>
>>
>> *Moreover, when the authors mention ASME V&V40 procedure, it could be
>> useful to spend few words about the model credibility pipeline drawn for
>> predictive models in biomedicine, such as the one suggested in "Credibility
>> of in Silico Trial Technologies-A Theoretical Framing", Viceconti, M.,
>> Juarez, M.A., Curreli, C., ...Russo, G., Pappalardo, F., IEEE Journal of
>> Biomedical and Health Informatics, 2020, 24(1), pp. 4–13, 8884189. For this
>> specific aspect, the authors should talk about "specific question of
>> interest" and not "specific purpose" in the paragraph called "Credibility
>> of Models" to follow in such a way a standardised language. *
>>
>> I have worked on the followng document in "track changes" mode:
>>
>> https://docs.google.com/document/d/1IMEgmdNkx-EsnOjGuegpenSIMmKIkK00Lc8Gred3QxM/edit
>> <https://urldefense.com/v3/__https://docs.google.com/document/d/1IMEgmdNkx-EsnOjGuegpenSIMmKIkK00Lc8Gred3QxM/edit__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0hrtFQ-XI$>
>> Actually, I ended up in changing rather drastically the section. I think
>> that all ideas and all text is conserved, but I would very much like to
>> encourage you to read, comment and challenge this revision. Please tell me
>> if that goes to far. No problem to revert to an earlier version if
>> necessary.
>>
>> I suggest the following formulation in the rebuttal letter:
>>
>> When trying to better describe the "credibility pipeline" (CoU
>> definition, verification, validation, uncertainty quantification, e.g. as
>> described in Viceconti et al. 2019) or others, we noticed that the section
>> about credibility could be restructured in order to put the reader (not
>> necessarily familiar with regulatory assessment of models) in the position
>> to follow the argumentation.
>>
>> We now roughly follow the following flow of writing:
>>
>> - Every model needs a purpose and credibility is tied to that purpose as
>> well as repeatability and reproducibility
>> - Credibility is essential for models that have impact on regulated
>> areas or people's lives which is why regulatory authorities issue guidance
>> - The probably most advanced guidance ASME V&V40 suggests a risk-based
>> approach and pipeline to establish credibility of a model. It is
>> overarching and widely applicable
>> - Regulators and modelers work together (for example in frame of the
>> model informed drug development program MIDD) in order to better
>> understand, better apply and better uptake different kinds of models in
>> regulatory applications
>> - More work is needed to harmonize, stay up to date and to be more
>> inclusive
>>
>> We hope that with the rather extensive change of the wording and
>> additional passages as well as citations (see below) we could address the
>> concern of reviewer 2 (and the general remarks of reviewer 1).
>>
>> " In 2018 the American Society of Mechanical Engineers (ASME) issued an
>> important guidance ASME V&V 40 (ASME, 2018) of how to assess credibility of
>> computational models of medical devices through verification and validation
>> (V&V). The guideline is centered around the definition of the context of
>> use (CoU) of the model which is formulated based on the questions of
>> interest the model will answer. The CoU is then analyzed in terms of the
>> “model risk” - being the influence the model exerts on a decision and the
>> potential consequences these decisions might incur. Commensurate with this
>> model risk the modeler suggests establish the credibility goals, perform
>> verification validation and uncertainty quantification actions and then
>> assess the outcome of this exercise in order to allow judging the
>> acceptability of the model CoU. Key to this guidance is its overarching
>> nature which also allows adoption in other (e.g. drug development) fields
>> irrespective of the model type (Kuemmel 2020, Viceconti 2019)."
>>
>> We also feel that it is essential to underline the cross-discipline
>> viewpoint of the ASME V&V40 which is further elaborated by the cited paper
>> by formalizing the verification, validation and uncertainty quantification
>> VVUQ pipeline across model types.
>>
>> We therefore have added the following statement
>> " In the paper by (Viceconti 2019) the verification, validation and
>> uncertainty quantification (VVUQ) pipeline is streamlined to different
>> types of models. It is, perhaps, the closest to score credibility across
>> model types from mechanistic physics driven models to machine learning
>> models. However, it is still short of including very recent developments
>> such as ensemble models, although it touches upon the topic."
>>
>> We additionally strengthened our argumentation by citing additional
>> literature
>>
>> Bai, JPF, Earp, JC, Florian, J, et al. Quantitative systems
>> pharmacology: Landscape analysis of regulatory submissions to the US Food
>> and Drug Administration. CPT Pharmacometrics Syst Pharmacol. 2021; 10:
>> 1479– 1484. doi.org:10.1002/psp4.12709
>>
>> FDA (2021b). Assessing the Credibility of Computational Modeling and
>> Simulation in Medical Device Submissions. FDA
>> Available at:
>> https://www.fda.gov/regulatory-information/search-fda-guidance-documents/assessing-credibility-computational-modeling-and-simulation-medical-device-submissions
>> <https://urldefense.com/v3/__https://www.fda.gov/regulatory-information/search-fda-guidance-documents/assessing-credibility-computational-modeling-and-simulation-medical-device-submissions__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0harP7IHc$>
>> [Accessed January 2, 2022]
>>
>> Galluppi, G.R., Brar, S., Caro, L., Chen, Y., Frey, N., Grimm, H.P.,
>> Rudd, D.J., Li, C.-C., Magee, M., Mukherjee, A., Nagao, L., Purohit, V.S.,
>> Roy, A., Salem, A.H., Sinha, V., Suleiman, A.A., Taskar, K.S., Upreti,
>> V.V., Weber, B. and Cook, J. (2021), Industrial Perspective on the Benefits
>> Realized From the FDA’s Model-Informed Drug Development Paired Meeting
>> Pilot Program. Clin Pharmacol Ther, 110: 1172-1175. doi:10.1002/cpt.2265
>>
>> Kuemmel, C., Yang, Y., Zhang, X., Florian, J., Zhu, H., Tegenge, M.,
>> Huang, S.-M., Wang, Y., Morrison, T. and Zineh, I. (2020), Consideration
>> of a Credibility Assessment Framework in Model-Informed Drug Development:
>> Potential Application to Physiologically-Based Pharmacokinetic Modeling and
>> Simulation. CPT Pharmacometrics Syst. Pharmacol., 9: 21-28. doi.org
>> :10.1002/psp4.12479
>>
>> Musuamba, FT, Skottheim Rusten, I, Lesage, R, et al. Scientific and
>> regulatory evaluation of mechanistic in silico drug and disease models in
>> drug development: Building model credibility. CPT Pharmacometrics Syst
>> Pharmacol. 2021; 10: 804– 825. doi:10.1002/psp4.12669
>>
>> Viceconti M, Emili L, Afshari P, et al. Possible Contexts of Use for In
>> Silico Trials Methodologies: A Consensus-Based Review. IEEE J Biomed
>> Health Inform. 2021;25(10):3977-3982. doi:10.1109/JBHI.2021.3090469
>>
>>
>> All the best
>> Alexander
>>
>> On Sun, 19 Dec 2021 at 03:10, Jacob Barhak <jacob.barhak at gmail.com>
>> wrote:
>>
>>
>> Greetings to the paper contributors,
>>
>> You may have seen my other message where I posted the reviews to our
>> paper. I am sending those again below to start a new thread discussing
>> possible revisions.
>>
>> Here is a checklist for revisions required:
>>
>>
>> 1. We should discuss the references that reviewer 1 raised and add
>> them as reference
>> 2. Reviewer 1 asked for more discussion around V&V40 - I believe Alex
>> added the text there originally - Alex do you think you can address the
>> request of the reviewer?
>> 3. Reviewer 1 asked to focus on COVID-19 at the title - I am not sure
>> we wish to limit our scope since many of our ideas are applicable far
>> beyond COVID-19 - I leave it up for discussion in the group on how to
>> address this request by the reviewer
>> 4. Reviewer 1 asked for additional graphics - I am unsure how to
>> address this beyond ur current diagram - ideas will help
>> 5. We need to check the "Contribution to the field" section for
>> grammar and spelling - if someone can contribute more elegant text - now is
>> the time - I looked over it now and found no issues - yet I may have missed
>> something - this section may have been missed since it was added last and
>> perhaps vetted less than other sections. Also Reviewer 2 asked for grammar
>> corrections, so it is worthwhile proof reading the paper as a whole.
>> 6. Reviewer 2 asks for major rewrite to emphasize why the problems
>> are there - I think we should explain that the core of the paper is the
>> table and perhaps emphasize it in the paper beyond what the current text -
>> we also should reply to the reviewer and explain that the paper is composed
>> of contributions from a large group and we made an effort to include every
>> voice in the choir - each of those voices is important and needs to be
>> preserved - hopefully it will convince the reviewer that change we will add
>> will be sufficient
>> 7. Reviewer 2 asks to revise the introduction - I believe some
>> changes are possible - yet the introduction includes contributions from
>> many authors - at least 10 - and I fear losing something important someone
>> contributed - if someone has an idea on how to address this reviewer
>> without a painful transformation, please reply to this message.
>> 8. Reviewer 2 wants additional discussion around the
>> "reproducibility crisis" section and asks a valid question about
>> expectations we should address - "“Computational biomedical modeling… was
>> expected to be less affected by the reproducibility crisis.” Is this true,
>> and why would it be so? " - My answer to the reviewer is that unlike
>> biological processes that have random nature and experiments
>> would not repeat if repeated, while computer software should be
>> deterministic and it should be repeatable if designed well. Unfortunately
>> we are not experiencing this promise - yet I believe the reviewer wants
>> more discussion beyond this section so I am happy to discuss this.
>> 9. Reviewer 2- writes: "Models are Hard to Locate: Are the authors
>> suggesting that entire simulation workflows, from model construction to
>> analysis, should be publicly available? At what point does one consider
>> intellectual property? Do the authors advocate for such extensive
>> publishing for all models, or only ones that are intended to be widely
>> re-used? - those are important points we need to discuss - we need to
>> better explain the difficulties we are having when creating models and the
>> reviewer is absolutely correct about expanding the discussion to IP - I
>> suggest we create a thread for this discussion and reference it - I
>> suggest it we merged with the licensing issue that I will address later -
>> there is a strong connection there - the reviewer was very observant and
>> sees the bigger problem. However,we need to eventually distill our
>> discussion to recommendations that will inline.
>> 10. Reviewer 2 asks that we fix the unit standardization section - I
>> believe Hana and myself were the largest contributors there - Hana - I will
>> start a discussion on that topic in a separate email where we can publicly
>> discuss how to fix this - others will be welcome to contribute.
>> 11. Reviewer 2 asks that we better handle the section on Data
>> availability and measurement definitions: in think we need to emphasize
>> solutions and separate it from issues that may not be solvable. Ideas are
>> welcome.
>> 12. Reviewer 2 asks good questions with regards to licensing
>> following our text- I personally have good answers to the reviewer and
>> William and I had some discussion on the topic in this list I suggest we
>> expand this discussion in a separate thread - hopefully William and perhaps
>> others will join the discussion. This discussion should also address the IP
>> issues raised by the reviewer for the "models are hard to locate section."
>> 13. Reviewer 2 asks to handle the Model Application and
>> Implementation Barriers section. We should decide what to do there, the
>> section may need expansion since the ideas there are solid, yet the section
>> is short so perhaps enhancing it makes better sense. I am open to
>> suggestions.
>>
>>
>> Those are the items I located and my suggestions. It seems we need
>> attention from Alex, Hana, myself and william. However, anyone on the list
>> is welcome to participate and suggest changes.
>>
>> I will start the discussion threads on specific topics. Hopefully we can
>> get it done quickly.
>>
>> Jacob
>>
>>
>>
>>
>>
>>
>> -
>>
>>
>>
>>
>>
>> ########### Original Reviews #############
>>
>> There are 2 reviews - both require major changes. I am copying the
>> relevant text below. If more appear, I will let you know, yet I only got
>> this message today although the reviews are dated a few days ago.
>>
>> Reviewer 1:
>> Recommendation for the Editor: Substantial revision is required
>>
>> Please list your revision requests for the authors and provide your
>> detailed comments, including highlighting limitations and strengths of the
>> review. If you have additional comments based on Q2 and Q3 you can add them
>> as well.
>>
>> Karr and co-authors made an interesting and exaustive point about the
>> reproducibility crisis that leads to inability to reuse and integrate
>> models, especially about COVID-19 disease.
>> Within the manuscript some typos and missing information are present.
>> I'd suggest the authors to revise the entire manuscript especially in
>> terms of the state of the art, revising and updating the most relevant
>> examples of computational models dealing with COVID-19 and in general about
>> some semi-standardised proposals about the pipeline to follow for the
>> verification and validation of model credibility. In particular, the
>> authors failed to mention and cite some major results on in silico modeling
>> about COVID-19 up to now. See for example:
>> a."In silico trial to test COVID-19 candidate vaccines: a case study with
>> UISS platform", Russo, G., Pennisi, M., Fichera, E., ...Viceconti, M.,
>> Pappalardo, F., BMC Bioinformatics, 2020, 21, 527.
>> b. Russo G, Di Salvatore V, Sgroi G, Parasiliti Palumbo GA, Reche PA,
>> Pappalardo F. "A multi-step and multi-scale bioinformatic protocol to
>> investigate potential SARS-CoV-2 vaccine targets" [published online ahead
>> of print, 2021 Oct 5]. Brief Bioinform. 2021;bbab403.
>> doi:10.1093/bib/bbab403.
>> Moreover, when the authors mention ASME V&V40 procedure, it could be
>> useful to spend few words about the model credibility pipeline drawn for
>> predictive models in biomedicine, such as the one suggested in "Credibility
>> of in Silico Trial Technologies-A Theoretical Framing", Viceconti, M.,
>> Juarez, M.A., Curreli, C., ...Russo, G., Pappalardo, F., IEEE Journal of
>> Biomedical and Health Informatics, 2020, 24(1), pp. 4–13, 8884189.
>> For this specific aspect, the authors should talk about "specific
>> question of interest" and not "specific purpose" in the paragraph called
>> "Credibility of Models" to follow in such a way a standardised language.
>> Moreover, the authors should refer to COVID-19 also in the title, with a
>> specific mention about the fact that the main topic of model integration in
>> computational biology will be discussed inside the COVID-19 context.
>> Furthermore, for the model key concepts such as
>> Reusability-Extensibility-Extractability-Portability the authors should
>> described and outlined through a graphical sketch or visual representation
>> that summarises these key point.
>> The authors should also fix some grammar and writing typos present in the
>> "Contribution to the field" section.
>>
>> a. Is the quality of the figures and tables satisfactory?
>> - No
>>
>> b. Does the reference list cover the relevant literature adequately and
>> in an unbiased manner?
>> - No
>>
>> c. Does this manuscript refer only to published data? (unpublished or
>> original data is not allowed for this article type)
>> - Yes
>>
>> d. Does the review include a balanced, comprehensive, and critical view
>> of the research area?
>> - Yes
>>
>>
>> Reviewer 2
>>
>> Please list your revision requests for the authors and provide your
>> detailed comments, including highlighting limitations and strengths of the
>> review. If you have additional comments based on Q2 and Q3 you can add them
>> as well.
>>
>>
>> The article often reads as a stream-of-consciousness account of the
>> discussions that took place but lacks a clear thesis or recommendations. It
>> is not clear what, if anything, the authors are advocating for. It is not
>> always clear why the issues being discussed are problematic, or that they
>> can be reasonably addressed. Some of the issues raised are indeed important
>> and should be discussed, but the paper lacks focus and does not tell a
>> cohesive story. I believe this manuscript requires a major re-write to be
>> suitable for publication. The authors should consider narrowing the scope
>> of the discussion and focusing on a cohesive set of recommendations or open
>> questions. More specifically, I make a few suggestions below:
>>
>> Major comments
>> 1. The introduction is long and repeats itself (e.g., “much less is known
>> about how viral infections spread throughout the body…” is repeated
>> verbatim). It is not clear from the introduction what the main goal of the
>> paper is or why the “reproducibility crisis” is truly a crisis. Why is the
>> discussion of composition and black/white box models relevant to the
>> introduction? Further, this section is subtitled “the promise of modeling”,
>> which does not seem to match the content.
>> 2. The Reproducibility Crisis: “Computational biomedical modeling… was
>> expected to be less affected by the reproducibility crisis.” Is this true,
>> and why would it be so? One would think that more complex models would
>> suffer more from a lack of reproducibility. It may be helpful to define
>> what exactly the “reproducibility crisis” refers to.
>> 3. Models are Hard to Locate: Are the authors suggesting that entire
>> simulation workflows, from model construction to analysis, should be
>> publicly available? At what point does one consider intellectual property?
>> Do the authors advocate for such extensive publishing for all models, or
>> only ones that are intended to be widely re-used?
>> 4. Unit standardization: The conversion from PFU or TCID50 to individual
>> virions is likely to differ across viruses – are the authors focused on
>> COVID here? Are the authors advocating for a standard conversion factor? It
>> is not clear what the purpose of this discussion is. As the authors
>> mention, different scales require different units. Even at a single scale,
>> different models may require different units for numerical reasons. It is
>> not clear what the authors are advocating for here.
>> 5. Data availability and measurement definitions: This section seems to
>> outline limitations of available data, but again makes no recommendations
>> or proposed solution to any of the issues raised. Is this the intention?
>> Most of the issues raised here reflect limitations of experimental science
>> or data privacy, which likely cannot be meaningfully addressed by the
>> modeling community.
>> 6. Models are Not Consistently Licensed…: Are the authors implying here
>> that all modeling work should be published with no rights reserved? Is it
>> reasonable to expect modelers to make their work freely usable by others
>> for profit? Is it reasonable for institutions to allow this? How much does
>> this really contribute to reproducibility and utility?
>> 7. Model Application and Implementation Barriers: This section seems
>> unnecessary and out of place.
>> 8. There are grammar and punctuation errors scattered throughout; please
>> edit carefully.
>>
>>
>> a. Is the quality of the figures and tables satisfactory?
>> - Not Applicable
>>
>> b. Does the reference list cover the relevant literature adequately and
>> in an unbiased manner?
>> - Yes
>>
>> c. Does this manuscript refer only to published data? (unpublished or
>> original data is not allowed for this article type)
>> - Yes
>>
>> d. Does the review include a balanced, comprehensive, and critical view
>> of the research area?
>> No answer given.
>>
>>
>> ========
>> _______________________________________________
>> Vp-reproduce-subgroup mailing list
>> Vp-reproduce-subgroup at lists.simtk.org
>> https://lists.simtk.org/mailman/listinfo/vp-reproduce-subgroup
>> <https://urldefense.com/v3/__https://lists.simtk.org/mailman/listinfo/vp-reproduce-subgroup__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0hhiovw3M$>
>>
>>
>>
>> --
>> Alexander Kulesza
>> Team leader
>> Modeling & simulation / Biomodeling
>> alexander.kulesza at novadiscovery.com
>> +33 7 82 92 44 62
>> nova
>> DISCOVERY
>> www.novadiscovery.com
>> <https://urldefense.com/v3/__http://www.novadiscovery.com/__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0hwI_2O8g$>
>> 1 Place Verrazzano, 69009 Lyon
>> +33 9 72 53 13 01
>>
>> *This message is intended only for the personal and confidential use of
>> the designated recipient(s) named above. If you are not the intended
>> recipient of this message you are hereby notified that any review,
>> dissemination, distribution or copying of this message is strictly
>> prohibited. Email transmission cannot be guaranteed to be secure or
>> error-free. Therefore, we do not represent that this information is
>> complete or accurate and it should not be relied upon as such. All
>> information is subject to change without notice.*
>>
>>
>>
>> --
>> Alexander Kulesza
>> Team leader
>> Modeling & simulation / Biomodeling
>> alexander.kulesza at novadiscovery.com
>> +33 7 82 92 44 62
>> nova
>> DISCOVERY
>> www.novadiscovery.com
>> <https://urldefense.com/v3/__http://www.novadiscovery.com/__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0hwI_2O8g$>
>> 1 Place Verrazzano, 69009 Lyon
>> +33 9 72 53 13 01
>>
>> *This message is intended only for the personal and confidential use of
>> the designated recipient(s) named above. If you are not the intended
>> recipient of this message you are hereby notified that any review,
>> dissemination, distribution or copying of this message is strictly
>> prohibited. Email transmission cannot be guaranteed to be secure or
>> error-free. Therefore, we do not represent that this information is
>> complete or accurate and it should not be relied upon as such. All
>> information is subject to change without notice.*
>>
>>
>>
>> --
>> Alexander Kulesza
>> Team leader
>> Modeling & simulation / Biomodeling
>> alexander.kulesza at novadiscovery.com
>> +33 7 82 92 44 62
>> nova
>> DISCOVERY
>> www.novadiscovery.com
>> <https://urldefense.com/v3/__http://www.novadiscovery.com/__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0hwI_2O8g$>
>> 1 Place Verrazzano, 69009 Lyon
>> +33 9 72 53 13 01
>>
>> *This message is intended only for the personal and confidential use of
>> the designated recipient(s) named above. If you are not the intended
>> recipient of this message you are hereby notified that any review,
>> dissemination, distribution or copying of this message is strictly
>> prohibited. Email transmission cannot be guaranteed to be secure or
>> error-free. Therefore, we do not represent that this information is
>> complete or accurate and it should not be relied upon as such. All
>> information is subject to change without notice.*
>>
>> _______________________________________________
>> Vp-integration-subgroup mailing list
>> Vp-integration-subgroup at lists.simtk.org
>> https://lists.simtk.org/mailman/listinfo/vp-integration-subgroup
>>
>>
>> _______________________________________________
> Vp-integration-subgroup mailing list
> Vp-integration-subgroup at lists.simtk.org
> https://lists.simtk.org/mailman/listinfo/vp-integration-subgroup
>
--
-- ------------------
Yaling Liu, Professor
Department of Bioengineering, Director of Graduate Studies
Department of Mechanical Engineering and Mechanics
Lehigh University
www.lehigh.edu/~yal310
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.simtk.org/pipermail/vp-reproduce-subgroup/attachments/20220114/d4407a72/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 28718 bytes
Desc: not available
URL: <https://lists.simtk.org/pipermail/vp-reproduce-subgroup/attachments/20220114/d4407a72/attachment-0001.png>
More information about the Vp-reproduce-subgroup
mailing list