[Vp-integration-subgroup] [Vp-reproduce-subgroup] Paper review and revision check list
Tingting Tang
ttang2 at sdsu.edu
Fri Jan 14 12:51:27 PST 2022
I approve
On Fri, Jan 14, 2022 at 9:54 AM Torres, Marcella <mtorres at richmond.edu>
wrote:
> Approve, of course, thank you!
>
> Marcella
>
> Get Outlook for Android <https://aka.ms/AAb9ysg>
> ------------------------------
> *From:* Thompson, Robin <Robin.N.Thompson at warwick.ac.uk>
> *Sent:* Friday, January 14, 2022 11:03:18 AM
> *To:* Yaling Liu <yal310 at lehigh.edu>
> *Cc:* Jacob Barhak <jacob.barhak at gmail.com>;
> vp-reproduce-subgroup at lists.simtk.org <
> vp-reproduce-subgroup at lists.simtk.org>;
> vp-integration-subgroup at lists.simtk.org <
> vp-integration-subgroup at lists.simtk.org>; Torres, Marcella <
> mtorres at richmond.edu>; John Gennari <gennari at uw.edu>; Faeder, James R <
> faeder at pitt.edu>
> *Subject:* Re: [Vp-integration-subgroup] [Vp-reproduce-subgroup] Paper
> review and revision check list
>
> *External Email:* Use caution in opening links, attachments, and buying
> gift cards.
>
> APPROVE. Thanks all!
>
> Best wishes,
> Robin
> ----------------------------------------------------------------------
> Dr Robin Thompson
> Assistant Professor of Mathematical Epidemiology
> Mathematics Institute
> University of Warwick, UK
> www.robin-thompson.co.uk
> ----------------------------------------------------------------------
>
> On 14 Jan 2022, at 16:01, Yaling Liu <yal310 at lehigh.edu> wrote:
>
> I also APPROVE
>
> Thanks for all the efforts!
>
> Yaling
>
> On Fri, Jan 14, 2022 at 1:38 AM Jacob Barhak <jacob.barhak at gmail.com>
> wrote:
>
> Thanks Marcella,
>
> In that case, it seems we have no reviewer objection and hopefully the
> editor will be positive as well . To speed up the process we need
> the following::
> 1. Sheriff - please send a high resolution image for Figure 1
> 2. Alex - Please send a high resolution image for Figure 2
> 3. Everyone please review the manuscript and Reply to all in this email
> with one word - either APPROVE, DISAPPROVE, ABSTAIN
>
> Where:
>
> APPROVE means you are ok with the version we reached - please note that we
> do have some new elements there and your approval means you understand what
> it means - please read the paper well and understand the nuances of
> approval.
>
> DISAPPROVE- means that you no longer agree with the version we reached and
> we need to discuss the paper more and withdraw it - if you have some
> serious objection, it is the last chance to get the paper back to the
> drawing board
>
> ABSTAIN - You have no objection for the paper being published, yet you
> want your name withdrawn
>
> Also, if anyone else has any objections for this paper being published,
> please raise them now so we can discuss them.
>
> I call again to John Gennarri to join us as an author - he did
> substantial work and many of his contributions are in the final version -
> including the order in figure 2. I can trace back many of his contributions
> and I think he should get credit for it - hopefully my last call will be
> accepted and he will rejoin.
>
> William - I did not forget your second affiliation - I just could not
> change it in the system - I will write to the editor about that when we get
> the final acceptance notice.
>
> I look forward to seeing your final decisions - hopefully it will be fast
> this time.
>
> Jacob
>
>
> On Thu, Jan 13, 2022 at 8:50 PM Torres, Marcella <mtorres at richmond.edu>
> wrote:
>
> Sorry, I was not clear enough: no response is required for Reviewer 2, as
> they have endorsed the manuscript. A response to Reviewer 1 is overdue,
> however; see the email below.
>
> Frontiers in Systems Biology Editorial Office has sent you a message.
> Please click 'Reply' to send a direct response
>
> This is to remind you that your response to the reviewer 1 is overdue in
> the interactive review forum of your manuscript 822606 under consideration
> for publication in Frontiers in Systems Biology, section Multiscale
> Mechanistic Modeling.
> To access the review forum, click:
>
> http://www.frontiersin.org/Review/EnterReviewForum.aspx?activationno=5f3876ba-0de4-47c1-8dff-1c2bcb5c6b11&retab=1
> <http://links.email.frontiersin.org/ls/click?upn=99YMbkkQ8WNJiHLectmHSn5XnTmYe57aPeiziTpYNtyfClFbQqFlwPq2nUMZBzQ7srQvkLkPbc-2F-2BZbZ-2BczlOsMnoBdURYE7vBE6wrLrCJR4-2BCXMD-2FyylaGS-2FaxCSwEXxntm7RnNdDdmDL5k3W2WQo0ie3jVFLbWE6g0wuBJBzfQ-3DOLrv_B-2Fy9jbSVGoT8qhSwbw0AFtm6DlDsGMQJjIadnFC13-2FJyATqJzBD73VYfuDTc-2F-2BJaS4kVj6eazmIcCMUijXaF7cwmpIPfXo62oyUmieGoN2C1Y5Or-2Fqw1mDFet-2FbgSMXcJvyRgL8VJvJK-2Bw-2BULW-2FawzD0QPdwLV1hkLM4-2F9kG-2BJqcbrOIV4DgfEtFEs9z2qt9wTumRZRHdyMGfQyetT9cia1k0sDolZ7ukB5Ik6E4DTmj3yHv4voIEhFHR5lhm-2Bbs>
>
> Should you require a deadline extension for your resubmission, please do
> let us know by selecting "Request Extension" in the review forum, if you
> have not done so already. We encourage you to submit your revised
> manuscript with tracked changes to facilitate the review.
>
> We look forward to receiving your revised manuscript. For any issues,
> please immediately contact the Associate Editor as well as the Editorial
> Office.
>
> Best regards,
>
>
> <https://www.google.com/maps/search/Avenue+du+Tribunal+Federal+34+%0D%0A+Lausanne,+Switzerland?entry=gmail&source=g>Your
> Frontiers in Systems Biology team
>
> Frontiers in Systems Biology Editorial Office
> www.frontiersin.org
> Avenue du Tribunal Federal 34
> <https://www.google.com/maps/search/Avenue+du+Tribunal+Federal+34+%0D%0A+Lausanne,+Switzerland?entry=gmail&source=g>
> Lausanne, Switzerland
> <https://www.google.com/maps/search/Avenue+du+Tribunal+Federal+34+%0D%0A+Lausanne,+Switzerland?entry=gmail&source=g>
> | T 41(0)21 510 17 93
>
> For technical issues, please contact our IT Helpdesk -
> support at frontiersin.org
>
>
>
>
> Marcella Torres, Ph.D.
> Director of Mathematical Studies
>
> <https://www.google.com/maps/search/221+Richmond+Way+%0D%0A+%0D%0ARichmond,+VA+23173?entry=gmail&source=g>University
> of Richmond
> Jepson Hall Room 212
> 221 Richmond Way
> <https://www.google.com/maps/search/221+Richmond+Way+%0D%0A+%0D%0ARichmond,+VA+23173?entry=gmail&source=g>
> Richmond, VA 23173
> <https://www.google.com/maps/search/221+Richmond+Way+%0D%0A+%0D%0ARichmond,+VA+23173?entry=gmail&source=g>
> (804) 289-8081
> Pronouns: she/her
>
> ------------------------------
> *From:* Jacob Barhak <jacob.barhak at gmail.com>
> *Sent:* Thursday, January 13, 2022 8:14 PM
> *To:* Torres, Marcella <mtorres at richmond.edu>
> *Cc:* Rahuman Sheriff <sheriff at ebi.ac.uk>;
> vp-reproduce-subgroup at lists.simtk.org <
> vp-reproduce-subgroup at lists.simtk.org>;
> vp-integration-subgroup at lists.simtk.org <
> vp-integration-subgroup at lists.simtk.org>; John Gennari <gennari at uw.edu>;
> Faeder, James R <faeder at pitt.edu>
> *Subject:* Re: [Vp-reproduce-subgroup] [Vp-integration-subgroup] Paper
> review and revision check list
>
>
> *External Email:* Use caution in opening links, attachments, and buying
> gift cards.
>
> Thanks Marcella, Thanks Gilberto,
>
> Since the second reviewer did not respond, I think we have more time - we
> did respond in time however.
>
> The figures were provided by Alex and Sheriff - they seem pretty good
> resolution to me, yet if we want increased resolution, they will have to
> generate those again from the same software that generated thm originally -
> Basically the images have to be at least 300 dpi and export it as pdf and
> extract it as tiff or eps - not jpg.
>
> Section 2.2.2 in the instruction provides information about the image
> quality:
> https://www.frontiersin.org/about/author-guidelines
>
> Hopefully we can have those ready.
>
> However, I do ask that all authors be ready to approve the paper assuming
> the other reviewer will be positive - we worked on thai for a year and
> there were many discussions - before this paper gets published I want all
> authors to be good with the final outcome. We will do this formally like we
> did before - yet I want everyone to be ready to avoid any delays. Hopefully
> it will take us less than 3 weeks this time.
>
> Jacob
>
>
>
>
>
>
>
>
>
> On Thu, Jan 13, 2022 at 1:05 PM Torres, Marcella <mtorres at richmond.edu>
> wrote:
>
> Hi again all,
>
> On the 7th I shared comments from Reviewer 1 that we are now late
> responding to. Is it possible to provide high resolution figures, or no
> (see below)?
>
> Reviewer 2 has endorsed the manuscript.
>
> <image.png>
>
> Reviewer 1 comment:
> *Reviewer 1* | 07 Jan 2022 | 14:36
> #3
>
> I would like to thank the authors for having fully addressed my comments.
> I'm satisfied about the quality the manuscript has been reached in this
> latest version. Finally, I would only suggest the authors to provide an
> high quality resolution of both new figures. Thank you.
>
>
> Marcella Torres, Ph.D.
> Director of Mathematical Studies
> University of Richmond
>
> <https://www.google.com/maps/search/221+Richmond+Way+%0D%0A+%0D%0ARichmond,+VA+23173?entry=gmail&source=g>
> Jepson Hall Room 212
> 221 Richmond Way
> <https://www.google.com/maps/search/221+Richmond+Way+%0D%0A+%0D%0ARichmond,+VA+23173?entry=gmail&source=g>
> Richmond, VA 23173
> <https://www.google.com/maps/search/221+Richmond+Way+%0D%0A+%0D%0ARichmond,+VA+23173?entry=gmail&source=g>
> (804) 289-8081
> Pronouns: she/her
>
> ------------------------------
> *From:* Vp-reproduce-subgroup <
> vp-reproduce-subgroup-bounces at lists.simtk.org> on behalf of Rahuman
> Sheriff <sheriff at ebi.ac.uk>
> *Sent:* Monday, January 10, 2022 3:09 PM
> *To:* Jacob Barhak <jacob.barhak at gmail.com>
> *Cc:* vp-reproduce-subgroup at lists.simtk.org <
> vp-reproduce-subgroup at lists.simtk.org>;
> vp-integration-subgroup at lists.simtk.org <
> vp-integration-subgroup at lists.simtk.org>; John Gennari <gennari at uw.edu>;
> Faeder, James R <faeder at pitt.edu>
> *Subject:* Re: [Vp-reproduce-subgroup] [Vp-integration-subgroup] Paper
> review and revision check list
>
> *External Email:* Use caution in opening links, attachments, and buying
> gift cards.
>
> Thanks Jakob for submitting the revision and others for the support.
> Thanks William for editing my response.
>
> Best
> Sheriff
>
>
> On 7 Jan 2022, at 04:52, Jacob Barhak <jacob.barhak at gmail.com> wrote:
>
> Thanks Hana, Thanks William, and thanks to Gilberto, Alex, and Sheriff,
>
> With your help and some effort, the revised paper was resubmitted in time
> - just now.
>
> William, please keep on reminding me about your second affiliation - I
> kept a comment as a reminder to do this - the submission system did
> not allow making any changes in Author information - not sure why - if this
> gets accepted we should ask the editor to add your affiliation manually.
>
> I urge all other authors to look at the changes we made and our response.
> We will have to approve the manuscript with everyone before it gets
> published - hopefully we will convince the reviewers this review round.
>
> Jacob
>
> On Thu, Jan 6, 2022 at 3:51 PM Dobrovolny, Hana <h.dobrovolny at tcu.edu>
> wrote:
>
> I've read through the paper and fixed a few grammar mistakes.
>
> Hana
>
>
> *******************************************************
> Dr. Hana Dobrovolny
> Associate Professor of Biophysics
> Texas Christian University
> TCU Box 298840
> Fort Worth, TX 76129
>
> phone: (817) 257-6379 fax: (817) 257-7742
> email: h.dobrovolny at tcu.edu
> *******************************************************
>
>
> ------------------------------
> *From:* Vp-integration-subgroup <
> vp-integration-subgroup-bounces at lists.simtk.org> on behalf of Jacob
> Barhak <jacob.barhak at gmail.com>
> *Sent:* January 5, 2022 11:48 PM
> *To:* William Waites; James Glazier
> *Cc:* vp-reproduce-subgroup at lists.simtk.org;
> vp-integration-subgroup at lists.simtk.org; John Gennari; Faeder, James R
> *Subject:* Re: [Vp-integration-subgroup] [Vp-reproduce-subgroup] Paper
> review and revision check list
>
>
> *[EXTERNAL EMAIL WARNING]* DO NOT CLICK LINKS or open attachments unless
> you recognize the sender and know the content is safe.
> Greetings paper contributors,
>
> Gilberto has addressed the final point and added text in the paper and
> review, thus concluding our response.
>
> I went over the paper from start to finish again and cleaned it up.
>
> There are two last comments in the paper addressed to William Waites and
> James Glazier - if you can quickly check those, it would help.
>
> If someone else wants to read the paper once more, please do this today -
> otherwise I will convert the paper and our response to reviewers and submit
> it to the Journal portal.
>
> If anyone has any objections, now is the time to raise them so we can
> withdraw rather than submit.
>
> Here is the link to the revised paper:
>
> https://docs.google.com/document/d/1IMEgmdNkx-EsnOjGuegpenSIMmKIkK00Lc8Gred3QxM/edit?usp=sharing
> <https://urldefense.com/v3/__https://docs.google.com/document/d/1IMEgmdNkx-EsnOjGuegpenSIMmKIkK00Lc8Gred3QxM/edit?usp=sharing__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0hvK3OmJ8$>
>
> Here is the link to response for reviewers:
>
> https://docs.google.com/document/d/1bO1CDbMMlyyweYayP9rGohPIfgvIhfe14sFchtJMJ-s/edit?usp=sharing
> <https://urldefense.com/v3/__https://docs.google.com/document/d/1bO1CDbMMlyyweYayP9rGohPIfgvIhfe14sFchtJMJ-s/edit?usp=sharing__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0haRkeB0o$>
>
>
> Hopefully there will be no objections and can submit this tomorrow.
>
> Jacob
>
>
>
>
>
>
>
>
> On Tue, Jan 4, 2022 at 11:30 PM Jacob Barhak <jacob.barhak at gmail.com>
> wrote:
>
> Hi Alex, Hi William, Hi Gilberto, and all other contributors.
>
> Alex, William and Gilberto are mentioned since we are the most active
> contributors.
>
> We have 2 days to finish edits.
>
> William - I attempted to add your suggestions to the paper regarding issue
> 11. You are free to revise further.
>
> Gilberto - I merged some of your texts, yet it seems you are the only one
> I saw interested in resolving issue 12 where the reviewer asks to remove
> the "Model Application and Implementation Barriers" section. Please suggest
> your corrections as soon as possible. If you cannot do this by tomorrow I
> will do my best to add a response based on your idea. Yet I really hope you
> can do this before me.
>
> Please finalize any edits you have - and if anyone has any objection to
> the text, please raise it within one day since after tomorrow I will start
> closing down all issues.
>
> I would ask for a volunteer that will go over the paper once more after
> its finalized and check for typos and grammar - we do not want the paper to
> return again due to this issue. Time is tight, so please volunteer fast.
>
> Hopefully we can get it done in time.
>
> Jacob
>
>
>
>
>
> On Tue, Jan 4, 2022 at 7:25 AM Alexander Kulesza <
> alexander.kulesza at novadiscovery.com> wrote:
>
> Dear Jacob, dear all,
> thanks for endorsing my figure. I'll gladly add this. Please find enclosed
> the .png.
>
> What you say makes sense and I agree that exaggerating changes is not
> speeding up approval. Please still consider it as a tool to perform "major"
> change (ordering of sections is a major change of a paper in my opinion)
> without changing too much the sections as such. But I agree that this could
> be tried out should the reviewers insist.
>
> I therefore reverted the table change (disconnecting ist from Figure 1)
> and suggest the following Figure caption:
>
> "Figure 2: Sketch of difficulties persistent that impede reproducibility,
> credibility, utility and integration of models, especially in computational
> biology. An assignment of these difficulties to the four different
> concepts interpreting them as hurdles is attempted. We would like to point
> out that this sequential assessment model indicated in the graph is only
> one of the possibilities a modeler would assess the suitability of
> available models. In the rest of the paper we therefore address each
> difficulty separately. "
>
> With that sentence we can discuss the difficulties in any order.
>
> Best
> Alexander
>
> On Tue, 4 Jan 2022 at 13:24, Jacob Barhak <jacob.barhak at gmail.com> wrote:
>
> Yes Alex,
>
> Your image is much better than mine, please use it instead in the paper
> and send a link to the image itself in an acceptable image format so I can
> upload it to the submission system later on.
>
> As for changing the table and order, I am not sure. Although it may be
> possible. We need to think about it. Yet we really are crunched in time, so
> I am leaning to say no.
>
> First, because some of the difficulties span more than one category, for
> example unit standardization affects integration as well as
> reproducibility, only it appears first at reproducibility, therefore it was
> placed there in the figure.
>
> Second, if you look at the text there are some transition sentences among
> sections, that connect them, so swapping those around may just mess things.
> I am not sure it if will also mess up acronym definitions on the first
> appearance as well, so shuffling order is not something's I would do unless
> necessary. And it was not requested by reviewers anyway.
>
> I am also trying to minimize changes and focus only on what was requested
> rather than change a lot more. Otherwise we may never finish changing
> things since each one of the 18 of us will have a one last item they MUST
> have corrected. I personally did not add some recent publications I had,
> although I added one publication James asked to add in the previous
> approval round and corrected one that Sheriff mentioned. Yet I really want
> to avoid opening the paper to significant changes.
>
> If you start opening corrections beyond what was requested, this will
> drag on forever since we do have a large list of contributors and each one
> of us will generate a new idea. So please focus on what was asked unless
> absolutely necessary.
>
> So I suggest not changing the table, and changing only the figure and
> keeping order intact. I hope I am projecting the sense of urgency - we had
> the paper lying around for a couple of weeks during the holiday without
> much progress, I ask not to escalate things just because a deadline is
> looming.
>
> If you have extra time on your hands in the next couple of days, I suggest
> you focus on addressing the last two points in the response that we have
> not resolved yet. I know Gilberto wanted to fix one of them, yet you are
> welcome to do it. Those are things requested by the review
>
> Hopefully my arguments make sense.
>
> Jacob
>
> On Tue, Jan 4, 2022, 06:45 Alexander Kulesza <
> alexander.kulesza at novadiscovery.com> wrote:
>
> Dear all,
>
> I have thought about the additional Figure requested by the reviewer.
> Jacob your Figure nicely shows the assignment of the difficulties we detail
> in our central table with the 4 big concepts.
>
> I feel however that the text-heavy figure is a lot o redundancy with the
> rest of the elements (existing figure & table). Therefore I wanted to throw
> another suggestion forward (inspired by systematic reviews narrowing down
> eligible studies/data).
>
> @all please notice that the assignment I would do is a bit different to
> what is currently done in the paper. The order of sections would have to be
> altered. Not sure if that is acceptable.
> <image.png>
> The table with Difficulties (hurdles) and solutions would then be
>
> Difficulty
> Potential Solutions
>
> Reproducibility
> Models are written in different languages
> Common transport specifications such as SBML or CellML, and proper
> documentation and annotation
> Models are hard to locate
> Archive web sites such as: BioModels, SimTK, IMAGWiki, and the future
> modeleXchange
> Lack of common platforms for executing models and simulations
> Platforms such as BioSimulators, and runBioSimulations
> Unit standardization
> Standardization efforts, and machine learning solutions such as
> ClinicalUnitMapping.com <http://clinicalunitmapping.com/>
> Credibility
> Models have built-in barriers to Evaluating model credibility
> Better modeling practices, documentation, and tests.
> Data availability and measurement definitions
> Models that merge human interpretation, and newer measurement devices
>
> Missing annotations in models
> Adoption of policies such as those COMBINE suggests
> Utility
> Models are not consistently licensed to allow for reuse
> Abandoning some old school open source licenses and promoting licenses
> that release to public domain
> Different scales and modeling paradigms
> Standardization effort and centralization tools
> Stochastic modeling difficulties
> Development of tools that guarantee repeatability and standards to address
> stochastic simulations
> Integration
> Model application and implementation barriers
> Education of modelers, users, and the public
> Modeling requires adaptation towards integration
> Tools for composing models such as SBML-Comp, and SemGen
>
> Following this assignment.
>
> Please let me know what you think. I have added this suggestion in track
> changes mode to the revision draft.
>
> Best
> Alexander
>
>
> On Mon, 3 Jan 2022 at 07:27, Jacob Barhak <jacob.barhak at gmail.com> wrote:
>
> Thanks Alex,
>
> You did great by adding the new material that the reviewers actually asked
> for. This is great - I found myself adding the new FDA document to find out
> that you already added it - I just voice read it ona long drive and
> figured it fits well - it seems you were faster - great.
>
> I merged your edits and I am ok with adding your response to the reviewers
> - I only had an issue with one example paragraph that you deleted - I left
> it with a comment - please check if it is possible to keep this example
> somewhere in the text.
>
> I will assemble a letter that includes all those responses point by point
> and will try to use your text almost verbatim if possible. You really bring
> important knowledge with you.
>
> It seems most points have been answered to some level - even a figure was
> created - although I wish I was a better artist.
>
> I will try to assemble the response letter for everyone within the next
> few days. And unless there will be objections I will try to pass it by
> reviewers with a new version. If the new version is accepted, we will do a
> final approval round before publication.
>
> Many thanks to all those who contributed.
>
> Jacob
>
>
>
>
>
> On Sun, Jan 2, 2022 at 7:27 AM Alexander Kulesza <
> alexander.kulesza at novadiscovery.com> wrote:
>
> Dear all,
>
> Happy new year to everyone. I hope that you are all well and safe and have
> spent a nice holiday season.
>
> I have tried to address
>
> *Reviewer 2 *
>
>
> *Moreover, when the authors mention ASME V&V40 procedure, it could be
> useful to spend few words about the model credibility pipeline drawn for
> predictive models in biomedicine, such as the one suggested in "Credibility
> of in Silico Trial Technologies-A Theoretical Framing", Viceconti, M.,
> Juarez, M.A., Curreli, C., ...Russo, G., Pappalardo, F., IEEE Journal of
> Biomedical and Health Informatics, 2020, 24(1), pp. 4–13, 8884189. For this
> specific aspect, the authors should talk about "specific question of
> interest" and not "specific purpose" in the paragraph called "Credibility
> of Models" to follow in such a way a standardised language. *
>
> I have worked on the followng document in "track changes" mode:
>
> https://docs.google.com/document/d/1IMEgmdNkx-EsnOjGuegpenSIMmKIkK00Lc8Gred3QxM/edit
> <https://urldefense.com/v3/__https://docs.google.com/document/d/1IMEgmdNkx-EsnOjGuegpenSIMmKIkK00Lc8Gred3QxM/edit__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0hrtFQ-XI$>
> Actually, I ended up in changing rather drastically the section. I think
> that all ideas and all text is conserved, but I would very much like to
> encourage you to read, comment and challenge this revision. Please tell me
> if that goes to far. No problem to revert to an earlier version if
> necessary.
>
> I suggest the following formulation in the rebuttal letter:
>
> When trying to better describe the "credibility pipeline" (CoU definition,
> verification, validation, uncertainty quantification, e.g. as described in
> Viceconti et al. 2019) or others, we noticed that the section about
> credibility could be restructured in order to put the reader (not
> necessarily familiar with regulatory assessment of models) in the position
> to follow the argumentation.
>
> We now roughly follow the following flow of writing:
>
> - Every model needs a purpose and credibility is tied to that purpose as
> well as repeatability and reproducibility
> - Credibility is essential for models that have impact on regulated areas
> or people's lives which is why regulatory authorities issue guidance
> - The probably most advanced guidance ASME V&V40 suggests a risk-based
> approach and pipeline to establish credibility of a model. It is
> overarching and widely applicable
> - Regulators and modelers work together (for example in frame of the model
> informed drug development program MIDD) in order to better understand,
> better apply and better uptake different kinds of models in regulatory
> applications
> - More work is needed to harmonize, stay up to date and to be more
> inclusive
>
> We hope that with the rather extensive change of the wording and
> additional passages as well as citations (see below) we could address the
> concern of reviewer 2 (and the general remarks of reviewer 1).
>
> " In 2018 the American Society of Mechanical Engineers (ASME) issued an
> important guidance ASME V&V 40 (ASME, 2018) of how to assess credibility of
> computational models of medical devices through verification and validation
> (V&V). The guideline is centered around the definition of the context of
> use (CoU) of the model which is formulated based on the questions of
> interest the model will answer. The CoU is then analyzed in terms of the
> “model risk” - being the influence the model exerts on a decision and the
> potential consequences these decisions might incur. Commensurate with this
> model risk the modeler suggests establish the credibility goals, perform
> verification validation and uncertainty quantification actions and then
> assess the outcome of this exercise in order to allow judging the
> acceptability of the model CoU. Key to this guidance is its overarching
> nature which also allows adoption in other (e.g. drug development) fields
> irrespective of the model type (Kuemmel 2020, Viceconti 2019)."
>
> We also feel that it is essential to underline the cross-discipline
> viewpoint of the ASME V&V40 which is further elaborated by the cited paper
> by formalizing the verification, validation and uncertainty quantification
> VVUQ pipeline across model types.
>
> We therefore have added the following statement
> " In the paper by (Viceconti 2019) the verification, validation and
> uncertainty quantification (VVUQ) pipeline is streamlined to different
> types of models. It is, perhaps, the closest to score credibility across
> model types from mechanistic physics driven models to machine learning
> models. However, it is still short of including very recent developments
> such as ensemble models, although it touches upon the topic."
>
> We additionally strengthened our argumentation by citing additional
> literature
>
> Bai, JPF, Earp, JC, Florian, J, et al. Quantitative systems pharmacology:
> Landscape analysis of regulatory submissions to the US Food and Drug
> Administration. CPT Pharmacometrics Syst Pharmacol. 2021; 10: 1479– 1484.
> doi.org:10.1002/psp4.12709
>
> FDA (2021b). Assessing the Credibility of Computational Modeling and
> Simulation in Medical Device Submissions. FDA
> Available at:
> https://www.fda.gov/regulatory-information/search-fda-guidance-documents/assessing-credibility-computational-modeling-and-simulation-medical-device-submissions
> <https://urldefense.com/v3/__https://www.fda.gov/regulatory-information/search-fda-guidance-documents/assessing-credibility-computational-modeling-and-simulation-medical-device-submissions__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0harP7IHc$>
> [Accessed January 2, 2022]
>
> Galluppi, G.R., Brar, S., Caro, L., Chen, Y., Frey, N., Grimm, H.P., Rudd,
> D.J., Li, C.-C., Magee, M., Mukherjee, A., Nagao, L., Purohit, V.S., Roy,
> A., Salem, A.H., Sinha, V., Suleiman, A.A., Taskar, K.S., Upreti, V.V.,
> Weber, B. and Cook, J. (2021), Industrial Perspective on the Benefits
> Realized From the FDA’s Model-Informed Drug Development Paired Meeting
> Pilot Program. Clin Pharmacol Ther, 110: 1172-1175. doi:10.1002/cpt.2265
>
> Kuemmel, C., Yang, Y., Zhang, X., Florian, J., Zhu, H., Tegenge, M.,
> Huang, S.-M., Wang, Y., Morrison, T. and Zineh, I. (2020), Consideration
> of a Credibility Assessment Framework in Model-Informed Drug Development:
> Potential Application to Physiologically-Based Pharmacokinetic Modeling and
> Simulation. CPT Pharmacometrics Syst. Pharmacol., 9: 21-28. doi.org
> :10.1002/psp4.12479
>
> Musuamba, FT, Skottheim Rusten, I, Lesage, R, et al. Scientific and
> regulatory evaluation of mechanistic in silico drug and disease models in
> drug development: Building model credibility. CPT Pharmacometrics Syst
> Pharmacol. 2021; 10: 804– 825. doi:10.1002/psp4.12669
>
> Viceconti M, Emili L, Afshari P, et al. Possible Contexts of Use for In
> Silico Trials Methodologies: A Consensus-Based Review. IEEE J Biomed
> Health Inform. 2021;25(10):3977-3982. doi:10.1109/JBHI.2021.3090469
>
>
> All the best
> Alexander
>
> On Sun, 19 Dec 2021 at 03:10, Jacob Barhak <jacob.barhak at gmail.com> wrote:
>
>
> Greetings to the paper contributors,
>
> You may have seen my other message where I posted the reviews to our
> paper. I am sending those again below to start a new thread discussing
> possible revisions.
>
> Here is a checklist for revisions required:
>
>
> 1. We should discuss the references that reviewer 1 raised and add
> them as reference
> 2. Reviewer 1 asked for more discussion around V&V40 - I believe Alex
> added the text there originally - Alex do you think you can address the
> request of the reviewer?
> 3. Reviewer 1 asked to focus on COVID-19 at the title - I am not sure
> we wish to limit our scope since many of our ideas are applicable far
> beyond COVID-19 - I leave it up for discussion in the group on how to
> address this request by the reviewer
> 4. Reviewer 1 asked for additional graphics - I am unsure how to
> address this beyond ur current diagram - ideas will help
> 5. We need to check the "Contribution to the field" section for
> grammar and spelling - if someone can contribute more elegant text - now is
> the time - I looked over it now and found no issues - yet I may have missed
> something - this section may have been missed since it was added last and
> perhaps vetted less than other sections. Also Reviewer 2 asked for grammar
> corrections, so it is worthwhile proof reading the paper as a whole.
> 6. Reviewer 2 asks for major rewrite to emphasize why the problems are
> there - I think we should explain that the core of the paper is the table
> and perhaps emphasize it in the paper beyond what the current text - we
> also should reply to the reviewer and explain that the paper is composed of
> contributions from a large group and we made an effort to include every
> voice in the choir - each of those voices is important and needs to be
> preserved - hopefully it will convince the reviewer that change we will add
> will be sufficient
> 7. Reviewer 2 asks to revise the introduction - I believe some
> changes are possible - yet the introduction includes contributions from
> many authors - at least 10 - and I fear losing something important someone
> contributed - if someone has an idea on how to address this reviewer
> without a painful transformation, please reply to this message.
> 8. Reviewer 2 wants additional discussion around the "reproducibility
> crisis" section and asks a valid question about expectations we should
> address - "“Computational biomedical modeling… was expected to be less
> affected by the reproducibility crisis.” Is this true, and why would it be
> so? " - My answer to the reviewer is that unlike biological processes that
> have random nature and experiments would not repeat if repeated, while
> computer software should be deterministic and it should be repeatable if
> designed well. Unfortunately we are not experiencing this promise - yet I
> believe the reviewer wants more discussion beyond this section so I am
> happy to discuss this.
> 9. Reviewer 2- writes: "Models are Hard to Locate: Are the authors
> suggesting that entire simulation workflows, from model construction to
> analysis, should be publicly available? At what point does one consider
> intellectual property? Do the authors advocate for such extensive
> publishing for all models, or only ones that are intended to be widely
> re-used? - those are important points we need to discuss - we need to
> better explain the difficulties we are having when creating models and the
> reviewer is absolutely correct about expanding the discussion to IP - I
> suggest we create a thread for this discussion and reference it - I
> suggest it we merged with the licensing issue that I will address later -
> there is a strong connection there - the reviewer was very observant and
> sees the bigger problem. However,we need to eventually distill our
> discussion to recommendations that will inline.
> 10. Reviewer 2 asks that we fix the unit standardization section - I
> believe Hana and myself were the largest contributors there - Hana - I will
> start a discussion on that topic in a separate email where we can publicly
> discuss how to fix this - others will be welcome to contribute.
> 11. Reviewer 2 asks that we better handle the section on Data
> availability and measurement definitions: in think we need to emphasize
> solutions and separate it from issues that may not be solvable. Ideas are
> welcome.
> 12. Reviewer 2 asks good questions with regards to licensing following
> our text- I personally have good answers to the reviewer and William and I
> had some discussion on the topic in this list I suggest we expand this
> discussion in a separate thread - hopefully William and perhaps others will
> join the discussion. This discussion should also address the IP issues
> raised by the reviewer for the "models are hard to locate section."
> 13. Reviewer 2 asks to handle the Model Application and Implementation
> Barriers section. We should decide what to do there, the section may need
> expansion since the ideas there are solid, yet the section is short so
> perhaps enhancing it makes better sense. I am open to suggestions.
>
>
> Those are the items I located and my suggestions. It seems we need
> attention from Alex, Hana, myself and william. However, anyone on the list
> is welcome to participate and suggest changes.
>
> I will start the discussion threads on specific topics. Hopefully we can
> get it done quickly.
>
> Jacob
>
>
>
>
>
>
> -
>
>
>
>
>
> ########### Original Reviews #############
>
> There are 2 reviews - both require major changes. I am copying the
> relevant text below. If more appear, I will let you know, yet I only got
> this message today although the reviews are dated a few days ago.
>
> Reviewer 1:
> Recommendation for the Editor: Substantial revision is required
>
> Please list your revision requests for the authors and provide your
> detailed comments, including highlighting limitations and strengths of the
> review. If you have additional comments based on Q2 and Q3 you can add them
> as well.
>
> Karr and co-authors made an interesting and exaustive point about the
> reproducibility crisis that leads to inability to reuse and integrate
> models, especially about COVID-19 disease.
> Within the manuscript some typos and missing information are present.
> I'd suggest the authors to revise the entire manuscript especially in
> terms of the state of the art, revising and updating the most relevant
> examples of computational models dealing with COVID-19 and in general about
> some semi-standardised proposals about the pipeline to follow for the
> verification and validation of model credibility. In particular, the
> authors failed to mention and cite some major results on in silico modeling
> about COVID-19 up to now. See for example:
> a."In silico trial to test COVID-19 candidate vaccines: a case study with
> UISS platform", Russo, G., Pennisi, M., Fichera, E., ...Viceconti, M.,
> Pappalardo, F., BMC Bioinformatics, 2020, 21, 527.
> b. Russo G, Di Salvatore V, Sgroi G, Parasiliti Palumbo GA, Reche PA,
> Pappalardo F. "A multi-step and multi-scale bioinformatic protocol to
> investigate potential SARS-CoV-2 vaccine targets" [published online ahead
> of print, 2021 Oct 5]. Brief Bioinform. 2021;bbab403.
> doi:10.1093/bib/bbab403.
> Moreover, when the authors mention ASME V&V40 procedure, it could be
> useful to spend few words about the model credibility pipeline drawn for
> predictive models in biomedicine, such as the one suggested in "Credibility
> of in Silico Trial Technologies-A Theoretical Framing", Viceconti, M.,
> Juarez, M.A., Curreli, C., ...Russo, G., Pappalardo, F., IEEE Journal of
> Biomedical and Health Informatics, 2020, 24(1), pp. 4–13, 8884189.
> For this specific aspect, the authors should talk about "specific question
> of interest" and not "specific purpose" in the paragraph called
> "Credibility of Models" to follow in such a way a standardised language.
> Moreover, the authors should refer to COVID-19 also in the title, with a
> specific mention about the fact that the main topic of model integration in
> computational biology will be discussed inside the COVID-19 context.
> Furthermore, for the model key concepts such as
> Reusability-Extensibility-Extractability-Portability the authors should
> described and outlined through a graphical sketch or visual representation
> that summarises these key point.
> The authors should also fix some grammar and writing typos present in the
> "Contribution to the field" section.
>
> a. Is the quality of the figures and tables satisfactory?
> - No
>
> b. Does the reference list cover the relevant literature adequately and in
> an unbiased manner?
> - No
>
> c. Does this manuscript refer only to published data? (unpublished or
> original data is not allowed for this article type)
> - Yes
>
> d. Does the review include a balanced, comprehensive, and critical view of
> the research area?
> - Yes
>
>
> Reviewer 2
>
> Please list your revision requests for the authors and provide your
> detailed comments, including highlighting limitations and strengths of the
> review. If you have additional comments based on Q2 and Q3 you can add them
> as well.
>
>
> The article often reads as a stream-of-consciousness account of the
> discussions that took place but lacks a clear thesis or recommendations. It
> is not clear what, if anything, the authors are advocating for. It is not
> always clear why the issues being discussed are problematic, or that they
> can be reasonably addressed. Some of the issues raised are indeed important
> and should be discussed, but the paper lacks focus and does not tell a
> cohesive story. I believe this manuscript requires a major re-write to be
> suitable for publication. The authors should consider narrowing the scope
> of the discussion and focusing on a cohesive set of recommendations or open
> questions. More specifically, I make a few suggestions below:
>
> Major comments
> 1. The introduction is long and repeats itself (e.g., “much less is known
> about how viral infections spread throughout the body…” is repeated
> verbatim). It is not clear from the introduction what the main goal of the
> paper is or why the “reproducibility crisis” is truly a crisis. Why is the
> discussion of composition and black/white box models relevant to the
> introduction? Further, this section is subtitled “the promise of modeling”,
> which does not seem to match the content.
> 2. The Reproducibility Crisis: “Computational biomedical modeling… was
> expected to be less affected by the reproducibility crisis.” Is this true,
> and why would it be so? One would think that more complex models would
> suffer more from a lack of reproducibility. It may be helpful to define
> what exactly the “reproducibility crisis” refers to.
> 3. Models are Hard to Locate: Are the authors suggesting that entire
> simulation workflows, from model construction to analysis, should be
> publicly available? At what point does one consider intellectual property?
> Do the authors advocate for such extensive publishing for all models, or
> only ones that are intended to be widely re-used?
> 4. Unit standardization: The conversion from PFU or TCID50 to individual
> virions is likely to differ across viruses – are the authors focused on
> COVID here? Are the authors advocating for a standard conversion factor? It
> is not clear what the purpose of this discussion is. As the authors
> mention, different scales require different units. Even at a single scale,
> different models may require different units for numerical reasons. It is
> not clear what the authors are advocating for here.
> 5. Data availability and measurement definitions: This section seems to
> outline limitations of available data, but again makes no recommendations
> or proposed solution to any of the issues raised. Is this the intention?
> Most of the issues raised here reflect limitations of experimental science
> or data privacy, which likely cannot be meaningfully addressed by the
> modeling community.
> 6. Models are Not Consistently Licensed…: Are the authors implying here
> that all modeling work should be published with no rights reserved? Is it
> reasonable to expect modelers to make their work freely usable by others
> for profit? Is it reasonable for institutions to allow this? How much does
> this really contribute to reproducibility and utility?
> 7. Model Application and Implementation Barriers: This section seems
> unnecessary and out of place.
> 8. There are grammar and punctuation errors scattered throughout; please
> edit carefully.
>
>
> a. Is the quality of the figures and tables satisfactory?
> - Not Applicable
>
> b. Does the reference list cover the relevant literature adequately and in
> an unbiased manner?
> - Yes
>
> c. Does this manuscript refer only to published data? (unpublished or
> original data is not allowed for this article type)
> - Yes
>
> d. Does the review include a balanced, comprehensive, and critical view of
> the research area?
> No answer given.
>
>
> ========
> _______________________________________________
> Vp-reproduce-subgroup mailing list
> Vp-reproduce-subgroup at lists.simtk.org
> https://lists.simtk.org/mailman/listinfo/vp-reproduce-subgroup
> <https://urldefense.com/v3/__https://lists.simtk.org/mailman/listinfo/vp-reproduce-subgroup__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0hhiovw3M$>
>
>
>
> --
> Alexander Kulesza
> Team leader
> Modeling & simulation / Biomodeling
> alexander.kulesza at novadiscovery.com
> +33 7 82 92 44 62
> nova
> DISCOVERY
> www.novadiscovery.com
> <https://urldefense.com/v3/__http://www.novadiscovery.com/__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0hwI_2O8g$>
> 1 Place Verrazzano, 69009 Lyon
> <https://www.google.com/maps/search/1+Place+Verrazzano,+69009+Lyon?entry=gmail&source=g>
> +33 9 72 53 13 01
>
> *This message is intended only for the personal and confidential use of
> the designated recipient(s) named above. If you are not the intended
> recipient of this message you are hereby notified that any review,
> dissemination, distribution or copying of this message is strictly
> prohibited. Email transmission cannot be guaranteed to be secure or
> error-free. Therefore, we do not represent that this information is
> complete or accurate and it should not be relied upon as such. All
> information is subject to change without notice.*
>
>
>
> --
> Alexander Kulesza
> Team leader
> Modeling & simulation / Biomodeling
> alexander.kulesza at novadiscovery.com
> +33 7 82 92 44 62
> nova
> DISCOVERY
> www.novadiscovery.com
> <https://urldefense.com/v3/__http://www.novadiscovery.com/__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0hwI_2O8g$>
> 1 Place Verrazzano, 69009 Lyon
> <https://www.google.com/maps/search/1+Place+Verrazzano,+69009+Lyon?entry=gmail&source=g>
> +33 9 72 53 13 01
>
> *This message is intended only for the personal and confidential use of
> the designated recipient(s) named above. If you are not the intended
> recipient of this message you are hereby notified that any review,
> dissemination, distribution or copying of this message is strictly
> prohibited. Email transmission cannot be guaranteed to be secure or
> error-free. Therefore, we do not represent that this information is
> complete or accurate and it should not be relied upon as such. All
> information is subject to change without notice.*
>
>
>
> --
> Alexander Kulesza
> Team leader
> Modeling & simulation / Biomodeling
> alexander.kulesza at novadiscovery.com
> +33 7 82 92 44 62
> nova
> DISCOVERY
> www.novadiscovery.com
> <https://urldefense.com/v3/__http://www.novadiscovery.com/__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0hwI_2O8g$>
> 1 Place Verrazzano, 69009 Lyon
> <https://www.google.com/maps/search/1+Place+Verrazzano,+69009+Lyon?entry=gmail&source=g>
> +33 9 72 53 13 01
>
> *This message is intended only for the personal and confidential use of
> the designated recipient(s) named above. If you are not the intended
> recipient of this message you are hereby notified that any review,
> dissemination, distribution or copying of this message is strictly
> prohibited. Email transmission cannot be guaranteed to be secure or
> error-free. Therefore, we do not represent that this information is
> complete or accurate and it should not be relied upon as such. All
> information is subject to change without notice.*
>
> _______________________________________________
> Vp-integration-subgroup mailing list
> Vp-integration-subgroup at lists.simtk.org
> https://lists.simtk.org/mailman/listinfo/vp-integration-subgroup
>
>
> _______________________________________________
> Vp-integration-subgroup mailing list
> Vp-integration-subgroup at lists.simtk.org
> https://lists.simtk.org/mailman/listinfo/vp-integration-subgroup
>
>
>
> --
> -- ------------------
> Yaling Liu, Professor
> Department of Bioengineering, Director of Graduate Studies
> Department of Mechanical Engineering and Mechanics
> Lehigh University
> www.lehigh.edu/~yal310
> ______
>
> --
Tingting Tang
Assistant Professor
San Diego State University Imperial Valley
Office: FOBE 110
Phone: 760-768-5531
720 Heber Ave
Calexico, CA 92231
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.simtk.org/pipermail/vp-integration-subgroup/attachments/20220114/f2e57f2c/attachment-0001.html>
More information about the Vp-integration-subgroup
mailing list