[Vp-integration-subgroup] [Vp-reproduce-subgroup] Paper review and revision check list
Jacob Barhak
jacob.barhak at gmail.com
Thu Jan 6 20:52:44 PST 2022
Thanks Hana, Thanks William, and thanks to Gilberto, Alex, and Sheriff,
With your help and some effort, the revised paper was resubmitted in time -
just now.
William, please keep on reminding me about your second affiliation - I kept
a comment as a reminder to do this - the submission system did not allow
making any changes in Author information - not sure why - if this gets
accepted we should ask the editor to add your affiliation manually.
I urge all other authors to look at the changes we made and our response.
We will have to approve the manuscript with everyone before it gets
published - hopefully we will convince the reviewers this review round.
Jacob
On Thu, Jan 6, 2022 at 3:51 PM Dobrovolny, Hana <h.dobrovolny at tcu.edu>
wrote:
> I've read through the paper and fixed a few grammar mistakes.
>
> Hana
>
>
> *******************************************************
> Dr. Hana Dobrovolny
> Associate Professor of Biophysics
> Texas Christian University
> TCU Box 298840
> Fort Worth, TX 76129
>
> phone: (817) 257-6379 fax: (817) 257-7742
> email: h.dobrovolny at tcu.edu
> *******************************************************
>
>
> ------------------------------
> *From:* Vp-integration-subgroup <
> vp-integration-subgroup-bounces at lists.simtk.org> on behalf of Jacob
> Barhak <jacob.barhak at gmail.com>
> *Sent:* January 5, 2022 11:48 PM
> *To:* William Waites; James Glazier
> *Cc:* vp-reproduce-subgroup at lists.simtk.org;
> vp-integration-subgroup at lists.simtk.org; John Gennari; Faeder, James R
> *Subject:* Re: [Vp-integration-subgroup] [Vp-reproduce-subgroup] Paper
> review and revision check list
>
>
> *[EXTERNAL EMAIL WARNING]* DO NOT CLICK LINKS or open attachments unless
> you recognize the sender and know the content is safe.
> Greetings paper contributors,
>
> Gilberto has addressed the final point and added text in the paper and
> review, thus concluding our response.
>
> I went over the paper from start to finish again and cleaned it up.
>
> There are two last comments in the paper addressed to William Waites and
> James Glazier - if you can quickly check those, it would help.
>
> If someone else wants to read the paper once more, please do this today -
> otherwise I will convert the paper and our response to reviewers and submit
> it to the Journal portal.
>
> If anyone has any objections, now is the time to raise them so we can
> withdraw rather than submit.
>
> Here is the link to the revised paper:
>
> https://docs.google.com/document/d/1IMEgmdNkx-EsnOjGuegpenSIMmKIkK00Lc8Gred3QxM/edit?usp=sharing
> <https://urldefense.com/v3/__https://docs.google.com/document/d/1IMEgmdNkx-EsnOjGuegpenSIMmKIkK00Lc8Gred3QxM/edit?usp=sharing__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0hvK3OmJ8$>
>
> Here is the link to response for reviewers:
>
> https://docs.google.com/document/d/1bO1CDbMMlyyweYayP9rGohPIfgvIhfe14sFchtJMJ-s/edit?usp=sharing
> <https://urldefense.com/v3/__https://docs.google.com/document/d/1bO1CDbMMlyyweYayP9rGohPIfgvIhfe14sFchtJMJ-s/edit?usp=sharing__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0haRkeB0o$>
>
>
> Hopefully there will be no objections and can submit this tomorrow.
>
> Jacob
>
>
>
>
>
>
>
>
> On Tue, Jan 4, 2022 at 11:30 PM Jacob Barhak <jacob.barhak at gmail.com>
> wrote:
>
>> Hi Alex, Hi William, Hi Gilberto, and all other contributors.
>>
>> Alex, William and Gilberto are mentioned since we are the most active
>> contributors.
>>
>> We have 2 days to finish edits.
>>
>> William - I attempted to add your suggestions to the paper
>> regarding issue 11. You are free to revise further.
>>
>> Gilberto - I merged some of your texts, yet it seems you are the only one
>> I saw interested in resolving issue 12 where the reviewer asks to remove
>> the "Model Application and Implementation Barriers" section. Please suggest
>> your corrections as soon as possible. If you cannot do this by tomorrow I
>> will do my best to add a response based on your idea. Yet I really hope you
>> can do this before me.
>>
>> Please finalize any edits you have - and if anyone has any objection to
>> the text, please raise it within one day since after tomorrow I will start
>> closing down all issues.
>>
>> I would ask for a volunteer that will go over the paper once more after
>> its finalized and check for typos and grammar - we do not want the paper to
>> return again due to this issue. Time is tight, so please volunteer fast.
>>
>> Hopefully we can get it done in time.
>>
>> Jacob
>>
>>
>>
>>
>>
>> On Tue, Jan 4, 2022 at 7:25 AM Alexander Kulesza <
>> alexander.kulesza at novadiscovery.com> wrote:
>>
>>> Dear Jacob, dear all,
>>> thanks for endorsing my figure. I'll gladly add this. Please find
>>> enclosed the .png.
>>>
>>> What you say makes sense and I agree that exaggerating changes is not
>>> speeding up approval. Please still consider it as a tool to perform "major"
>>> change (ordering of sections is a major change of a paper in my opinion)
>>> without changing too much the sections as such. But I agree that this could
>>> be tried out should the reviewers insist.
>>>
>>> I therefore reverted the table change (disconnecting ist from Figure 1)
>>> and suggest the following Figure caption:
>>>
>>> "Figure 2: Sketch of difficulties persistent that impede
>>> reproducibility, credibility, utility and integration of models, especially
>>> in computational biology. An assignment of these difficulties to the
>>> four different concepts interpreting them as hurdles is attempted. We would
>>> like to point out that this sequential assessment model indicated in the
>>> graph is only one of the possibilities a modeler would assess the
>>> suitability of available models. In the rest of the paper we therefore
>>> address each difficulty separately. "
>>>
>>> With that sentence we can discuss the difficulties in any order.
>>>
>>> Best
>>> Alexander
>>>
>>> On Tue, 4 Jan 2022 at 13:24, Jacob Barhak <jacob.barhak at gmail.com>
>>> wrote:
>>>
>>>> Yes Alex,
>>>>
>>>> Your image is much better than mine, please use it instead in the paper
>>>> and send a link to the image itself in an acceptable image format so I can
>>>> upload it to the submission system later on.
>>>>
>>>> As for changing the table and order, I am not sure. Although it may be
>>>> possible. We need to think about it. Yet we really are crunched in time, so
>>>> I am leaning to say no.
>>>>
>>>> First, because some of the difficulties span more than one category,
>>>> for example unit standardization affects integration as well as
>>>> reproducibility, only it appears first at reproducibility, therefore it was
>>>> placed there in the figure.
>>>>
>>>> Second, if you look at the text there are some transition sentences
>>>> among sections, that connect them, so swapping those around may just mess
>>>> things. I am not sure it if will also mess up acronym definitions on the
>>>> first appearance as well, so shuffling order is not something's I would do
>>>> unless necessary. And it was not requested by reviewers anyway.
>>>>
>>>> I am also trying to minimize changes and focus only on what was
>>>> requested rather than change a lot more. Otherwise we may never finish
>>>> changing things since each one of the 18 of us will have a one last item
>>>> they MUST have corrected. I personally did not add some recent publications
>>>> I had, although I added one publication James asked to add in the previous
>>>> approval round and corrected one that Sheriff mentioned. Yet I really want
>>>> to avoid opening the paper to significant changes.
>>>>
>>>> If you start opening corrections beyond what was requested, this will
>>>> drag on forever since we do have a large list of contributors and each one
>>>> of us will generate a new idea. So please focus on what was asked unless
>>>> absolutely necessary.
>>>>
>>>> So I suggest not changing the table, and changing only the figure and
>>>> keeping order intact. I hope I am projecting the sense of urgency - we had
>>>> the paper lying around for a couple of weeks during the holiday without
>>>> much progress, I ask not to escalate things just because a deadline is
>>>> looming.
>>>>
>>>> If you have extra time on your hands in the next couple of days, I
>>>> suggest you focus on addressing the last two points in the response that we
>>>> have not resolved yet. I know Gilberto wanted to fix one of them, yet you
>>>> are welcome to do it. Those are things requested by the review
>>>>
>>>> Hopefully my arguments make sense.
>>>>
>>>> Jacob
>>>>
>>>> On Tue, Jan 4, 2022, 06:45 Alexander Kulesza <
>>>> alexander.kulesza at novadiscovery.com> wrote:
>>>>
>>>>> Dear all,
>>>>>
>>>>> I have thought about the additional Figure requested by the reviewer.
>>>>> Jacob your Figure nicely shows the assignment of the difficulties we detail
>>>>> in our central table with the 4 big concepts.
>>>>>
>>>>> I feel however that the text-heavy figure is a lot o redundancy with
>>>>> the rest of the elements (existing figure & table). Therefore I wanted to
>>>>> throw another suggestion forward (inspired by systematic reviews narrowing
>>>>> down eligible studies/data).
>>>>>
>>>>> @all please notice that the assignment I would do is a bit different
>>>>> to what is currently done in the paper. The order of sections would have to
>>>>> be altered. Not sure if that is acceptable.
>>>>> [image: image.png]
>>>>> The table with Difficulties (hurdles) and solutions would then be
>>>>>
>>>>> Difficulty
>>>>>
>>>>> Potential Solutions
>>>>>
>>>>> Reproducibility
>>>>>
>>>>> Models are written in different languages
>>>>>
>>>>> Common transport specifications such as SBML or CellML, and proper
>>>>> documentation and annotation
>>>>>
>>>>> Models are hard to locate
>>>>>
>>>>> Archive web sites such as: BioModels, SimTK, IMAGWiki, and the future
>>>>> modeleXchange
>>>>>
>>>>> Lack of common platforms for executing models and simulations
>>>>>
>>>>> Platforms such as BioSimulators, and runBioSimulations
>>>>>
>>>>> Unit standardization
>>>>>
>>>>> Standardization efforts, and machine learning solutions such as
>>>>> ClinicalUnitMapping.com
>>>>>
>>>>> Credibility
>>>>>
>>>>> Models have built-in barriers to Evaluating model credibility
>>>>>
>>>>> Better modeling practices, documentation, and tests.
>>>>>
>>>>> Data availability and measurement definitions
>>>>>
>>>>> Models that merge human interpretation, and newer measurement devices
>>>>>
>>>>> Missing annotations in models
>>>>>
>>>>> Adoption of policies such as those COMBINE suggests
>>>>>
>>>>> Utility
>>>>>
>>>>> Models are not consistently licensed to allow for reuse
>>>>>
>>>>> Abandoning some old school open source licenses and promoting licenses
>>>>> that release to public domain
>>>>>
>>>>> Different scales and modeling paradigms
>>>>>
>>>>> Standardization effort and centralization tools
>>>>>
>>>>> Stochastic modeling difficulties
>>>>>
>>>>> Development of tools that guarantee repeatability and standards to
>>>>> address stochastic simulations
>>>>>
>>>>> Integration
>>>>>
>>>>> Model application and implementation barriers
>>>>>
>>>>> Education of modelers, users, and the public
>>>>>
>>>>> Modeling requires adaptation towards integration
>>>>>
>>>>> Tools for composing models such as SBML-Comp, and SemGen
>>>>>
>>>>> Following this assignment.
>>>>>
>>>>> Please let me know what you think. I have added this suggestion in
>>>>> track changes mode to the revision draft.
>>>>>
>>>>> Best
>>>>> Alexander
>>>>>
>>>>>
>>>>> On Mon, 3 Jan 2022 at 07:27, Jacob Barhak <jacob.barhak at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Thanks Alex,
>>>>>>
>>>>>> You did great by adding the new material that the reviewers actually
>>>>>> asked for. This is great - I found myself adding the new FDA document to
>>>>>> find out that you already added it - I just voice read it ona long drive
>>>>>> and figured it fits well - it seems you were faster - great.
>>>>>>
>>>>>> I merged your edits and I am ok with adding your response to the
>>>>>> reviewers - I only had an issue with one example paragraph that you deleted
>>>>>> - I left it with a comment - please check if it is possible to keep this
>>>>>> example somewhere in the text.
>>>>>>
>>>>>> I will assemble a letter that includes all those responses point by
>>>>>> point and will try to use your text almost verbatim if possible. You really
>>>>>> bring important knowledge with you.
>>>>>>
>>>>>> It seems most points have been answered to some level - even a figure
>>>>>> was created - although I wish I was a better artist.
>>>>>>
>>>>>> I will try to assemble the response letter for everyone within the
>>>>>> next few days. And unless there will be objections I will try to pass it by
>>>>>> reviewers with a new version. If the new version is accepted, we will do a
>>>>>> final approval round before publication.
>>>>>>
>>>>>> Many thanks to all those who contributed.
>>>>>>
>>>>>> Jacob
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sun, Jan 2, 2022 at 7:27 AM Alexander Kulesza <
>>>>>> alexander.kulesza at novadiscovery.com> wrote:
>>>>>>
>>>>>>> Dear all,
>>>>>>>
>>>>>>> Happy new year to everyone. I hope that you are all well and safe
>>>>>>> and have spent a nice holiday season.
>>>>>>>
>>>>>>> I have tried to address
>>>>>>>
>>>>>>> *Reviewer 2 *
>>>>>>>
>>>>>>>
>>>>>>> *Moreover, when the authors mention ASME V&V40 procedure, it could
>>>>>>> be useful to spend few words about the model credibility pipeline drawn for
>>>>>>> predictive models in biomedicine, such as the one suggested in "Credibility
>>>>>>> of in Silico Trial Technologies-A Theoretical Framing", Viceconti, M.,
>>>>>>> Juarez, M.A., Curreli, C., ...Russo, G., Pappalardo, F., IEEE Journal of
>>>>>>> Biomedical and Health Informatics, 2020, 24(1), pp. 4–13, 8884189. For this
>>>>>>> specific aspect, the authors should talk about "specific question of
>>>>>>> interest" and not "specific purpose" in the paragraph called "Credibility
>>>>>>> of Models" to follow in such a way a standardised language. *
>>>>>>>
>>>>>>> I have worked on the followng document in "track changes" mode:
>>>>>>>
>>>>>>> https://docs.google.com/document/d/1IMEgmdNkx-EsnOjGuegpenSIMmKIkK00Lc8Gred3QxM/edit
>>>>>>> <https://urldefense.com/v3/__https://docs.google.com/document/d/1IMEgmdNkx-EsnOjGuegpenSIMmKIkK00Lc8Gred3QxM/edit__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0hrtFQ-XI$>
>>>>>>> Actually, I ended up in changing rather drastically the section. I
>>>>>>> think that all ideas and all text is conserved, but I would very much like
>>>>>>> to encourage you to read, comment and challenge this revision. Please tell
>>>>>>> me if that goes to far. No problem to revert to an earlier version if
>>>>>>> necessary.
>>>>>>>
>>>>>>> I suggest the following formulation in the rebuttal letter:
>>>>>>>
>>>>>>> When trying to better describe the "credibility pipeline" (CoU
>>>>>>> definition, verification, validation, uncertainty quantification, e.g. as
>>>>>>> described in Viceconti et al. 2019) or others, we noticed that the section
>>>>>>> about credibility could be restructured in order to put the reader (not
>>>>>>> necessarily familiar with regulatory assessment of models) in the position
>>>>>>> to follow the argumentation.
>>>>>>>
>>>>>>> We now roughly follow the following flow of writing:
>>>>>>>
>>>>>>> - Every model needs a purpose and credibility is tied to that
>>>>>>> purpose as well as repeatability and reproducibility
>>>>>>> - Credibility is essential for models that have impact on regulated
>>>>>>> areas or people's lives which is why regulatory authorities issue guidance
>>>>>>> - The probably most advanced guidance ASME V&V40 suggests a
>>>>>>> risk-based approach and pipeline to establish credibility of a model. It is
>>>>>>> overarching and widely applicable
>>>>>>> - Regulators and modelers work together (for example in frame of the
>>>>>>> model informed drug development program MIDD) in order to better
>>>>>>> understand, better apply and better uptake different kinds of models in
>>>>>>> regulatory applications
>>>>>>> - More work is needed to harmonize, stay up to date and to be more
>>>>>>> inclusive
>>>>>>>
>>>>>>> We hope that with the rather extensive change of the wording and
>>>>>>> additional passages as well as citations (see below) we could address the
>>>>>>> concern of reviewer 2 (and the general remarks of reviewer 1).
>>>>>>>
>>>>>>> " In 2018 the American Society of Mechanical Engineers (ASME) issued
>>>>>>> an important guidance ASME V&V 40 (ASME, 2018) of how to assess credibility
>>>>>>> of computational models of medical devices through verification and
>>>>>>> validation (V&V). The guideline is centered around the definition of the
>>>>>>> context of use (CoU) of the model which is formulated based on the
>>>>>>> questions of interest the model will answer. The CoU is then analyzed in
>>>>>>> terms of the “model risk” - being the influence the model exerts on a
>>>>>>> decision and the potential consequences these decisions might incur.
>>>>>>> Commensurate with this model risk the modeler suggests establish the
>>>>>>> credibility goals, perform verification validation and uncertainty
>>>>>>> quantification actions and then assess the outcome of this exercise in
>>>>>>> order to allow judging the acceptability of the model CoU. Key to this
>>>>>>> guidance is its overarching nature which also allows adoption in other
>>>>>>> (e.g. drug development) fields irrespective of the model type (Kuemmel
>>>>>>> 2020, Viceconti 2019)."
>>>>>>>
>>>>>>> We also feel that it is essential to underline the cross-discipline
>>>>>>> viewpoint of the ASME V&V40 which is further elaborated by the cited paper
>>>>>>> by formalizing the verification, validation and uncertainty quantification
>>>>>>> VVUQ pipeline across model types.
>>>>>>>
>>>>>>> We therefore have added the following statement
>>>>>>> " In the paper by (Viceconti 2019) the verification, validation and
>>>>>>> uncertainty quantification (VVUQ) pipeline is streamlined to different
>>>>>>> types of models. It is, perhaps, the closest to score credibility across
>>>>>>> model types from mechanistic physics driven models to machine learning
>>>>>>> models. However, it is still short of including very recent developments
>>>>>>> such as ensemble models, although it touches upon the topic."
>>>>>>>
>>>>>>> We additionally strengthened our argumentation by citing additional
>>>>>>> literature
>>>>>>>
>>>>>>> Bai, JPF, Earp, JC, Florian, J, et al. Quantitative systems
>>>>>>> pharmacology: Landscape analysis of regulatory submissions to the US Food
>>>>>>> and Drug Administration. CPT Pharmacometrics Syst Pharmacol. 2021;
>>>>>>> 10: 1479– 1484. doi.org:10.1002/psp4.12709
>>>>>>>
>>>>>>> FDA (2021b). Assessing the Credibility of Computational Modeling and
>>>>>>> Simulation in Medical Device Submissions. FDA
>>>>>>> Available at:
>>>>>>> https://www.fda.gov/regulatory-information/search-fda-guidance-documents/assessing-credibility-computational-modeling-and-simulation-medical-device-submissions
>>>>>>> <https://urldefense.com/v3/__https://www.fda.gov/regulatory-information/search-fda-guidance-documents/assessing-credibility-computational-modeling-and-simulation-medical-device-submissions__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0harP7IHc$>
>>>>>>> [Accessed January 2, 2022]
>>>>>>>
>>>>>>> Galluppi, G.R., Brar, S., Caro, L., Chen, Y., Frey, N., Grimm, H.P.,
>>>>>>> Rudd, D.J., Li, C.-C., Magee, M., Mukherjee, A., Nagao, L., Purohit, V.S.,
>>>>>>> Roy, A., Salem, A.H., Sinha, V., Suleiman, A.A., Taskar, K.S.,
>>>>>>> Upreti, V.V., Weber, B. and Cook, J. (2021), Industrial Perspective on the
>>>>>>> Benefits Realized From the FDA’s Model-Informed Drug Development Paired
>>>>>>> Meeting Pilot Program. Clin Pharmacol Ther, 110: 1172-1175.
>>>>>>> doi:10.1002/cpt.2265
>>>>>>>
>>>>>>> Kuemmel, C., Yang, Y., Zhang, X., Florian, J., Zhu, H., Tegenge, M.,
>>>>>>> Huang, S.-M., Wang, Y., Morrison, T. and Zineh, I. (2020), Consideration
>>>>>>> of a Credibility Assessment Framework in Model-Informed Drug Development:
>>>>>>> Potential Application to Physiologically-Based Pharmacokinetic Modeling and
>>>>>>> Simulation. CPT Pharmacometrics Syst. Pharmacol., 9: 21-28.
>>>>>>> doi.org:10.1002/psp4.12479
>>>>>>>
>>>>>>> Musuamba, FT, Skottheim Rusten, I, Lesage, R, et al. Scientific and
>>>>>>> regulatory evaluation of mechanistic in silico drug and disease models in
>>>>>>> drug development: Building model credibility. CPT Pharmacometrics Syst
>>>>>>> Pharmacol. 2021; 10: 804– 825. doi:10.1002/psp4.12669
>>>>>>>
>>>>>>> Viceconti M, Emili L, Afshari P, et al. Possible Contexts of Use for
>>>>>>> In Silico Trials Methodologies: A Consensus-Based Review. IEEE J
>>>>>>> Biomed Health Inform. 2021;25(10):3977-3982.
>>>>>>> doi:10.1109/JBHI.2021.3090469
>>>>>>>
>>>>>>>
>>>>>>> All the best
>>>>>>> Alexander
>>>>>>>
>>>>>>> On Sun, 19 Dec 2021 at 03:10, Jacob Barhak <jacob.barhak at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> Greetings to the paper contributors,
>>>>>>>>
>>>>>>>> You may have seen my other message where I posted the reviews to
>>>>>>>> our paper. I am sending those again below to start a new thread discussing
>>>>>>>> possible revisions.
>>>>>>>>
>>>>>>>> Here is a checklist for revisions required:
>>>>>>>>
>>>>>>>>
>>>>>>>> 1. We should discuss the references that reviewer 1 raised and
>>>>>>>> add them as reference
>>>>>>>> 2. Reviewer 1 asked for more discussion around V&V40 - I
>>>>>>>> believe Alex added the text there originally - Alex do you think you can
>>>>>>>> address the request of the reviewer?
>>>>>>>> 3. Reviewer 1 asked to focus on COVID-19 at the title - I am
>>>>>>>> not sure we wish to limit our scope since many of our ideas are applicable
>>>>>>>> far beyond COVID-19 - I leave it up for discussion in the group on how to
>>>>>>>> address this request by the reviewer
>>>>>>>> 4. Reviewer 1 asked for additional graphics - I am unsure how
>>>>>>>> to address this beyond ur current diagram - ideas will help
>>>>>>>> 5. We need to check the "Contribution to the field" section for
>>>>>>>> grammar and spelling - if someone can contribute more elegant text - now is
>>>>>>>> the time - I looked over it now and found no issues - yet I may have missed
>>>>>>>> something - this section may have been missed since it was added last and
>>>>>>>> perhaps vetted less than other sections. Also Reviewer 2 asked for grammar
>>>>>>>> corrections, so it is worthwhile proof reading the paper as a whole.
>>>>>>>> 6. Reviewer 2 asks for major rewrite to emphasize why the
>>>>>>>> problems are there - I think we should explain that the core of the paper
>>>>>>>> is the table and perhaps emphasize it in the paper beyond what the current
>>>>>>>> text - we also should reply to the reviewer and explain that the paper is
>>>>>>>> composed of contributions from a large group and we made an effort to
>>>>>>>> include every voice in the choir - each of those voices is important and
>>>>>>>> needs to be preserved - hopefully it will convince the reviewer that change
>>>>>>>> we will add will be sufficient
>>>>>>>> 7. Reviewer 2 asks to revise the introduction - I believe some
>>>>>>>> changes are possible - yet the introduction includes contributions from
>>>>>>>> many authors - at least 10 - and I fear losing something important someone
>>>>>>>> contributed - if someone has an idea on how to address this reviewer
>>>>>>>> without a painful transformation, please reply to this message.
>>>>>>>> 8. Reviewer 2 wants additional discussion around the
>>>>>>>> "reproducibility crisis" section and asks a valid question about
>>>>>>>> expectations we should address - "“Computational biomedical modeling… was
>>>>>>>> expected to be less affected by the reproducibility crisis.” Is this true,
>>>>>>>> and why would it be so? " - My answer to the reviewer is that unlike
>>>>>>>> biological processes that have random nature and experiments
>>>>>>>> would not repeat if repeated, while computer software should be
>>>>>>>> deterministic and it should be repeatable if designed well. Unfortunately
>>>>>>>> we are not experiencing this promise - yet I believe the reviewer wants
>>>>>>>> more discussion beyond this section so I am happy to discuss this.
>>>>>>>> 9. Reviewer 2- writes: "Models are Hard to Locate: Are the
>>>>>>>> authors suggesting that entire simulation workflows, from model
>>>>>>>> construction to analysis, should be publicly available? At what point does
>>>>>>>> one consider intellectual property? Do the authors advocate for such
>>>>>>>> extensive publishing for all models, or only ones that are intended to be
>>>>>>>> widely re-used? - those are important points we need to discuss - we need
>>>>>>>> to better explain the difficulties we are having when creating models and
>>>>>>>> the reviewer is absolutely correct about expanding the discussion to IP - I
>>>>>>>> suggest we create a thread for this discussion and reference it - I
>>>>>>>> suggest it we merged with the licensing issue that I will address later -
>>>>>>>> there is a strong connection there - the reviewer was very observant and
>>>>>>>> sees the bigger problem. However,we need to eventually distill our
>>>>>>>> discussion to recommendations that will inline.
>>>>>>>> 10. Reviewer 2 asks that we fix the unit standardization
>>>>>>>> section - I believe Hana and myself were the largest contributors there -
>>>>>>>> Hana - I will start a discussion on that topic in a separate email where we
>>>>>>>> can publicly discuss how to fix this - others will be welcome to
>>>>>>>> contribute.
>>>>>>>> 11. Reviewer 2 asks that we better handle the section on Data
>>>>>>>> availability and measurement definitions: in think we need to emphasize
>>>>>>>> solutions and separate it from issues that may not be solvable. Ideas are
>>>>>>>> welcome.
>>>>>>>> 12. Reviewer 2 asks good questions with regards to licensing
>>>>>>>> following our text- I personally have good answers to the reviewer and
>>>>>>>> William and I had some discussion on the topic in this list I suggest we
>>>>>>>> expand this discussion in a separate thread - hopefully William and perhaps
>>>>>>>> others will join the discussion. This discussion should also address the IP
>>>>>>>> issues raised by the reviewer for the "models are hard to locate section."
>>>>>>>> 13. Reviewer 2 asks to handle the Model Application and
>>>>>>>> Implementation Barriers section. We should decide what to do there, the
>>>>>>>> section may need expansion since the ideas there are solid, yet the section
>>>>>>>> is short so perhaps enhancing it makes better sense. I am open to
>>>>>>>> suggestions.
>>>>>>>>
>>>>>>>>
>>>>>>>> Those are the items I located and my suggestions. It seems we need
>>>>>>>> attention from Alex, Hana, myself and william. However, anyone on the list
>>>>>>>> is welcome to participate and suggest changes.
>>>>>>>>
>>>>>>>> I will start the discussion threads on specific topics. Hopefully
>>>>>>>> we can get it done quickly.
>>>>>>>>
>>>>>>>> Jacob
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> -
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ########### Original Reviews #############
>>>>>>>>
>>>>>>>> There are 2 reviews - both require major changes. I am copying the
>>>>>>>> relevant text below. If more appear, I will let you know, yet I only got
>>>>>>>> this message today although the reviews are dated a few days ago.
>>>>>>>>
>>>>>>>> Reviewer 1:
>>>>>>>> Recommendation for the Editor: Substantial revision is required
>>>>>>>>
>>>>>>>> Please list your revision requests for the authors and provide your
>>>>>>>> detailed comments, including highlighting limitations and strengths of the
>>>>>>>> review. If you have additional comments based on Q2 and Q3 you can add them
>>>>>>>> as well.
>>>>>>>>
>>>>>>>> Karr and co-authors made an interesting and exaustive point about
>>>>>>>> the reproducibility crisis that leads to inability to reuse and integrate
>>>>>>>> models, especially about COVID-19 disease.
>>>>>>>> Within the manuscript some typos and missing information are
>>>>>>>> present.
>>>>>>>> I'd suggest the authors to revise the entire manuscript especially
>>>>>>>> in terms of the state of the art, revising and updating the most relevant
>>>>>>>> examples of computational models dealing with COVID-19 and in general about
>>>>>>>> some semi-standardised proposals about the pipeline to follow for the
>>>>>>>> verification and validation of model credibility. In particular, the
>>>>>>>> authors failed to mention and cite some major results on in silico modeling
>>>>>>>> about COVID-19 up to now. See for example:
>>>>>>>> a."In silico trial to test COVID-19 candidate vaccines: a case
>>>>>>>> study with UISS platform", Russo, G., Pennisi, M., Fichera, E.,
>>>>>>>> ...Viceconti, M., Pappalardo, F., BMC Bioinformatics, 2020, 21, 527.
>>>>>>>> b. Russo G, Di Salvatore V, Sgroi G, Parasiliti Palumbo GA, Reche
>>>>>>>> PA, Pappalardo F. "A multi-step and multi-scale bioinformatic protocol to
>>>>>>>> investigate potential SARS-CoV-2 vaccine targets" [published online ahead
>>>>>>>> of print, 2021 Oct 5]. Brief Bioinform. 2021;bbab403.
>>>>>>>> doi:10.1093/bib/bbab403.
>>>>>>>> Moreover, when the authors mention ASME V&V40 procedure, it could
>>>>>>>> be useful to spend few words about the model credibility pipeline drawn for
>>>>>>>> predictive models in biomedicine, such as the one suggested in "Credibility
>>>>>>>> of in Silico Trial Technologies-A Theoretical Framing", Viceconti, M.,
>>>>>>>> Juarez, M.A., Curreli, C., ...Russo, G., Pappalardo, F., IEEE Journal of
>>>>>>>> Biomedical and Health Informatics, 2020, 24(1), pp. 4–13, 8884189.
>>>>>>>> For this specific aspect, the authors should talk about "specific
>>>>>>>> question of interest" and not "specific purpose" in the paragraph called
>>>>>>>> "Credibility of Models" to follow in such a way a standardised language.
>>>>>>>> Moreover, the authors should refer to COVID-19 also in the title,
>>>>>>>> with a specific mention about the fact that the main topic of model
>>>>>>>> integration in computational biology will be discussed inside the COVID-19
>>>>>>>> context.
>>>>>>>> Furthermore, for the model key concepts such as
>>>>>>>> Reusability-Extensibility-Extractability-Portability the authors should
>>>>>>>> described and outlined through a graphical sketch or visual representation
>>>>>>>> that summarises these key point.
>>>>>>>> The authors should also fix some grammar and writing typos present
>>>>>>>> in the "Contribution to the field" section.
>>>>>>>>
>>>>>>>> a. Is the quality of the figures and tables satisfactory?
>>>>>>>> - No
>>>>>>>>
>>>>>>>> b. Does the reference list cover the relevant literature adequately
>>>>>>>> and in an unbiased manner?
>>>>>>>> - No
>>>>>>>>
>>>>>>>> c. Does this manuscript refer only to published data? (unpublished
>>>>>>>> or original data is not allowed for this article type)
>>>>>>>> - Yes
>>>>>>>>
>>>>>>>> d. Does the review include a balanced, comprehensive, and critical
>>>>>>>> view of the research area?
>>>>>>>> - Yes
>>>>>>>>
>>>>>>>>
>>>>>>>> Reviewer 2
>>>>>>>>
>>>>>>>> Please list your revision requests for the authors and provide your
>>>>>>>> detailed comments, including highlighting limitations and strengths of the
>>>>>>>> review. If you have additional comments based on Q2 and Q3 you can add them
>>>>>>>> as well.
>>>>>>>>
>>>>>>>>
>>>>>>>> The article often reads as a stream-of-consciousness account of the
>>>>>>>> discussions that took place but lacks a clear thesis or recommendations. It
>>>>>>>> is not clear what, if anything, the authors are advocating for. It is not
>>>>>>>> always clear why the issues being discussed are problematic, or that they
>>>>>>>> can be reasonably addressed. Some of the issues raised are indeed important
>>>>>>>> and should be discussed, but the paper lacks focus and does not tell a
>>>>>>>> cohesive story. I believe this manuscript requires a major re-write to be
>>>>>>>> suitable for publication. The authors should consider narrowing the scope
>>>>>>>> of the discussion and focusing on a cohesive set of recommendations or open
>>>>>>>> questions. More specifically, I make a few suggestions below:
>>>>>>>>
>>>>>>>> Major comments
>>>>>>>> 1. The introduction is long and repeats itself (e.g., “much less is
>>>>>>>> known about how viral infections spread throughout the body…” is repeated
>>>>>>>> verbatim). It is not clear from the introduction what the main goal of the
>>>>>>>> paper is or why the “reproducibility crisis” is truly a crisis. Why is the
>>>>>>>> discussion of composition and black/white box models relevant to the
>>>>>>>> introduction? Further, this section is subtitled “the promise of modeling”,
>>>>>>>> which does not seem to match the content.
>>>>>>>> 2. The Reproducibility Crisis: “Computational biomedical modeling…
>>>>>>>> was expected to be less affected by the reproducibility crisis.” Is this
>>>>>>>> true, and why would it be so? One would think that more complex models
>>>>>>>> would suffer more from a lack of reproducibility. It may be helpful to
>>>>>>>> define what exactly the “reproducibility crisis” refers to.
>>>>>>>> 3. Models are Hard to Locate: Are the authors suggesting that
>>>>>>>> entire simulation workflows, from model construction to analysis, should be
>>>>>>>> publicly available? At what point does one consider intellectual property?
>>>>>>>> Do the authors advocate for such extensive publishing for all models, or
>>>>>>>> only ones that are intended to be widely re-used?
>>>>>>>> 4. Unit standardization: The conversion from PFU or TCID50 to
>>>>>>>> individual virions is likely to differ across viruses – are the authors
>>>>>>>> focused on COVID here? Are the authors advocating for a standard conversion
>>>>>>>> factor? It is not clear what the purpose of this discussion is. As the
>>>>>>>> authors mention, different scales require different units. Even at a single
>>>>>>>> scale, different models may require different units for numerical reasons.
>>>>>>>> It is not clear what the authors are advocating for here.
>>>>>>>> 5. Data availability and measurement definitions: This section
>>>>>>>> seems to outline limitations of available data, but again makes no
>>>>>>>> recommendations or proposed solution to any of the issues raised. Is this
>>>>>>>> the intention? Most of the issues raised here reflect limitations of
>>>>>>>> experimental science or data privacy, which likely cannot be meaningfully
>>>>>>>> addressed by the modeling community.
>>>>>>>> 6. Models are Not Consistently Licensed…: Are the authors implying
>>>>>>>> here that all modeling work should be published with no rights reserved? Is
>>>>>>>> it reasonable to expect modelers to make their work freely usable by others
>>>>>>>> for profit? Is it reasonable for institutions to allow this? How much does
>>>>>>>> this really contribute to reproducibility and utility?
>>>>>>>> 7. Model Application and Implementation Barriers: This section
>>>>>>>> seems unnecessary and out of place.
>>>>>>>> 8. There are grammar and punctuation errors scattered throughout;
>>>>>>>> please edit carefully.
>>>>>>>>
>>>>>>>>
>>>>>>>> a. Is the quality of the figures and tables satisfactory?
>>>>>>>> - Not Applicable
>>>>>>>>
>>>>>>>> b. Does the reference list cover the relevant literature adequately
>>>>>>>> and in an unbiased manner?
>>>>>>>> - Yes
>>>>>>>>
>>>>>>>> c. Does this manuscript refer only to published data? (unpublished
>>>>>>>> or original data is not allowed for this article type)
>>>>>>>> - Yes
>>>>>>>>
>>>>>>>> d. Does the review include a balanced, comprehensive, and critical
>>>>>>>> view of the research area?
>>>>>>>> No answer given.
>>>>>>>>
>>>>>>>>
>>>>>>>> ========
>>>>>>>> _______________________________________________
>>>>>>>> Vp-reproduce-subgroup mailing list
>>>>>>>> Vp-reproduce-subgroup at lists.simtk.org
>>>>>>>> https://lists.simtk.org/mailman/listinfo/vp-reproduce-subgroup
>>>>>>>> <https://urldefense.com/v3/__https://lists.simtk.org/mailman/listinfo/vp-reproduce-subgroup__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0hhiovw3M$>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Alexander Kulesza
>>>>>>> Team leader
>>>>>>> Modeling & simulation / Biomodeling
>>>>>>> alexander.kulesza at novadiscovery.com
>>>>>>> +33 7 82 92 44 62
>>>>>>> nova
>>>>>>> DISCOVERY
>>>>>>> www.novadiscovery.com
>>>>>>> <https://urldefense.com/v3/__http://www.novadiscovery.com/__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0hwI_2O8g$>
>>>>>>> 1 Place Verrazzano, 69009 Lyon
>>>>>>> +33 9 72 53 13 01
>>>>>>>
>>>>>>> *This message is intended only for the personal and confidential use
>>>>>>> of the designated recipient(s) named above. If you are not the intended
>>>>>>> recipient of this message you are hereby notified that any review,
>>>>>>> dissemination, distribution or copying of this message is strictly
>>>>>>> prohibited. Email transmission cannot be guaranteed to be secure or
>>>>>>> error-free. Therefore, we do not represent that this information is
>>>>>>> complete or accurate and it should not be relied upon as such. All
>>>>>>> information is subject to change without notice.*
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Alexander Kulesza
>>>>> Team leader
>>>>> Modeling & simulation / Biomodeling
>>>>> alexander.kulesza at novadiscovery.com
>>>>> +33 7 82 92 44 62
>>>>> nova
>>>>> DISCOVERY
>>>>> www.novadiscovery.com
>>>>> <https://urldefense.com/v3/__http://www.novadiscovery.com/__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0hwI_2O8g$>
>>>>> 1 Place Verrazzano, 69009 Lyon
>>>>> +33 9 72 53 13 01
>>>>>
>>>>> *This message is intended only for the personal and confidential use
>>>>> of the designated recipient(s) named above. If you are not the intended
>>>>> recipient of this message you are hereby notified that any review,
>>>>> dissemination, distribution or copying of this message is strictly
>>>>> prohibited. Email transmission cannot be guaranteed to be secure or
>>>>> error-free. Therefore, we do not represent that this information is
>>>>> complete or accurate and it should not be relied upon as such. All
>>>>> information is subject to change without notice.*
>>>>
>>>>
>>>
>>> --
>>> Alexander Kulesza
>>> Team leader
>>> Modeling & simulation / Biomodeling
>>> alexander.kulesza at novadiscovery.com
>>> +33 7 82 92 44 62
>>> nova
>>> DISCOVERY
>>> www.novadiscovery.com
>>> <https://urldefense.com/v3/__http://www.novadiscovery.com/__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0hwI_2O8g$>
>>> 1 Place Verrazzano, 69009 Lyon
>>> +33 9 72 53 13 01
>>>
>>> *This message is intended only for the personal and confidential use of
>>> the designated recipient(s) named above. If you are not the intended
>>> recipient of this message you are hereby notified that any review,
>>> dissemination, distribution or copying of this message is strictly
>>> prohibited. Email transmission cannot be guaranteed to be secure or
>>> error-free. Therefore, we do not represent that this information is
>>> complete or accurate and it should not be relied upon as such. All
>>> information is subject to change without notice.*
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.simtk.org/pipermail/vp-integration-subgroup/attachments/20220106/f56e3397/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 144605 bytes
Desc: not available
URL: <https://lists.simtk.org/pipermail/vp-integration-subgroup/attachments/20220106/f56e3397/attachment-0001.png>
More information about the Vp-integration-subgroup
mailing list