[Vp-reproduce-subgroup] Paper review and revision check list
Jacob Barhak
jacob.barhak at gmail.com
Mon Dec 20 15:50:05 PST 2021
Thanks Alex,
The paper opened back for editing. If someone really feels they have to
edit and they are not on the list - let me know and I will add them to the
list of editors. - yet for now - let us form a plan to answer the reviewers
and only then touch text again - otherwise this will get out of hand since
there are so many of us.
I appreciate your support.
Jacob
On Mon, Dec 20, 2021 at 3:50 PM Alexander Kulesza <
alexander.kulesza at novadiscovery.com> wrote:
> Dear Jacob, dear all,
> Yes will take care of 2. and happy to help wherever I can.
> Best
>
> On Sun, 19 Dec 2021 at 03:10, Jacob Barhak <jacob.barhak at gmail.com> wrote:
>
>>
>> Greetings to the paper contributors,
>>
>> You may have seen my other message where I posted the reviews to our
>> paper. I am sending those again below to start a new thread discussing
>> possible revisions.
>>
>> Here is a checklist for revisions required:
>>
>>
>> 1. We should discuss the references that reviewer 1 raised and add
>> them as reference
>> 2. Reviewer 1 asked for more discussion around V&V40 - I believe Alex
>> added the text there originally - Alex do you think you can address the
>> request of the reviewer?
>> 3. Reviewer 1 asked to focus on COVID-19 at the title - I am not sure
>> we wish to limit our scope since many of our ideas are applicable far
>> beyond COVID-19 - I leave it up for discussion in the group on how to
>> address this request by the reviewer
>> 4. Reviewer 1 asked for additional graphics - I am unsure how to
>> address this beyond ur current diagram - ideas will help
>> 5. We need to check the "Contribution to the field" section for
>> grammar and spelling - if someone can contribute more elegant text - now is
>> the time - I looked over it now and found no issues - yet I may have missed
>> something - this section may have been missed since it was added last and
>> perhaps vetted less than other sections. Also Reviewer 2 asked for grammar
>> corrections, so it is worthwhile proof reading the paper as a whole.
>> 6. Reviewer 2 asks for major rewrite to emphasize why the problems
>> are there - I think we should explain that the core of the paper is the
>> table and perhaps emphasize it in the paper beyond what the current text -
>> we also should reply to the reviewer and explain that the paper is composed
>> of contributions from a large group and we made an effort to include every
>> voice in the choir - each of those voices is important and needs to be
>> preserved - hopefully it will convince the reviewer that change we will add
>> will be sufficient
>> 7. Reviewer 2 asks to revise the introduction - I believe some
>> changes are possible - yet the introduction includes contributions from
>> many authors - at least 10 - and I fear losing something important someone
>> contributed - if someone has an idea on how to address this reviewer
>> without a painful transformation, please reply to this message.
>> 8. Reviewer 2 wants additional discussion around the
>> "reproducibility crisis" section and asks a valid question about
>> expectations we should address - "“Computational biomedical modeling… was
>> expected to be less affected by the reproducibility crisis.” Is this true,
>> and why would it be so? " - My answer to the reviewer is that unlike
>> biological processes that have random nature and experiments
>> would not repeat if repeated, while computer software should be
>> deterministic and it should be repeatable if designed well. Unfortunately
>> we are not experiencing this promise - yet I believe the reviewer wants
>> more discussion beyond this section so I am happy to discuss this.
>> 9. Reviewer 2- writes: "Models are Hard to Locate: Are the authors
>> suggesting that entire simulation workflows, from model construction to
>> analysis, should be publicly available? At what point does one consider
>> intellectual property? Do the authors advocate for such extensive
>> publishing for all models, or only ones that are intended to be widely
>> re-used? - those are important points we need to discuss - we need to
>> better explain the difficulties we are having when creating models and the
>> reviewer is absolutely correct about expanding the discussion to IP - I
>> suggest we create a thread for this discussion and reference it - I
>> suggest it we merged with the licensing issue that I will address later -
>> there is a strong connection there - the reviewer was very observant and
>> sees the bigger problem. However,we need to eventually distill our
>> discussion to recommendations that will inline.
>> 10. Reviewer 2 asks that we fix the unit standardization section - I
>> believe Hana and myself were the largest contributors there - Hana - I will
>> start a discussion on that topic in a separate email where we can publicly
>> discuss how to fix this - others will be welcome to contribute.
>> 11. Reviewer 2 asks that we better handle the section on Data
>> availability and measurement definitions: in think we need to emphasize
>> solutions and separate it from issues that may not be solvable. Ideas are
>> welcome.
>> 12. Reviewer 2 asks good questions with regards to licensing
>> following our text- I personally have good answers to the reviewer and
>> William and I had some discussion on the topic in this list I suggest we
>> expand this discussion in a separate thread - hopefully William and perhaps
>> others will join the discussion. This discussion should also address the IP
>> issues raised by the reviewer for the "models are hard to locate section."
>> 13. Reviewer 2 asks to handle the Model Application and
>> Implementation Barriers section. We should decide what to do there, the
>> section may need expansion since the ideas there are solid, yet the section
>> is short so perhaps enhancing it makes better sense. I am open to
>> suggestions.
>>
>>
>> Those are the items I located and my suggestions. It seems we need
>> attention from Alex, Hana, myself and william. However, anyone on the list
>> is welcome to participate and suggest changes.
>>
>> I will start the discussion threads on specific topics. Hopefully we can
>> get it done quickly.
>>
>> Jacob
>>
>>
>>
>>
>>
>>
>> -
>>
>>
>>
>>
>>
>> ########### Original Reviews #############
>>
>> There are 2 reviews - both require major changes. I am copying the
>> relevant text below. If more appear, I will let you know, yet I only got
>> this message today although the reviews are dated a few days ago.
>>
>> Reviewer 1:
>> Recommendation for the Editor: Substantial revision is required
>>
>> Please list your revision requests for the authors and provide your
>> detailed comments, including highlighting limitations and strengths of the
>> review. If you have additional comments based on Q2 and Q3 you can add them
>> as well.
>>
>> Karr and co-authors made an interesting and exaustive point about the
>> reproducibility crisis that leads to inability to reuse and integrate
>> models, especially about COVID-19 disease.
>> Within the manuscript some typos and missing information are present.
>> I'd suggest the authors to revise the entire manuscript especially in
>> terms of the state of the art, revising and updating the most relevant
>> examples of computational models dealing with COVID-19 and in general about
>> some semi-standardised proposals about the pipeline to follow for the
>> verification and validation of model credibility. In particular, the
>> authors failed to mention and cite some major results on in silico modeling
>> about COVID-19 up to now. See for example:
>> a."In silico trial to test COVID-19 candidate vaccines: a case study with
>> UISS platform", Russo, G., Pennisi, M., Fichera, E., ...Viceconti, M.,
>> Pappalardo, F., BMC Bioinformatics, 2020, 21, 527.
>> b. Russo G, Di Salvatore V, Sgroi G, Parasiliti Palumbo GA, Reche PA,
>> Pappalardo F. "A multi-step and multi-scale bioinformatic protocol to
>> investigate potential SARS-CoV-2 vaccine targets" [published online ahead
>> of print, 2021 Oct 5]. Brief Bioinform. 2021;bbab403.
>> doi:10.1093/bib/bbab403.
>> Moreover, when the authors mention ASME V&V40 procedure, it could be
>> useful to spend few words about the model credibility pipeline drawn for
>> predictive models in biomedicine, such as the one suggested in "Credibility
>> of in Silico Trial Technologies-A Theoretical Framing", Viceconti, M.,
>> Juarez, M.A., Curreli, C., ...Russo, G., Pappalardo, F., IEEE Journal of
>> Biomedical and Health Informatics, 2020, 24(1), pp. 4–13, 8884189.
>> For this specific aspect, the authors should talk about "specific
>> question of interest" and not "specific purpose" in the paragraph called
>> "Credibility of Models" to follow in such a way a standardised language.
>> Moreover, the authors should refer to COVID-19 also in the title, with a
>> specific mention about the fact that the main topic of model integration in
>> computational biology will be discussed inside the COVID-19 context.
>> Furthermore, for the model key concepts such as
>> Reusability-Extensibility-Extractability-Portability the authors should
>> described and outlined through a graphical sketch or visual representation
>> that summarises these key point.
>> The authors should also fix some grammar and writing typos present in the
>> "Contribution to the field" section.
>>
>> a. Is the quality of the figures and tables satisfactory?
>> - No
>>
>> b. Does the reference list cover the relevant literature adequately and
>> in an unbiased manner?
>> - No
>>
>> c. Does this manuscript refer only to published data? (unpublished or
>> original data is not allowed for this article type)
>> - Yes
>>
>> d. Does the review include a balanced, comprehensive, and critical view
>> of the research area?
>> - Yes
>>
>>
>> Reviewer 2
>>
>> Please list your revision requests for the authors and provide your
>> detailed comments, including highlighting limitations and strengths of the
>> review. If you have additional comments based on Q2 and Q3 you can add them
>> as well.
>>
>>
>> The article often reads as a stream-of-consciousness account of the
>> discussions that took place but lacks a clear thesis or recommendations. It
>> is not clear what, if anything, the authors are advocating for. It is not
>> always clear why the issues being discussed are problematic, or that they
>> can be reasonably addressed. Some of the issues raised are indeed important
>> and should be discussed, but the paper lacks focus and does not tell a
>> cohesive story. I believe this manuscript requires a major re-write to be
>> suitable for publication. The authors should consider narrowing the scope
>> of the discussion and focusing on a cohesive set of recommendations or open
>> questions. More specifically, I make a few suggestions below:
>>
>> Major comments
>> 1. The introduction is long and repeats itself (e.g., “much less is known
>> about how viral infections spread throughout the body…” is repeated
>> verbatim). It is not clear from the introduction what the main goal of the
>> paper is or why the “reproducibility crisis” is truly a crisis. Why is the
>> discussion of composition and black/white box models relevant to the
>> introduction? Further, this section is subtitled “the promise of modeling”,
>> which does not seem to match the content.
>> 2. The Reproducibility Crisis: “Computational biomedical modeling… was
>> expected to be less affected by the reproducibility crisis.” Is this true,
>> and why would it be so? One would think that more complex models would
>> suffer more from a lack of reproducibility. It may be helpful to define
>> what exactly the “reproducibility crisis” refers to.
>> 3. Models are Hard to Locate: Are the authors suggesting that entire
>> simulation workflows, from model construction to analysis, should be
>> publicly available? At what point does one consider intellectual property?
>> Do the authors advocate for such extensive publishing for all models, or
>> only ones that are intended to be widely re-used?
>> 4. Unit standardization: The conversion from PFU or TCID50 to individual
>> virions is likely to differ across viruses – are the authors focused on
>> COVID here? Are the authors advocating for a standard conversion factor? It
>> is not clear what the purpose of this discussion is. As the authors
>> mention, different scales require different units. Even at a single scale,
>> different models may require different units for numerical reasons. It is
>> not clear what the authors are advocating for here.
>> 5. Data availability and measurement definitions: This section seems to
>> outline limitations of available data, but again makes no recommendations
>> or proposed solution to any of the issues raised. Is this the intention?
>> Most of the issues raised here reflect limitations of experimental science
>> or data privacy, which likely cannot be meaningfully addressed by the
>> modeling community.
>> 6. Models are Not Consistently Licensed…: Are the authors implying here
>> that all modeling work should be published with no rights reserved? Is it
>> reasonable to expect modelers to make their work freely usable by others
>> for profit? Is it reasonable for institutions to allow this? How much does
>> this really contribute to reproducibility and utility?
>> 7. Model Application and Implementation Barriers: This section seems
>> unnecessary and out of place.
>> 8. There are grammar and punctuation errors scattered throughout; please
>> edit carefully.
>>
>>
>> a. Is the quality of the figures and tables satisfactory?
>> - Not Applicable
>>
>> b. Does the reference list cover the relevant literature adequately and
>> in an unbiased manner?
>> - Yes
>>
>> c. Does this manuscript refer only to published data? (unpublished or
>> original data is not allowed for this article type)
>> - Yes
>>
>> d. Does the review include a balanced, comprehensive, and critical view
>> of the research area?
>> No answer given.
>>
>>
>> ========
>> _______________________________________________
>> Vp-reproduce-subgroup mailing list
>> Vp-reproduce-subgroup at lists.simtk.org
>> https://lists.simtk.org/mailman/listinfo/vp-reproduce-subgroup
>>
>
>
> --
> Alexander Kulesza
> Team leader
> Modeling & simulation / Biomodeling
> alexander.kulesza at novadiscovery.com
> +33 7 82 92 44 62
> nova
> DISCOVERY
> www.novadiscovery.com
> 1 Place Verrazzano, 69009 Lyon
> +33 9 72 53 13 01
>
> *This message is intended only for the personal and confidential use of
> the designated recipient(s) named above. If you are not the intended
> recipient of this message you are hereby notified that any review,
> dissemination, distribution or copying of this message is strictly
> prohibited. Email transmission cannot be guaranteed to be secure or
> error-free. Therefore, we do not represent that this information is
> complete or accurate and it should not be relied upon as such. All
> information is subject to change without notice.*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.simtk.org/pipermail/vp-reproduce-subgroup/attachments/20211220/5b73dcf6/attachment-0001.html>
More information about the Vp-reproduce-subgroup
mailing list