[Vp-reproduce-subgroup] Paper review and revision check list
Jacob Barhak
jacob.barhak at gmail.com
Sat Dec 18 18:09:43 PST 2021
Greetings to the paper contributors,
You may have seen my other message where I posted the reviews to our
paper. I am sending those again below to start a new thread discussing
possible revisions.
Here is a checklist for revisions required:
1. We should discuss the references that reviewer 1 raised and add them
as reference
2. Reviewer 1 asked for more discussion around V&V40 - I believe Alex
added the text there originally - Alex do you think you can address the
request of the reviewer?
3. Reviewer 1 asked to focus on COVID-19 at the title - I am not sure we
wish to limit our scope since many of our ideas are applicable far
beyond COVID-19 - I leave it up for discussion in the group on how to
address this request by the reviewer
4. Reviewer 1 asked for additional graphics - I am unsure how to address
this beyond ur current diagram - ideas will help
5. We need to check the "Contribution to the field" section for grammar
and spelling - if someone can contribute more elegant text - now is the
time - I looked over it now and found no issues - yet I may have missed
something - this section may have been missed since it was added last and
perhaps vetted less than other sections. Also Reviewer 2 asked for grammar
corrections, so it is worthwhile proof reading the paper as a whole.
6. Reviewer 2 asks for major rewrite to emphasize why the problems are
there - I think we should explain that the core of the paper is the table
and perhaps emphasize it in the paper beyond what the current text - we
also should reply to the reviewer and explain that the paper is composed of
contributions from a large group and we made an effort to include every
voice in the choir - each of those voices is important and needs to be
preserved - hopefully it will convince the reviewer that change we will add
will be sufficient
7. Reviewer 2 asks to revise the introduction - I believe some changes
are possible - yet the introduction includes contributions from many
authors - at least 10 - and I fear losing something important someone
contributed - if someone has an idea on how to address this reviewer
without a painful transformation, please reply to this message.
8. Reviewer 2 wants additional discussion around the "reproducibility
crisis" section and asks a valid question about expectations we should
address - "“Computational biomedical modeling… was expected to be less
affected by the reproducibility crisis.” Is this true, and why would it be
so? " - My answer to the reviewer is that unlike biological processes that
have random nature and experiments would not repeat if repeated, while
computer software should be deterministic and it should be repeatable if
designed well. Unfortunately we are not experiencing this promise - yet I
believe the reviewer wants more discussion beyond this section so I am
happy to discuss this.
9. Reviewer 2- writes: "Models are Hard to Locate: Are the authors
suggesting that entire simulation workflows, from model construction to
analysis, should be publicly available? At what point does one consider
intellectual property? Do the authors advocate for such extensive
publishing for all models, or only ones that are intended to be widely
re-used? - those are important points we need to discuss - we need to
better explain the difficulties we are having when creating models and the
reviewer is absolutely correct about expanding the discussion to IP - I
suggest we create a thread for this discussion and reference it - I
suggest it we merged with the licensing issue that I will address later -
there is a strong connection there - the reviewer was very observant and
sees the bigger problem. However,we need to eventually distill our
discussion to recommendations that will inline.
10. Reviewer 2 asks that we fix the unit standardization section - I
believe Hana and myself were the largest contributors there - Hana - I will
start a discussion on that topic in a separate email where we can publicly
discuss how to fix this - others will be welcome to contribute.
11. Reviewer 2 asks that we better handle the section on Data
availability and measurement definitions: in think we need to emphasize
solutions and separate it from issues that may not be solvable. Ideas are
welcome.
12. Reviewer 2 asks good questions with regards to licensing following
our text- I personally have good answers to the reviewer and William and I
had some discussion on the topic in this list I suggest we expand this
discussion in a separate thread - hopefully William and perhaps others will
join the discussion. This discussion should also address the IP issues
raised by the reviewer for the "models are hard to locate section."
13. Reviewer 2 asks to handle the Model Application and Implementation
Barriers section. We should decide what to do there, the section may need
expansion since the ideas there are solid, yet the section is short so
perhaps enhancing it makes better sense. I am open to suggestions.
Those are the items I located and my suggestions. It seems we need
attention from Alex, Hana, myself and william. However, anyone on the list
is welcome to participate and suggest changes.
I will start the discussion threads on specific topics. Hopefully we can
get it done quickly.
Jacob
-
########### Original Reviews #############
There are 2 reviews - both require major changes. I am copying the relevant
text below. If more appear, I will let you know, yet I only got this
message today although the reviews are dated a few days ago.
Reviewer 1:
Recommendation for the Editor: Substantial revision is required
Please list your revision requests for the authors and provide your
detailed comments, including highlighting limitations and strengths of the
review. If you have additional comments based on Q2 and Q3 you can add them
as well.
Karr and co-authors made an interesting and exaustive point about the
reproducibility crisis that leads to inability to reuse and integrate
models, especially about COVID-19 disease.
Within the manuscript some typos and missing information are present.
I'd suggest the authors to revise the entire manuscript especially in terms
of the state of the art, revising and updating the most relevant examples
of computational models dealing with COVID-19 and in general about some
semi-standardised proposals about the pipeline to follow for the
verification and validation of model credibility. In particular, the
authors failed to mention and cite some major results on in silico modeling
about COVID-19 up to now. See for example:
a."In silico trial to test COVID-19 candidate vaccines: a case study with
UISS platform", Russo, G., Pennisi, M., Fichera, E., ...Viceconti, M.,
Pappalardo, F., BMC Bioinformatics, 2020, 21, 527.
b. Russo G, Di Salvatore V, Sgroi G, Parasiliti Palumbo GA, Reche PA,
Pappalardo F. "A multi-step and multi-scale bioinformatic protocol to
investigate potential SARS-CoV-2 vaccine targets" [published online ahead
of print, 2021 Oct 5]. Brief Bioinform. 2021;bbab403.
doi:10.1093/bib/bbab403.
Moreover, when the authors mention ASME V&V40 procedure, it could be useful
to spend few words about the model credibility pipeline drawn for
predictive models in biomedicine, such as the one suggested in "Credibility
of in Silico Trial Technologies-A Theoretical Framing", Viceconti, M.,
Juarez, M.A., Curreli, C., ...Russo, G., Pappalardo, F., IEEE Journal of
Biomedical and Health Informatics, 2020, 24(1), pp. 4–13, 8884189.
For this specific aspect, the authors should talk about "specific question
of interest" and not "specific purpose" in the paragraph called
"Credibility of Models" to follow in such a way a standardised language.
Moreover, the authors should refer to COVID-19 also in the title, with a
specific mention about the fact that the main topic of model integration in
computational biology will be discussed inside the COVID-19 context.
Furthermore, for the model key concepts such as
Reusability-Extensibility-Extractability-Portability the authors should
described and outlined through a graphical sketch or visual representation
that summarises these key point.
The authors should also fix some grammar and writing typos present in the
"Contribution to the field" section.
a. Is the quality of the figures and tables satisfactory?
- No
b. Does the reference list cover the relevant literature adequately and in
an unbiased manner?
- No
c. Does this manuscript refer only to published data? (unpublished or
original data is not allowed for this article type)
- Yes
d. Does the review include a balanced, comprehensive, and critical view of
the research area?
- Yes
Reviewer 2
Please list your revision requests for the authors and provide your
detailed comments, including highlighting limitations and strengths of the
review. If you have additional comments based on Q2 and Q3 you can add them
as well.
The article often reads as a stream-of-consciousness account of the
discussions that took place but lacks a clear thesis or recommendations. It
is not clear what, if anything, the authors are advocating for. It is not
always clear why the issues being discussed are problematic, or that they
can be reasonably addressed. Some of the issues raised are indeed important
and should be discussed, but the paper lacks focus and does not tell a
cohesive story. I believe this manuscript requires a major re-write to be
suitable for publication. The authors should consider narrowing the scope
of the discussion and focusing on a cohesive set of recommendations or open
questions. More specifically, I make a few suggestions below:
Major comments
1. The introduction is long and repeats itself (e.g., “much less is known
about how viral infections spread throughout the body…” is repeated
verbatim). It is not clear from the introduction what the main goal of the
paper is or why the “reproducibility crisis” is truly a crisis. Why is the
discussion of composition and black/white box models relevant to the
introduction? Further, this section is subtitled “the promise of modeling”,
which does not seem to match the content.
2. The Reproducibility Crisis: “Computational biomedical modeling… was
expected to be less affected by the reproducibility crisis.” Is this true,
and why would it be so? One would think that more complex models would
suffer more from a lack of reproducibility. It may be helpful to define
what exactly the “reproducibility crisis” refers to.
3. Models are Hard to Locate: Are the authors suggesting that entire
simulation workflows, from model construction to analysis, should be
publicly available? At what point does one consider intellectual property?
Do the authors advocate for such extensive publishing for all models, or
only ones that are intended to be widely re-used?
4. Unit standardization: The conversion from PFU or TCID50 to individual
virions is likely to differ across viruses – are the authors focused on
COVID here? Are the authors advocating for a standard conversion factor? It
is not clear what the purpose of this discussion is. As the authors
mention, different scales require different units. Even at a single scale,
different models may require different units for numerical reasons. It is
not clear what the authors are advocating for here.
5. Data availability and measurement definitions: This section seems to
outline limitations of available data, but again makes no recommendations
or proposed solution to any of the issues raised. Is this the intention?
Most of the issues raised here reflect limitations of experimental science
or data privacy, which likely cannot be meaningfully addressed by the
modeling community.
6. Models are Not Consistently Licensed…: Are the authors implying here
that all modeling work should be published with no rights reserved? Is it
reasonable to expect modelers to make their work freely usable by others
for profit? Is it reasonable for institutions to allow this? How much does
this really contribute to reproducibility and utility?
7. Model Application and Implementation Barriers: This section seems
unnecessary and out of place.
8. There are grammar and punctuation errors scattered throughout; please
edit carefully.
a. Is the quality of the figures and tables satisfactory?
- Not Applicable
b. Does the reference list cover the relevant literature adequately and in
an unbiased manner?
- Yes
c. Does this manuscript refer only to published data? (unpublished or
original data is not allowed for this article type)
- Yes
d. Does the review include a balanced, comprehensive, and critical view of
the research area?
No answer given.
========
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.simtk.org/pipermail/vp-reproduce-subgroup/attachments/20211218/2c122822/attachment-0001.html>
More information about the Vp-reproduce-subgroup
mailing list