[Vp-reproduce-subgroup] [Vp-integration-subgroup] Shayn Peirce-Cottler via Frontiers: Manuscript
Jacob Barhak
jacob.barhak at gmail.com
Thu Dec 16 22:00:50 PST 2021
Ok Jonatan,
There are 2 reviews - both require major changes. I am copying the relevant
text below. If more appear, I will let you know, yet I only got this
message today although the reviews are dated a few days ago.
Reviewer 1:
*Recommendation for the Editor: *Substantial revision is required
Please list your revision requests for the authors and provide your
detailed comments, including highlighting limitations and strengths of the
review. If you have additional comments based on Q2 and Q3 you can add them
as well.
Karr and co-authors made an interesting and exaustive point about the
reproducibility crisis that leads to inability to reuse and integrate
models, especially about COVID-19 disease.
Within the manuscript some typos and missing information are present.
I'd suggest the authors to revise the entire manuscript especially in terms
of the state of the art, revising and updating the most relevant examples
of computational models dealing with COVID-19 and in general about some
semi-standardised proposals about the pipeline to follow for the
verification and validation of model credibility. In particular, the
authors failed to mention and cite some major results on in silico modeling
about COVID-19 up to now. See for example:
a."In silico trial to test COVID-19 candidate vaccines: a case study with
UISS platform", Russo, G., Pennisi, M., Fichera, E., ...Viceconti, M.,
Pappalardo, F., BMC Bioinformatics, 2020, 21, 527.
b. Russo G, Di Salvatore V, Sgroi G, Parasiliti Palumbo GA, Reche PA,
Pappalardo F. "A multi-step and multi-scale bioinformatic protocol to
investigate potential SARS-CoV-2 vaccine targets" [published online ahead
of print, 2021 Oct 5]. Brief Bioinform. 2021;bbab403.
doi:10.1093/bib/bbab403.
Moreover, when the authors mention ASME V&V40 procedure, it could be useful
to spend few words about the model credibility pipeline drawn for
predictive models in biomedicine, such as the one suggested in "Credibility
of in Silico Trial Technologies-A Theoretical Framing", Viceconti, M.,
Juarez, M.A., Curreli, C., ...Russo, G., Pappalardo, F., IEEE Journal of
Biomedical and Health Informatics, 2020, 24(1), pp. 4–13, 8884189.
For this specific aspect, the authors should talk about "specific question
of interest" and not "specific purpose" in the paragraph called
"Credibility of Models" to follow in such a way a standardised language.
Moreover, the authors should refer to COVID-19 also in the title, with a
specific mention about the fact that the main topic of model integration in
computational biology will be discussed inside the COVID-19 context.
Furthermore, for the model key concepts such as
Reusability-Extensibility-Extractability-Portability the authors should
described and outlined through a graphical sketch or visual representation
that summarises these key point.
The authors should also fix some grammar and writing typos present in the
"Contribution to the field" section.
a. Is the quality of the figures and tables satisfactory?
- No
b. Does the reference list cover the relevant literature adequately and in
an unbiased manner?
- No
c. Does this manuscript refer only to published data? (unpublished or
original data is not allowed for this article type)
- Yes
d. Does the review include a balanced, comprehensive, and critical view of
the research area?
- Yes
Reviewer 2
Please list your revision requests for the authors and provide your
detailed comments, including highlighting limitations and strengths of the
review. If you have additional comments based on Q2 and Q3 you can add them
as well.
The article often reads as a stream-of-consciousness account of the
discussions that took place but lacks a clear thesis or recommendations. It
is not clear what, if anything, the authors are advocating for. It is not
always clear why the issues being discussed are problematic, or that they
can be reasonably addressed. Some of the issues raised are indeed important
and should be discussed, but the paper lacks focus and does not tell a
cohesive story. I believe this manuscript requires a major re-write to be
suitable for publication. The authors should consider narrowing the scope
of the discussion and focusing on a cohesive set of recommendations or open
questions. More specifically, I make a few suggestions below:
Major comments
1. The introduction is long and repeats itself (e.g., “much less is known
about how viral infections spread throughout the body…” is repeated
verbatim). It is not clear from the introduction what the main goal of the
paper is or why the “reproducibility crisis” is truly a crisis. Why is the
discussion of composition and black/white box models relevant to the
introduction? Further, this section is subtitled “the promise of modeling”,
which does not seem to match the content.
2. The Reproducibility Crisis: “Computational biomedical modeling… was
expected to be less affected by the reproducibility crisis.” Is this true,
and why would it be so? One would think that more complex models would
suffer more from a lack of reproducibility. It may be helpful to define
what exactly the “reproducibility crisis” refers to.
3. Models are Hard to Locate: Are the authors suggesting that entire
simulation workflows, from model construction to analysis, should be
publicly available? At what point does one consider intellectual property?
Do the authors advocate for such extensive publishing for all models, or
only ones that are intended to be widely re-used?
4. Unit standardization: The conversion from PFU or TCID50 to individual
virions is likely to differ across viruses – are the authors focused on
COVID here? Are the authors advocating for a standard conversion factor? It
is not clear what the purpose of this discussion is. As the authors
mention, different scales require different units. Even at a single scale,
different models may require different units for numerical reasons. It is
not clear what the authors are advocating for here.
5. Data availability and measurement definitions: This section seems to
outline limitations of available data, but again makes no recommendations
or proposed solution to any of the issues raised. Is this the intention?
Most of the issues raised here reflect limitations of experimental science
or data privacy, which likely cannot be meaningfully addressed by the
modeling community.
6. Models are Not Consistently Licensed…: Are the authors implying here
that all modeling work should be published with no rights reserved? Is it
reasonable to expect modelers to make their work freely usable by others
for profit? Is it reasonable for institutions to allow this? How much does
this really contribute to reproducibility and utility?
7. Model Application and Implementation Barriers: This section seems
unnecessary and out of place.
8. There are grammar and punctuation errors scattered throughout; please
edit carefully.
a. Is the quality of the figures and tables satisfactory?
- Not Applicable
b. Does the reference list cover the relevant literature adequately and in
an unbiased manner?
- Yes
c. Does this manuscript refer only to published data? (unpublished or
original data is not allowed for this article type)
- Yes
d. Does the review include a balanced, comprehensive, and critical view of
the research area?
No answer given.
=========
Those were individual reviewers. I have not seen any editorial
recommendations. - The title says interactive review so I guess we can
write back to the reviewers. Macella might know better since she knows the
Journal
We did go over the paper multiple times and reviewers still point out
errors - so either I copied the wrong version when we submitted, or we
missed something major that disrupts both reviewers - can someone see any
grammatical errors the reviewers are pointing to?
The first reviewer seems to focus on more literature and his requests are
fairly easy to accommodate to my understanding . The second reviewer goes
into detail and calls for some necessary discussion points - I actually
think we should discuss his points in more detail in the group - the
discussion itself is worth it to clarify details - beyond any publication.
I am especially interested in the license issue he discussed - We had some
discussion about it in the past, it is time to do it again and answer the
reviewer. Yet I assume some of you may want to discuss other points.
I am not sure if everyone has access to the system to comment internally,
so I suggest we have our discussions in this group publicly and I will
periodically copy those responses to the submission system if allowed -
another option is to withdraw the paper, have our discussions and send a
revised version. If we decide to revise we will need another approval
round. I am unsure of the best course of action so let us start
discussions.
The thing is, if we revise we need to go through another long approval
round - the last 2 times it took us 3 weeks only to get all approvals in
place and I assume it is the time it will take if we need to do it again.
So the sooner we discuss the situation, the better.
Jacob
On Thu, Dec 16, 2021 at 8:44 PM Jonathan Karr <jonrkarr at gmail.com> wrote:
> I haven't seen anything. If there's feedback, can you share?
>
> Jonathan
>
> On Thu, Dec 16, 2021, 8:56 PM Torres, Marcella <mtorres at richmond.edu>
> wrote:
>
>> Hi all,
>>
>> Just checking in: am I the only one seeing the reviewer's comments on the
>> manuscript?
>>
>> Thanks,
>> Marcella
>>
>> Marcella Torres, Ph.D.
>> Director of Mathematical Studies
>> University of Richmond
>> Jepson Hall Room 212
>> 221 Richmond Way
>> Richmond, VA 23173
>> (804) 289-8081
>> Pronouns: she/her
>>
>> ------------------------------
>> *From:* Vp-reproduce-subgroup <
>> vp-reproduce-subgroup-bounces at lists.simtk.org> on behalf of Jacob Barhak
>> <jacob.barhak at gmail.com>
>> *Sent:* Monday, November 29, 2021 2:39 AM
>> *To:* James A Glazier <jaglazier at gmail.com>
>> *Cc:* Thompson, Robin <Robin.N.Thompson at warwick.ac.uk>;
>> vp-reproduce-subgroup at lists.simtk.org <
>> vp-reproduce-subgroup at lists.simtk.org>;
>> vp-integration-subgroup at lists.simtk.org <
>> vp-integration-subgroup at lists.simtk.org>; Jonathan Karr <
>> jonrkarr at gmail.com>; John Gennari <gennari at uw.edu>; Winston Garira <
>> Winston.Garira at univen.ac.za>; Faeder, James R <faeder at pitt.edu>
>> *Subject:* Re: [Vp-reproduce-subgroup] [Vp-integration-subgroup] Shayn
>> Peirce-Cottler via Frontiers: Manuscript
>>
>> *External Email:* Use caution in opening links, attachments, and buying
>> gift cards.
>>
>> Thanks James, And thanks to all contributors,
>>
>> The paper was submitted on Thanksgiving evening and confirmation was
>> received.
>>
>> Hopefully it will be processed quickly.
>>
>> Jacob
>>
>>
>>
>> On Wed, Nov 24, 2021 at 12:22 PM James A Glazier <jaglazier at gmail.com>
>> wrote:
>>
>> Dear Jacob:
>>
>> Thank you for all your hard work and patience on this paper. I know you
>> have made a lot of effort to integrate so many differing ideas and voices.
>>
>> JAG
>> On 11/24/2021 9:14 AM, Jacob Barhak wrote:
>>
>> Thanks James,
>>
>> Unless there are any last minute objections, I will submit the paper this
>> holiday weekend.
>>
>> I will update you all on progress.
>>
>> Hopefully it will be smooth.
>>
>> Happy thanksgiving.
>>
>> Jacob
>>
>> On Wed, Nov 24, 2021 at 8:09 AM Faeder, James R <faeder at pitt.edu> wrote:
>>
>> APPROVE. I apologize for the delay. These messages were being routed to a
>> folder that I do not check frequently. I finally found out from a voicemail
>> message. Thanks everyone for the efforts you have put into this work and
>> especially Jacob for all of the work organizing on top of it. Good luck
>> with the submission and Happy Thanksgiving!
>>
>>
>>
>> Jim
>>
>>
>>
>> *From: *Jacob Barhak <jacob.barhak at gmail.com>
>> *Date: *Wednesday, November 24, 2021 at 6:47 AM
>> *To: *John Rice <john.rice at noboxes.org>, Jonathan Karr <
>> jonrkarr at gmail.com>, William Waites <wwaites at ieee.org>, Tingting Tang <
>> ttang2 at sdsu.edu>, vp-reproduce-subgroup at lists.simtk.org <
>> vp-reproduce-subgroup at lists.simtk.org>, James A Glazier <
>> jaglazier at gmail.com>, vp-integration-subgroup at lists.simtk.org <
>> vp-integration-subgroup at lists.simtk.org>, John Gennari <gennari at uw.edu>,
>> Winston Garira <Winston.Garira at univen.ac.za>, Faeder, James R <
>> faeder at pitt.edu>, Thompson, Robin <Robin.N.Thompson at warwick.ac.uk>
>> *Subject: *Re: [Vp-integration-subgroup] [Vp-reproduce-subgroup] Shayn
>> Peirce-Cottler via Frontiers: Manuscript
>>
>> Greetings James Faeder,
>>
>>
>>
>> You are the last one that needs to Approve. DISAPPROVE , or ABSTAIN from
>> submission of the paper.
>>
>>
>>
>> I apologize if you already sent a response yet I have not located any
>> response from you- it may be my error, yet I need your public response
>> before I can continue to submit the paper.
>>
>>
>>
>> 17 of us already approved submission and you are the last one on my list.
>>
>>
>>
>> I tried to contact your department in the last few days and left voice
>> messages to your administrators since I could not locate your direct phone.
>>
>>
>>
>> I will really appreciate it if someone can contact you and make sure you
>> get this message.
>>
>>
>>
>> I really hope we get an answer so I can process the paper by thanksgiving
>> weekend.
>>
>>
>>
>> Jacob
>>
>>
>>
>>
>>
>>
>>
>> On Fri, Nov 19, 2021 at 12:22 AM Jacob Barhak <jacob.barhak at gmail.com>
>> wrote:
>>
>> Thanks John, Hi James Faedar,
>>
>>
>>
>> John, with your approval below we only need James Faedar to respond so I
>> can proceed.
>>
>>
>>
>> Can someone contact him to make sure he saw this correspondence? He
>> should receive these messages, yet if someone is in contact with him
>> regularly, please ask him to respond so we can move on.
>>
>>
>>
>> One last thing before submission. If there is anyone who thinks this
>> paper should not be published, I would ask them to write to this mailing
>> list publicly - if there is any error or omission that anyone thinks is
>> important, we want to know about it before submitting the article. This
>> request is public and not targeted at the authors - if there is
>> something we missed, we want to know it before submission.
>>
>>
>>
>> Hopefully James Feedar will respond quickly and there will be no public
>> objection and we can get this paper published quickly.
>>
>>
>>
>> Jacob
>>
>>
>>
>>
>>
>>
>>
>> On Thu, Nov 18, 2021 at 4:34 PM John Rice <john.rice at noboxes.org> wrote:
>>
>> I approve submission.
>>
>> Typed with two thumbs on my iPhone. (757) 318-0671
>>
>>
>>
>> “Upon this gifted age, in its dark hour,
>>
>> Rains from the sky a meteoric shower
>>
>> Of facts . . . they lie unquestioned, uncombined.
>>
>> Wisdom enough to leech us of our ill
>>
>> Is daily spun; but there exists no loom
>>
>> To weave it into fabric.”
>>
>>
>>
>> –Edna St. Vincent Millay,
>>
>>
>>
>>
>>
>>
>> On Nov 17, 2021, at 22:04, Jacob Barhak <jacob.barhak at gmail.com> wrote:
>>
>>
>>
>> Hi James Faedar, Hi John Rice,
>>
>>
>>
>> Please check the manuscript and declare by REPLY ALL if you APPROVE,
>> DISAPPROVE or ABSTAIN.
>>
>>
>>
>> James, if you responded and I did not get the email, I apologize - there
>> are some communication hiccups occasionally. Yet I need your public
>> approval to proceed and use your name.
>>
>>
>>
>> John, I know you allowed us to send the paper, yet you need to be
>> specific and declare if you want to be included as an Author or just moved
>> to acknowledgements. You did have a major contribution, so I hope you will
>> APPROVE.
>>
>>
>>
>> Once I have your answer I can proceed. Hopefully you will both APPROVE
>> quickly.
>>
>>
>>
>> Jacob
>>
>>
>>
>> On Wed, Nov 17, 2021 at 3:02 PM sheriff <sheriff at ebi.ac.uk> wrote:
>>
>> Hi Jacob and all,
>>
>> I approve the submission of the manuscript.
>>
>> Please update the citation Tiwari et 2020 Preprint to
>>
>> Tiwari K, Kananathan S, Roberts MG, Meyer JP, Sharif Shohan MU, Xavier A,
>> Maire M, Zyoud A, Men J, Ng S, Nguyen TVN, Glont M, Hermjakob H,
>> Malik-Sheriff RS. Reproducibility in systems biology modelling. Mol Syst
>> Biol. 2021 Feb;17(2):e9982. doi: 10.15252/msb.20209982. PMID: 33620773;
>> PMCID: PMC7901289.
>>
>> Also my name and affiliation below
>>
>> Rahuman S. *Malik Sheriff*
>>
>> European Molecular Biology Laboratory, European Bioinformatics Institute
>> (EMBL-EBI), Wellcome Genome Campus, Hinxton, Cambridge, UK
>>
>>
>>
>> Thanks for pushing this work forward
>>
>> Best regards
>>
>> Sheriff
>>
>>
>>
>> On 2021-11-17 18:58, Jonathan Karr wrote:
>>
>> Hi Jacob,
>>
>>
>>
>> Thank you again for spearheading this effort.
>>
>>
>>
>> I approve the submission. I think the paper organizes a variety of
>> important issues toward more credible models, and the content is sound.
>>
>>
>>
>> My affiliation is Department of Genetics and Genomic Sciences, Icahn
>> School of Medicine at Mount Sinai, New York, NY, USA.
>>
>>
>>
>> As others have mentioned, I think the paper would be more impactful with
>> further editing to make the paper easier to read:
>>
>> - More clearly articulate the goals/motivation for more credible
>> models
>> - Be more concise and focused
>> - Remove redundancies by grouping related text
>> - Fill in the gaps in logic in the introduction with more
>> transitions, less extraneous information, and/or a brief outline.
>> - Change the title of the "Utility of models" section to something
>> like "Challenges with using multiscale models" (James' suggestion)
>> - Order the "Utility of models" subsections to be easier to follow,
>> perhaps from model construction through to reuse
>> - Rephrase the titles of the "Utility of models" subsections to
>> focus on opportunities for improvement rather than current problems
>>
>> Regards,
>>
>> Jonathan
>>
>>
>>
>> On Wed, Nov 10, 2021 at 11:25 PM Jacob Barhak <jacob.barhak at gmail.com>
>> wrote:
>>
>> Thanks to all contributors who answered,
>>
>>
>>
>> Most contributors approved by now.
>>
>>
>>
>> James and Wiliam, I am not sure how many more revisions there will be for
>> this publication. I did not add several important references of my own in
>> this revision. Frankly I wanted to keep reference changes to a minimum and
>> kept it as close as possible to the origin . However, we are transferring
>> the paper to the publisher under CC license, so it will not be hard to
>> create another derived version with all sorts of additions and
>> modifications in the future where people can edit it as they see fit and
>> add whatever references they want - so after it gets published you can
>> continue working on other versions as you see fit. For now I ask we focus
>> on publishing this version alone and I really hope the editors will not put
>> us through another revision round since it makes no sense at this point -
>> any change is not substantial compared to what we have already and any
>> delay is more damaging than productive.
>>
>>
>>
>> To be productive I ask that the following people who did not answer so
>> far take the time to APPROVE, DISAPPROVE, or ABSTAIN.:
>>
>> Robin Thompson
>>
>> James R. Faeder
>>
>> Jonathan Karr
>> Rahuman Sheriff
>>
>> John Rice
>>
>>
>>
>> John Rice - thanks for your support, yet you will have to be specific:
>> You can either choose APPROVE or ABSTAIN - both of these options will move
>> the paper forwards. APPROVE will add your name to the list of authors with
>> all responsibilities and privileges listed or implied , ABSTAIN will put
>> you in acknowledgements without any obligation. So John, please choose if
>> you want your name on the paper or not in this specific version - any
>> decision you make will be ok.
>>
>>
>>
>> I hope we can father all support quickly so I can move forward and submit
>> the paper.
>>
>>
>>
>> Jacob
>>
>>
>>
>>
>>
>> On Tue, Nov 9, 2021 at 5:46 AM William Waites <wwaites at ieee.org> wrote:
>>
>> I'm just chasing down references for use in another context and I notice
>> that the Ke et al paper is still a pre-print after more than a year. There
>> is no problem in my opinion citing preprints but it is suspect in this
>> case: there's been a ton of work on SARS-CoV-2 infectiousness, why didn't
>> this get published? Probably not best to rely on it as an example of
>> practice. Perhaps to fix on revision.
>>
>> -w
>>
>> _______________________________________________
>> Vp-integration-subgroup mailing list
>> Vp-integration-subgroup at lists.simtk.org
>> https://lists.simtk.org/mailman/listinfo/vp-integration-subgroup
>> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.simtk.org%2Fmailman%2Flistinfo%2Fvp-integration-subgroup&data=04%7C01%7Cfaeder%40pitt.edu%7Ca2976628735e4c785b7c08d9af401018%7C9ef9f489e0a04eeb87cc3a526112fd0d%7C1%7C0%7C637733512383015705%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=z6LRVCOqE9rXT2%2BQvwuNdmgWM4uodEUNGMP5VO0C5to%3D&reserved=0>
>>
>>
>>
>> --
>> James A. Glazier, PhD
>> Professor of Intelligent Systems Engineering, Adjunct Professor of Physics
>> Director, Biocomplexity Institute
>> Indiana University, Bloomington
>> (812) 391-2159 (cell)
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.simtk.org/pipermail/vp-reproduce-subgroup/attachments/20211217/7de25d69/attachment-0001.html>
More information about the Vp-reproduce-subgroup
mailing list