[Vp-reproduce-subgroup] [Vp-integration-subgroup] Paper review and revision check list
sheriff
sheriff at ebi.ac.uk
Thu Dec 30 12:28:17 PST 2021
Hi Jacob,
One crude idea for the graphics could be a cartoon of hurdling, where
each hurdle corresponds to the one we listed in the table or the dozen
challenges version. Under each, we could add a sentence about it and
this could be a summary figure.
Probably there are better ideas.
Best
Sheriff
On 2021-12-30 20:18, Jacob Barhak wrote:
> Thanks Sheriff,
>
> You comments would be more helpful within a different thread
> https://lists.simtk.org/pipermail/vp-integration-subgroup/2021-December/000324.html
>
> I will copy your text there for continuity. I assumed many are on
> vacation, yet I was trying to buy time in hope some of us are active.
>
> As for a rewrite. We have gone this road before. No one in the group
> had a solution on how to preserve all the ideas we discussed and
> represent all contributions. We lost precious time going in that route.
>
> If anyone knows how to solve it, please suggest, yet know that this may
> be lost effort that may only delay things.
>
> My opinion is that the reviewers will have to weigh the importance of
> our summary table compared to any other aspect. If they decide against
> it, there is always another venue. The price is delay.
>
> And Sheriff, frankly, if a publisher will not publish while preserving
> all ideas, then what value is being served in publishing?
>
> If a reviewer does not like a particular idea, they should argue
> against it and we will respond, yet a "general rewrite" argument is not
> sufficient. A good editor should spot it and act accordingly.
>
> The paper is readable, and has a clear summary for those who need it .
> Any argument against this "not being good enough" requires proof by
> example - if none can produce this example, then the argument should be
> shelved.
>
> For now, I ask that others attempt to join this discussion and attempt
> to fix the issues we still need attention for.
>
> Specifically, if anyone has an idea on more graphics, please suggest a
> figure. Sheriff, you created our first figure, if you have ideas for
> another figure, please help here. Otherwise we have some solutions for
> the review that we can debate.
>
> Thanks for answering during the holiday season and a happy new year.
>
> Jacob
>
> On Thu, Dec 30, 2021, 08:42 sheriff <sheriff at ebi.ac.uk> wrote:
>
> Hi Jacob,
>
> Thanks for leading the discussion.
>
> You may have to wait until early Jan for everyone to get back from the
> holidays and respond.
>
> Also, I think, it may not be possible to get away without improving the
> flow and readability of the manuscript.
>
> Here are my comments on model sharing and license
>
> Indeed often models are hard to locate. Hence in our reproducibility
> study ( Tiwari et al. 2021, https://doi.org/10.15252/msb.20209982), we
> strongly recommend authors to make their models public, but more
> specifically through public repositories such as BioModels
> (https://www.ebi.ac.uk/biomodels). We recommend authors to submit model
> codes, parameter sets, and simulation conditions needed to reproduce
> simulation studies.
>
> BioModels provides model curation and annotation service, where the
> models' reproducibility is assessed and annotated with controlled
> vocabularies. The BioModels offers sophisticated search engines to
> search and locate models and model components
> (https://www.ebi.ac.uk/biomodels/parameterSearch). Overall this will
> greatly facilitate model reuse.
>
> More importantly, all models are disseminated under the permissive CC0
> license in BioModels. This is because it is crucial to openly release
> the models developed through public funding. Also, curation services
> done at BioModels are supported by public funding.
>
> Although few users suggested we use CC-BY, we decided to continue with
> CC0. However, we strongly recommend our users to properly cite the
> model when reused and most researchers normally do. Also, If a modeler
> collects and integrates components from several hundred models, it may
> become difficult to properly acknowledge them all.
>
> Best regards
>
> Sheriff
>
> On 2021-12-27 10:59, Jacob Barhak wrote:
>
> Greetings Paper contributors,
>
> Especially Alex, Hana, William, and Sheriff should read this since I
> name them below.
>
> To move things forwards I started editing the paper and am sending an
> update of our checklist with responses to the reviewers. Please check
> the list if you think you want to contribute more to the paper - it is
> open for editing - the time window will close once we answered all
> reviewers - however, please focus on what reviewers asked for and do
> nto add anything else - I am restraining myself from adding more
> otherwise there will be no end to editing.
>
> The answers for the reviews so far are:
>
> 1. We thank reviewer 1 for pointing out the papers - we added those
> references into the discussion. Especially we thank the reviewer for
> pointing out the (Viceconti et al., 2019) paper - it is indeed unique
> in its attempt to classify and quantify model credibility with some
> good examples. We agree it should be cited in our paper since it is an
> important step forward.
>
> 2. TBD - Alex will provide an answer to the reviewer - he confirmed he
> got the message
>
> 3. I suggest that the reviewer will have the following answer to the
> request to focus on COVID-19 in the title. We thank the reviewer for
> the suggestion to focus on COVId-19 and put it in the title. However,
> we intend the paper to cover more than just one disease - in fact we
> have examples in the paper that span diabetes, heart disease as well as
> COVID - We did add recent COVID-19 models as the reviewer suggested,
> yet we would lie to keep the broader approach and keep the title
> without change.
>
> 4. TBD - I have no idea for more graphics - this is still an open item
> - please suggest solutions.
>
> 5. The "Contribution to the Field Statement" section was revised. Any
> contributor is welcome to edit it further.
>
> 6. Regarding a major rewrite I suggest the following answer to the
> reviewer: "We thank the reviewer for advice to improve the paper
> readability and indeed the group is aware that there are many ideas
> presented in the paper - yet this is expected from a large and diverse
> group that provided contributions. The reviewer should be aware that we
> made every attempt to keep all the ideas intact and what seems
> sometimes as noise, is actually a choir where every voice is
> important. This choire attempts to explain the modeling complexity. We
> did have attempts to reduce the work to something easier to digest -
> see the versions we quote in the paper. However, those versions were
> never approved by all contributors as necessary for publication - this
> is actually the law. What the reviewer saw was a compromise that keeps
> all ideas as best we could while providing a summary that allows
> readers to jump to sections of interest. Specifically table 1 is
> perhaps the most important artifact in the paper. and do ask the
> reviewer to weigh the importance of its quick publication. We can spend
> months on trying to revise the paper and reach agreement, yet this will
> probably not change the core of the paper and will waste resources on
> 18 contributors needing to constantly change things. Therefore we ask
> that the reviewer consider the benefit of releasing this information
> fast so it will make an impact rather than sending is back to rewrites
> that will delay impact in the name of style - an agreement among such a
> large and diverse group has a weight and the discussions around this
> paper are transparent so that readers can go back and clarify things if
> needed. " - beyond this response, if someone wants to spend time and
> improve the introduction without omitting ideas or references other
> than their own, then please let us know - I am open to suggestions.
>
> 7. The answer above in 6. should answer the reviewer that requested
> changing the introduction - however, if anyone has another suggestion,
> I am open.
>
> 8. I modified the specific section that the reviewer asked to address.
> Reviewer 2 wants additional discussion around the "reproducibility
> crisis" section and asks a valid question about expectations we should
> address - ""Computational biomedical modeling... was expected to be
> less affected by the reproducibility crisis." Is this true, and why
> would it be so? " - My answer to the reviewer is that unlike
> biological processes that have random nature and experiments would not
> repeat if repeated, while computer software should be deterministic and
> it should be repeatable if designed well. Unfortunately we are not
> experiencing this promise. If anyone wants to expand the discussion,
> please feel free to do so.
>
> 9. TBD - See also point 12. below. A discussion was started in this
> thread:
> https://lists.simtk.org/pipermail/vp-integration-subgroup/2021-December/000324.html
> - suggested discussors are William, and Sheriff - yet anyone else is
> welcome to join and contributes. So far no confirmation or response
> was recorded.
>
> 10. A discussion was started in this thread:
> https://lists.simtk.org/pipermail/vp-integration-subgroup/2021-December/000325.html
> - suggested discussor is Hana, So far no confirmation or response was
> recorded.
>
> 11. TBD - Reviewer 2 asks that we better handle the section on Data
> availability and measurement definitions: I think we need to emphasize
> solutions and separate them from issues that may not be solvable.
> However, Hana and I wrote most of this section with some text from Alex
> I believe, so we perhaps should meet and discuss how to improve it - it
> is indeed long and needs fixing. Ideas from anyone are welcome.
>
> 12. TBD - See - point 9 above - A discussion was started in this
> thread:
> https://lists.simtk.org/pipermail/vp-integration-subgroup/2021-December/000324.html
> - suggested discussors are William, and Sheriff - yet anyone else is
> welcome to join and contributes. So far no confirmation or response
> was recorded.
>
> 13. TBD - Reviewer 2 asks to handle the Model Application and
> Implementation Barriers section. We should decide what to do there, the
> section may need expansion since the ideas there are solid, yet the
> section is short so perhaps enhancing it makes better sense. I am open
> to suggestions.
>
> The revised paper is available in:
>
> https://docs.google.com/document/d/1IMEgmdNkx-EsnOjGuegpenSIMmKIkK00Lc8Gred3QxM/edit?usp=sharing
>
> I ask that all contributors interested in making modifications write
> those ideas on the mailing list so that we can move forward with this
> paper.
>
> I really hope some of you will use the time before the year ends to
> advance this.
>
> Jacob
>
> On Mon, Dec 20, 2021 at 5:50 PM Jacob Barhak <jacob.barhak at gmail.com>
> wrote:
> Thanks Alex,
>
> The paper opened back for editing. If someone really feels they have to
> edit and they are not on the list - let me know and I will add them to
> the list of editors. - yet for now - let us form a plan to answer the
> reviewers and only then touch text again - otherwise this will get out
> of hand since there are so many of us.
>
> I appreciate your support.
>
> Jacob
>
> On Mon, Dec 20, 2021 at 3:50 PM Alexander Kulesza
> <alexander.kulesza at novadiscovery.com> wrote:
>
> Dear Jacob, dear all,
> Yes will take care of 2. and happy to help wherever I can.
> Best
>
> On Sun, 19 Dec 2021 at 03:10, Jacob Barhak <jacob.barhak at gmail.com>
> wrote:
>
> Greetings to the paper contributors,
>
> You may have seen my other message where I posted the reviews to our
> paper. I am sending those again below to start a new thread discussing
> possible revisions.
>
> Here is a checklist for revisions required:
>
> * We should discuss the references that reviewer 1 raised and add them
> as reference
> * Reviewer 1 asked for more discussion around V&V40 - I believe Alex
> added the text there originally - Alex do you think you can address the
> request of the reviewer?
> * Reviewer 1 asked to focus on COVID-19 at the title - I am not sure we
> wish to limit our scope since many of our ideas are applicable far
> beyond COVID-19 - I leave it up for discussion in the group on how to
> address this request by the reviewer
> * Reviewer 1 asked for additional graphics - I am unsure how to address
> this beyond ur current diagram - ideas will help
> * We need to check the "Contribution to the field" section for grammar
> and spelling - if someone can contribute more elegant text - now is the
> time - I looked over it now and found no issues - yet I may have missed
> something - this section may have been missed since it was added last
> and perhaps vetted less than other sections. Also Reviewer 2 asked for
> grammar corrections, so it is worthwhile proof reading the paper as a
> whole.
> * Reviewer 2 asks for major rewrite to emphasize why the problems are
> there - I think we should explain that the core of the paper is the
> table and perhaps emphasize it in the paper beyond what the current
> text - we also should reply to the reviewer and explain that the paper
> is composed of contributions from a large group and we made an effort
> to include every voice in the choir - each of those voices is important
> and needs to be preserved - hopefully it will convince the reviewer
> that change we will add will be sufficient
> * Reviewer 2 asks to revise the introduction - I believe some changes
> are possible - yet the introduction includes contributions from many
> authors - at least 10 - and I fear losing something important someone
> contributed - if someone has an idea on how to address this reviewer
> without a painful transformation, please reply to this message.
> * Reviewer 2 wants additional discussion around the "reproducibility
> crisis" section and asks a valid question about expectations we should
> address - ""Computational biomedical modeling... was expected to be
> less affected by the reproducibility crisis." Is this true, and why
> would it be so? " - My answer to the reviewer is that unlike
> biological processes that have random nature and experiments would not
> repeat if repeated, while computer software should be deterministic and
> it should be repeatable if designed well. Unfortunately we are not
> experiencing this promise - yet I believe the reviewer wants more
> discussion beyond this section so I am happy to discuss this.
> * Reviewer 2- writes: "Models are Hard to Locate: Are the authors
> suggesting that entire simulation workflows, from model construction to
> analysis, should be publicly available? At what point does one consider
> intellectual property? Do the authors advocate for such extensive
> publishing for all models, or only ones that are intended to be widely
> re-used? - those are important points we need to discuss - we need to
> better explain the difficulties we are having when creating models and
> the reviewer is absolutely correct about expanding the discussion to IP
> - I suggest we create a thread for this discussion and reference it - I
> suggest it we merged with the licensing issue that I will address
> later - there is a strong connection there - the reviewer was very
> observant and sees the bigger problem. However,we need to eventually
> distill our discussion to recommendations that will inline.
> * Reviewer 2 asks that we fix the unit standardization section - I
> believe Hana and myself were the largest contributors there - Hana - I
> will start a discussion on that topic in a separate email where we can
> publicly discuss how to fix this - others will be welcome to
> contribute.
> * Reviewer 2 asks that we better handle the section on Data
> availability and measurement definitions: in think we need to emphasize
> solutions and separate it from issues that may not be solvable. Ideas
> are welcome.
> * Reviewer 2 asks good questions with regards to licensing following
> our text- I personally have good answers to the reviewer and William
> and I had some discussion on the topic in this list I suggest we expand
> this discussion in a separate thread - hopefully William and perhaps
> others will join the discussion. This discussion should also address
> the IP issues raised by the reviewer for the "models are hard to locate
> section."
> * Reviewer 2 asks to handle the Model Application and Implementation
> Barriers section. We should decide what to do there, the section may
> need expansion since the ideas there are solid, yet the section is
> short so perhaps enhancing it makes better sense. I am open to
> suggestions.
>
> Those are the items I located and my suggestions. It seems we need
> attention from Alex, Hana, myself and william. However, anyone on the
> list is welcome to participate and suggest changes.
>
> I will start the discussion threads on specific topics. Hopefully we
> can get it done quickly.
>
> Jacob
>
> -
>
> ########### Original Reviews #############
>
> There are 2 reviews - both require major changes. I am copying the
> relevant text below. If more appear, I will let you know, yet I only
> got this message today although the reviews are dated a few days ago.
>
> Reviewer 1:
> Recommendation for the Editor: Substantial revision is required
>
> Please list your revision requests for the authors and provide your
> detailed comments, including highlighting limitations and strengths of
> the review. If you have additional comments based on Q2 and Q3 you can
> add them as well.
>
> Karr and co-authors made an interesting and exaustive point about the
> reproducibility crisis that leads to inability to reuse and integrate
> models, especially about COVID-19 disease.
> Within the manuscript some typos and missing information are present.
> I'd suggest the authors to revise the entire manuscript especially in
> terms of the state of the art, revising and updating the most relevant
> examples of computational models dealing with COVID-19 and in general
> about some semi-standardised proposals about the pipeline to follow for
> the verification and validation of model credibility. In particular,
> the authors failed to mention and cite some major results on in silico
> modeling about COVID-19 up to now. See for example:
> a."In silico trial to test COVID-19 candidate vaccines: a case study
> with UISS platform", Russo, G., Pennisi, M., Fichera, E., ...Viceconti,
> M., Pappalardo, F., BMC Bioinformatics, 2020, 21, 527.
> b. Russo G, Di Salvatore V, Sgroi G, Parasiliti Palumbo GA, Reche PA,
> Pappalardo F. "A multi-step and multi-scale bioinformatic protocol to
> investigate potential SARS-CoV-2 vaccine targets" [published online
> ahead of print, 2021 Oct 5]. Brief Bioinform. 2021;bbab403.
> doi:10.1093/bib/bbab403.
> Moreover, when the authors mention ASME V&V40 procedure, it could be
> useful to spend few words about the model credibility pipeline drawn
> for predictive models in biomedicine, such as the one suggested in
> "Credibility of in Silico Trial Technologies-A Theoretical Framing",
> Viceconti, M., Juarez, M.A., Curreli, C., ...Russo, G., Pappalardo, F.,
> IEEE Journal of Biomedical and Health Informatics, 2020, 24(1), pp.
> 4-13, 8884189.
> For this specific aspect, the authors should talk about "specific
> question of interest" and not "specific purpose" in the paragraph
> called "Credibility of Models" to follow in such a way a standardised
> language.
> Moreover, the authors should refer to COVID-19 also in the title, with
> a specific mention about the fact that the main topic of model
> integration in computational biology will be discussed inside the
> COVID-19 context.
> Furthermore, for the model key concepts such as
> Reusability-Extensibility-Extractability-Portability the authors should
> described and outlined through a graphical sketch or visual
> representation that summarises these key point.
> The authors should also fix some grammar and writing typos present in
> the "Contribution to the field" section.
>
> a. Is the quality of the figures and tables satisfactory?
> - No
>
> b. Does the reference list cover the relevant literature adequately and
> in an unbiased manner?
> - No
>
> c. Does this manuscript refer only to published data? (unpublished or
> original data is not allowed for this article type)
> - Yes
>
> d. Does the review include a balanced, comprehensive, and critical view
> of the research area?
> - Yes
>
> Reviewer 2
>
> Please list your revision requests for the authors and provide your
> detailed comments, including highlighting limitations and strengths of
> the review. If you have additional comments based on Q2 and Q3 you can
> add them as well.
>
> The article often reads as a stream-of-consciousness account of the
> discussions that took place but lacks a clear thesis or
> recommendations. It is not clear what, if anything, the authors are
> advocating for. It is not always clear why the issues being discussed
> are problematic, or that they can be reasonably addressed. Some of the
> issues raised are indeed important and should be discussed, but the
> paper lacks focus and does not tell a cohesive story. I believe this
> manuscript requires a major re-write to be suitable for publication.
> The authors should consider narrowing the scope of the discussion and
> focusing on a cohesive set of recommendations or open questions. More
> specifically, I make a few suggestions below:
>
> Major comments
> 1. The introduction is long and repeats itself (e.g., "much less is
> known about how viral infections spread throughout the body..." is
> repeated verbatim). It is not clear from the introduction what the main
> goal of the paper is or why the "reproducibility crisis" is truly a
> crisis. Why is the discussion of composition and black/white box models
> relevant to the introduction? Further, this section is subtitled "the
> promise of modeling", which does not seem to match the content.
> 2. The Reproducibility Crisis: "Computational biomedical modeling...
> was expected to be less affected by the reproducibility crisis." Is
> this true, and why would it be so? One would think that more complex
> models would suffer more from a lack of reproducibility. It may be
> helpful to define what exactly the "reproducibility crisis" refers to.
> 3. Models are Hard to Locate: Are the authors suggesting that entire
> simulation workflows, from model construction to analysis, should be
> publicly available? At what point does one consider intellectual
> property? Do the authors advocate for such extensive publishing for all
> models, or only ones that are intended to be widely re-used?
> 4. Unit standardization: The conversion from PFU or TCID50 to
> individual virions is likely to differ across viruses - are the authors
> focused on COVID here? Are the authors advocating for a standard
> conversion factor? It is not clear what the purpose of this discussion
> is. As the authors mention, different scales require different units.
> Even at a single scale, different models may require different units
> for numerical reasons. It is not clear what the authors are advocating
> for here.
> 5. Data availability and measurement definitions: This section seems to
> outline limitations of available data, but again makes no
> recommendations or proposed solution to any of the issues raised. Is
> this the intention? Most of the issues raised here reflect limitations
> of experimental science or data privacy, which likely cannot be
> meaningfully addressed by the modeling community.
> 6. Models are Not Consistently Licensed...: Are the authors implying
> here that all modeling work should be published with no rights
> reserved? Is it reasonable to expect modelers to make their work freely
> usable by others for profit? Is it reasonable for institutions to allow
> this? How much does this really contribute to reproducibility and
> utility?
> 7. Model Application and Implementation Barriers: This section seems
> unnecessary and out of place.
> 8. There are grammar and punctuation errors scattered throughout;
> please edit carefully.
>
> a. Is the quality of the figures and tables satisfactory?
> - Not Applicable
>
> b. Does the reference list cover the relevant literature adequately and
> in an unbiased manner?
> - Yes
>
> c. Does this manuscript refer only to published data? (unpublished or
> original data is not allowed for this article type)
> - Yes
>
> d. Does the review include a balanced, comprehensive, and critical view
> of the research area?
> No answer given.
>
> ======== _______________________________________________
> Vp-reproduce-subgroup mailing list
> Vp-reproduce-subgroup at lists.simtk.org
> https://lists.simtk.org/mailman/listinfo/vp-reproduce-subgroup
> --
>
> Alexander Kulesza
> Team leader
> Modeling & simulation / Biomodeling
> alexander.kulesza at novadiscovery.com
> +33 7 82 92 44 62
>
> nova
> DISCOVERY
> www.novadiscovery.com [1]
> 1 Place Verrazzano, 69009 Lyon
> +33 9 72 53 13 01
> _This message is intended only for the personal and confidential use of
> the designated recipient(s) named above. If you are not the intended
> recipient of this message you are hereby notified that any review,
> dissemination, distribution or copying of this message is strictly
> prohibited. Email transmission cannot be guaranteed to be secure or
> error-free. Therefore, we do not represent that this information is
> complete or accurate and it should not be relied upon as such. All
> information is subject to change without notice._
_______________________________________________
Vp-integration-subgroup mailing list
Vp-integration-subgroup at lists.simtk.org
https://lists.simtk.org/mailman/listinfo/vp-integration-subgroup
Pls
Links:
------
[1] http://www.novadiscovery.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.simtk.org/pipermail/vp-reproduce-subgroup/attachments/20211230/05caf1a4/attachment-0001.html>
More information about the Vp-reproduce-subgroup
mailing list