[Vp-integration-subgroup] [Vp-reproduce-subgroup] Paper review and revision check list

sheriff sheriff at ebi.ac.uk
Thu Dec 30 12:28:17 PST 2021


Hi Jacob,

One crude idea for the graphics could be a cartoon of hurdling, where 
each hurdle corresponds to the one we listed in the table or the dozen 
challenges version. Under each, we could add a sentence about it and 
this could be a summary figure.

Probably there are better ideas.

Best

Sheriff

On 2021-12-30 20:18, Jacob Barhak wrote:

> Thanks Sheriff,
> 
> You comments would be more helpful within a different thread 
> https://lists.simtk.org/pipermail/vp-integration-subgroup/2021-December/000324.html
> 
> I will copy your text there for continuity. I assumed many are on 
> vacation, yet I was trying to buy time in hope some of us are active.
> 
> As for a rewrite. We have gone this road before. No one in the group 
> had a solution on how to preserve all the ideas we discussed and 
> represent all contributions. We lost precious time going in that route.
> 
> If anyone knows how to solve it, please suggest, yet know that this may 
> be lost effort that may only delay things.
> 
> My opinion is that the reviewers will have to weigh the importance of 
> our summary table compared to any other aspect. If they decide against 
> it, there is always another venue. The price is delay.
> 
> And Sheriff, frankly, if a publisher will not publish while preserving 
> all ideas, then what value is being served in publishing?
> 
> If a reviewer does not like a particular idea, they should argue 
> against it and we will respond, yet a "general rewrite" argument is not 
> sufficient. A good editor should spot it and act accordingly.
> 
> The paper is readable, and has a clear summary for those who need it . 
> Any argument against this "not being good enough" requires proof by 
> example - if none can produce this example, then the argument should be 
> shelved.
> 
> For now, I ask that others attempt to join this discussion and attempt 
> to fix the issues we still need attention for.
> 
> Specifically, if anyone has an idea on more graphics, please suggest a 
> figure. Sheriff, you created our first figure, if you have ideas for 
> another figure, please help here. Otherwise we have some solutions for 
> the review that we can debate.
> 
> Thanks for answering during the holiday season and a happy new year.
> 
> Jacob
> 
> On Thu, Dec 30, 2021, 08:42 sheriff <sheriff at ebi.ac.uk> wrote:
> 
> Hi Jacob,
> 
> Thanks for leading the discussion.
> 
> You may have to wait until early Jan for everyone to get back from the 
> holidays and respond.
> 
> Also, I think, it may not be possible to get away without improving the 
> flow and readability of the manuscript.
> 
> Here are my comments on model sharing and license
> 
> Indeed often models are hard to locate. Hence in our reproducibility 
> study ( Tiwari et al. 2021, https://doi.org/10.15252/msb.20209982), we 
> strongly recommend authors to make their models public, but more 
> specifically through public repositories such as BioModels 
> (https://www.ebi.ac.uk/biomodels). We recommend authors to submit model 
> codes, parameter sets, and simulation conditions needed to reproduce 
> simulation studies.
> 
> BioModels provides model curation and annotation service, where the 
> models' reproducibility is assessed and annotated with controlled 
> vocabularies. The BioModels offers sophisticated search engines to 
> search and locate models and model components 
> (https://www.ebi.ac.uk/biomodels/parameterSearch). Overall this will 
> greatly facilitate model reuse.
> 
> More importantly, all models are disseminated under the permissive CC0 
> license in BioModels. This is because it is crucial to openly release 
> the models developed through public funding. Also, curation services 
> done at BioModels are supported by public funding.
> 
> Although few users suggested we use CC-BY, we decided to continue with 
> CC0. However, we strongly recommend our users to properly cite the 
> model when reused and most researchers normally do. Also, If a modeler 
> collects and integrates components from several hundred models, it may 
> become difficult to properly acknowledge them all.
> 
> Best regards
> 
> Sheriff
> 
> On 2021-12-27 10:59, Jacob Barhak wrote:
> 
> Greetings Paper contributors,
> 
> Especially Alex, Hana, William, and Sheriff should read this since I 
> name them below.
> 
> To move things forwards I started editing the paper and am sending an 
> update of our checklist with responses to the reviewers. Please check 
> the list if you think you want to contribute more to the paper - it is 
> open for editing - the time window will close once we answered all 
> reviewers - however, please focus on what reviewers asked for and do 
> nto add anything else - I am restraining myself from adding more 
> otherwise there will be no end to editing.
> 
> The answers for the reviews so far are:
> 
> 1. We thank reviewer 1 for pointing out the papers - we added those 
> references into the discussion. Especially we thank the reviewer for 
> pointing out the (Viceconti et al., 2019) paper - it is indeed unique 
> in its attempt to classify and quantify model credibility with some 
> good examples. We agree it should be cited in our paper since it is an 
> important step forward.
> 
> 2. TBD - Alex will provide an answer to the reviewer - he confirmed he 
> got the message
> 
> 3. I suggest that the reviewer will have the following answer to the 
> request to focus on COVID-19 in the title. We thank the reviewer for 
> the suggestion to focus on COVId-19 and put it in the title. However, 
> we intend the paper to cover more than just one disease - in fact we 
> have examples in the paper that span diabetes, heart disease as well as 
> COVID - We did add recent COVID-19 models as the reviewer suggested, 
> yet we would lie to keep the broader approach and keep the title 
> without change.
> 
> 4. TBD - I have no idea for more graphics - this is still an open item 
> - please suggest solutions.
> 
> 5. The "Contribution to the Field Statement" section was revised. Any 
> contributor is welcome to edit it further.
> 
> 6. Regarding a major rewrite I suggest the following answer to the 
> reviewer:  "We thank the reviewer for advice to improve the paper 
> readability and indeed the group is aware that there are many ideas 
> presented in the paper - yet this is expected from a large and diverse 
> group that provided contributions. The reviewer should be aware that we 
> made every attempt to keep all the ideas intact and what seems 
> sometimes as noise, is actually a choir  where every voice is 
> important. This choire attempts to explain the modeling complexity. We 
> did have attempts to reduce the work to something easier to digest - 
> see the versions we quote in the paper. However, those versions were 
> never approved by all contributors as necessary for publication - this 
> is actually the law. What the reviewer saw was a compromise that keeps 
> all ideas as best we could while providing a summary that allows 
> readers to jump to sections of interest. Specifically table 1 is 
> perhaps the most important artifact in the paper. and do ask the 
> reviewer to weigh the importance of its quick publication. We can spend 
> months on trying to revise the paper and reach agreement, yet this will 
> probably not change the core of the paper and will waste resources on 
> 18 contributors needing to constantly change things. Therefore we ask 
> that the reviewer consider the benefit of releasing this information 
> fast so it will make an impact rather than sending is back to rewrites 
> that will delay impact in the name of style - an agreement among such a 
>  large and diverse group has a weight and the discussions around this 
> paper are transparent so that readers can go back and clarify things if 
> needed. "  - beyond this response, if someone wants to spend time and 
> improve the introduction without omitting ideas or references other 
> than their own, then please let us know - I am open to suggestions.
> 
> 7. The answer above in 6. should answer the reviewer that requested 
> changing the introduction - however, if anyone has another suggestion, 
> I am open.
> 
> 8. I modified the specific section that the reviewer asked to address. 
> Reviewer 2 wants additional discussion around the "reproducibility 
> crisis" section  and asks a valid question about expectations we should 
> address - ""Computational biomedical modeling... was expected to be 
> less affected by the reproducibility crisis." Is this true, and why 
> would it be so? "  - My answer to the reviewer is that unlike 
> biological processes that have random nature and experiments would not 
> repeat if repeated, while computer software should be deterministic and 
> it should be repeatable if designed well.  Unfortunately we are not 
> experiencing this promise. If anyone wants to expand the discussion, 
> please feel free to do so.
> 
> 9. TBD - See also point 12. below. A discussion was started in this 
> thread: 
> https://lists.simtk.org/pipermail/vp-integration-subgroup/2021-December/000324.html 
> - suggested discussors are William, and Sheriff - yet anyone else is 
> welcome to join and contributes.  So far no confirmation or response 
> was recorded.
> 
> 10.  A discussion  was started in this thread: 
> https://lists.simtk.org/pipermail/vp-integration-subgroup/2021-December/000325.html 
>  - suggested discussor is Hana, So far no confirmation or response was 
> recorded.
> 
> 11. TBD - Reviewer 2 asks that we better handle the section on Data 
> availability and measurement definitions: I think we need to emphasize 
> solutions and separate them from issues that may not be solvable. 
> However, Hana and I wrote most of this section with some text from Alex 
> I believe, so we perhaps should meet and discuss how to improve it - it 
> is indeed long and needs fixing. Ideas from anyone are welcome.
> 
> 12. TBD - See - point 9 above - A discussion was started in this 
> thread: 
> https://lists.simtk.org/pipermail/vp-integration-subgroup/2021-December/000324.html 
> - suggested discussors are William, and Sheriff - yet anyone else is 
> welcome to join and contributes.  So far no confirmation or response 
> was recorded.
> 
> 13. TBD - Reviewer 2 asks to handle the Model Application and 
> Implementation Barriers section. We should decide what to do there, the 
> section may need expansion since the ideas there are solid, yet the 
> section is short so perhaps enhancing it makes better sense. I am open 
> to suggestions.
> 
> The revised paper is available in:
> 
> https://docs.google.com/document/d/1IMEgmdNkx-EsnOjGuegpenSIMmKIkK00Lc8Gred3QxM/edit?usp=sharing
> 
> I ask that all contributors interested in making modifications write 
> those ideas on the mailing list so that we can move forward with this 
> paper.
> 
> I really hope some of you will use the time before the year ends to 
> advance this.
> 
> Jacob
> 
> On Mon, Dec 20, 2021 at 5:50 PM Jacob Barhak <jacob.barhak at gmail.com> 
> wrote:
> Thanks Alex,
> 
> The paper opened back for editing. If someone really feels they have to 
> edit and they are not on the list - let me know and I will add them to 
> the list of editors. - yet for now - let us form a plan to answer the 
> reviewers and only then touch text again - otherwise this will get out 
> of hand since there are so many of us.
> 
> I appreciate your support.
> 
> Jacob
> 
> On Mon, Dec 20, 2021 at 3:50 PM Alexander Kulesza 
> <alexander.kulesza at novadiscovery.com> wrote:
> 
> Dear Jacob, dear all,
> Yes will take care of 2. and happy to help wherever I can.
> Best
> 
> On Sun, 19 Dec 2021 at 03:10, Jacob Barhak <jacob.barhak at gmail.com> 
> wrote:
> 
> Greetings to the paper contributors,
> 
> You may have seen my other message where I posted the reviews to our 
> paper.  I am sending those again below to start a new thread discussing 
> possible revisions.
> 
> Here is a checklist for revisions required:
> 
> *  We should discuss the references that reviewer 1 raised and add them 
> as reference
> * Reviewer 1 asked for more discussion around V&V40 - I believe Alex 
> added the text there originally - Alex do you think you can address the 
> request of the reviewer?
> * Reviewer 1 asked to focus on COVID-19 at the title - I am not sure we 
> wish to limit our scope since many of our ideas are applicable far 
> beyond COVID-19 - I leave it up for discussion in the group on how to 
> address this request by the reviewer
> * Reviewer 1 asked for additional graphics - I am unsure how to address 
> this beyond ur current diagram - ideas will help
> * We need to check the "Contribution to the field" section for grammar 
> and spelling - if someone can contribute more elegant text - now is the 
> time - I looked over it now and found no issues - yet I may have missed 
> something - this section may have been missed since it was added last 
> and perhaps vetted less than other sections. Also Reviewer 2 asked for 
> grammar corrections, so it is worthwhile proof reading the paper as a 
> whole.
> * Reviewer 2 asks for major rewrite to emphasize why the problems are 
> there - I think we should explain that the core of the paper is the 
> table and perhaps emphasize it in the paper beyond what the current 
> text - we also should reply to the reviewer and explain that the paper 
> is composed of contributions from a large group and we made an effort 
> to include every voice in the choir - each of those voices is important 
> and needs to be preserved - hopefully it will convince the reviewer 
> that change we will add will be sufficient
> * Reviewer 2 asks to revise the introduction  - I believe some changes 
> are possible - yet the introduction includes contributions from many 
> authors - at least 10 - and I fear losing something important someone 
> contributed - if someone has an idea on how to address this reviewer 
> without a painful transformation, please reply to this message.
> *  Reviewer 2 wants additional discussion around the "reproducibility 
> crisis" section  and asks a valid question about expectations we should 
> address - ""Computational biomedical modeling... was expected to be 
> less affected by the reproducibility crisis." Is this true, and why 
> would it be so? "  - My answer to the reviewer is that unlike 
> biological processes that have random nature and experiments would not 
> repeat if repeated, while computer software should be deterministic and 
> it should be repeatable if designed well.  Unfortunately we are not 
> experiencing this promise - yet I believe the reviewer wants more 
> discussion beyond this section so I am happy to discuss this.
> * Reviewer 2- writes:  "Models are Hard to Locate: Are the authors 
> suggesting that entire simulation workflows, from model construction to 
> analysis, should be publicly available? At what point does one consider 
> intellectual property? Do the authors advocate for such extensive 
> publishing for all models, or only ones that are intended to be widely 
> re-used? - those are important points we need to discuss - we need to 
> better explain the difficulties we are having when creating models and 
> the reviewer is absolutely correct about expanding the discussion to IP 
> - I suggest we create a thread for this discussion and reference it - I 
> suggest it we merged with the licensing  issue that I will address 
> later - there is a strong connection there - the reviewer was very 
> observant and sees the bigger problem. However,we need to eventually 
> distill our discussion to recommendations that will inline.
> * Reviewer 2 asks that we fix the unit standardization section - I 
> believe Hana and myself were the largest contributors there - Hana - I 
> will start a discussion on that topic in a separate email where we can 
> publicly discuss how to fix this - others will be welcome to 
> contribute.
> * Reviewer 2 asks that we better handle the section on Data 
> availability and measurement definitions: in think we need to emphasize 
> solutions and separate it from issues that may not be solvable. Ideas 
> are welcome.
> * Reviewer 2 asks good questions with regards to licensing following 
> our text- I personally have good answers to the reviewer and William 
> and I had some discussion on the topic in this list I suggest we expand 
> this discussion in a separate thread - hopefully William and perhaps 
> others will join the discussion. This discussion should also address 
> the IP issues raised by the reviewer for the "models are hard to locate 
> section."
> * Reviewer 2 asks to handle the Model Application and Implementation 
> Barriers section. We should decide what to do there, the section may 
> need expansion since the ideas there are solid, yet the section is 
> short so perhaps enhancing it makes better sense. I am open to 
> suggestions.
> 
> Those are the items I located and my suggestions. It seems we need 
> attention from Alex, Hana, myself and william. However, anyone on the 
> list is welcome to participate and suggest changes.
> 
> I will start the discussion threads on specific topics. Hopefully we 
> can get it done quickly.
> 
> Jacob
> 
> -
> 
> ########### Original Reviews #############
> 
> There are 2 reviews - both require major changes. I am copying the 
> relevant text below.  If more appear, I will let you know, yet I only 
> got this message today although the reviews are dated a few days ago.
> 
> Reviewer 1:
> Recommendation for the Editor: Substantial revision is required
> 
> Please list your revision requests for the authors and provide your 
> detailed comments, including highlighting limitations and strengths of 
> the review. If you have additional comments based on Q2 and Q3 you can 
> add them as well.
> 
> Karr and co-authors made an interesting and exaustive point about the 
> reproducibility crisis that leads to inability to reuse and integrate 
> models, especially about COVID-19 disease.
> Within the manuscript some typos and missing information are present.
> I'd suggest the authors to revise the entire manuscript especially in 
> terms of the state of the art, revising and updating the most relevant 
> examples of computational models dealing with COVID-19 and in general 
> about some semi-standardised proposals about the pipeline to follow for 
> the verification and validation of model credibility. In particular, 
> the authors failed to mention and cite some major results on in silico 
> modeling about COVID-19 up to now. See for example:
> a."In silico trial to test COVID-19 candidate vaccines: a case study 
> with UISS platform", Russo, G., Pennisi, M., Fichera, E., ...Viceconti, 
> M., Pappalardo, F., BMC Bioinformatics, 2020, 21, 527.
> b. Russo G, Di Salvatore V, Sgroi G, Parasiliti Palumbo GA, Reche PA, 
> Pappalardo F. "A multi-step and multi-scale bioinformatic protocol to 
> investigate potential SARS-CoV-2 vaccine targets" [published online 
> ahead of print, 2021 Oct 5]. Brief Bioinform. 2021;bbab403. 
> doi:10.1093/bib/bbab403.
> Moreover, when the authors mention ASME V&V40 procedure, it could be 
> useful to spend few words about the model credibility pipeline drawn 
> for predictive models in biomedicine, such as the one suggested in 
> "Credibility of in Silico Trial Technologies-A Theoretical Framing", 
> Viceconti, M., Juarez, M.A., Curreli, C., ...Russo, G., Pappalardo, F., 
> IEEE Journal of Biomedical and Health Informatics, 2020, 24(1), pp. 
> 4-13, 8884189.
> For this specific aspect, the authors should talk about "specific 
> question of interest" and not "specific purpose" in the paragraph 
> called "Credibility of Models" to follow in such a way a standardised 
> language.
> Moreover, the authors should refer to COVID-19 also in the title, with 
> a specific mention about the fact that the main topic of model 
> integration in computational biology will be discussed inside the 
> COVID-19 context.
> Furthermore, for the model key concepts such as 
> Reusability-Extensibility-Extractability-Portability the authors should 
> described and outlined through a graphical sketch or visual 
> representation that summarises these key point.
> The authors should also fix some grammar and writing typos present in 
> the "Contribution to the field" section.
> 
> a. Is the quality of the figures and tables satisfactory?
> - No
> 
> b. Does the reference list cover the relevant literature adequately and 
> in an unbiased manner?
> - No
> 
> c. Does this manuscript refer only to published data? (unpublished or 
> original data is not allowed for this article type)
> - Yes
> 
> d. Does the review include a balanced, comprehensive, and critical view 
> of the research area?
> - Yes
> 
> Reviewer 2
> 
> Please list your revision requests for the authors and provide your 
> detailed comments, including highlighting limitations and strengths of 
> the review. If you have additional comments based on Q2 and Q3 you can 
> add them as well.
> 
> The article often reads as a stream-of-consciousness account of the 
> discussions that took place but lacks a clear thesis or 
> recommendations. It is not clear what, if anything, the authors are 
> advocating for. It is not always clear why the issues being discussed 
> are problematic, or that they can be reasonably addressed. Some of the 
> issues raised are indeed important and should be discussed, but the 
> paper lacks focus and does not tell a cohesive story. I believe this 
> manuscript requires a major re-write to be suitable for publication. 
> The authors should consider narrowing the scope of the discussion and 
> focusing on a cohesive set of recommendations or open questions. More 
> specifically, I make a few suggestions below:
> 
> Major comments
> 1. The introduction is long and repeats itself (e.g., "much less is 
> known about how viral infections spread throughout the body..." is 
> repeated verbatim). It is not clear from the introduction what the main 
> goal of the paper is or why the "reproducibility crisis" is truly a 
> crisis. Why is the discussion of composition and black/white box models 
> relevant to the introduction? Further, this section is subtitled "the 
> promise of modeling", which does not seem to match the content.
> 2. The Reproducibility Crisis: "Computational biomedical modeling... 
> was expected to be less affected by the reproducibility crisis." Is 
> this true, and why would it be so? One would think that more complex 
> models would suffer more from a lack of reproducibility. It may be 
> helpful to define what exactly the "reproducibility crisis" refers to.
> 3. Models are Hard to Locate: Are the authors suggesting that entire 
> simulation workflows, from model construction to analysis, should be 
> publicly available? At what point does one consider intellectual 
> property? Do the authors advocate for such extensive publishing for all 
> models, or only ones that are intended to be widely re-used?
> 4. Unit standardization: The conversion from PFU or TCID50 to 
> individual virions is likely to differ across viruses - are the authors 
> focused on COVID here? Are the authors advocating for a standard 
> conversion factor? It is not clear what the purpose of this discussion 
> is. As the authors mention, different scales require different units. 
> Even at a single scale, different models may require different units 
> for numerical reasons. It is not clear what the authors are advocating 
> for here.
> 5. Data availability and measurement definitions: This section seems to 
> outline limitations of available data, but again makes no 
> recommendations or proposed solution to any of the issues raised. Is 
> this the intention? Most of the issues raised here reflect limitations 
> of experimental science or data privacy, which likely cannot be 
> meaningfully addressed by the modeling community.
> 6. Models are Not Consistently Licensed...: Are the authors implying 
> here that all modeling work should be published with no rights 
> reserved? Is it reasonable to expect modelers to make their work freely 
> usable by others for profit? Is it reasonable for institutions to allow 
> this? How much does this really contribute to reproducibility and 
> utility?
> 7. Model Application and Implementation Barriers: This section seems 
> unnecessary and out of place.
> 8. There are grammar and punctuation errors scattered throughout; 
> please edit carefully.
> 
> a. Is the quality of the figures and tables satisfactory?
> - Not Applicable
> 
> b. Does the reference list cover the relevant literature adequately and 
> in an unbiased manner?
> - Yes
> 
> c. Does this manuscript refer only to published data? (unpublished or 
> original data is not allowed for this article type)
> - Yes
> 
> d. Does the review include a balanced, comprehensive, and critical view 
> of the research area?
> No answer given.
> 
> ======== _______________________________________________
> Vp-reproduce-subgroup mailing list
> Vp-reproduce-subgroup at lists.simtk.org
> https://lists.simtk.org/mailman/listinfo/vp-reproduce-subgroup
> --
> 
> Alexander Kulesza
> Team leader
> Modeling & simulation / Biomodeling
> alexander.kulesza at novadiscovery.com
> +33 7 82 92 44 62
> 
> nova
> DISCOVERY
> www.novadiscovery.com [1]
> 1 Place Verrazzano, 69009 Lyon
> +33 9 72 53 13 01
> _This message is intended only for the personal and confidential use of 
> the designated recipient(s) named above. If you are not the intended 
> recipient of this message you are hereby notified that any review, 
> dissemination, distribution or copying of this message is strictly 
> prohibited. Email transmission cannot be guaranteed to be secure or 
> error-free. Therefore, we do not represent that this information is 
> complete or accurate and it should not be relied upon as such. All 
> information is subject to change without notice._

_______________________________________________
Vp-integration-subgroup mailing list
Vp-integration-subgroup at lists.simtk.org
https://lists.simtk.org/mailman/listinfo/vp-integration-subgroup

  Pls



Links:
------
[1] http://www.novadiscovery.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.simtk.org/pipermail/vp-integration-subgroup/attachments/20211230/05caf1a4/attachment-0001.html>


More information about the Vp-integration-subgroup mailing list