[Vp-reproduce-subgroup] Figure requested by the reviewer

Jacob Barhak jacob.barhak at gmail.com
Fri Dec 31 20:25:48 PST 2021 

Hi Sheriff, Hi John,

How about this figure to satisfy the reviewer:
https://drive.google.com/file/d/13__2yL442yDD92DMfOO-ulqIpPxzbYTu/view?usp=sharing

My graphics are not the best, yet if this is what you want, perhaps someone
more artistic can create it -here is the original file I used with
libreoffice:
https://drive.google.com/file/d/1PikwST54XcezSSwMVIGcccm_XineJsJo/view?usp=sharing

Please remember we need to move fast since our extension is only for about
a week and we need to approve the response and revised version.

Hopefully we can quickly get this figure to satisfy the reviewer.

             Jacob


On Thu, Dec 30, 2021 at 3:14 PM John Rice <john.rice at noboxes.org> wrote:

> I like that!
>
> On Dec 30, 2021, at 15:28, sheriff <sheriff at ebi.ac.uk> wrote:
>
> 
>
> Hi Jacob,
>
> One crude idea for the graphics could be a cartoon of hurdling, where each
> hurdle corresponds to the one we listed in the table or the dozen
> challenges version. Under each, we could add a sentence about it and this
> could be a summary figure.
>
> Probably there are better ideas.
>
> Best
>
> Sheriff
>
>
> On 2021-12-30 20:18, Jacob Barhak wrote:
>
> Thanks Sheriff,
>
> You comments would be more helpful within a different thread
> https://lists.simtk.org/pipermail/vp-integration-subgroup/2021-December/000324.html
>
>  I will copy your text there for continuity. I assumed many are on
> vacation, yet I was trying to buy time in hope some of us are active.
>
> As for a rewrite. We have gone this road before. No one in the group had a
> solution on how to preserve all the ideas we discussed and represent all
> contributions. We lost precious time going in that route.
>
> If anyone knows how to solve it, please suggest, yet know that this may be
> lost effort that may only delay things.
>
> My opinion is that the reviewers will have to weigh the importance of our
> summary table compared to any other aspect. If they decide against it,
> there is always another venue. The price is delay.
>
> And Sheriff, frankly, if a publisher will not publish while preserving all
> ideas, then what value is being served in publishing?
>
> If a reviewer does not like a particular idea, they should argue against
> it and we will respond, yet a "general rewrite" argument is not sufficient.
> A good editor should spot it and act accordingly.
>
> The paper is readable, and has a clear summary for those who need it . Any
> argument against this "not being good enough" requires proof by example -
> if none can produce this example, then the argument should be shelved.
>
> For now, I ask that others attempt to join this discussion and attempt to
> fix the issues we still need attention for.
>
> Specifically, if anyone has an idea on more graphics, please suggest a
> figure. Sheriff, you created our first figure, if you have ideas for
> another figure, please help here. Otherwise we have some solutions for the
> review that we can debate.
>
> Thanks for answering during the holiday season and a happy new year.
>
>         Jacob
>
>
>
>
>
>
>
>
>
> On Thu, Dec 30, 2021, 08:42 sheriff <sheriff at ebi.ac.uk> wrote:
>
> Hi Jacob,
>
> Thanks for leading the discussion.
>
> You may have to wait until early Jan for everyone to get back from the
> holidays and respond.
>
> Also, I think, it may not be possible to get away without improving the
> flow and readability of the manuscript.
>
> Here are my comments on model sharing and license
>
> Indeed often models are hard to locate. Hence in our reproducibility study
> ( Tiwari et al. 2021, https://doi.org/10.15252/msb.20209982), we strongly
> recommend authors to make their models public, but more specifically
> through public repositories such as BioModels (
> https://www.ebi.ac.uk/biomodels). We recommend authors to submit model
> codes, parameter sets, and simulation conditions needed to reproduce
> simulation studies.
>
> BioModels provides model curation and annotation service, where the
> models' reproducibility is assessed and annotated with controlled
> vocabularies. The BioModels offers sophisticated search engines to search
> and locate models and model components (
> https://www.ebi.ac.uk/biomodels/parameterSearch). Overall this will
> greatly facilitate model reuse.
>
> More importantly, all models are disseminated under the permissive CC0
> license in BioModels. This is because it is crucial to openly release the
> models developed through public funding. Also, curation services done at
> BioModels are supported by public funding.
>
> Although few users suggested we use CC-BY, we decided to continue with
> CC0. However, we strongly recommend our users to properly cite the model
> when reused and most researchers normally do. Also, If a modeler collects
> and integrates components from several hundred models, it may become
> difficult to properly acknowledge them all.
>
> Best regards
>
> Sheriff
>
>
> On 2021-12-27 10:59, Jacob Barhak wrote:
>
> Greetings Paper contributors,
>
> Especially Alex, Hana, William, and Sheriff should read this since I name
> them below.
>
> To move things forwards I started editing the paper and am sending an
> update of our checklist with responses to the reviewers. Please check the
> list if you think you want to contribute more to the paper - it is open for
> editing - the time window will close once we answered all reviewers -
> however, please focus on what reviewers asked for and do nto add anything
> else - I am restraining myself from adding more otherwise there will be no
> end to editing.
>
> The answers for the reviews so far are:
>
> 1. We thank reviewer 1 for pointing out the papers - we added those
> references into the discussion. Especially we thank the reviewer
> for pointing out the (Viceconti et al., 2019) paper - it is indeed unique
> in its attempt to classify and quantify model credibility with some good
> examples. We agree it should be cited in our paper since it is an important
> step forward.
>
> 2. TBD - Alex will provide an answer to the reviewer - he confirmed he got
> the message
>
> 3. I suggest that the reviewer will have the following answer to the
> request to focus on COVID-19 in the title. We thank the reviewer for the
> suggestion to focus on COVId-19 and put it in the title. However, we intend
> the paper to cover more than just one disease - in fact we have examples in
> the paper that span diabetes, heart disease as well as COVID - We did add
> recent COVID-19 models as the reviewer suggested, yet we would lie to keep
> the broader approach and keep the title without change.
>
> 4. TBD - I have no idea for more graphics - this is still an open item -
> please suggest solutions.
>
> 5. The "Contribution to the Field Statement" section was revised. Any
> contributor is welcome to edit it further.
>
> 6. Regarding a major rewrite I suggest the following answer to the
> reviewer:  "We thank the reviewer for advice to improve the paper
> readability and indeed the group is aware that there are many ideas
> presented in the paper - yet this is expected from a large and diverse
> group that provided contributions. The reviewer should be aware that we
> made every attempt to keep all the ideas intact and what seems sometimes as
> noise, is actually a choir  where every voice is important. This choire
> attempts to explain the modeling complexity. We did have attempts to reduce
> the work to something easier to digest - see the versions we quote in the
> paper. However, those versions were never approved by all contributors as
> necessary for publication - this is actually the law. What the reviewer saw
> was a compromise that keeps all ideas as best we could while providing a
> summary that allows readers to jump to sections of interest.
> Specifically table 1 is perhaps the most important artifact in the paper.
> and do ask the reviewer to weigh the importance of its quick publication.
> We can spend months on trying to revise the paper and reach agreement, yet
> this will probably not change the core of the paper and will waste
> resources on 18 contributors needing to constantly change things. Therefore
> we ask that the reviewer consider the benefit of releasing this information
> fast so it will make an impact rather than sending is back to rewrites that
> will delay impact in the name of style - an agreement among such a  large
> and diverse group has a weight and the discussions around this paper are
> transparent so that readers can go back and clarify things if needed. "  -
> beyond this response, if someone wants to spend time and improve the
> introduction without omitting ideas or references other than their own,
> then please let us know - I am open to suggestions.
>
> 7. The answer above in 6. should answer the reviewer that requested
> changing the introduction - however, if anyone has another suggestion, I am
> open.
>
> 8. I modified the specific section that the reviewer asked to address.
> Reviewer 2 wants additional discussion around the "reproducibility crisis"
> section  and asks a valid question about expectations we should address
> - ""Computational biomedical modeling... was expected to be less affected
> by the reproducibility crisis." Is this true, and why would it be so? "  -
> My answer to the reviewer is that unlike biological processes that have
> random nature and experiments would not repeat if repeated, while computer
> software should be deterministic and it should be repeatable if designed
> well.  Unfortunately we are not experiencing this promise. If anyone wants
> to expand the discussion, please feel free to do so.
>
> 9. TBD - See also point 12. below. A discussion was started in this
> thread:
> https://lists.simtk.org/pipermail/vp-integration-subgroup/2021-December/000324.html
> - suggested discussors are William, and Sheriff - yet anyone else is
> welcome to join and contributes.  So far no confirmation or response was
> recorded.
>
> 10.  A discussion  was started in this thread:
> https://lists.simtk.org/pipermail/vp-integration-subgroup/2021-December/000325.html
> - suggested discussor is Hana, So far no confirmation or response was
> recorded.
>
> 11. TBD - Reviewer 2 asks that we better handle the section on Data
> availability and measurement definitions: I think we need to emphasize
> solutions and separate them from issues that may not be solvable. However,
> Hana and I wrote most of this section with some text from Alex I believe,
> so we perhaps should meet and discuss how to improve it - it is indeed long
> and needs fixing. Ideas from anyone are welcome.
>
> 12. TBD - See - point 9 above - A discussion was started in this thread:
> https://lists.simtk.org/pipermail/vp-integration-subgroup/2021-December/000324.html
> - suggested discussors are William, and Sheriff - yet anyone else is
> welcome to join and contributes.  So far no confirmation or response was
> recorded.
>
> 13. TBD - Reviewer 2 asks to handle the Model Application and
> Implementation Barriers section. We should decide what to do there, the
> section may need expansion since the ideas there are solid, yet the section
> is short so perhaps enhancing it makes better sense. I am open to
> suggestions.
>
> The revised paper is available in:
>
>
> https://docs.google.com/document/d/1IMEgmdNkx-EsnOjGuegpenSIMmKIkK00Lc8Gred3QxM/edit?usp=sharing
>
> I ask that all contributors interested in making modifications write those
> ideas on the mailing list so that we can move forward with this paper.
>
> I really hope some of you will use the time before the year ends to
> advance this.
>
>
>                   Jacob
>
>
>
>
> On Mon, Dec 20, 2021 at 5:50 PM Jacob Barhak <jacob.barhak at gmail.com>
> wrote:
>
> Thanks Alex,
>
> The paper opened back for editing. If someone really feels they have to
> edit and they are not on the list - let me know and I will add them to the
> list of editors. - yet for now - let us form a plan to answer the reviewers
> and only then touch text again - otherwise this will get out of hand since
> there are so many of us.
>
> I appreciate your support.
>
>             Jacob
>
> On Mon, Dec 20, 2021 at 3:50 PM Alexander Kulesza <
> alexander.kulesza at novadiscovery.com> wrote:
>
> Dear Jacob, dear all,
> Yes will take care of 2. and happy to help wherever I can.
> Best
>
> On Sun, 19 Dec 2021 at 03:10, Jacob Barhak <jacob.barhak at gmail.com> wrote:
>
>
> Greetings to the paper contributors,
>
> You may have seen my other message where I posted the reviews to our
> paper.  I am sending those again below to start a new thread discussing
> possible revisions.
>
> Here is a checklist for revisions required:
>
>
>    1.  We should discuss the references that reviewer 1 raised and add
>    them as reference
>    2. Reviewer 1 asked for more discussion around V&V40 - I believe Alex
>    added the text there originally - Alex do you think you can address the
>    request of the reviewer?
>    3. Reviewer 1 asked to focus on COVID-19 at the title - I am not sure
>    we wish to limit our scope since many of our ideas are applicable far
>    beyond COVID-19 - I leave it up for discussion in the group on how to
>    address this request by the reviewer
>    4. Reviewer 1 asked for additional graphics - I am unsure how to
>    address this beyond ur current diagram - ideas will help
>    5. We need to check the "Contribution to the field" section for
>    grammar and spelling - if someone can contribute more elegant text - now is
>    the time - I looked over it now and found no issues - yet I may have missed
>    something - this section may have been missed since it was added last and
>    perhaps vetted less than other sections. Also Reviewer 2 asked for grammar
>    corrections, so it is worthwhile proof reading the paper as a whole.
>    6. Reviewer 2 asks for major rewrite to emphasize why the problems are
>    there - I think we should explain that the core of the paper is the table
>    and perhaps emphasize it in the paper beyond what the current text - we
>    also should reply to the reviewer and explain that the paper is composed of
>    contributions from a large group and we made an effort to include every
>    voice in the choir - each of those voices is important and needs to be
>    preserved - hopefully it will convince the reviewer that change we will add
>    will be sufficient
>    7. Reviewer 2 asks to revise the introduction  - I believe some
>    changes are possible - yet the introduction includes contributions from
>    many authors - at least 10 - and I fear losing something important someone
>    contributed - if someone has an idea on how to address this reviewer
>    without a painful transformation, please reply to this message.
>    8.  Reviewer 2 wants additional discussion around the "reproducibility
>    crisis" section  and asks a valid question about expectations we should
>    address - ""Computational biomedical modeling... was expected to be less
>    affected by the reproducibility crisis." Is this true, and why would it be
>    so? "  - My answer to the reviewer is that unlike biological processes that
>    have random nature and experiments would not repeat if repeated, while
>    computer software should be deterministic and it should be repeatable if
>    designed well.  Unfortunately we are not experiencing this promise - yet I
>    believe the reviewer wants more discussion beyond this section so I am
>    happy to discuss this.
>    9. Reviewer 2- writes:  "Models are Hard to Locate: Are the authors
>    suggesting that entire simulation workflows, from model construction to
>    analysis, should be publicly available? At what point does one consider
>    intellectual property? Do the authors advocate for such extensive
>    publishing for all models, or only ones that are intended to be widely
>    re-used? - those are important points we need to discuss - we need to
>    better explain the difficulties we are having when creating models and the
>    reviewer is absolutely correct about expanding the discussion to IP - I
>    suggest we create a thread for this discussion and reference it - I
>    suggest it we merged with the licensing  issue that I will address later -
>    there is a strong connection there - the reviewer was very observant and
>    sees the bigger problem. However,we need to eventually distill our
>    discussion to recommendations that will inline.
>    10. Reviewer 2 asks that we fix the unit standardization section - I
>    believe Hana and myself were the largest contributors there - Hana - I will
>    start a discussion on that topic in a separate email where we can publicly
>    discuss how to fix this - others will be welcome to contribute.
>    11. Reviewer 2 asks that we better handle the section on Data
>    availability and measurement definitions: in think we need to emphasize
>    solutions and separate it from issues that may not be solvable. Ideas are
>    welcome.
>    12. Reviewer 2 asks good questions with regards to licensing following
>    our text- I personally have good answers to the reviewer and William and I
>    had some discussion on the topic in this list I suggest we expand this
>    discussion in a separate thread - hopefully William and perhaps others will
>    join the discussion. This discussion should also address the IP issues
>    raised by the reviewer for the "models are hard to locate section."
>    13. Reviewer 2 asks to handle the Model Application and Implementation
>    Barriers section. We should decide what to do there, the section may need
>    expansion since the ideas there are solid, yet the section is short so
>    perhaps enhancing it makes better sense. I am open to suggestions.
>
>
> Those are the items I located and my suggestions. It seems we need
> attention from Alex, Hana, myself and william. However, anyone on the list
> is welcome to participate and suggest changes.
>
> I will start the discussion threads on specific topics. Hopefully we can
> get it done quickly.
>
>          Jacob
>
>
>
>
>
>
> -
>
>
>
>
>
> ########### Original Reviews #############
>
> There are 2 reviews - both require major changes. I am copying the
> relevant text below.  If more appear, I will let you know, yet I only got
> this message today although the reviews are dated a few days ago.
>
> Reviewer 1:
> Recommendation for the Editor: Substantial revision is required
>
> Please list your revision requests for the authors and provide your
> detailed comments, including highlighting limitations and strengths of the
> review. If you have additional comments based on Q2 and Q3 you can add them
> as well.
>
> Karr and co-authors made an interesting and exaustive point about the
> reproducibility crisis that leads to inability to reuse and integrate
> models, especially about COVID-19 disease.
> Within the manuscript some typos and missing information are present.
> I'd suggest the authors to revise the entire manuscript especially in
> terms of the state of the art, revising and updating the most relevant
> examples of computational models dealing with COVID-19 and in general about
> some semi-standardised proposals about the pipeline to follow for the
> verification and validation of model credibility. In particular, the
> authors failed to mention and cite some major results on in silico modeling
> about COVID-19 up to now. See for example:
> a."In silico trial to test COVID-19 candidate vaccines: a case study with
> UISS platform", Russo, G., Pennisi, M., Fichera, E., ...Viceconti, M.,
> Pappalardo, F., BMC Bioinformatics, 2020, 21, 527.
> b. Russo G, Di Salvatore V, Sgroi G, Parasiliti Palumbo GA, Reche PA,
> Pappalardo F. "A multi-step and multi-scale bioinformatic protocol to
> investigate potential SARS-CoV-2 vaccine targets" [published online ahead
> of print, 2021 Oct 5]. Brief Bioinform. 2021;bbab403.
> doi:10.1093/bib/bbab403.
> Moreover, when the authors mention ASME V&V40 procedure, it could be
> useful to spend few words about the model credibility pipeline drawn for
> predictive models in biomedicine, such as the one suggested in "Credibility
> of in Silico Trial Technologies-A Theoretical Framing", Viceconti, M.,
> Juarez, M.A., Curreli, C., ...Russo, G., Pappalardo, F., IEEE Journal of
> Biomedical and Health Informatics, 2020, 24(1), pp. 4–13, 8884189.
> For this specific aspect, the authors should talk about "specific question
> of interest" and not "specific purpose" in the paragraph called
> "Credibility of Models" to follow in such a way a standardised language.
> Moreover, the authors should refer to COVID-19 also in the title, with a
> specific mention about the fact that the main topic of model integration in
> computational biology will be discussed inside the COVID-19 context.
> Furthermore, for the model key concepts such as
> Reusability-Extensibility-Extractability-Portability the authors should
> described and outlined through a graphical sketch or visual representation
> that summarises these key point.
> The authors should also fix some grammar and writing typos present in the
> "Contribution to the field" section.
>
> a. Is the quality of the figures and tables satisfactory?
> - No
>
> b. Does the reference list cover the relevant literature adequately and in
> an unbiased manner?
> - No
>
> c. Does this manuscript refer only to published data? (unpublished or
> original data is not allowed for this article type)
> - Yes
>
> d. Does the review include a balanced, comprehensive, and critical view of
> the research area?
> - Yes
>
>
>
> Reviewer 2
>
> Please list your revision requests for the authors and provide your
> detailed comments, including highlighting limitations and strengths of the
> review. If you have additional comments based on Q2 and Q3 you can add them
> as well.
>
>
> The article often reads as a stream-of-consciousness account of the
> discussions that took place but lacks a clear thesis or recommendations. It
> is not clear what, if anything, the authors are advocating for. It is not
> always clear why the issues being discussed are problematic, or that they
> can be reasonably addressed. Some of the issues raised are indeed important
> and should be discussed, but the paper lacks focus and does not tell a
> cohesive story. I believe this manuscript requires a major re-write to be
> suitable for publication. The authors should consider narrowing the scope
> of the discussion and focusing on a cohesive set of recommendations or open
> questions. More specifically, I make a few suggestions below:
>
> Major comments
> 1. The introduction is long and repeats itself (e.g., "much less is known
> about how viral infections spread throughout the body..." is repeated
> verbatim). It is not clear from the introduction what the main goal of the
> paper is or why the "reproducibility crisis" is truly a crisis. Why is the
> discussion of composition and black/white box models relevant to the
> introduction? Further, this section is subtitled "the promise of modeling",
> which does not seem to match the content.
> 2. The Reproducibility Crisis: "Computational biomedical modeling... was
> expected to be less affected by the reproducibility crisis." Is this true,
> and why would it be so? One would think that more complex models would
> suffer more from a lack of reproducibility. It may be helpful to define
> what exactly the "reproducibility crisis" refers to.
> 3. Models are Hard to Locate: Are the authors suggesting that entire
> simulation workflows, from model construction to analysis, should be
> publicly available? At what point does one consider intellectual property?
> Do the authors advocate for such extensive publishing for all models, or
> only ones that are intended to be widely re-used?
> 4. Unit standardization: The conversion from PFU or TCID50 to individual
> virions is likely to differ across viruses – are the authors focused on
> COVID here? Are the authors advocating for a standard conversion factor? It
> is not clear what the purpose of this discussion is. As the authors
> mention, different scales require different units. Even at a single scale,
> different models may require different units for numerical reasons. It is
> not clear what the authors are advocating for here.
> 5. Data availability and measurement definitions: This section seems to
> outline limitations of available data, but again makes no recommendations
> or proposed solution to any of the issues raised. Is this the intention?
> Most of the issues raised here reflect limitations of experimental science
> or data privacy, which likely cannot be meaningfully addressed by the
> modeling community.
> 6. Models are Not Consistently Licensed...: Are the authors implying here
> that all modeling work should be published with no rights reserved? Is it
> reasonable to expect modelers to make their work freely usable by others
> for profit? Is it reasonable for institutions to allow this? How much does
> this really contribute to reproducibility and utility?
> 7. Model Application and Implementation Barriers: This section seems
> unnecessary and out of place.
> 8. There are grammar and punctuation errors scattered throughout; please
> edit carefully.
>
>
> a. Is the quality of the figures and tables satisfactory?
> - Not Applicable
>
> b. Does the reference list cover the relevant literature adequately and in
> an unbiased manner?
> - Yes
>
> c. Does this manuscript refer only to published data? (unpublished or
> original data is not allowed for this article type)
> - Yes
>
> d. Does the review include a balanced, comprehensive, and critical view of
> the research area?
> No answer given.
>
>
> ========
> _______________________________________________
> Vp-reproduce-subgroup mailing list
> Vp-reproduce-subgroup at lists.simtk.org
> https://lists.simtk.org/mailman/listinfo/vp-reproduce-subgroup
>
>
>
> --
> Alexander Kulesza
> Team leader
> Modeling & simulation / Biomodeling
> alexander.kulesza at novadiscovery.com
> +33 7 82 92 44 62
> nova
> DISCOVERY
> www.novadiscovery.com
> 1 Place Verrazzano, 69009 Lyon
> +33 9 72 53 13 01
>
> *This message is intended only for the personal and confidential use of
> the designated recipient(s) named above. If you are not the intended
> recipient of this message you are hereby notified that any review,
> dissemination, distribution or copying of this message is strictly
> prohibited. Email transmission cannot be guaranteed to be secure or
> error-free. Therefore, we do not represent that this information is
> complete or accurate and it should not be relied upon as such. All
> information is subject to change without notice.*
>
>
> _______________________________________________
> Vp-integration-subgroup mailing list
> Vp-integration-subgroup at lists.simtk.org
> https://lists.simtk.org/mailman/listinfo/vp-integration-subgroup
>
>
>  Pls
>
>
> _______________________________________________
> Vp-reproduce-subgroup mailing list
> Vp-reproduce-subgroup at lists.simtk.org
> https://lists.simtk.org/mailman/listinfo/vp-reproduce-subgroup
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.simtk.org/pipermail/vp-reproduce-subgroup/attachments/20211231/d7ee5e66/attachment-0001.html>

More information about the Vp-reproduce-subgroup mailing list