<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Thanks Jakob for submitting the revision and others for the support. Thanks William for editing my response.<div class=""><br class=""></div><div class="">Best</div><div class="">Sheriff</div><div class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 7 Jan 2022, at 04:52, Jacob Barhak <<a href="mailto:jacob.barhak@gmail.com" class="">jacob.barhak@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Thanks Hana, Thanks William, and thanks to Gilberto, Alex, and Sheriff,<div class=""><br class=""></div><div class="">With your help and some effort, the revised paper was resubmitted in time - just now.</div><div class=""><br class=""></div><div class="">William, please keep on reminding me about your second affiliation - I kept a comment as a reminder to do this - the submission system did not allow making any changes in Author information - not sure why - if this gets accepted we should ask the editor to add your affiliation manually. </div><div class=""><br class=""></div><div class="">I urge all other authors to look at the changes we made and our response. We will have to approve the manuscript with everyone before it gets published - hopefully we will convince the reviewers this review round.</div><div class=""><br class=""></div><div class=""> Jacob</div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jan 6, 2022 at 3:51 PM Dobrovolny, Hana <<a href="mailto:h.dobrovolny@tcu.edu" class="">h.dobrovolny@tcu.edu</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr" class="">
<div id="gmail-m_8456291084743863762divtagdefaultwrapper" style="font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;" dir="ltr" class=""><p class="">I've read through the paper and fixed a few grammar mistakes.</p><p class="">Hana<br class="">
</p><p class=""><br class="">
</p>
<div id="gmail-m_8456291084743863762Signature" class="">
<div name="divtagdefaultwrapper" class="">
<div class=""><font size="2" class="">
<div class="">*******************************************************<br class="">
Dr. Hana Dobrovolny<br class="">
Associate Professor of Biophysics<br class="">
Texas Christian University<br class="">
TCU Box 298840<br class="">
Fort Worth, TX 76129<br class="">
<br class="">
phone: (817) 257-6379 fax: (817) 257-7742<br class="">
email: <a href="mailto:h.dobrovolny@tcu.edu" target="_blank" class="">h.dobrovolny@tcu.edu</a><br class="">
*******************************************************</div>
</font></div>
</div>
</div>
<br class="">
<br class="">
<div style="" class="">
<hr style="display:inline-block;width:98%" class="">
<div id="gmail-m_8456291084743863762divRplyFwdMsg" dir="ltr" class=""><font style="font-size:11pt" face="Calibri, sans-serif" class=""><b class="">From:</b> Vp-integration-subgroup <<a href="mailto:vp-integration-subgroup-bounces@lists.simtk.org" target="_blank" class="">vp-integration-subgroup-bounces@lists.simtk.org</a>> on behalf of Jacob Barhak <<a href="mailto:jacob.barhak@gmail.com" target="_blank" class="">jacob.barhak@gmail.com</a>><br class="">
<b class="">Sent:</b> January 5, 2022 11:48 PM<br class="">
<b class="">To:</b> William Waites; James Glazier<br class="">
<b class="">Cc:</b> <a href="mailto:vp-reproduce-subgroup@lists.simtk.org" target="_blank" class="">vp-reproduce-subgroup@lists.simtk.org</a>; <a href="mailto:vp-integration-subgroup@lists.simtk.org" target="_blank" class="">vp-integration-subgroup@lists.simtk.org</a>; John Gennari; Faeder, James R<br class="">
<b class="">Subject:</b> Re: [Vp-integration-subgroup] [Vp-reproduce-subgroup] Paper review and revision check list</font>
<div class=""> </div>
</div>
<div class=""><p class=""><span style="background-color:rgb(255,255,0)" class=""><strong class="">[EXTERNAL EMAIL WARNING]</strong> DO NOT CLICK LINKS or open attachments unless you recognize the sender and know the content is safe.</span></p>
<div class="">
<div dir="ltr" class="">Greetings paper contributors,
<div class=""><br class="">
</div>
<div class="">Gilberto has addressed the final point and added text in the paper and review, thus concluding our response.</div>
<div class=""><br class="">
</div>
<div class="">I went over the paper from start to finish again and cleaned it up.</div>
<div class=""><br class="">
</div>
<div class="">There are two last comments in the paper addressed to William Waites and James Glazier - if you can quickly check those, it would help. </div>
<div class=""><br class="">
</div>
<div class="">If someone else wants to read the paper once more, please do this today - otherwise I will convert the paper and our response to reviewers and submit it to the Journal portal.</div>
<div class=""><br class="">
</div>
<div class="">If anyone has any objections, now is the time to raise them so we can withdraw rather than submit. </div>
<div class=""><br class="">
</div>
<div class="">Here is the link to the revised paper:</div>
<div class=""><a href="https://urldefense.com/v3/__https://docs.google.com/document/d/1IMEgmdNkx-EsnOjGuegpenSIMmKIkK00Lc8Gred3QxM/edit?usp=sharing__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0hvK3OmJ8$" target="_blank" class="">https://docs.google.com/document/d/1IMEgmdNkx-EsnOjGuegpenSIMmKIkK00Lc8Gred3QxM/edit?usp=sharing</a><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Here is the link to response for reviewers:</div>
<div class=""><a href="https://urldefense.com/v3/__https://docs.google.com/document/d/1bO1CDbMMlyyweYayP9rGohPIfgvIhfe14sFchtJMJ-s/edit?usp=sharing__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0haRkeB0o$" target="_blank" class="">https://docs.google.com/document/d/1bO1CDbMMlyyweYayP9rGohPIfgvIhfe14sFchtJMJ-s/edit?usp=sharing</a><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Hopefully there will be no objections and can submit this tomorrow.</div>
<div class=""><br class="">
</div>
<div class=""> Jacob</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
</div>
<br class="">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, Jan 4, 2022 at 11:30 PM Jacob Barhak <<a href="mailto:jacob.barhak@gmail.com" target="_blank" class="">jacob.barhak@gmail.com</a>> wrote:<br class="">
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr" class="">Hi Alex, Hi William, Hi Gilberto, and all other contributors.
<div class=""><br class="">
</div>
<div class="">Alex, William and Gilberto are mentioned since we are the most active contributors.</div>
<div class=""><br class="">
</div>
<div class="">We have 2 days to finish edits. </div>
<div class=""><br class="">
</div>
<div class="">William - I attempted to add your suggestions to the paper regarding issue 11. You are free to revise further. </div>
<div class=""><br class="">
</div>
<div class="">Gilberto - I merged some of your texts, yet it seems you are the only one I saw interested in resolving issue 12 where the reviewer asks to remove the "Model Application and Implementation Barriers" section. Please suggest your corrections as soon as possible.
If you cannot do this by tomorrow I will do my best to add a response based on your idea. Yet I really hope you can do this before me.</div>
<div class=""><br class="">
</div>
<div class="">Please finalize any edits you have - and if anyone has any objection to the text, please raise it within one day since after tomorrow I will start closing down all issues. </div>
<div class=""><br class="">
</div>
<div class="">I would ask for a volunteer that will go over the paper once more after its finalized and check for typos and grammar - we do not want the paper to return again due to this issue. Time is tight, so please volunteer fast. </div>
<div class=""><br class="">
</div>
<div class="">Hopefully we can get it done in time.</div>
<div class=""><br class="">
</div>
<div class=""> Jacob</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
</div>
<br class="">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, Jan 4, 2022 at 7:25 AM Alexander Kulesza <<a href="mailto:alexander.kulesza@novadiscovery.com" target="_blank" class="">alexander.kulesza@novadiscovery.com</a>> wrote:<br class="">
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr" class="">Dear Jacob, dear all,
<div class="">thanks for endorsing my figure. I'll gladly add this. Please find enclosed the .png. </div>
<div class=""><br class="">
</div>
<div class="">What you say makes sense and I agree that exaggerating changes is not speeding up approval. Please still consider it as a tool to perform "major" change (ordering of sections is a major change of a paper in my opinion) without changing too much the sections
as such. But I agree that this could be tried out should the reviewers insist.</div>
<div class=""><br class="">
</div>
<div class="">I therefore reverted the table change (disconnecting ist from Figure 1) and suggest the following Figure caption: </div>
<div class=""><br class="">
</div>
<div class="">"Figure 2: <span style="background-color: transparent; font-size: 11pt; font-family: Arial; vertical-align: baseline; white-space: pre-wrap;" class="">Sketch of difficulties persistent that impede reproducibility, credibility, utility and integration
of models, especially in computational biology. An </span><span style="background-color: transparent; font-size: 11pt; font-family: Arial; vertical-align: baseline; white-space: pre-wrap;" class="">assignment of these difficulties to the four different
concepts interpreting them as hurdles is attempted. We would like to point out that this sequential assessment model indicated in the graph is only one of the possibilities a modeler would assess the suitability of available models. In the rest of the paper
we therefore address each difficulty separately. " </span></div>
<div class=""><span style="background-color: transparent; font-size: 11pt; font-family: Arial; vertical-align: baseline; white-space: pre-wrap;" class=""><br class="">
</span></div>
<div class=""><span style="background-color: transparent; font-size: 11pt; font-family: Arial; vertical-align: baseline; white-space: pre-wrap;" class="">With that sentence we can discuss the difficulties in any order.
</span></div>
<div class=""><span style="background-color: transparent; font-size: 11pt; font-family: Arial; vertical-align: baseline; white-space: pre-wrap;" class=""><br class="">
</span></div>
<div class=""><span style="background-color: transparent; font-size: 11pt; font-family: Arial; vertical-align: baseline; white-space: pre-wrap;" class="">Best
</span></div>
<div class=""><span style="background-color: transparent; font-size: 11pt; font-family: Arial; vertical-align: baseline; white-space: pre-wrap;" class="">Alexander
</span></div>
</div>
<br class="">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, 4 Jan 2022 at 13:24, Jacob Barhak <<a href="mailto:jacob.barhak@gmail.com" target="_blank" class="">jacob.barhak@gmail.com</a>> wrote:<br class="">
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="auto" class="">
<div dir="auto" class="">Yes Alex,</div>
<div dir="auto" class=""><br class="">
</div>
<div dir="auto" class="">Your image is much better than mine, please use it instead in the paper and send a link to the image itself in an acceptable image format so I can upload it to the submission system later on.</div>
<div dir="auto" class=""><br class="">
</div>
<div dir="auto" class="">As for changing the table and order, I am not sure. Although it may be possible. We need to think about it. Yet we really are crunched in time, so I am leaning to say no.</div>
<div dir="auto" class=""><br class="">
</div>
<div dir="auto" class="">First, because some of the difficulties span more than one category, for example unit standardization affects integration as well as reproducibility, only it appears first at reproducibility, therefore it was placed there in the figure.</div>
<div dir="auto" class=""><br class="">
</div>
<div dir="auto" class="">Second, if you look at the text there are some transition sentences among sections, that connect them, so swapping those around may just mess things. I am not sure it if will also mess up acronym definitions on the first appearance as well,
so shuffling order is not something's I would do unless necessary. And it was not requested by reviewers anyway.</div>
<div dir="auto" class=""><br class="">
</div>
<div dir="auto" class="">I am also trying to minimize changes and focus only on what was requested rather than change a lot more. Otherwise we may never finish changing things since each one of the 18 of us will have a one last item they MUST have corrected. I personally
did not add some recent publications I had, although I added one publication James asked to add in the previous approval round and corrected one that Sheriff mentioned. Yet I really want to avoid opening the paper to significant changes.</div>
<div dir="auto" class=""><br class="">
</div>
<div dir="auto" class=""> If you start opening corrections beyond what was requested, this will drag on forever since we do have a large list of contributors and each one of us will generate a new idea. So please focus on what was asked unless absolutely necessary. </div>
<div dir="auto" class=""><br class="">
</div>
<div dir="auto" class="">So I suggest not changing the table, and changing only the figure and keeping order intact. I hope I am projecting the sense of urgency - we had the paper lying around for a couple of weeks during the holiday without much progress, I ask not
to escalate things just because a deadline is looming.</div>
<div dir="auto" class=""><br class="">
</div>
<div dir="auto" class="">If you have extra time on your hands in the next couple of days, I suggest you focus on addressing the last two points in the response that we have not resolved yet. I know Gilberto wanted to fix one of them, yet you are welcome to do it. Those
are things requested by the review </div>
<div dir="auto" class=""><br class="">
</div>
<div dir="auto" class="">Hopefully my arguments make sense.</div>
<div dir="auto" class=""><br class="">
</div>
<div dir="auto" class=""> Jacob</div>
</div>
<br class="">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, Jan 4, 2022, 06:45 Alexander Kulesza <<a href="mailto:alexander.kulesza@novadiscovery.com" target="_blank" class="">alexander.kulesza@novadiscovery.com</a>> wrote:<br class="">
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr" class="">Dear all,
<div class=""><br class="">
</div>
<div class="">I have thought about the additional Figure requested by the reviewer. Jacob your Figure nicely shows the assignment of the difficulties we detail in our central table with the 4 big concepts. </div>
<div class=""><br class="">
</div>
<div class="">I feel however that the text-heavy figure is a lot o redundancy with the rest of the elements (existing figure & table). Therefore I wanted to throw another suggestion forward (inspired by systematic reviews narrowing down eligible studies/data). </div>
<div class=""><br class="">
</div>
<div class="">@all please notice that the assignment I would do is a bit different to what is currently done in the paper. The order of sections would have to be altered. Not sure if that is acceptable.<br class="">
</div>
<span id="cid:17e32dbd4e4cb971f161"><image.png></span>
<div class="">The table with Difficulties (hurdles) and solutions would then be </div>
<div class=""><span id="gmail-m_8456291084743863762gmail-m_-8605773720728109558gmail-m_8835093414540775309gmail-m_-7489562775149064076m_-5298176866822363800gmail-docs-internal-guid-f3cb25dd-7fff-3774-33b8-9f03072225ed" class="">
<div dir="ltr" style="margin-left:0pt" align="left" class="">
<table style="border:none;border-collapse:collapse" class="">
<colgroup class=""><col width="172" class=""><col width="172" class=""><col width="280" class=""></colgroup>
<tbody class="">
<tr style="height:0pt" class="">
<td style="border-width:1pt;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:5pt;overflow:hidden" class="">
<br class="">
</td>
<td style="border-width:1pt;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:5pt;overflow:hidden" class=""><div style="line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;" class="">Difficulty</span></div>
</td>
<td style="border-width:1pt;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:5pt;overflow:hidden" class=""><div style="line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;" class="">Potential Solutions</span></div>
</td>
</tr>
<tr style="height:21pt" class="">
<td rowspan="4" style="border-width:1pt;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:5pt;overflow:hidden" class="">
<br class=""><div style="line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; vertical-align: baseline; white-space: pre-wrap;" class="">Reproducibility </span></div>
</td>
<td style="border-width:1pt;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:5pt;overflow:hidden" class=""><div style="line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; vertical-align: baseline; white-space: pre-wrap;" class="">Models are written in different languages</span></div>
</td>
<td style="border-width:1pt;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:5pt;overflow:hidden" class=""><div style="line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; vertical-align: baseline; white-space: pre-wrap;" class="">Common transport specifications such as SBML
or CellML, and proper documentation and annotation</span></div>
</td>
</tr>
<tr style="height:21pt" class="">
<td style="border-width:1pt;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:5pt;overflow:hidden" class=""><div style="line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; vertical-align: baseline; white-space: pre-wrap;" class="">Models are hard to locate </span></div>
</td>
<td style="border-width:1pt;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:5pt;overflow:hidden" class=""><div style="line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; vertical-align: baseline; white-space: pre-wrap;" class="">Archive web sites such as: BioModels, SimTK,
IMAGWiki, and the future modeleXchange</span></div>
</td>
</tr>
<tr style="height:21pt" class="">
<td style="border-width:1pt;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:5pt;overflow:hidden" class=""><div style="line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; vertical-align: baseline; white-space: pre-wrap;" class="">Lack of common platforms for executing models
and simulations</span></div>
</td>
<td style="border-width:1pt;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:5pt;overflow:hidden" class=""><div style="line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; vertical-align: baseline; white-space: pre-wrap;" class="">Platforms such as BioSimulators, and runBioSimulations</span></div>
</td>
</tr>
<tr style="height:21pt" class="">
<td style="border-width:1pt;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:5pt;overflow:hidden" class=""><div style="line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; vertical-align: baseline; white-space: pre-wrap;" class="">Unit standardization</span></div>
</td>
<td style="border-width:1pt;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:5pt;overflow:hidden" class=""><div style="line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; vertical-align: baseline; white-space: pre-wrap;" class="">Standardization efforts, and machine learning
solutions such as <a href="http://ClinicalUnitMapping.com" class="">ClinicalUnitMapping.com</a> </span></div>
</td>
</tr>
<tr style="height:21pt" class="">
<td rowspan="3" style="border-width:1pt;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:5pt;overflow:hidden" class=""><div style="line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; vertical-align: baseline; white-space: pre-wrap;" class="">Credibility</span></div>
</td>
<td style="border-width:1pt;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:5pt;overflow:hidden" class=""><div style="line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; vertical-align: baseline; white-space: pre-wrap;" class="">Models have built-in barriers to Evaluating
model credibility </span></div>
</td>
<td style="border-width:1pt;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:5pt;overflow:hidden" class=""><div style="line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; vertical-align: baseline; white-space: pre-wrap;" class="">Better modeling practices, documentation,
and tests.</span></div>
</td>
</tr>
<tr style="height:21pt" class="">
<td style="border-width:1pt;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:5pt;overflow:hidden" class=""><div style="line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; vertical-align: baseline; white-space: pre-wrap;" class="">Data availability and measurement definitions</span></div>
</td>
<td style="border-width:1pt;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:5pt;overflow:hidden" class=""><div style="line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; vertical-align: baseline; white-space: pre-wrap;" class="">Models that merge human interpretation, and
newer measurement devices</span></div>
<br class="">
</td>
</tr>
<tr style="height:21pt" class="">
<td style="border-width:1pt;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:5pt;overflow:hidden" class=""><div style="line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; vertical-align: baseline; white-space: pre-wrap;" class="">Missing annotations in models</span></div>
</td>
<td style="border-width:1pt;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:5pt;overflow:hidden" class=""><div style="line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; vertical-align: baseline; white-space: pre-wrap;" class="">Adoption of policies such as those COMBINE
suggests</span></div>
</td>
</tr>
<tr style="height:21pt" class="">
<td rowspan="3" style="border-width:1pt;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:5pt;overflow:hidden" class=""><div style="line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; vertical-align: baseline; white-space: pre-wrap;" class="">Utility </span></div>
</td>
<td style="border-width:1pt;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:5pt;overflow:hidden" class=""><div style="line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; vertical-align: baseline; white-space: pre-wrap;" class="">Models are not consistently licensed to allow
for reuse</span></div>
</td>
<td style="border-width:1pt;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:5pt;overflow:hidden" class=""><div style="line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; vertical-align: baseline; white-space: pre-wrap;" class="">Abandoning some old school open source licenses
and promoting licenses that release to public domain </span></div>
</td>
</tr>
<tr style="height:21pt" class="">
<td style="border-width:1pt;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:5pt;overflow:hidden" class=""><div style="line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; vertical-align: baseline; white-space: pre-wrap;" class="">Different scales and modeling paradigms</span></div>
</td>
<td style="border-width:1pt;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:5pt;overflow:hidden" class=""><div style="line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; vertical-align: baseline; white-space: pre-wrap;" class="">Standardization effort and centralization
tools</span></div>
</td>
</tr>
<tr style="height:21pt" class="">
<td style="border-width:1pt;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:5pt;overflow:hidden" class=""><div style="line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; vertical-align: baseline; white-space: pre-wrap;" class="">Stochastic modeling difficulties</span></div>
</td>
<td style="border-width:1pt;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:5pt;overflow:hidden" class=""><div style="line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; vertical-align: baseline; white-space: pre-wrap;" class="">Development of tools that guarantee repeatability
and standards to address stochastic simulations </span></div>
</td>
</tr>
<tr style="height:21pt" class="">
<td rowspan="2" style="border-width:1pt;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:5pt;overflow:hidden" class=""><div style="line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; vertical-align: baseline; white-space: pre-wrap;" class="">Integration </span></div>
</td>
<td style="border-width:1pt;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:5pt;overflow:hidden" class=""><div style="line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; vertical-align: baseline; white-space: pre-wrap;" class="">Model application and implementation barriers</span></div>
</td>
<td style="border-width:1pt;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:5pt;overflow:hidden" class=""><div style="line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; vertical-align: baseline; white-space: pre-wrap;" class="">Education of modelers, users, and the public</span></div>
</td>
</tr>
<tr style="height:21pt" class="">
<td style="border-width:1pt;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:5pt;overflow:hidden" class=""><div style="line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; vertical-align: baseline; white-space: pre-wrap;" class="">Modeling requires adaptation towards integration</span></div>
</td>
<td style="border-width:1pt;border-style:solid;border-color:rgb(0,0,0);vertical-align:top;padding:5pt;overflow:hidden" class=""><div style="line-height: 1.2; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; vertical-align: baseline; white-space: pre-wrap;" class="">Tools for composing models such as SBML-Comp,
and SemGen</span></div>
</td>
</tr>
</tbody>
</table>
<br class="">
</div>
<div style="margin-left:0pt" align="left" class="">Following this assignment.</div>
<div style="margin-left:0pt" align="left" class=""><br class="">
</div>
<div style="margin-left:0pt" align="left" class="">Please let me know what you think. I have added this suggestion in track changes mode to the revision draft. </div>
<div style="margin-left:0pt" align="left" class=""><br class="">
</div>
<div style="margin-left:0pt" align="left" class="">Best </div>
<div style="margin-left:0pt" align="left" class="">Alexander </div>
</span>
<div class=""> </div>
</div>
</div>
<br class="">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, 3 Jan 2022 at 07:27, Jacob Barhak <<a href="mailto:jacob.barhak@gmail.com" rel="noreferrer" target="_blank" class="">jacob.barhak@gmail.com</a>> wrote:<br class="">
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr" class="">Thanks Alex,
<div class=""><br class="">
</div>
<div class="">You did great by adding the new material that the reviewers actually asked for. This is great - I found myself adding the new FDA document to find out that you already added it - I just voice read it ona long drive and figured it fits well - it seems
you were faster - great.</div>
<div class=""><br class="">
</div>
<div class="">I merged your edits and I am ok with adding your response to the reviewers - I only had an issue with one example paragraph that you deleted - I left it with a comment - please check if it is possible to keep this example somewhere in the text. </div>
<div class=""><br class="">
</div>
<div class="">I will assemble a letter that includes all those responses point by point and will try to use your text almost verbatim if possible. You really bring important knowledge with you. </div>
<div class=""><br class="">
</div>
<div class="">It seems most points have been answered to some level - even a figure was created - although I wish I was a better artist. </div>
<div class=""><br class="">
</div>
<div class="">I will try to assemble the response letter for everyone within the next few days. And unless there will be objections I will try to pass it by reviewers with a new version. If the new version is accepted, we will do a final approval round before publication.</div>
<div class=""><br class="">
</div>
<div class="">Many thanks to all those who contributed. </div>
<div class=""><br class="">
</div>
<div class=""> Jacob</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""> </div>
<div class=""><br class="">
</div>
</div>
<br class="">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Sun, Jan 2, 2022 at 7:27 AM Alexander Kulesza <<a href="mailto:alexander.kulesza@novadiscovery.com" rel="noreferrer" target="_blank" class="">alexander.kulesza@novadiscovery.com</a>> wrote:<br class="">
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr" class="">Dear all,
<div class=""><br class="">
</div>
<div class="">Happy new year to everyone. I hope that you are all well and safe and have spent a nice holiday season. </div>
<div class=""><br class="">
</div>
<div class="">I have tried to address </div>
<div class=""><br class="">
</div>
<div class=""><i class=""><font color="#ff0000" class="">Reviewer 2 </font></i></div>
<div class=""><i class=""><font color="#ff0000" class="">Moreover, when the authors mention ASME V&V40 procedure, it could be useful to spend few words about the model credibility pipeline drawn for predictive models in biomedicine, such as the one suggested in "Credibility of in Silico
Trial Technologies-A Theoretical Framing", Viceconti, M., Juarez, M.A., Curreli, C., ...Russo, G., Pappalardo, F., IEEE Journal of Biomedical and Health Informatics, 2020, 24(1), pp. 4–13, 8884189.<br class="">
For this specific aspect, the authors should talk about "specific question of interest" and not "specific purpose" in the paragraph called "Credibility of Models" to follow in such a way a standardised language.<br class="">
</font></i></div>
<div class=""><i class=""><br class="">
</i></div>
<div class="">I have worked on the followng document in "track changes" mode: <br class="">
<a href="https://urldefense.com/v3/__https://docs.google.com/document/d/1IMEgmdNkx-EsnOjGuegpenSIMmKIkK00Lc8Gred3QxM/edit__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0hrtFQ-XI$" rel="noreferrer" target="_blank" class="">https://docs.google.com/document/d/1IMEgmdNkx-EsnOjGuegpenSIMmKIkK00Lc8Gred3QxM/edit</a><br class="">
</div>
<div class="">Actually, I ended up in changing rather drastically the section. I think that all ideas and all text is conserved, but I would very much like to encourage you to read, comment and challenge this revision. Please tell me if that goes to far. No problem
to revert to an earlier version if necessary. </div>
<div class=""><br class="">
</div>
<div class="">I suggest the following formulation in the rebuttal letter: </div>
<div class=""><i class=""><br class="">
</i></div>
<div class="">When trying to better describe the "credibility pipeline" (CoU definition, verification, validation, uncertainty quantification, e.g. as described in Viceconti et al. 2019) or others, we noticed that the section about credibility could be restructured
in order to put the reader (not necessarily familiar with regulatory assessment of models) in the position to follow the argumentation. <br class="">
</div>
<div class=""><br class="">
</div>
<div class="">We now roughly follow the following flow of writing: </div>
<div class=""><br class="">
</div>
<div class=""> - Every model needs a purpose and credibility is tied to that purpose as well as repeatability and reproducibility</div>
<div class="">- Credibility is essential for models that have impact on regulated areas or people's lives which is why regulatory authorities issue guidance</div>
<div class="">- The probably most advanced guidance ASME V&V40 suggests a risk-based approach and pipeline to establish credibility of a model. It is overarching and widely applicable</div>
<div class="">- Regulators and modelers work together (for example in frame of the model informed drug development program MIDD) in order to better understand, better apply and better uptake different kinds of models in regulatory applications</div>
<div class="">- More work is needed to harmonize, stay up to date and to be more inclusive </div>
<div class=""><br class="">
</div>
<div class="">We hope that with the rather extensive change of the wording and additional passages as well as citations (see below) we could address the concern of reviewer 2 (and the general remarks of reviewer 1).</div>
<div class=""><br class="">
</div>
<div class=""><span style="background-color:rgb(255,255,0)" class="">" In 2018 the American Society of Mechanical Engineers (ASME) issued an important guidance ASME V&V 40 (ASME, 2018) of how to assess credibility of computational models of medical devices through verification
and validation (V&V). The guideline is centered around the definition of the context of use (CoU) of the model which is formulated based on the questions of interest the model will answer. The CoU is then analyzed in terms of the “model risk” - being the influence
the model exerts on a decision and the potential consequences these decisions might incur. Commensurate with this model risk the modeler suggests establish the credibility goals, perform verification validation and uncertainty quantification actions and then
assess the outcome of this exercise in order to allow judging the acceptability of the model CoU. Key to this guidance is its overarching nature which also allows adoption in other (e.g. drug development) fields irrespective of the model type (Kuemmel 2020,
Viceconti 2019)." </span><br class="">
</div>
<div class=""><span style="background-color:rgb(255,255,0)" class=""><br class="">
</span></div>
<div class=""><span style="background-color:rgb(255,255,255)" class="">We also feel that it is essential to underline the cross-discipline viewpoint of the ASME V&V40 which is further elaborated by the cited paper by formalizing the verification, validation and uncertainty quantification
VVUQ pipeline across model types. </span></div>
<div class=""><span style="background-color:rgb(255,255,255)" class=""><br class="">
</span></div>
<div class=""><span style="background-color:rgb(255,255,255)" class="">We therefore have added the following statement </span></div>
<div class=""><span style="background-color:rgb(255,255,0)" class="">" In the paper by (Viceconti 2019) the verification, validation and uncertainty quantification (VVUQ) pipeline is streamlined to different types of models. It is, perhaps, the closest to score credibility across
model types from mechanistic physics driven models to machine learning models. However, it is still short of including very recent developments such as ensemble models, although it touches upon the topic." </span></div>
<div class=""><span style="background-color:rgb(255,255,0)" class=""><br class="">
</span></div>
<div class=""><span style="background-color:rgb(255,255,255)" class="">We additionally strengthened our argumentation by citing additional literature </span></div>
<div class=""><span style="background-color:rgb(255,255,0)" class=""><br class="">
</span></div>
<div class=""><span id="gmail-m_8456291084743863762gmail-m_-8605773720728109558gmail-m_8835093414540775309gmail-m_-7489562775149064076m_-5298176866822363800gmail-m_-8284483344536722196gmail-m_5263574797771020238gmail-docs-internal-guid-b84f7d0e-7fff-85c4-e4a6-564288d5358b" class=""><span style="font-size:12pt;font-family:Arial;vertical-align:baseline;white-space:pre-wrap" class="">Bai,
JPF, Earp, JC, Florian, J, et al. </span><span style="font-size:12pt;font-family:Arial;font-style:italic;vertical-align:baseline;white-space:pre-wrap" class="">Quantitative systems pharmacology: Landscape analysis of regulatory submissions to the US Food and Drug
Administration. </span><span style="font-size:12pt;font-family:Arial;vertical-align:baseline;white-space:pre-wrap" class="">CPT Pharmacometrics Syst Pharmacol. 2021; 10: 1479– 1484. <a href="http://doi.org" class="">doi.org</a>:10.1002/psp4.12709</span></span><span style="background-color:rgb(255,255,0)" class=""><br class="">
</span></div>
<div class=""><span style="background-color:rgb(255,255,0)" class=""><br class="">
</span></div>
<div class=""><span id="gmail-m_8456291084743863762gmail-m_-8605773720728109558gmail-m_8835093414540775309gmail-m_-7489562775149064076m_-5298176866822363800gmail-m_-8284483344536722196gmail-m_5263574797771020238gmail-docs-internal-guid-61d5263c-7fff-2486-9431-9ec8fddb8250" class=""><span style="font-size:12pt;font-family:Arial;vertical-align:baseline;white-space:pre-wrap" class="">FDA
(2021b). Assessing the Credibility of Computational Modeling and Simulation in Medical Device Submissions.
</span><span style="font-size:12pt;font-family:Arial;font-style:italic;vertical-align:baseline;white-space:pre-wrap" class="">FDA</span><span style="font-size:12pt;font-family:Arial;vertical-align:baseline;white-space:pre-wrap" class=""><br class="">
</span><span style="font-size:12pt;font-family:Arial;vertical-align:baseline;white-space:pre-wrap" class="">Available at:
</span><a href="https://urldefense.com/v3/__https://www.fda.gov/regulatory-information/search-fda-guidance-documents/assessing-credibility-computational-modeling-and-simulation-medical-device-submissions__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0harP7IHc$" rel="noreferrer" target="_blank" class=""><span style="font-size:12pt;font-family:Arial;vertical-align:baseline;white-space:pre-wrap" class="">https://www.fda.gov/regulatory-information/search-fda-guidance-documents/assessing-credibility-computational-modeling-and-simulation-medical-device-submissions</span></a><span style="font-size:12pt;font-family:Arial;vertical-align:baseline;white-space:pre-wrap" class="">
[Accessed January 2, 2022]</span></span><span style="font-family:Arial;font-size:12pt;white-space:pre-wrap" class=""><br class="">
</span></div>
<div class=""><span class=""><span style="font-size:12pt;font-family:Arial;vertical-align:baseline;white-space:pre-wrap" class=""><br class="">
</span></span></div>
<div class=""><span id="gmail-m_8456291084743863762gmail-m_-8605773720728109558gmail-m_8835093414540775309gmail-m_-7489562775149064076m_-5298176866822363800gmail-m_-8284483344536722196gmail-m_5263574797771020238gmail-docs-internal-guid-363fdda7-7fff-f3a1-7eb6-90b27eb97bf9" class=""><span style="font-size:12pt;font-family:Arial;vertical-align:baseline;white-space:pre-wrap" class="">Galluppi,
G.R., Brar, S., Caro, L., Chen, Y., Frey, N., Grimm, H.P., Rudd, D.J., Li, C.-C., Magee, M., Mukherjee, A., Nagao, L., Purohit, V.S., Roy, A., Salem, A.H., Sinha,
</span><span style="font-size:12pt;font-family:Arial;font-style:italic;vertical-align:baseline;white-space:pre-wrap" class="">V., Suleiman, A.A., Taskar, K.S., Upreti, V.V., Weber, B. and Cook, J. (2021), Industrial Perspective on the Benefits Realized From the
FDA’s Model-Informed Drug Development Paired Meeting Pilot Program</span><span style="font-size:12pt;font-family:Arial;vertical-align:baseline;white-space:pre-wrap" class="">. Clin Pharmacol Ther, 110: 1172-1175. doi:10.1002/cpt.2265</span></span><span style="font-family:Arial;font-size:12pt;white-space:pre-wrap" class=""><br class="">
</span></div>
<div class=""><span class=""><span style="font-size:12pt;font-family:Arial;vertical-align:baseline;white-space:pre-wrap" class=""><br class="">
</span></span></div>
<div class=""><span id="gmail-m_8456291084743863762gmail-m_-8605773720728109558gmail-m_8835093414540775309gmail-m_-7489562775149064076m_-5298176866822363800gmail-m_-8284483344536722196gmail-m_5263574797771020238gmail-docs-internal-guid-9ff33aa8-7fff-740f-e8c1-2cb295c030c8" class=""><span style="font-size:12pt;font-family:Arial;vertical-align:baseline;white-space:pre-wrap" class="">Kuemmel,
C., Yang, Y., Zhang, X., Florian, J., Zhu, H., Tegenge, M., Huang, S.-M., Wang, Y., Morrison, T. and Zineh, I. (2020),
</span><span style="font-size:12pt;font-family:Arial;font-style:italic;vertical-align:baseline;white-space:pre-wrap" class="">Consideration of a Credibility Assessment Framework in Model-Informed Drug Development: Potential Application to Physiologically-Based Pharmacokinetic
Modeling and Simulation.</span><span style="font-size:12pt;font-family:Arial;vertical-align:baseline;white-space:pre-wrap" class=""> CPT Pharmacometrics Syst. Pharmacol., 9: 21-28. <a href="http://doi.org" class="">doi.org</a>:10.1002/psp4.12479</span></span><span class=""><span style="font-size:12pt;font-family:Arial;vertical-align:baseline;white-space:pre-wrap" class=""><br class="">
</span></span></div>
<div class=""><span style="font-family:Arial;font-size:12pt;white-space:pre-wrap" class=""><br class="">
</span></div>
<div class=""><span style="font-family:Arial;font-size:12pt;white-space:pre-wrap" class="">Musuamba, FT, Skottheim Rusten, I, Lesage, R, et al. Scientific and regulatory evaluation of mechanistic in silico drug and disease models in drug development: Building model credibility.
CPT Pharmacometrics Syst Pharmacol. 2021; 10: 804– 825. doi:10.1002/psp4.12669</span><span style="background-color:rgb(255,255,0)" class=""> </span></div>
<div class=""><br class="">
</div>
<div class=""><span id="gmail-m_8456291084743863762gmail-m_-8605773720728109558gmail-m_8835093414540775309gmail-m_-7489562775149064076m_-5298176866822363800gmail-m_-8284483344536722196gmail-m_5263574797771020238gmail-docs-internal-guid-e2949d43-7fff-00c1-09ad-33e73c17438c" class=""><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(33,33,33);vertical-align:baseline;white-space:pre-wrap" class="">Viceconti
M, Emili L, Afshari P, et al. Possible Contexts of Use for In Silico Trials Methodologies: A Consensus-Based Review.
</span><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(33,33,33);font-style:italic;vertical-align:baseline;white-space:pre-wrap" class="">IEEE J Biomed Health Inform</span><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(33,33,33);vertical-align:baseline;white-space:pre-wrap" class="">.
2021;25(10):3977-3982. doi:10.1109/JBHI.2021.3090469</span></span><br class="">
</div>
<div class=""><span class=""><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(33,33,33);vertical-align:baseline;white-space:pre-wrap" class=""><br class="">
</span></span></div>
<div class=""><span class=""><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(33,33,33);vertical-align:baseline;white-space:pre-wrap" class=""><br class="">
</span></span></div>
<div class=""><span class=""><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(33,33,33);vertical-align:baseline;white-space:pre-wrap" class="">All the best
</span></span></div>
<div class=""><span class=""><span style="font-size:12pt;font-family:Roboto,sans-serif;color:rgb(33,33,33);vertical-align:baseline;white-space:pre-wrap" class="">Alexander</span></span></div>
</div>
<br class="">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Sun, 19 Dec 2021 at 03:10, Jacob Barhak <<a href="mailto:jacob.barhak@gmail.com" rel="noreferrer" target="_blank" class="">jacob.barhak@gmail.com</a>> wrote:<br class="">
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="auto" class="">
<div dir="ltr" class=""><br class="">
<div class="">Greetings to the paper contributors,</div>
<div class=""><br class="">
</div>
<div class="">You may have seen my other message where I posted the reviews to our paper. I am sending those again below to start a new thread discussing possible revisions.</div>
<div class=""><br class="">
</div>
<div class="">Here is a checklist for revisions required:</div>
<div class=""><br class="">
</div>
<div class="">
<ol class="">
<li class=""> We should discuss the references that reviewer 1 raised and add them as reference</li><li class="">Reviewer 1 asked for more discussion around V&V40 - I believe Alex added the text there originally - Alex do you think you can address the request of the reviewer?</li><li class="">Reviewer 1 asked to focus on COVID-19 at the title - I am not sure we wish to limit our scope since many of our ideas are applicable far beyond COVID-19 - I leave it up for discussion in the group on how to address this request by the reviewer </li><li class="">Reviewer 1 asked for additional graphics - I am unsure how to address this beyond ur current diagram - ideas will help</li><li class="">We need to check the "Contribution to the field" section for grammar and spelling - if someone can contribute more elegant text - now is the time - I looked over it now and found no issues - yet I may have missed something - this section may have been missed
since it was added last and perhaps vetted less than other sections. Also Reviewer 2 asked for grammar corrections, so it is worthwhile proof reading the paper as a whole.</li><li class="">Reviewer 2 asks for major rewrite to emphasize why the problems are there - I think we should explain that the core of the paper is the table and perhaps emphasize it in the paper beyond what the current text - we also should reply to the reviewer and explain
that the paper is composed of contributions from a large group and we made an effort to include every voice in the choir - each of those voices is important and needs to be preserved - hopefully it will convince the reviewer that change we will add will be
sufficient</li><li class="">Reviewer 2 asks to revise the introduction - I believe some changes are possible - yet the introduction includes contributions from many authors - at least 10 - and I fear losing something important someone contributed - if someone has an idea on how to
address this reviewer without a painful transformation, please reply to this message.</li><li class=""> Reviewer 2 wants additional discussion around the "reproducibility crisis" section and asks a valid question about expectations we should address - "“Computational biomedical modeling… was expected to be less affected by the reproducibility crisis.” Is
this true, and why would it be so? " - My answer to the reviewer is that unlike biological processes that have random nature and experiments would not repeat if repeated, while computer software should be deterministic and it should be repeatable if designed
well. Unfortunately we are not experiencing this promise - yet I believe the reviewer wants more discussion beyond this section so I am happy to discuss this.</li><li class="">Reviewer 2- writes: "Models are Hard to Locate: Are the authors suggesting that entire simulation workflows, from model construction to analysis, should be publicly available? At what point does one consider intellectual property? Do the authors advocate
for such extensive publishing for all models, or only ones that are intended to be widely re-used? - those are important points we need to discuss - we need to better explain the difficulties we are having when creating models and the reviewer is absolutely correct
about expanding the discussion to IP - I suggest we create a thread for this discussion and reference it - I suggest it we merged with the licensing issue that I will address later - there is a strong connection there - the reviewer was very observant and
sees the bigger problem. However,we need to eventually distill our discussion to recommendations that will inline. </li><li class="">Reviewer 2 asks that we fix the unit standardization section - I believe Hana and myself were the largest contributors there - Hana - I will start a discussion on that topic in a separate email where we can publicly discuss how to fix this - others will
be welcome to contribute. </li><li class="">Reviewer 2 asks that we better handle the section on Data availability and measurement definitions: in think we need to emphasize solutions and separate it from issues that may not be solvable. Ideas are welcome.</li><li class="">Reviewer 2 asks good questions with regards to licensing following our text- I personally have good answers to the reviewer and William and I had some discussion on the topic in this list I suggest we expand this discussion in a separate thread - hopefully
William and perhaps others will join the discussion. This discussion should also address the IP issues raised by the reviewer for the "models are hard to locate section."</li><li class="">Reviewer 2 asks to handle the Model Application and Implementation Barriers section. We should decide what to do there, the section may need expansion since the ideas there are solid, yet the section is short so perhaps enhancing it makes better sense.
I am open to suggestions. </li></ol>
<div dir="auto" class=""><br class="">
</div>
<div class="">Those are the items I located and my suggestions. It seems we need attention from Alex, Hana, myself and william. However, anyone on the list is welcome to participate and suggest changes.</div>
<div dir="auto" class=""><br class="">
</div>
<div dir="auto" class="">I will start the discussion threads on specific topics. Hopefully we can get it done quickly.</div>
<div dir="auto" class=""><br class="">
</div>
<div dir="auto" class=""> Jacob</div>
<div dir="auto" class=""><br class="">
</div>
<div dir="auto" class=""><br class="">
</div>
<div class=""><br class="">
</div>
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">- </div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">########### Original Reviews #############</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">There are 2 reviews - both require major changes. I am copying the relevant text below. If more appear, I will let you know, yet I only got this message today although the reviews are dated a few days ago. </div>
<div class=""><br class="">
</div>
<div class="">Reviewer 1: </div>
<div class=""><font face="Lucida Grande, Trebuchet MS, Arial, sans-serif" color="#3e3d40" class=""><span style="font-size:12px;outline-color:initial;outline-style:initial" class="">Recommendation for the Editor: </span></font>Substantial revision is required<br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Please list your revision requests for the authors and provide your detailed comments, including highlighting limitations and strengths of the review. If you have additional comments based on Q2 and Q3 you can add them as well.<br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Karr and co-authors made an interesting and exaustive point about the reproducibility crisis that leads to inability to reuse and integrate models, especially about COVID-19 disease.<br style="outline:0px" class="">
Within the manuscript some typos and missing information are present.<br style="outline:0px" class="">
I'd suggest the authors to revise the entire manuscript especially in terms of the state of the art, revising and updating the most relevant examples of computational models dealing with COVID-19 and in general about some semi-standardised proposals about the
pipeline to follow for the verification and validation of model credibility. In particular, the authors failed to mention and cite some major results on in silico modeling about COVID-19 up to now. See for example:<br style="outline:0px" class="">
a."In silico trial to test COVID-19 candidate vaccines: a case study with UISS platform", Russo, G., Pennisi, M., Fichera, E., ...Viceconti, M., Pappalardo, F., BMC Bioinformatics, 2020, 21, 527.<br style="outline:0px" class="">
b. Russo G, Di Salvatore V, Sgroi G, Parasiliti Palumbo GA, Reche PA, Pappalardo F. "A multi-step and multi-scale bioinformatic protocol to investigate potential SARS-CoV-2 vaccine targets" [published online ahead of print, 2021 Oct 5]. Brief Bioinform. 2021;bbab403.
doi:10.1093/bib/bbab403.<br style="outline:0px" class="">
Moreover, when the authors mention ASME V&V40 procedure, it could be useful to spend few words about the model credibility pipeline drawn for predictive models in biomedicine, such as the one suggested in "Credibility of in Silico Trial Technologies-A Theoretical
Framing", Viceconti, M., Juarez, M.A., Curreli, C., ...Russo, G., Pappalardo, F., IEEE Journal of Biomedical and Health Informatics, 2020, 24(1), pp. 4–13, 8884189.<br style="outline:0px" class="">
For this specific aspect, the authors should talk about "specific question of interest" and not "specific purpose" in the paragraph called "Credibility of Models" to follow in such a way a standardised language.<br style="outline:0px" class="">
Moreover, the authors should refer to COVID-19 also in the title, with a specific mention about the fact that the main topic of model integration in computational biology will be discussed inside the COVID-19 context.<br style="outline:0px" class="">
Furthermore, for the model key concepts such as Reusability-Extensibility-Extractability-Portability the authors should described and outlined through a graphical sketch or visual representation that summarises these key point.<br style="outline:0px" class="">
The authors should also fix some grammar and writing typos present in the "Contribution to the field" section.<br class="">
</div>
<div class=""><br class="">
</div>
<div class="">
<div style="border-top:1px solid rgb(217,217,217);outline:0px" class="">
<div style="padding-top:15px;outline:0px" class="">
<div style="outline:0px" class="">a. Is the quality of the figures and tables satisfactory?<br style="outline:0px" class="">
- No<br style="outline:0px" class="">
<br style="outline:0px" class="">
b. Does the reference list cover the relevant literature adequately and in an unbiased manner?<br style="outline:0px" class="">
- No<br style="outline:0px" class="">
<br style="outline:0px" class="">
c. Does this manuscript refer only to published data? (unpublished or original data is not allowed for this article type)<br style="outline:0px" class="">
- Yes<br style="outline:0px" class="">
<br style="outline:0px" class="">
d. Does the review include a balanced, comprehensive, and critical view of the research area?<br style="outline:0px" class="">
- Yes</div>
</div>
<div id="gmail-m_8456291084743863762gmail-m_-8605773720728109558gmail-m_8835093414540775309gmail-m_-7489562775149064076m_-5298176866822363800gmail-m_-8284483344536722196gmail-m_5263574797771020238gmail-m_-1584538599559411493m_6090386399552853868m_5335031614454254218m_6805043092736084016m_476535936230498237gmail-m_-8345716752565666099gmail-7672" style="outline:0px" class="">
<div style="padding:5px 0px 5px 5px;outline:0px" class="">
<div style="outline:0px" class=""></div>
</div>
</div>
</div>
<div style="height:20px;margin-top:10px;margin-left:20px;outline:0px" class=""><br class="">
</div>
</div>
<div style="height:20px;margin-top:10px;margin-left:20px;outline:0px" class=""><br class="">
</div>
<div style="height:20px;margin-top:10px;margin-left:20px;outline:0px" class="">Reviewer 2</div>
<div style="height:20px;margin-top:10px;margin-left:20px;outline:0px" class=""><br class="">
</div>
<div style="height:20px;margin-top:10px;margin-left:20px;outline:0px" class="">Please list your revision requests for the authors and provide your detailed comments, including highlighting limitations and strengths of the review. If you have additional comments based
on Q2 and Q3 you can add them as well.<br class="">
</div>
<div style="height:20px;margin-top:10px;margin-left:20px;outline:0px" class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">The article often reads as a stream-of-consciousness account of the discussions that took place but lacks a clear thesis or recommendations. It is not clear what, if anything, the authors are advocating for. It is not always clear why the issues being
discussed are problematic, or that they can be reasonably addressed. Some of the issues raised are indeed important and should be discussed, but the paper lacks focus and does not tell a cohesive story. I believe this manuscript requires a major re-write to
be suitable for publication. The authors should consider narrowing the scope of the discussion and focusing on a cohesive set of recommendations or open questions. More specifically, I make a few suggestions below:<br style="outline:0px" class="">
<br style="outline:0px" class="">
Major comments<br style="outline:0px" class="">
1. The introduction is long and repeats itself (e.g., “much less is known about how viral infections spread throughout the body…” is repeated verbatim). It is not clear from the introduction what the main goal of the paper is or why the “reproducibility crisis”
is truly a crisis. Why is the discussion of composition and black/white box models relevant to the introduction? Further, this section is subtitled “the promise of modeling”, which does not seem to match the content.<br style="outline:0px" class="">
2. The Reproducibility Crisis: “Computational biomedical modeling… was expected to be less affected by the reproducibility crisis.” Is this true, and why would it be so? One would think that more complex models would suffer more from a lack of reproducibility.
It may be helpful to define what exactly the “reproducibility crisis” refers to.<br style="outline:0px" class="">
3. Models are Hard to Locate: Are the authors suggesting that entire simulation workflows, from model construction to analysis, should be publicly available? At what point does one consider intellectual property? Do the authors advocate for such extensive publishing
for all models, or only ones that are intended to be widely re-used?<br style="outline:0px" class="">
4. Unit standardization: The conversion from PFU or TCID50 to individual virions is likely to differ across viruses – are the authors focused on COVID here? Are the authors advocating for a standard conversion factor? It is not clear what the purpose of this
discussion is. As the authors mention, different scales require different units. Even at a single scale, different models may require different units for numerical reasons. It is not clear what the authors are advocating for here.<br style="outline:0px" class="">
5. Data availability and measurement definitions: This section seems to outline limitations of available data, but again makes no recommendations or proposed solution to any of the issues raised. Is this the intention? Most of the issues raised here reflect
limitations of experimental science or data privacy, which likely cannot be meaningfully addressed by the modeling community.<br style="outline:0px" class="">
6. Models are Not Consistently Licensed…: Are the authors implying here that all modeling work should be published with no rights reserved? Is it reasonable to expect modelers to make their work freely usable by others for profit? Is it reasonable for institutions
to allow this? How much does this really contribute to reproducibility and utility?<br style="outline:0px" class="">
7. Model Application and Implementation Barriers: This section seems unnecessary and out of place.<br style="outline:0px" class="">
8. There are grammar and punctuation errors scattered throughout; please edit carefully.<br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">a. Is the quality of the figures and tables satisfactory?<br style="outline:0px" class="">
- Not Applicable<br style="outline:0px" class="">
<br style="outline:0px" class="">
b. Does the reference list cover the relevant literature adequately and in an unbiased manner?<br style="outline:0px" class="">
- Yes<br style="outline:0px" class="">
<br style="outline:0px" class="">
c. Does this manuscript refer only to published data? (unpublished or original data is not allowed for this article type)<br style="outline:0px" class="">
- Yes<br style="outline:0px" class="">
<br style="outline:0px" class="">
d. Does the review include a balanced, comprehensive, and critical view of the research area?<br style="outline:0px" class="">
No answer given.<br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">========</div>
</div>
</div>
</div>
_______________________________________________<br class="">
Vp-reproduce-subgroup mailing list<br class="">
<a href="mailto:Vp-reproduce-subgroup@lists.simtk.org" rel="noreferrer" target="_blank" class="">Vp-reproduce-subgroup@lists.simtk.org</a><br class="">
<a href="https://urldefense.com/v3/__https://lists.simtk.org/mailman/listinfo/vp-reproduce-subgroup__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0hhiovw3M$" rel="noreferrer noreferrer" target="_blank" class="">https://lists.simtk.org/mailman/listinfo/vp-reproduce-subgroup</a><br class="">
</blockquote>
</div>
<br clear="all" class="">
<div class=""><br class="">
</div>
-- <br class="">
<div dir="ltr" class="">
<div dir="ltr" class="">
<div style="font-family:sans-serif;font-size:13px;white-space:pre-wrap;padding:2px 0px 2px 13px;color:rgb(54,57,79)" class="">
<div style="line-height:normal;white-space:normal;padding:2px 0px;color:rgb(35,35,73)" class="">
Alexander Kulesza</div>
<div style="font-size:11px;line-height:normal;white-space:normal;padding:2px 0px;color:rgb(52,70,115)" class="">
Team leader</div>
<div style="font-size:11px;line-height:normal;white-space:normal;padding:2px 0px;color:rgb(82,135,189)" class="">
Modeling & simulation / Biomodeling</div>
<div style="font-size:11px;line-height:normal;white-space:normal;padding:2px 0px;color:rgb(128,134,150)" class="">
<a href="mailto:alexander.kulesza@novadiscovery.com" rel="noreferrer" style="color:rgb(128,134,150)" target="_blank" class="">alexander.kulesza@novadiscovery.com</a></div>
<div style="font-size:11px;line-height:normal;white-space:normal;padding:2px 0px;color:rgb(128,134,150)" class="">
+33 7 82 92 44 62</div>
<div style="white-space:normal;width:100px;margin:15px 0px;font-size:28px;color:rgb(35,35,73)" class="">
<div style="font-family:Montserrat,arial;font-weight:600;letter-spacing:-1px" class="">nova</div>
<div style="font-size:9px" class="">DISCOVERY</div>
</div>
<div style="color:rgb(51,51,51);font-size:11px;line-height:normal;white-space:normal;padding:2px 0px;margin:0px" class="">
<a href="https://urldefense.com/v3/__http://www.novadiscovery.com/__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0hwI_2O8g$" title="Visit our website" rel="noreferrer" style="color:rgb(128,134,150)" target="_blank" class="">www.novadiscovery.com</a></div>
<div style="font-size:11px;line-height:normal;white-space:normal;padding:2px 0px;color:rgb(128,134,150)" class="">
1 Place Verrazzano, 69009 Lyon<br class="">
+33 9 72 53 13 01</div>
</div>
</div>
</div>
<br class="">
<i style="color:rgb(187,187,187);font-family:sans-serif;font-size:10px;text-align:justify" class="">This message is intended only for the personal and confidential use of the designated recipient(s) named above. If you are not the intended recipient of this message
you are hereby notified that any review, dissemination, distribution or copying of this message is strictly prohibited. Email transmission cannot be guaranteed to be secure or error-free. Therefore, we do not represent that this information is complete or
accurate and it should not be relied upon as such. All information is subject to change without notice.</i></blockquote>
</div>
</blockquote>
</div>
<br clear="all" class="">
<div class=""><br class="">
</div>
-- <br class="">
<div dir="ltr" class="">
<div dir="ltr" class="">
<div style="font-family:sans-serif;font-size:13px;white-space:pre-wrap;padding:2px 0px 2px 13px;color:rgb(54,57,79)" class="">
<div style="line-height:normal;white-space:normal;padding:2px 0px;color:rgb(35,35,73)" class="">
Alexander Kulesza</div>
<div style="font-size:11px;line-height:normal;white-space:normal;padding:2px 0px;color:rgb(52,70,115)" class="">
Team leader</div>
<div style="font-size:11px;line-height:normal;white-space:normal;padding:2px 0px;color:rgb(82,135,189)" class="">
Modeling & simulation / Biomodeling</div>
<div style="font-size:11px;line-height:normal;white-space:normal;padding:2px 0px;color:rgb(128,134,150)" class="">
<a href="mailto:alexander.kulesza@novadiscovery.com" rel="noreferrer" style="color:rgb(128,134,150)" target="_blank" class="">alexander.kulesza@novadiscovery.com</a></div>
<div style="font-size:11px;line-height:normal;white-space:normal;padding:2px 0px;color:rgb(128,134,150)" class="">
+33 7 82 92 44 62</div>
<div style="white-space:normal;width:100px;margin:15px 0px;font-size:28px;color:rgb(35,35,73)" class="">
<div style="font-family:Montserrat,arial;font-weight:600;letter-spacing:-1px" class="">nova</div>
<div style="font-size:9px" class="">DISCOVERY</div>
</div>
<div style="color:rgb(51,51,51);font-size:11px;line-height:normal;white-space:normal;padding:2px 0px;margin:0px" class="">
<a href="https://urldefense.com/v3/__http://www.novadiscovery.com/__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0hwI_2O8g$" title="Visit our website" rel="noreferrer" style="color:rgb(128,134,150)" target="_blank" class="">www.novadiscovery.com</a></div>
<div style="font-size:11px;line-height:normal;white-space:normal;padding:2px 0px;color:rgb(128,134,150)" class="">
1 Place Verrazzano, 69009 Lyon<br class="">
+33 9 72 53 13 01</div>
</div>
</div>
</div>
<br class="">
<i style="color:rgb(187,187,187);font-family:sans-serif;font-size:10px;text-align:justify" class="">This message is intended only for the personal and confidential use of the designated recipient(s) named above. If you are not the intended recipient of this message
you are hereby notified that any review, dissemination, distribution or copying of this message is strictly prohibited. Email transmission cannot be guaranteed to be secure or error-free. Therefore, we do not represent that this information is complete or
accurate and it should not be relied upon as such. All information is subject to change without notice.</i></blockquote>
</div>
</blockquote>
</div>
<br clear="all" class="">
<div class=""><br class="">
</div>
-- <br class="">
<div dir="ltr" class="">
<div dir="ltr" class="">
<div style="font-family:sans-serif;font-size:13px;white-space:pre-wrap;padding:2px 0px 2px 13px;color:rgb(54,57,79)" class="">
<div style="line-height:normal;white-space:normal;padding:2px 0px;color:rgb(35,35,73)" class="">
Alexander Kulesza</div>
<div style="font-size:11px;line-height:normal;white-space:normal;padding:2px 0px;color:rgb(52,70,115)" class="">
Team leader</div>
<div style="font-size:11px;line-height:normal;white-space:normal;padding:2px 0px;color:rgb(82,135,189)" class="">
Modeling & simulation / Biomodeling</div>
<div style="font-size:11px;line-height:normal;white-space:normal;padding:2px 0px;color:rgb(128,134,150)" class="">
<a href="mailto:alexander.kulesza@novadiscovery.com" style="color:rgb(128,134,150)" target="_blank" class="">alexander.kulesza@novadiscovery.com</a></div>
<div style="font-size:11px;line-height:normal;white-space:normal;padding:2px 0px;color:rgb(128,134,150)" class="">
+33 7 82 92 44 62</div>
<div style="white-space:normal;width:100px;margin:15px 0px;font-size:28px;color:rgb(35,35,73)" class="">
<div style="font-family:Montserrat,arial;font-weight:600;letter-spacing:-1px" class="">nova</div>
<div style="font-size:9px" class="">DISCOVERY</div>
</div>
<div style="color:rgb(51,51,51);font-size:11px;line-height:normal;white-space:normal;padding:2px 0px;margin:0px" class="">
<a href="https://urldefense.com/v3/__http://www.novadiscovery.com/__;!!K6Z8K8YTIA!Qmy11SwUfgLnvEtA_Yy8DDA-OCSuTnTFPcnArqqv3zx410koRSJz-doddC0hwI_2O8g$" title="Visit our website" style="color:rgb(128,134,150)" target="_blank" class="">www.novadiscovery.com</a></div>
<div style="font-size:11px;line-height:normal;white-space:normal;padding:2px 0px;color:rgb(128,134,150)" class="">
1 Place Verrazzano, 69009 Lyon<br class="">
+33 9 72 53 13 01</div>
</div>
</div>
</div>
<br class="">
<i style="color:rgb(187,187,187);font-family:sans-serif;font-size:10px;text-align:justify" class="">This message is intended only for the personal and confidential use of the designated recipient(s) named above. If you are not the intended recipient of this message
you are hereby notified that any review, dissemination, distribution or copying of this message is strictly prohibited. Email transmission cannot be guaranteed to be secure or error-free. Therefore, we do not represent that this information is complete or
accurate and it should not be relied upon as such. All information is subject to change without notice.</i></blockquote>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote></div>
_______________________________________________<br class="">Vp-integration-subgroup mailing list<br class=""><a href="mailto:Vp-integration-subgroup@lists.simtk.org" class="">Vp-integration-subgroup@lists.simtk.org</a><br class="">https://lists.simtk.org/mailman/listinfo/vp-integration-subgroup<br class=""></div></blockquote></div><br class=""></div></body></html>