<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div><br></div><div>Monday morning thoughts:</div><div><br></div><div>Nobody, modeler, peer reviewer, editor, software club, government agency, Government, NOBODY but ME can accredit ANY model for what I’M going to DO with it or its output. Current language implies that, for example NASA or FDA, because they have guidance and are working on how to do it for themselves (EACH of their uses) can accredit a model for MY use/ purpose. An increased risk is presented if anyone believes that because FDA accredited it for a piece of a clinical trial for a device for a purpose, that it can be used again without REACCREDITING for the new use. Of course, if both the model and the first accreditation process and decisions are VERY WELL documented, the reuse accreditation MAY be pretty easy. Repeated reaccreditations whether the decisions are to use or NOT use it build a sort of pedigree (?), which may make some models much easier to reaccredit for many uses. </div><div><br></div><div>Therefor in the paper, consider a global search and replace: “accreditation for each use” in place of accreditation. (Then change back the few where that does not fit the sentence.)<br><div><br></div><div>What has to change is that the burden for accreditation for use (AfU) is 99% on the USER. The burden on the modeler is to DOCUMENT as you work (std practice for any reliable software development), wright and when asked questions speak the truth, the whole truth, and nothing but the truth about the MODEL 1st (validation), then the math (verification), then the code (repeatability); after which it is on the potential user to ACCREDIT it for there USE.</div><div><br></div><div>Stop making it sound like somehow models EASILY make things easy. </div><div><br></div><div>User accountability may be a paradigm shift despite a few people having tried to make that point for 40 years. </div><div><br></div><div>That said there may be some First Principle in some sciences that do not require reinvestigation as a practical matter. </div><div><br></div><div>Companies know. That’s why they don’t collaborate well together. Their models are understood well enough that they can reaccredit quickly so those are proprietary high value easily reusable compete easily cost saving treasures. </div><div><br></div><div>Libraries of “accredited models” are another dangerous idea as most people seem to think that they would work.</div><div> <br><br><div dir="ltr"><div><div> </div></div></div><div dir="ltr"><br>On May 17, 2021, at 06:33, Jacob Barhak <jacob.barhak@gmail.com> wrote:<br><br></div><div dir="ltr"><div dir="auto">Well William, <div dir="auto"><br></div><div dir="auto"><div dir="auto">Jonathan and you criticize some work done. However, in a larger perspective, let us remember that although disease models existed for about a century, this is still emerging technology. </div><div dir="auto"><br></div><div dir="auto">I have been working in the field for over a decade and I am a great critic of our current state and believe we can do better. </div><div dir="auto"><br></div><div dir="auto">Our technologies are still not good enough for prediction. We really cannot predict, moreover we still cannot fully explain the phenomena we see computationally. So every technique will have difficulties in forecasting. </div><div dir="auto"><br></div><div dir="auto">And when I write we, I also include myself. I really wish we can do better. </div><div dir="auto"><br></div><div dir="auto">This does not mean people should stop trying. So instead of trying to dismiss other methods, perhaps we should try to suggest how to learn from mistakes and improve things for the future. </div><div dir="auto"><br></div><div dir="auto">We have the collective responsibility of preparing tools for the future. And I think a positive tone in our message of what to do to improve things will do better than pointing fingers in this situation. </div><div dir="auto"><br></div><div dir="auto">I think we were able to do it ok in the paper so far, for each deficiency we were able to show a potential solution. Perhaps we should keep that concept. </div><div dir="auto"><br></div><div dir="auto">The credibility section in the paper attempts to address the issue you discuss in a subtle way. It hints that we should do better as modelers so that regulators will trust us. I think it serves the purpose you are both aiming for. Yet if you think a stronger message is needed, the paper is now open for revisions and for discussions and you are welcome to make those. </div><div dir="auto"><br></div><div dir="auto"> Jacob</div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, May 17, 2021, 03:13 William Waites <<a href="mailto:wwaites@ieee.org">wwaites@ieee.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">> Also, to motivate the focus on credibility, it could be helpful to cite an instance where the pandemic models were wrong or where lack of credibility (trust) inhibited use of a model. For the former, one example that comes to mind is the widely cited IHME model which was substantially off in the Spring. <br>
<br>
Another good example is Friston’s dynamic causal model which is interesting in itself and apparently a useful technique in neuroscience but does badly for infectious disease, famously leading to the assertion that the reason the model results’ divergence from reality must be do to mysterious “epidemiological dark matter”. The debunking of this sucked up a lot of time from people who know better…<br>
<br>
<br>
_______________________________________________<br>
Vp-reproduce-subgroup mailing list<br>
<a href="mailto:Vp-reproduce-subgroup@lists.simtk.org" target="_blank" rel="noreferrer">Vp-reproduce-subgroup@lists.simtk.org</a><br>
<a href="https://lists.simtk.org/mailman/listinfo/vp-reproduce-subgroup" rel="noreferrer noreferrer" target="_blank">https://lists.simtk.org/mailman/listinfo/vp-reproduce-subgroup</a><br>
</blockquote></div>
<span>_______________________________________________</span><br><span>Vp-reproduce-subgroup mailing list</span><br><span>Vp-reproduce-subgroup@lists.simtk.org</span><br><span>https://lists.simtk.org/mailman/listinfo/vp-reproduce-subgroup</span><br></div></div></div></body></html>