[Vp-integration-subgroup] [EXT] [Vp-reproduce-subgroup] Credibility “for EACH USE”Re: White paper revision

[John Rice]

So talking to device industry guy who says there is software available and being improved to capture a lot of information as models are being developed.  He is sending some links. 

But why is writing good computational models any less that writing any other MATURE software.  Maybe the compbio mathematicians are NOT SOFTWARE people at all. ???

If they were trained as “software” engineers to use good general software development methods and processes where there are lots of “Stds” for good process practices, they would be way ahead when it comes to developing model unique practices built on good software engineering practices.

Once asked in a very large meeting:
How many people worked in companies that developed models for company use. (Marketing, finance, Qa, product design.
(quite a few yes hands).

Then asked how many work in companies that developed software on contract to Government.
(Many since it was a defense conference).

Then asked if they developed the gov software using all the same Stds of practice that they explicitly or tacitly used to develop their own software.  
Long silence.  

The Carnegie Mellon Software Capability Maturity Model was hailed as a huge step in preventing extramarital high cost of faulty software.  DOD began to require CMM independent accreditation of bidders. Price went up.  Still are paying cost of bad software. 

But again, companies can’t afford bad software, their own or any they buy.  Gov can’t afford good software from operating budget or research funds.  
Dollars ARE the life blood of research.  So why do we keep using blood that has on platelets?   

John



Typed with two thumbs on my iPhone.  (757) 318-0671

“Upon this gifted age, in its dark hour,
Rains from the sky a meteoric shower
Of facts . . . they lie unquestioned, uncombined.
Wisdom enough to leech us of our ill
Is daily spun; but there exists no loom
To weave it into fabric.”

–Edna St. Vincent Millay,

 

On May 17, 2021, at 10:58, Morse, Katherine L. <Katherine.Morse at jhuapl.edu> wrote:

The defense simulation community has been struggling with this problem (like so many others) for decades. We haven't solved it, but we've made some progress. Further progress would require political will and funding that isn't forthcoming. The approach includes metadata about the underlying models such as William identified below as well as interfaces and types and formats of required data. That metadata gets stored in a searchable catalog. Using the catalog and the metadata, a potential user can rapidly do a first order culling of models / simulations that are clearly no applicable. This doesn't rise to the level of solution, but it's better than guessing or sending out mass emails.

BTW, one of the things you could put in the metadata is links to verification, validation, and accreditation (VV&A) reports. Despite the provable impossibility of asserting the validity of the composition, it can get a potential user a couple of steps closer to finding potentially interoperable components.

KLM
---
Katherine L. Morse, PhD
IEEE Fellow
Principal Professional Staff, JHU/APL
11100 Johns Hopkins Road
Laurel, MD  20723-6099
(240)917-9602 (w)
(858)775-8651 (m) 


On 5/17/21, 7:11 AM, "William Waites" <wwaites at ieee.org> wrote:

   APL external email warning: Verify sender wwaites at ieee.org before clicking links or attachments 

> I might have to do some digging to find it, but Dr. Mikel Petty* (UAH) did a proof that you cannot make any assertions about the validity of the composition of two valid models.

   In general, that is very likely true. I’d like to see the proof!

   This gets into the weeds a bit, but in some specific cases we can say some things. Two examples.

   1. If we have a system of differential equations of the form F’(x) = M(x) = A(x) + B(x). Call the two models A and B. We can make propagators φA and φB separately and show that φM(x) = φA(φB(x)) + O(h) where h is a chosen time-step. (A propagator in this context is a function that takes the state of the system at time t and gives you the state at time t+h.) This is a simple result that is quite specific about the properties of the composition and its error. This is a building block that lets you construct integration schemes from the individual propagators that can be cheaper and more stable than the one you get for the big propagator directly. This is used, for example, in calculating trajectories of particles around synchrotrons where there is one model for the behaviour far from the magnets and one for the behaviour near to the magnets.

   2. Models formulated as Petri nets or generalisations like graph rewriting systems can be freely composed to get well-defined behaviour. Whether this is the behaviour that you want is a slightly different question and some care is required, but it is well-defined. This is why we can study systems of chemical reactions for cascades. There is also a connection to differential equations because there is also a completely mechanical way to get the equations for the moments of observables of such systems (which coincide with the standard ODE formulation of many of the SBML models in the limit of large systems). This also applies to some kinds of electronics design and is why we can build circuits out of linear components and they behave as we thing they should. It also works for some of the kinds of models we like to make for infectious diseases.

   Note that both examples require pretty strong conditions on the form of the models to be able to say anything useful about how they compose. It seems likely that there are other kinds of models where we can say useful things. All is not lost, there is some hope!

   I am completely willing to believe, however, that we cannot say much more for general models other than “the interfaces match”, particularly for heterogeneous bunches of models.

   Cheers,
   -w