[Vp-integration-subgroup] About using a multi-scale mortality model in the ensemble
Jacob Barhak
jacob.barhak at gmail.com
Sun Jan 3 16:54:50 PST 2021
Thanks Fil,
You will find the code available under CC0:
https://github.com/Jacob-Barhak/COVID19Models/tree/main/COVID19_Mortality_Castiglione
This essentially makes it public domain so people can use it with less
restrictions.
As for integration - I already integrated your model into the ensemble and
executed a small simulation that checks that it works. The results are
preliminary and showed that your model has similar influence to the
previous models combined. However, those results should not be trusted
since the simulation was minimal and was not executed sufficient times to
produce enough mortality for the equations to take effect. To get such
results, there is a need for a larger cloud simulation. I am working on
such a simulation now and will update you when I know more.
I was able to find a way to integrate the equations without resorting to
random techniques like you suggested - I used my patent pending technology
to merge the different types of models - yet I thank you for sending the
link - it may come useful in the future and I think it will be helpful for
others reading this conversation.
I just want to point out that our public conversation regarding integrating
your equation revealed many issues that are relevant to several subgroups
of the MSM viral pandemic working group:
- how to define a comorbidity and to interpret a continuous model towards a
medical condition - relevant to the comorbidities
- comprehension of model purpose/use - relevant to integration
- licenses - relevant to integration / reproducibility
- units of measure - relevant to integration / reproducibility /
standardization
- adaptation of model - relevant to integration
- how to generate event probabilities from models - relevant to
dissemination and integration.
The fact that you agreed to public and transparent communications will help
others learn of difficulties and best practices around model reuse. I truly
appreciate your willingness to be open and allow this demonstration and
sharing of knowledge.
Jacob
On Sun, Jan 3, 2021 at 6:35 AM Filippo Castiglione <
filippo.castiglione at gmail.com> wrote:
> Hi Jacob. Happy New Year!
>
> > Since this function we discussed is a density function, the probability
> of mortality on a certain day should be 3 times larger as well. I corrected
> the Github repository to reflect this.
>
> That’s correct! thanks for noticing it.
>
> As for the license question it is ok with me if you put it on CC0 license.
> Please go ahead.
>
> Concerning the issue of how to plug the death model into your ensemble
> model it is not yet too clear to me how I can be of help. I am not sure if
> I understood correctly but if the question is how to generate an event with
> probability given by a prob.distr.function maybe the answer is to use the
> Rejection Method explained (for example) in
>
> https://web.mit.edu/urban_or_book/www/book/chapter7/7.1.3.html
>
> However in this case please note that the integral of the function I gave
> you between 0-30 days doesn’t sum up to one. In fact it sums up (approx.)
> to the value of Panel A of Fig. 3.
>
> Thanks and best wishes
> Fil
>
>
> >
> https://github.com/Jacob-Barhak/COVID19Models/tree/main/COVID19_Mortality_Castiglione
> >
> > Please verify this is correct and the numbers the plots are showing make
> sense.
> >
> > Meanwhile I wish you a happy new year.
> >
> > Jacob
> >
> >
> > On Thu, Dec 31, 2020 at 3:27 AM Jacob Barhak <jacob.barhak at gmail.com>
> wrote:
> > Ok Fil,
> >
> > Your corrected model code is now on Github at:
> >
> https://github.com/Jacob-Barhak/COVID19Models/tree/main/COVID19_Mortality_Castiglione
> >
> > I chose MIT license for now, just to make sure you are comfortable with
> this initially. I would have liked to put it under CC0 license so that it
> can be more easily reused - however, I cannot really do it without you
> understanding the consequences - after all it means you giving away your IP
> - think about it as a gift to the world - it will make any type of reuse
> much easier for various purposes. In fact I am not the only one
> recommending this license, yet I cannot make the decision unless the
> creators approve.
> >
> > Here is a link to the license language:
> > https://creativecommons.org/choose/zero/
> >
> > If after you read the language you approve the change, I will change the
> license to CC0.
> >
> > Yet for now I am more interested in how to plug your model into the
> ensemble.
> >
> > The Reference Model was recently using a certain approach to calculate
> mortality - it had the probability of mortality extracted as a function of
> age from 3 models without deciding on time and the time of mortality
> extracted from merging 2 models. The model would know ahead of time when
> the individual might die according to the mortality time model and then
> test a random number against this threshold mortality probability at that
> time during simulation for infected individuals. I did not have a density
> function I can build upon like the one you provided. Basically what I had
> is a spike function for each individual so using your function will make
> the ensemble much richer since it replaces a combination of multiple model
> components, yet it requires a different approach to modeling mortality and
> recovery since I need to merge the spike functions with your smoother
> distribution somehow. I will need to think of the best way to do this and
> it may take a few days.
> >
> > Since I need now to also remodel recovery, it bring us back to the issue
> you raised of not having a defined state and instead having a continuous
> function with a cutoff required.
> >
> > Also, recall that in an ensemble there are multiple other models that
> influence the result - the system will pick a mixture of models that work
> together best to explain the phenomenon observed. So it is more about the
> contribution of each model and sometimes it is hard to predict which models
> will work well together. Yet I think that having more models will help us
> better understand things, so your model will for sure contribute to
> understanding once I figure out how to merge it.
> >
> > Hopefully I can get it done soon. Until then thank you for your fast
> responses.
> >
> > Jacob
> >
> >
> >
> >
> > On Wed, Dec 30, 2020 at 2:33 AM Filippo Castiglione <
> filippo.castiglione at gmail.com> wrote:
> > Hi Jacob, it looks all good to me but the definition of time (which I
> forgot to tell you before, sorry ;) that in the simulator (therefore in the
> data I sent you and therefore in the mortality model) is 8 hours: it just
> means that you have to divide time by 3 to get days. So the period is 30
> days, not 90 days. Apart from that, it all looks great. Thanks for the
> copyright. Feel free to put it on you git with the license you see fit.
> >
> > I am curious to see how it works in your ensamble model.
> >
> > Best wishes
> > Fil
> >
> > > On 30 Dec 2020, at 08:47, Jacob Barhak <jacob.barhak at gmail.com> wrote:
> > >
> > > Thanks Fil,
> > >
> > > You helped a lot. However, to make sure we are on the same page, I
> decided to implement the function using python and show you the plot - just
> to make sure I implemented your instructions correctly - I rather show you
> my understanding so you can tell me if this looks reasonable to you.
> > >
> > > Attached is the link to an archive that contains the code and the
> output.
> > >
> https://drive.google.com/file/d/12AU3xsjUk8xVBbb7UmO4hE-ldGj_QJ08/view?usp=sharing
> > >
> > > Once you open the archive you should see the interactive plot inside
> as an html file - you should be able to open it with a modern browser like
> chrome, edge, or firefox
> > >
> > > As for the code I assigned copyright to you and added some
> documentation. If you have someone in your team familiar with python , you
> just need to install holoviews and panel to create the plots. I did not put
> the code in Github - I leave this to you - I am just interested in using
> the function in my ensemble and need verification that this is correct. Yet
> if you want, I will be happy to upload it to my github account if we decide
> on a proper license like MIT / BSD or even preferred CC0 to release this to
> the public without copyright restrictions.
> > >
> > > For now, please just verify this is correct - I noticed peak
> mortality increases by age from 50 days to 61 days - is this correct or did
> I misunderstand something?
> > >
> > > Hopefully this is correct and I can continue to plug in the model into
> the ensemble.
> > >
> > > Jacob
> > >
> > >
> > >
> > >
> > > On Sun, Dec 27, 2020 at 11:13 PM Filippo Castiglione <
> filippo.castiglione at gmail.com> wrote:
> > > Jacob,
> > >
> > > I forgot to say that after you have calculated the function F’, which
> is the probability-density-function of the event “death”, you have to
> multiply to an age-dependent coefficient which can be found in fig.3 panel
> A of my paper divided by 100 (because there it is expressed in percent).
> > >
> > > In other words for instance the probability to die at day x with
> respect to the age-class 40-49 is
> > >
> > > 1.25/100 * F’(x,a,b,c) with
> > >
> > > a=95.351153808322
> > > b=0.173066025988826
> > > c=51.92498287875
> > >
> > > Sorry for the lack of precision.
> > > I hope it is clearer now.
> > > Fil
> > >
> > >
> > >
> > > > On 27 Dec 2020, at 15:51, Filippo Castiglione <
> filippo.castiglione at gmail.com> wrote:
> > > >
> > > > Hi Jacob, let me reply below
> > > >
> > > >> I was thinking of a function that defines the probability of
> mortality as a function of time since infection, with the good chance of
> surviving. This function should look like figure 2 in your paper where
> higher age individuals have higher chance of dying, yet after a certain
> point in time those who survived no longer can die, indicating recovery.
> > > >>
> > > >> It seems that the function you suggested is a little different. If
> I understand you correctly, you already stratified the group that will die
> and you are trying to show the probability of an individual dying in a
> certain day from infection knowing they will die.
> > > >
> > > > Ok (in my last reply I was incorrect), then the function you are
> looking for is the derivative of the one I sent you (normalised dividing by
> 100), namely,
> > > >
> > > > F(x,a,b,c) = S(x,a,b,c) / 100
> > > >
> > > > and its derivative is
> > > >
> > > > F’(x,a,b,c) = (a * b exp(-b * (-c + x))) / (100 (1 + exp(-b *
> (-c + x)))^2)
> > > >
> > > > Now the values of the parameters a,b,c are those I sent you last
> time IF
> > > > we consider the age-UNstratified population.
> > > >
> > > > Otherwise, if we go by age (age ranges as in the paper), then the
> parameters are:
> > > >
> > > > age range 0-39
> > > > a=96.56299632152
> > > > b=0.185777403544838
> > > > c=49.8164210315686
> > > >
> > > > age range 40-49
> > > > a=95.351153808322
> > > > b=0.173066025988826
> > > > c=51.92498287875
> > > >
> > > > age range 50-59
> > > > a=94.2976479245251
> > > > b=0.165018635066338
> > > > c=54.1494851021362
> > > >
> > > > age range 60-69
> > > > a=91.3990353489634
> > > > b=0.158299883031816
> > > > c=55.5935133562513
> > > >
> > > > age range 70-79
> > > > a=88.9048923236574
> > > > b=0.152924609509616
> > > > c=57.7447222979953
> > > >
> > > > age range 80+
> > > > a=85.6674967019556
> > > > b=0.14040123117131
> > > > c=61.383806850923
> > > >
> > > >> Please verify that I am correct. Since I need to know how to use
> your function.
> > > >
> > > > Let me know if F’ (F-prime) is the function you were looking for. It
> should be it.
> > > >
> > > >> You have two more interesting elements in the paper that can start
> interesting discussions:
> > > >>
> > > >> 1. You approach the definition of a comorbidity by a level of
> damage to tissue.
> > > >
> > > > Yes, given the elements in my simulation, that is a possible choice
> for the definition (not really of the comorbidity) of the “severity” of the
> disease. It is a tricky point to link some computer output to a “clinical
> endpoint”, that is, something used by clinician to assess the disease
> status. But a very important point.
> > > >
> > > > All the best
> > > > Fil
> > > >
> > > >> 2. You end the paper by discussing what needs to be done to make
> models used in medical settings.
> > > >> Those topics deserve discussion in separate email threads. Yet for
> now I wish to make sure I understood your model correctly so I can
> implement it properly.
> > > >>
> > > >> Happy holidays
> > > >>
> > > >>
> > > >> Jacob
> > > >>
> > > >>
> > > >> On Tue, Dec 22, 2020, 13:38 Filippo Castiglione <
> filippo.castiglione at gmail.com> wrote:
> > > >> Dear Jacob, the preprint is out. You can have a look at it at
> > > >>
> > > >> https://biorxiv.org/cgi/content/short/2020.12.20.423670v1
> > > >>
> > > >> However you might not need to read it to figure out your
> probability function.
> > > >> In fact, the curve I sent you last time is
> > > >>
> > > >> f(t) = 1-P(t)
> > > >>
> > > >> where P(t) is the probability of dying at time t. To be precise you
> have to take
> > > >>
> > > >> P(t) = 1- f(t)/100
> > > >>
> > > >> because f(t) is expressed in percent.
> > > >>
> > > >> it is of course a rough approximation with rough assumptions (e.g.,
> the choice of the cutoff theta I was mentioning in my last mail) but might
> be a starting point (!?)
> > > >>
> > > >> Hope it helps.
> > > >> All the best
> > > >> Fil
> > > >>
> > > >>
> > > >>> On 22 Dec 2020, at 10:42, Jacob Barhak <jacob.barhak at gmail.com>
> wrote:
> > > >>>
> > > >>> Thanks Filippo,
> > > >>>
> > > >>> Your example is nice since you include a functional form that can
> be reproduced fairly easily by plugging in the coefficients. However, it
> seems I will have to wait until you upload the paper to biorxiv so I can
> better understand what you are showing and if I can reuse it. Allow me to
> explain my thoughts.
> > > >>>
> > > >>> Your plot indicates that after 80 days only about 10% of the
> population have a viral load larger than theta. Also you indicate that your
> arbitrary death definition is viral load theta above 1000 in day 30.
> > > >>>
> > > >>> What I am looking for is a little bit different, What I want is
> the probability of death as a function on time. For example, I assume the
> probability at the day of infection will be zero since the virus did not
> spread in the body and the probability will then gradually rise as days
> pass until a peak and then drop again and will be close to zero after
> several weeks when subjects recover and can no longer die from the disease.
> > > >>>
> > > >>> The question is if it is possible to derive such a function from
> your data? - and if yes, exactly how?
> > > >>>
> > > >>> When integrating models, many time the issue is that the models
> presented in the publication need some manipulation, Many times this
> manipulation is trivial, yet many other times it requires some
> transformation and deep comprehension of the details and sometimes the
> model is unusable for some purposes because it calculates something
> different. This is what I am trying to figure out in this case and I need
> more information.
> > > >>>
> > > >>> Unless you can help extract the function I am looking for before
> publication, I will have to wait for the preprint and will be happy to
> continue this public conversation on this thread.
> > > >>>
> > > >>> Jacob
> > > >>>
> > > >>> On Mon, Dec 21, 2020 at 4:46 AM Filippo Castiglione <
> filippo.castiglione at gmail.com> wrote:
> > > >>> Dear Jacob, please see attachment as a reply to your mail.
> > > >>>
> > > >>> I could parametrise the “model” further with respect to patient’s
> age but I though the reply in attachment might be already ok for your
> purpose. You tell me.
> > > >>>
> > > >>> As for the permission to use it for commercial purposes I can
> declare no restrictions (so far ;).
> > > >>>
> > > >>> Best wishes
> > > >>> Fil
> > > >>>
> > > >>>
> > > >>>
> > > >>>> On 20 Dec 2020, at 23:53, Jacob Barhak <jacob.barhak at gmail.com>
> wrote:
> > > >>>>
> > > >>>> Well Filippo,
> > > >>>>
> > > >>>> Since the discussion is relevant to the comorbidities subgroup
> and to the integration subgroup, you will find both mailing lists CCd. I
> suggest we try to keep this discussion public and merge it with the working
> group activities. I believe others will find interest in this discussion as
> it may be relevant to their activities. I am copying the entire email
> conversation below.
> > > >>>>
> > > >>>> To your question about if I am suggesting to use your whole model
> to simulate mortality?
> > > >>>>
> > > >>>> The answer is: almost.
> > > >>>>
> > > >>>> Your model has important utility towards calculating mortality.
> However, I only need your output at this point. What I am asking for is the
> output of your model as a function. Something like:
> > > >>>>
> > > >>>> probability_death = f (days since infection)
> > > >>>>
> > > >>>> And at this initial point f does not have to be in functional
> form. It is enough for simulation purposes to have an x,y table where x is
> from 0 to 60 in days for each day and the associated mortality probability.
> > > >>>>
> > > >>>> And it is ok if you give me several assumptions with different
> cutoff values in your model.
> > > >>>>
> > > >>>> The ensemble model will balance those assumptions with other
> models and assumptions. If the assumptions do not merge well with other
> assumptions/models, the ensemble model should assign them less influence.
> > > >>>>
> > > >>>> So if you can use the model you showed at the webinar to derive
> different mortality functions and send those to me as a table - it will be
> great.
> > > >>>>
> > > >>>> However, please also make sure you give me permission to use
> those for commercial purposes - I do intend to monetize my technology and
> therefore avoid using information that has restrictions. I did declare a
> conflict of interest and I am making it public. However, I hope we can
> still collaborate.
> > > >>>>
> > > >>>> Jacob
> > > >>>>
> > > >>>> --
> > > >>>> Jacob Barhak Ph.D.
> > > >>>> Sole Proprietor, Software Developer, and Computational Disease
> Modeler
> > > >>>> https://sites.google.com/view/jacob-barhak/home
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> On Sun, Dec 20, 2020 at 2:40 AM Filippo Castiglione <
> filippo.castiglione at gmail.com> wrote:
> > > >>>> Dear Jacob, sure this conversation is open.
> > > >>>>
> > > >>>> Coming back to the mortality model question are you suggesting to
> use my whole model to simulate mortality? If yes, then It is possible at
> the condition that we define when the simulation should stop for “death”.
> > > >>>>
> > > >>>>
> > > >>>> Best
> > > >>>>
> > > >>>> --- Filippo (@iPhone)
> > > >>>>
> > > >>>>> On 19 Dec 2020, at 21:48, Jacob Barhak <jacob.barhak at gmail.com>
> wrote:
> > > >>>>>
> > > >>>>>
> > > >>>>> Thanks Filippo,
> > > >>>>>
> > > >>>>> Yes, you are correct, the mortality models are not accurate. All
> models are assumptions - none of them are really accurate. The Reference
> Model was built to figure out how well they perform and provide a reference
> modelers can compare their models to.
> > > >>>>>
> > > >>>>> Your suggestions make sense and in fact I already have several
> mortality models plugged into my model. Yet those are simplified and not
> multi scale models.
> > > >>>>>
> > > >>>>> The value in your approach is that it makes this multi scale -
> if you would plug your model in the ensemble and the ensemble will prefer
> it to other models - this will give the multi scale model validation of
> sorts.
> > > >>>>>
> > > >>>>> And with your permission, I would like to add this conversation
> to the co-morbidities mailing list that I plan to create or to the
> integration mailing list in the integration working group portal:
> > > >>>>> https://lists.simtk.org/pipermail/vp-integration-subgroup/
> > > >>>>>
> > > >>>>> Hopefully you are ok with making this conversation public and
> part of the working group discussions.
> > > >>>>>
> > > >>>>> Jacob
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>> On Sat, Dec 19, 2020 at 2:48 AM Filippo Castiglione <
> f.castiglione at iac.cnr.it> wrote:
> > > >>>>> Hi Jacob.
> > > >>>>>
> > > >>>>> The only thing I feel to suggest is to use survival curves of
> covid patients. I don’t know if you can recall them somewhere from clinical
> available data (e.g.,
> https://bmcpublichealth.biomedcentral.com/articles/10.1186/s12889-020-09721-2).
>
> > > >>>>> You might be interested in age-related of such curves
> (Kaplan-Meier curves plot 1-P(death)).
> > > >>>>>
> > > >>>>> I might calculated them starting from my simulations but they
> would be inaccurate because based on an artificial definition of “death”
> that, of course, I don’t have. Let me know if you want to try it anyway but
> I suggest to look for them around.
> > > >>>>>
> > > >>>>> Another very simple stylised model of mortality would be to use
> a reverse sigmoid function to represent the upside-down sigmoid shape (long
> right tail) of a generic Kaplan-Meier plot centred around the "typical
> value” time of death. For instance such t could be in the range 7-15 days,
> for instance, 1-0.2/(1+exp(7-x)) where 1-0.2 is the asymptotic value and 7
> is the typical death t.
> > > >>>>>
> > > >>>>> Let me know if I expressed my thoughts clearly.
> > > >>>>>
> > > >>>>> All the best
> > > >>>>> Fil
> > > >>>>>
> > > >>>>>> On 18 Dec 2020, at 18:38, Jacob Barhak <jacob.barhak at gmail.com>
> wrote:
> > > >>>>>>
> > > >>>>>> Hi Filippo,
> > > >>>>>>
> > > >>>>>> Following your talk at the webinar yesterday, you may recall
> that I was interested in a mortality model based on time.
> > > >>>>>>
> > > >>>>>> I wanted to communicate with you about it to see what is
> possible.
> > > >>>>>>
> > > >>>>>> I am looking for models to plug into my ensemble model. You can
> see quickly how it looks in this video - I am pointing at the point where I
> show some ensemble results:
> > > >>>>>> https://youtu.be/aTB8-XEZheU?t=1000
> > > >>>>>>
> > > >>>>>> I am actively looking for several types of models to integrate
> into my model. One of the models can be mortality models. Currently I use
> two different models for mortality - one of overall risk and one for death.
> I do not have a probability of death per day from infection and looking at
> your plots I figured you may have something - even at a level of an
> assumption.
> > > >>>>>>
> > > >>>>>> Do you think it is possible to get such models from you and if
> so, under what terms?
> > > >>>>>>
> > > >>>>>> Hopefully you can help.
> > > >>>>>>
> > > >>>>>> Jacob
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> On Mon, Dec 7, 2020 at 7:57 AM Jacob Barhak <
> jacob.barhak at gmail.com> wrote:
> > > >>>>>> Ok Filippo,
> > > >>>>>>
> > > >>>>>> You are in a similar situation to many others.
> > > >>>>>>
> > > >>>>>> Your situation is very common and well understood.
> > > >>>>>>
> > > >>>>>> Jacob
> > > >>>>>>
> > > >>>>>> On Mon, Dec 7, 2020, 02:37 Filippo Castiglione <
> f.castiglione at iac.cnr.it> wrote:
> > > >>>>>> Dear Jacob, sorry for the late response.
> > > >>>>>>
> > > >>>>>> I recently realised I had subscribed to too many groups to be
> practically able to follow, so opted out from many of them included the
> comorbidity. I asked Bruce to update the groups’ lists.
> > > >>>>>>
> > > >>>>>> All the best
> > > >>>>>> Fil
> > > >>>>>>
> > > >>>>>>> On 6 Dec 2020, at 06:23, Jacob Barhak <jacob.barhak at gmail.com>
> wrote:
> > > >>>>>>>
> > > >>>>>>> Hi Filippo,
> > > >>>>>>>
> > > >>>>>>> You were listed as interested in comorbidities within the
> viral pandemic working group. I am going over each member personally to see
> why they signed up.
> > > >>>>>>>
> > > >>>>>>> I looked at your web site:
> > > >>>>>>>
> > > >>>>>>> https://www.iac.cnr.it/~filippo/about-me.html
> > > >>>>>>>
> > > >>>>>>> Your interest in co-morbidities is visible directly from what
> you state in your web site:
> > > >>>>>>> "particular interest in the immune system and related
> pathologies"
> > > >>>>>>>
> > > >>>>>>> I wonder why you did not respond to my email to step up and
> co-lead this sub-working group. After all you have many publications and
> experience in leadership.
> > > >>>>>>>
> > > >>>>>>> Hopefully my personal email will get a response.
> > > >>>>>>>
> > > >>>>>>> Jacob
> > > >>>>>>>
> > > >>>>>>> --
> > > >>>>>>> Jacob Barhak Ph.D.
> > > >>>>>>> https://sites.google.com/view/jacob-barhak/home
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>
> > > >>
> > > >
> > >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.simtk.org/pipermail/vp-integration-subgroup/attachments/20210103/636a1bcb/attachment-0001.html>
More information about the Vp-integration-subgroup
mailing list