{"id":1440,"date":"2024-11-28T13:54:48","date_gmt":"2024-11-28T13:54:48","guid":{"rendered":"https:\/\/www.allendowney.com\/blog\/?p=1440"},"modified":"2024-11-29T15:10:26","modified_gmt":"2024-11-29T15:10:26","slug":"hazard-and-survival","status":"publish","type":"post","link":"https:\/\/www.allendowney.com\/blog\/2024\/11\/28\/hazard-and-survival\/","title":{"rendered":"Hazard and Survival"},"content":{"rendered":"\n

Here\u2019s a question from the Reddit statistics forum<\/a>.<\/p>\n\n\n\n

\n

If I have a tumor that I\u2019ve been told has a malignancy rate of 2% per year, does that compound? So after 5 years there\u2019s a 10% chance it will turn malignant?<\/p>\n<\/blockquote>\n\n\n\n

This turns out to be an interesting question, because the answer depends on what that 2% means. If we know that it\u2019s the same for everyone, and it doesn\u2019t vary over time, computing the compounded probability after 5 years is a relatively simple.<\/p>\n\n\n\n

But if that 2% is an average across people with different probabilities, the computation is a little more complicated \u2013 and the answer turns out to be substantially different, so this is not a negligible effect.<\/p>\n\n\n\n

To demonstrate both computations, I\u2019ll assume that the probability for a given patient doesn\u2019t change over time. This assumption is consistent with the multistage model of carcinogenesis<\/a>, which posits that normal cells become cancerous through a series of mutations, where the probability of any of those mutations is constant over time.<\/p>\n\n\n\n

Click here to run this notebook on Colab<\/a>.<\/p>\n\n\n\n

Constant Hazard<\/h2>\n\n\n\n

Let\u2019s start with the simpler calculation, where the probability that a tumor progresses to malignancy is known to be 2% per year and constant. In that case, we can answer OP\u2019s question by making a constant hazard function and using it to compute a survival function.<\/p>\n\n\n\n

empiricaldist<\/code> provides a Hazard<\/code> object that represents a hazard function. Here\u2019s one where the hazard is 2% per year for 20 years.<\/p>\n\n\n\n

from empiricaldist import Hazard\n\np = 0.02\nts = np.arange(1, 21)\nhazard = Hazard(p, ts, name='hazard1')\n<\/pre>\n\n\n\n
The probability that a tumor survives a given number of years without progressing is the cumulative product of the complements of the hazard, which we can compute like this.<\/p>\n\n\n\n
p_surv = (1 - hazard).cumprod()\n<\/pre>\n\n\n\n
Hazard<\/code> provides a make_surv<\/code> method that does this computation and returns a Surv<\/code> object that represents the corresponding survival function.<\/p>\n\n\n\n
surv = hazard.make_surv(name='surv1')\n<\/pre>\n\n\n\n
Here\u2019s what it looks like.<\/p>\n\n\n\n
surv.plot()\ndecorate(xlabel='Year', ylabel='P(survival > t)')\n<\/pre>\n\n\n\n
\"_images\/6520d3305e45a1d25be58e49729b7a380e68afd2546d79afcd9f462338ec02b0.png\"\/<\/figure>\n\n\n\n
The y-axis shows the probability that a tumor \u201csurvives\u201d for more than a given number of years without progressing. The probability of survival past Year 1 is 98%, as you might expect.<\/p>\n\n\n\n
surv.head()\n<\/pre>\n\n\n\n
<\/th>probs<\/th><\/tr><\/thead>
1<\/th>0.980000<\/td><\/tr>
2<\/th>0.960400<\/td><\/tr>
3<\/th>0.941192<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
And the probability of going more than 10 years without progressing is about 82%.<\/p>\n\n\n\n
surv(10)\n<\/pre>\n\n\n\n
array(0.81707281)\n<\/pre>\n\n\n\n
Because of the way the probabilities compound, the survival function drops off with decreasing slope, even though the hazard is constant.<\/p>\n\n\n\n
Knowledge is Power<\/h2>\n\n\n\n
Now let\u2019s add a little more realism to the model. Suppose that in the observed population the average rate of progression is 2% per year, but it varies from one person to another. As an example, suppose the actual rate is 1% for half the population and 3% for the other half. And for a given patient, suppose we don\u2019t know initially which group they are in.<\/p>\n\n\n\n
[UPDATE: If you read an earlier version of this article, there was an error in this section \u2013 I had the likelihood ratio wrong and it had a substantial effect on the results.]<\/p>\n\n\n\n
As in the previous example, the probability that the tumor goes a year without progressing is 98%. However, at the end of that year, if it has not progressed, we have evidence in favor of the hypothesis that the patient is in the low-progression group. Specifically, the likelihood ratio is 99\/97 in favor of that hypothesis.<\/p>\n\n\n\n
Now we can apply Bayes\u2019s rule in odds form<\/a>. Since the prior odds were 1:1 and the likelihood ratio is 99\/97, the posterior odds are 99:97 \u2013 so after one year we now believe the probability is 50.5% that the patient is in the low-progression group.<\/p>\n\n\n\n
p_low = 99 \/ (99 + 97)\np_low\n<\/pre>\n\n\n\n
0.5051020408163265\n<\/pre>\n\n\n\n
In that case we can update the probability that the tumor progresses in the second year:<\/p>\n\n\n\n
p1 = 0.01\np2 = 0.03\n\np_low * p1 + (1-p_low) * p2\n<\/pre>\n\n\n\n
0.019897959183673472\n<\/pre>\n\n\n\n
If the tumor survives a year without progressing, the probability it will progress in the second year is 1.99%, slightly less than the initial estimate of 2%. Note that this change is due to evidence that the patient is in the low progression group. It does not assume that anything has changed in the world \u2013 only that we have more information about which world we\u2019re in.<\/p>\n\n\n\n
If the tumor lasts another year without progressing, we would do the same update again. The following loop repeats this computation for 20 years.<\/p>\n\n\n\n
odds = 1\nratio = 99\/97\nres = []\n\nfor year in hazard.index:\n    p_low = odds \/ (odds + 1)\n    haz = p_low * p1 + (1-p_low) * p2\n    res.append((p_low, haz))\n    odds *= ratio\n<\/pre>\n\n\n\n
Here are the results in percentages.<\/p>\n\n\n\n
df = pd.DataFrame(res, columns=['p_low', 'hazard'], index=hazard.index)\n(df * 100).round(2).head()\n<\/pre>\n\n\n\n
<\/th>p_low<\/th>hazard<\/th><\/tr><\/thead>
1<\/th>50.00<\/td>2.00<\/td><\/tr>
2<\/th>50.51<\/td>1.99<\/td><\/tr>
3<\/th>51.02<\/td>1.98<\/td><\/tr>
4<\/th>51.53<\/td>1.97<\/td><\/tr>
5<\/th>52.04<\/td>1.96<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
If we put the hazard rates in a Hazard<\/code> object, we can compare them to the constant hazard model.<\/p>\n\n\n\n
hazard2 = Hazard(df['hazard'], name='hazard2')\n<\/pre>\n\n\n\n
hazard.plot(label='Known probability')\nhazard2.plot(label='Uncertain probability')\ndecorate(xlabel='Year', ylabel='Hazard')\n<\/pre>\n\n\n\n
\"_images\/5d7cb2951f8e4a949b31295ba14ad3f1c7353887e065b451b35488e1e943f5ff.png\"\/<\/figure>\n\n\n\n
Each year, the probability increases that the patient is in the low-progression group, so the probability of progressing in the next year is a little lower.<\/p>\n\n\n\n
Here\u2019s what the corresponding survival function looks like.<\/p>\n\n\n\n
surv2 = hazard2.make_surv(name='surv2')\n<\/pre>\n\n\n\n
surv.plot(label='Known probability')\nsurv2.plot(label='Uncertain probability')\ndecorate(xlabel='Year', ylabel='P(survival > t)')\n<\/pre>\n\n\n\n
\"_images\/30ecc5edc2c3c2fc40fb64a4110f570177450b0f2e5fa819f0404a4064c651fb.png\"\/<\/figure>\n\n\n\n
The difference in survival is small, but it accumulates over time. For example, the probability of going more than 20 years without progression increases from 67% to 68%.<\/p>\n\n\n\n
surv(20), surv2(20)\n<\/pre>\n\n\n\n
(array(0.66760797), array(0.68085064))\n<\/pre>\n\n\n\n
In this example, there are only two groups with different probabilities of progression. But we would see the same effect in a more realistic model with a range of probabilities. As time passes without progression, it becomes more likely that the patient is in a low-progression group, so their hazard during the next period is lower. The more variability there is in the probability of progression, the stronger this effect.<\/p>\n\n\n\n
Discussion<\/h2>\n\n\n\n
This example demonstrates a subtle point about a distribution of probabilities. To explain it, let\u2019s consider a more abstract scenario. Suppose you have two coin-flipping devices:<\/p>\n\n\n\n