https://jumpdata.co.uk/using-survival-analysis-to-estimate-product-lifetime/ Skip to content * Data and Analytics Consultancy in London, UK * Newsletter Sign up for Newsletter [ ] [Sign Up] * * * Data and Analytics Consultancy in London, UK Data Consultancy in London, UK >> JumpData Ltd.Data Consultancy in London, UK >> JumpData Ltd.Data Consultancy in London, UK >> JumpData Ltd.Data Consultancy in London, UK >> JumpData Ltd. * Menu * [ ] * Our Services [svg][ezgif] DATA-DRIVEN APPLICATIONs and SERVICES Data Driven Applications Dashboard Design [svg][ezgif] STATISTICAL MODELLING AND DATA ANALYTICS Advanced Statistical Analysis of Survey Data Business Intelligence Automation and Streamlining of Data Flows * About Us * Our Work * Blog * Key Driver Analysis * Contact Us * [ ] * Blog Using Survival Analysis to estimate product lifetime Posted on December 31, 2022May 9, 2023 by Paul Brown We needed to estimate the average number of years after purchase various consumer products broke down from a large dataset containing: * the product type; * the brand and age of the devices; * whether or not the devices were still in use; * if the device was no longer in use, did it breakdown or was it simply replaced. Such estimates are best provided using survival analysis. We outline below how we applied it in this case, and some of the issues that arose. Survival analysis Survival analysis is an area of statistics that studies the (expected) length of time until some particular "event" occurs. This involves estimating the probability distribution of the length of time until the event happens, i.e., constructing a function whereby for each time-period, one can estimate the chances of the event having occurred in the time-period. In our case, the event is the "failure" / "breakdown" of the machine - its (end of) lifetime - whilst time is the age of machine when the particular event happens. Clearly, we are actually interested in the cumulative probability distribution function, i.e., we wish to know, for any age, the probability that the breakdown has occurred at any point from purchase up to the stated age. For the object of interest is length of time before the event occurs. Indeed, since we are analysing product life-expectancy, it is far more useful to construct what's known as the survival function S(t); this is the probability of the event not having happened for each age period, i.e., the probability of surviving beyond that age point. The graph of the survival function is known as the survival curve, and this is what is commonly produced from SPSS, R, or other statistical packages. (The instantaneous probability of breakdown at a time t, given that the machine has survived up to t is called the hazard function, but we won't focus on that here). The main technique we used to analyse our data was Kaplan-Meier (KM). This is a well-respected technique that has been used since the 1950s in analysis of clinical survival and product failure rates. It is a non-parametric approach, i.e., it does not yield an age-dependent "formula" to obtain the probability of the event having occurred. However, since most of the data in these sorts of studies is discrete - we have time periods, as opposed to instantaneous moments in time - it is arguably more appropriate. Common parametric distributions include the Exponential and the Weibull, with the latter being the recommended by much of the literature. As alluded to above, all the survival probabilities we are interested in are conditional probabilities. For, if a product fails during some age-range 't', then it must also have survived through all the previous time periods, i.e., P(Breakdown during t) = P(Survive up to t) x P(Fail during t). Equivalently, we can simply note that the survival function at time 't' is the product of all survival probabilities up to end of age-range 't'. This is important since it follows then that we cannot just look at the individual breakdown proportions per time-period to ascertain estimates of life expectancy, irrespective of how precise your estimates of the individual proportions are. Censoring The major problem with all survival analysis techniques is the issue of censoring, i.e., cases for which it is not known when the event occurred. In medical studies, this could be because of the end of the study or simply patient drop-out. In our case, we do not know when a machine breaks down, nor do we know when a machine that was replaced for non-breakdown reasons, when or if it actually breakdowns in the future. Consequentially, our dataset is, in effect, incomplete. If we had the actual ages of breakdown / replacement of every machine, we could easily calculate expected age with confidence intervals. However, we do not. Both the KM non-parametric approach and other survival curve estimations take censoring into account. In either case, the result of both censoring and the number of products that breakdown / are replaced over time causes the sample size to reduce as time increases. Correspondingly, the standard errors for each survival probability increase as time increases. This makes estimates of survival probabilities past certain time-periods very uncertain. KM survival curve For any age t, the KM survival point estimate is calculated to be the number of machines that do not breakdown during t divided by the number that were alive at the start of t. Note here that the denominator has removed both the machines that broke down before t and those that have been censored before t (i.e., were replaced for non-faulty reason or were still alive and have age less than t). This is how KM deals with the censoring problem. The survival function S(t) at time t is then the product of all these point estimates up to time t. Standard errors can be easily calculated from the survival point estimates. There are two ways of calculating the confidence bounds, either the Greenwood estimates, given by (S(t) - 1.96*SE, S(t)- 1.96*SE). or the the log confidence interval, defined to be (S(t)*exp{- 1.96 * SE / S(t)}, S(t)*exp{+ 1.96 * SE / S(t)}). We believe the log confidence interval to be more bust, and indeed is the default for the statistical package R uses; SPSS defaults to the Greenwood ones. [svg][SurvPlot1-300x185] Stratification by brand We needed to produce an overall estimate of expected lifetime for a particular product type. However, it was highly likely that different brands would have expected lifetimes. So, for each product type, we first produced survival curves of the devices grouped by brand, to assess differences. It was clear by examining the KM survival curves by eye that there were indeed large variations from brand to brand. We therefore needed to consider as many brands as possible separately. However, due to the nature of this type of product data, there were far less older products in the dataset than younger ones. Hence, the confidence bounds around the survival curve became very large as age increased, meaning that we could not be confident that the brands were different over extended time periods. We therefore conducted a set of log-rank tests on the various brand KM survival curves, and considered the devices to be in the "same" brand group unless they differed sufficiently according to the test. We thus produced, for each product type, estimated life expectancies for a small number of brand groups. Average life expectancy using KM survival analysis We wished to construct estimates for the average life expectancy of a product, i.e., either the mean or the median length of time before the particular event occurred, plus produce confidence bounds around our estimate. KM median The KM median length of time is the age that the probability of survival is less than 50%, and it is generally recommended in the literature to use this measure for the average instead of the mean, if possible. The bounds of the confidence interval of the median is normally defined to be the first time that the lower, respectively, upper, end of any of the confidence intervals for the survival proportions is less than 50%. (The statistical package SPSS uses a different confidence interval for the KM median which we do not believe was appropriate). However, the problem with using the median was that for some of the product types, this level of breakdown was never reached. So for these products, we had to report the lower bound of the confidence interval of the probability of breakdown at a specified age. In all cases, we reported the (adjusted restricted KM) mean. KM mean Irrespective of whether you use a non-parametric or a parametric approach, the mean length of time until the event occurs is equal to the area under the survival curve. However, the standard errors of the mean naturally depend on the approach taken. With KM, they are relatively easily calculated from the survival probabilities; with a parametric curve they are taken from the distribution, assuming the estimates are normal. Two problems immediately arise when calculating the mean: the maximum expected lifetime is dependent on the age of the oldest product in the survey, and the survival proportion of those who are at this maximum age does not affect the mean. To deal with the first problem, we can produce a series of means called restricted KM survival means. For our final brand estimate, we therefore need to decide which is the most appropriate maximum age to use. To address the second problem, we used a slight amendment to the KM calculation of the mean: instead of using rectangles to calculate the area, we used trapeziums. This had the advantage of including the last age survival proportion into the calculation and is easy to hand-calculate. We called this the adjusted restricted KM survival mean. Standard errors are not obvious to calculate in this case, but since they are likely to be similar to the original KM estimate, we took an approximate standard error as SE (max age+1). Choosing a maximum age for the adjusted KM mean Finally, we needed to choose a maximum age for the adjusted restricted KM means. To guide us in our choice of cap - and to give use some confidence in our KM survival curves - we constructed sets of parametric survival curves on our (merged) brands, using the Weibull distribution. We then plotted both curves on the same graph and visually compared them. Where the curves diverged, i.e., where the Weibull curve lay a long way outside the KM confidence bound, gave us a good guide where to put the maximum age for the adjusted restricted KM mean. [svg][KMWeibull-300x185] [svg][paul-e1648043] Paul Brown Responsible for data analysis, modelling and automation jumpdata.co.uk This entry was posted in Blog and tagged survey analytics, survival analysis. Payroll Tool for Commission Calculation and Reporting for Recruiters The problem of multicollinearity in Key Driver Analysis [ ] Recent Posts * The problem of multicollinearity in Key Driver Analysis Comments Off on The problem of multicollinearity in Key Driver Analysis * Using Survival Analysis to estimate product lifetime Comments Off on Using Survival Analysis to estimate product lifetime * Payroll Tool for Commission Calculation and Reporting for Recruiters Comments Off on Payroll Tool for Commission Calculation and Reporting for Recruiters * Data Visualisation tools for BI and analytics: Six typical scenarios and possible solutions for SMEs and big corporations Comments Off on Data Visualisation tools for BI and analytics: Six typical scenarios and possible solutions for SMEs and big corporations Partner with JumpData We solve the data problems that seem unsolvable. Drop us a line Call Us * Privacy Policy * Cookie policy (UK) Copyright 2024 (c) Jumpdata Limited. Company No. 08241266. Web Design by MySuper.Site * Data and Analytics Consultancy in London, UK * Our Services * About Us * Our Work * Blog * Key Driver Analysis * Contact Us * Newsletter Manage Cookie Consent We use cookies to optimize our website and our service. Functional [ ] Functional Always active The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network. Preferences [ ] Preferences The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user. Statistics [ ] Statistics The technical storage or access that is used exclusively for statistical purposes. The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you. Marketing [ ] Marketing The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes. Manage options Manage services Manage {vendor_count} vendors Read more about these purposes Accept cookies Deny View preferences Save preferences View preferences {title} {title} {title} Manage consent