Skip to main content

How We Do Risk Analysis

BAUS has included all the data returned in our overall analysis but, when presenting the individual surgeons' results, we have excluded those surgeons who returned less than 5 cases for the year because any statistical analysis of such low numbers would be invalid.

What is risk adjustment?

Patients vary by age, sex and the number of other illnesses they have (known as co-morbidities). Some surgeons may have many patients with complex problems, others  far fewer. This is known as the patient casemix and needs to be taken into account when considering figures such as those shown, because higher risk patients are more likely to have complications. In addition, some procedures are inherently riskier than others. This also needs to be taken into account and is termed "risk adjustment".

It is important to note that risk adjustment of patients in these datasets (originally set up in 2001 to evaluate the "new" technique of laparoscopic nephrectomy) was never planned, so the predictive accuracy (and hence the ability to risk-adjust every patient) is inevitably imperfect.

We have now added a number of additional questions to the dataset, in order to improve risk-adjustment for future years, for example:

 

How did we set up the risk adjustment?

To attempt risk adjustment for the first published audit (of nephrectomy), Bayesian risk models were constructed using the following variables: gender, age, WHO performance status (a measure of a patient's general well-being and how they can carry out the normal activities of daily life), procedure performed (radical nephrectomy, simple nephrectomy, partial nephrectomy & nephroureterectomy) and the stage (or severity) of the tumour being removed, for each of the three outcomes selected for this analysis: 30-day mortality, transfusion rate and in-hospital complication rate. The risk models show how many patients with a particular variable have the outcome being measured and the resultant likelihood of a future patient with the same variable having the same outcome.

Table 1: Bayesian modelling components for nephrectomy

  VARIABLES   PARAMETERS
(G)ender   Male: Female
(A)ge group   <=10; 11-20; 21-30; 31-40; 41-50; 51-60; 61-70; 71-80; 81-90; >90
(W)HO performance status   0; 1; 2; 3; 4
(P)rocedure   Radical; Partial; Simple; Nephro-ureterectomy
(T)umour stage   Benign; 1; 2; 3; 4; a
   

Table 2: Variables included in risk models (where G = gender, A = age, 
W = WHO performance status, P = procedure & T = stage)

  RISK FACTOR GAW__ GAWP_ GAW_T GAWPT
Gender Yes Yes Yes Yes
Age Yes Yes Yes Yes
WHO performance status Yes Yes Yes Yes
Procedure No Yes No Yes
T-Stage No No Yes Yes
         
 

Receiver Operating Characteristic (ROC) Curves

Hanley & McNeill stated that the area under an ROC curve:

... represents the probability that a randomly chosen diseased subject (or a patient with an adverse event) is correctly rated or ranked with greater suspicion than a randomly-chosen, non-diseased subject (or a patient who does not have an adverse event)"

A simplistic re-working of this statement for mortality rates would be that, for example, the area under the mortality ROC represents the probability that the risk predictor accurately discriminates between patients who die and patients who survive. An area of 0.50 indicates that there is no discrimination (i.e. individuals in survivor-deceased patient pairings are allocated to the correct group by the risk predictor according to chance). An area of 1.00 would indicate that discrimination was perfect, and any intermediate value is a measure of the ability of the risk predictor to distinguish between survivors and non-survivors. Obviously, the closer the value is to 0.50, the less accurate the discrimination and the closer to 1.00, the better. The same analyses can, of course, be performed for patients who require a blood transfusion or suffer a complication.

Table 3:  Risk modelling table & graph for mortality

  ANALYSIS GAW__ GAWP_ GAW_T GAWPT
  Hosmer-Lemeshow chi-sq p-value 0.654 0.760 0.580 0.428
  ROC area 0.775 0.785 0.792 0.796
  ROC standard error 0.047 0.046 0.046 0.045
  Events 35 35 35 35
  TOTAL UNDER ANALYSIS 5,238 5,238 5,238 5,238
         


GAWP: mortality

Table 4: Risk modelling table & graph for complications

  ANALYSIS GAW__ GAWP_ GAW_T GAWPT
  Calibration p-value 0.564 0.544 0.326 0.161
  ROC area 0.619 0.642 0.641 0.660
  ROC standard error 0.021 0.021 0.021 0.021
  Events 204 204 204 204
  TOTAL UNDER ANALYSIS 5,014 5,014 5,014 5,014
         


GAWP: complications

Table 5: Risk modelling table for blood transfusion (graph not available)

  ANALYSIS GAW__ GAWP_ GAW_T GAWPT
  Calibration p-value 0.136 0.293 0.028 0.015
  ROC area 0.632 0.662 0.731 0.727
  ROC standard error 0.015 0.015 0.015 0.015
  Events 408 408 408 408
  TOTAL UNDER ANALYSIS 4,519 4,519 4,519 4,519
         
 

Calibration plots

Risk scores must also provide an estimate of the risk both for individual patients and for groups of patients. One way to test this component of a risk score is to plot the observed number of events against the predicted number of events; this is termed a calibration plot. To simplify the procedure, the data can be split into groups, according to risk, and the observed and predicted outcome rates plotted side by side. If the model accurately predicts the outcome, the two should match closely.

Whilst, at first sight, the areas below the ROC curves suggest that including all five variables in the models would provide the best risk model, the calibration plots suggested otherwise. It was, therefore, decided to use gender, age, WHO performance status & procedure (GAWP_) to calculate the final risk-adjusted rates for each nephrectomy outcome, for each patient and, subsequently, for each hospital and Consultant. The final GAWP_ model calibration plots are shown below.

Table 6: Calibration plot for mortality

Calibration: mortality

Table 7: Calibration plot for complications

Calibration: complications

Table 8: Calibration plot for transfusion

Calibration: transfusion

 

Presentation of risk-adjusted rates

Once the risk-adjusted rate for each centre and surgeon has been calculated, this is displayed as a simple bar chart (see below), including the number of risk-adjusted procedures and the exact binomial 99% and 99.9% control limits to indicate thresholds for alarm.

The horizontal dark blue bar represents the individual risk-adjusted rate (by Consultant or by unit), the light blue line is the average rate for all Consultants or units, and the red lines show the calculated 99% and 99.9% alarm rates for that surgeon or unit.

Risk Adjusted Rate for Transfusion

 

BAUS is indebted to Dendrite Clinical Systems Ltd for
their help in generating the risk-adjustment analyses