How accurate is clinical prognostication by oncologists during routine practice in a general hospital and can it be improved by a specific prognosis training programme: a prospective interventional study

Participants

The participants of the study consisted of the whole medical team (= physicians) of the department of oncology and haematology (n=21), who (individually) generated the prognostic estimates on the patients (described below). This ‘whole group’ was subdivided into ‘residents’ (physicians in training, less than 1 year of training in clinical oncology = ‘inexperienced’) and into ‘oncologists’ (board-certified oncologists with a minimum experience in clinical oncology of 13+years = ‘experienced’).

Procedure

Data collection

Every physician’s estimate and every hospitalisation of a patient were considered. When there were multiple estimates for a patient (eg, by resident, senior physician and chief physician), all three estimates were recorded as distinct data points for this respective patient for this hospital stay. Patients, who received a PE on more than one occasion, were included in the analysis for each of their PE. For analysis all physicians together formed the ‘whole group’. In addition, according to the clinical experience, we defined the subgroups ‘residents’ and ‘oncologists’ (board-certified specialists in haematology and oncology).

To avoid the possibility of non-representative values caused by outliers, only physicians who provided ≥10 PE per phase were included in the final analysis. Physicians, who did not work in the oncology wards in every phase (eg, due to rotation in residency training), were included only in the involved phase(s) if they had completed ≥10 PE. Real survival data (date of death or last recorded life status) were provided by the Tumor Center Regensburg in March 2023, with a follow-up of approximately 4 years. PE for patients, who did not have an oncological disease or who were not registered at the Tumor Center Regensburg, were not included in the analysis (because follow-up could not be ensured).

Phase 1 versus phases 2 and 3/interventions

A flow chart of the 3 phases with all participating physicians and patients is provided in online supplemental file 1. In phase 1, the PE was performed with no prior training and was thus based mainly on prior clinical experience and ‘gut feeling’ (‘thinking fast’). In contrast, during phase 2 ‘thinking slow’ (according to Kahneman³⁸) was encouraged—that is to consciously take time for the PE. To enable that, a prognosis training was provided for the physicians prior to the start of the second phase and repeated before phase 3 (see online supplemental file 2):

1. The prognosis training consisted of a brochure designed to support the physicians with the aid of epidemiological data on the 21 most common oncological diseases of the clinic. For each disease, general information was provided on the median survival time across all groups as well as information for different patient groups (eg, according to patients’ age, stage or mutation status). Corresponding Kaplan-Meier curves were provided with information on absolute and relative survival for these subgroups. Sources of the information were derived mostly from the Munich Tumor Registry, other sources were, for example, the website UpToDate or clinical or epidemiological studies. These epidemiological data were intended to be used as numerical ‘anchors’ during the (conscious) process of prognostication.
2. The brochure was introduced to the participating physicians within a teaching lecture during which a general approach to prognostication was provided (see online supplemental files). In brief, this approach assumes that prognostic accuracy can be improved by first taking an ‘outside view’ on a situation—for example, by using anchor numbers derived from the brochure (‘actuarial’ approach).³⁹ In subsequent steps, a more ‘personalised’ and multifaceted ‘inside view’ may follow in which relevant details of the individual patient can be used to adjust the prognostic estimate—using the principle of ‘Fermisation’ (referring to Enrico Fermi—see online supplemental files).

In addition, the SQ of the questionnaire was reworded in phase 2 to exclude the subjective factor of the term ‘surprise’. (‘Do you think it is more likely (> 50%) that your patient will be alive in 6 months than that he will have died by that time?’)

Prior to the start of phase 3, prognostic tools were provided for the most common oncological diseases of the clinic, if they were available either as a website or as an app (see online supplemental file 3). In this phase, physicians had to provide two separate PE on the same patient—the first without knowledge of the PE made by the respective (electronic) prognostic tool and the second while being aware of the result.

In phase 3, the questionnaire was complemented by another version of the SQ—that is, the ‘probabilistic SQ’: In addition to the answer option ‘yes/no’, the SQ was now to be answered probabilistically (‘Please indicate the probability (in 10% increments from 0% to 100%) that your patient will be alive in 6 months.’).

Statistical evaluations

CPS was evaluated using the three categories ‘accurate’, ‘neutral’ and ‘inaccurate’. When the real OS time was set as 100%, we set the definition of ‘accurate’ estimates as those that ranged from 67% to 133% of this value. ‘Neutral’ estimates were in the range of 34%–67% and 133%–166% and ‘inaccurate’ estimates were beyond these limits, that is, <34% and >166% of the actual observed survival time.

The SQ was considered ‘correct’ if the answer was ‘yes’ and the patient was alive at 6 months from the time of the PE or the answer was ‘no’ and the patient had died within 6 months. In the probabilistic form of the SQ, we defined estimates as ‘correct’ if the patient had died at 6 months when given answers ranging from 0% to 40% or lived at 6 months when given answers ranging from 50% to 100%.

The rate of correctly answered SQ and the rate of ‘accurate’ CPS PE of the ‘whole group’ were collected as the primary endpoints of this study.

As a secondary endpoint, we aimed to describe the potential impact of a ‘prognosis training’ on the prognostic performance of physicians. To analyse this intervention, performance in phase 1 (‘untrained’) was compared with the performance after training (both phases 2 and 3 after training—‘trained’) and tested for statistical significance by using a χ² test. These analyses were performed for the ‘whole group’ and for the two subgroups ‘residents’ and ‘oncologists’ for each of the 3 phases.

In addition, ‘residents’ and ‘oncologists’ were tested against each other for differences in prognostic performance in each phase.

For the probabilistic form of the SQ, we tested how far the estimated survival likelihood correlates with the actual survival (χ² test for linear trend). The basis of this analysis is the groups of patients who had in common their respective estimated survival likelihoods—that is, ‘0% group’, ‘10% group’, ‘20% group’, etc up to the ‘100% group’. For example, the ‘100% group’ comprises all patients, which had been given a survival likelihood of 100% at 6 months by their respective physicians. In this probabilistic setting we also calculated the Brier score for the overall group and the subgroups (oncologists and residents).

The programme Microsoft Excel was used for descriptive statistics of the 3 phases as well as for documentation, analysis and creation of diagrams. The presentation of Kaplan-Meier curves was realised with the statistical program IBM SPSS Statistics 29.0.

Patients

A total of 748 patients with a wide range of diseases were assessed—12 of which were excluded from the analysis because they either had no oncologic disease or no status on survival was available at the time of follow-up, so the final analysis consisted of 736 patients. The patients’ characteristics can be seen in table 1. Of 736 patients, 410 (55.7%) had died by the time of the last follow-up in March 2023. The median overall survival was 2.5 years (see figure 1A).

Open in a separate window

Figure 1

(A) Overall survival of patients in all phases (n=736) in blue—estimated survival (n=2427 estimates) in red. (B) Real survival (x-axis) with the annotated range (eg, 9–10 months) and the percentage of accurate prognoses (dark blue), overestimated prognoses (= high inaccurate + high moderate, light blue) and underestimated prognoses (= low inaccurate + low moderate, middle blue) in this cohort of patients. Accuracy changes with the real survival with best results between 8 and 10 months. (C) Overall survival of patients in all phases—real survival (x-axis) versus estimated survival (y-axis)—the diagonal line represents perfect prediction. Patients above diagonal are those in whom survival was overestimated; patients below line are those in whom survival was underestimated²¹.

Clinician prediction of survival

Main findings

In our study, the whole group of participating physicians achieved an accuracy of 76.8% when using the SQ. This result is fairly comparable to the summarised accuracy of 78.6% for oncological patients calculated by a systematic review in 2017.⁴⁰ It should be noted, however, that very different time intervals have been investigated so far when answering the SQ for oncological patients and that the patient cohorts differed considerably (see online supplemental file 5).^25–27 Quite evidently, the time interval chosen for the SQ has an impact on accuracy, since the longer the time interval the less certain the estimates will be and the accuracy will ‘suffer’. The selection of a time interval appropriate for the respective patient population is therefore essential.

Interestingly, the prognostic accuracy of the whole group of participating physicians increased after training from an initial 72.6% to 84.3% in the last phase. This effect was especially pronounced in residents who started from a lower level (60.7% in phase 1) and then achieved 81.1% in the final phase (+20.4%). The experienced oncologists started from a significantly higher level of accuracy (75.7%) but also benefited from training (85.6% in the final phase)—however, to a less pronounced degree in absolute terms (+9.9%).

Our data indicate that experienced physicians are better prognosticators than early career physicians which has so far been a controversial issue within the literature.^{8 16 21–23 41 42} Moreover, a prognosis-oriented training programme has a potential to improve the prognostic accuracy of both residents as well as of experienced oncologists. Such a focused training programme has been repeatedly recommended in the literature.^{8 42 43} While some guidelines and training concepts exist in the palliative setting.^33–35 This is—to our knowledge—the first study to have implemented and systematically tested such an approach in a large group of oncological patients with a median survival over 2 years. (Our intention was to develop and test a short training programme and find out whether there is an effect. Our aim was not to specifically identify and isolate each reasoning principle within the programme and determine its incremental effectiveness.)

Wider implications

Prognostication using the SQ for a 6-month period can be considered ‘short-term prognostication’. In this situation, physicians’ accuracy was substantially better than chance and it was trainable. This finding contrasted with the situation of ‘long-term prognostication’ in which physicians were required to estimate overall survival (which most of the times exceeded 6 months—as demonstrated by a median OS of 2.5 years). The CPS was mostly ‘off-target’ with only 25.9% ‘accurate’ PE. Our results are in accordance with smaller studies which used the same definition for accuracy and demonstrated accuracy values between 20% and 35%.^{6 7 11 13 15 16 18 20 21} As mentioned above, previous studies assessed very advanced cancer patients with an OS of only a few months so that survival data were available for nearly all patients.^{6 7 11 13 15 16 18 21} Since only 60% of our patients had died at the time of analysis further follow-up of our patients might change the numerical value for accuracy of CPS.

CPS and SQ at 6 months are different metrics and, therefore, the absolute values of these parameters cannot be directly compared. However, CPS can be ‘converted’ into the format of an SQ by assuming the following: The groups ‘low inaccurate’, ‘low moderate’ and ‘accurate’ will not have been ‘surprised’ that the patient has died at his/her specific time of death. To illustrate this, imagine the following thought experiment: If we (retrospectively) know that a patient has died 450 days after discharge from our care we can pose this constellation (discharge letter from that past discharge) to a group of physicians in the form of an individual SQ: ‘Would you be surprised if this patient had died within 450 days?’ A physician with a CPS of ‘200 days’ would not be ‘surprised’ as in our SQ definition and he would be ‘right’. In contrast, a physician stating a CPS of ‘700 days’ would be ‘surprised’ by the earlier demise of the patient and his estimate would have been ‘wrong’. This also illustrates that the SQ is the more ‘permissive’ metric because it considers estimates as correct (‘not surprised’) that are not in the ‘accurate’ category of the CPS definition—that is, ‘low inaccurate’ and ‘low moderate’.

However—even when using the more permissive ‘individual SQ’ definition (=consider the sum of ‘low inaccurate’, ‘low moderate’ and ‘accurate’ as ‘not surprised’ and therefore as ‘correct’)—the value for the CPS for the whole group is only 53.8%—hardly better than chance. Interestingly, there is no (significant) trend towards improvement by training and also there seems to be no difference between residents and experienced oncologists. These findings can be used to estimate the ‘horizon’ of individual clinical prognostication—which is a controversial issue in the literature^{12 17 19 21 22 24 42 43}—in our study: The group of patients on which definite survival data were available (ie, had died at the time of analysis) and on which our CPS analysis was performed had a median (50%) survival of 9 months and 60% of patients had died within 1 year (data not shown). Assuming that prognostication over longer periods (than 1 year) will be even worse, we conclude that whereas individual prognostication within a time frame of 6 months by physicians is better than chance (and is trainable) this seems no longer to be the case when a time horizon of circa 1 year has been reached.

It is relevant to differentiate between prognostication for an individual patient and prognostication for a group of patients. During phase 3, when we also analysed the ‘probabilistic SQ’. Its overall accuracy of 83.1% was nearly identical to that of the ‘standard SQ’ in that phase (84.3%) as expected. These are the numerical values that are relevant for the PE of individual patients. However, unlike the standard SQ, the probabilistic SQ also had a quantitative component (‘estimate the probability of survival at 6 months in 10% increments!’). This has two consequences:

1. The forecasts in this format convey a sense of decisiveness and indecisiveness and thereby contain additional prognostic information/certitude: This is demonstrated by the high accuracy of the extreme ends of the spectrum (0%–30% = likely to die within 6 months; 70%–100% = likely to survive the next 6 months) which reached 89.9%. This was substantially higher than in the middle (‘indecisive’) part of the spectrum (40%–60%) where the accuracy was only 54.4%.
2. This ‘indecisive’ accuracy of 54.4% in the mid-range is hardly better than chance and implies a ‘coin flip’ for the individual patient. However, this does not mean, that such a prognostic estimate contains no information at all and is basically worthless. In contrast, we could show that the collective prognostic estimates in 10% increments closely mirrored the de facto survival rates of groups at 6 months as demonstrated in figure 2B. For example, the ‘group 50%’ comprises a group of patients (all considered by their treating physician to have a 50% likelihood of survival at 6 months) of which roughly half will in fact die within 6 months.

Even though each physician performed discrete prognostic estimates on an individual patient the sum of all these forecasts nevertheless also contains prognostic meaning for groups. This ‘wisdom of the clinical crowd’ is also exemplified by the survival curve of our patient cohort (figure 1A) which is fitted quite well (though not perfectly) by the multitude of forecasts done on these patients by the participating physicians.

Another aspect that can be derived from these two Kaplan-Meier curves is that overestimation of prognosis tends to occur predominantly in the early phase (6–9 months) whereas underestimation tends to be predominant at the end of the first year of observation and thereafter. The two curves cross between 6 and 9 months after t₀ (= time of the prognostic estimate) with the crossing tending to be earlier in the later phases—which may have been due to more conservative forecasts after training. This finding is in accordance with Fairchild et al who also observed an overestimation for patients with a survival of less than 6 months and an underestimation for patients with a survival of more than 9 months.¹² We have also visualised this finding in the dot plot (figure 1C) and found a higher rate of overestimation for survival <9 months and a higher rate of underestimation for survival >9 months.

Established prognostic tools were only available for a minority of patients in their specific clinical situation (34%) and—when used alone—performed worse (accuracy 70.2%) than physicians (84.3%) in phase 3. They also did not add to prognostic accuracy of physicians. This might have been due to the fact, that often these tools are optimised for only one specific situation (eg, primary diagnosis and start of curative therapy). In these situations, some of the tools worked very well. However, most tools were not or less helpful in the many other situations when a clinical encounter takes place (eg, 3 months into second-line therapy). Also most of the time these tools convey ‘collective’ prognoses and place a patient into a prognostic group—similar to our results of the probabilistic SQ. This can of course be helpful, but it is a more permissive/easier task than individual prognostication.

Strengths and limitations

This is the first study that presents a structured training programme for oncologists that has a positive impact on clinical prognostication. This prospective study included a large cohort of patients with different oncological and malignant haematological diseases at all disease stages. Prospective follow-up data of almost 4 years were collected, which allowed for the assessment of longer PE. The effect of training was not demonstrated by a randomised comparison but by comparing baseline (untrained) prognostication abilities with the performance after training in the same group of physicians. The study is a single-centre study—even though performed at a large teaching hospital.

We thank all members of the medical team of the department of oncology and haematology at the hospital Barmherzige Brüder who participated in this project: Berberich Veronika, Böhm Valentin, Braess Jan, Braun Bernhard, Daum Susanne, Heilmeier Bernhard, Homann Arne, Keyßner Verena, Koch Leonie, Maguire Nadia, Minderjahn Lisa, Moosmann Nicolas, Reiß Kathrin, Schenk Michael, Schlenska-Lange Anke, Seiler Isabella, Stauder Heribert, Stigler Katharina, Stosiek Christoph, Thalmeier Susanne, Wirsching Andreas.