The Benefits of Significant Event Audit in Primary Care: a Case Study

Grace Sweeney, Richard Westcott, & Jonathan Stead

Introduction

Learning from experience is an essential part of any good quality improvement system, and therefore a crucial element in Clinical Governance¹. There is regrettably a poor history of systematic learning from experiences – be they good or bad - within the NHS. A culture of blame, coupled with an understandable fear of punishment, continue to discourage this type of learning. But there is no reason why such a situation need be the case. Even now, and too often, individuals are held responsible for errors, while it is the systems that are usually at fault². Critical Incident Analysis – the forerunner of SEA* – was developed by an organisation no less hierarchical than the N.H.S., the US Air Force during World War II. In those early accounts of SEA, both good and bad experiences were discussed on a systematic basis to ensure that people with the right skills were recruited and retained as bomber crews³.

Three aspects of this early work can now be seen to be important for the successful implementation of SEA. Firstly, the incidents which were recorded were not just ‘bad’ events: the practice of SEA created a work-based forum for identifying, analysing, replicating and indeed celebrating successes – an activity not traditionally noted within the N.H.S., past or present. Secondly, the approach was systematic. Calling a meeting only when something goes wrong stresses the inquisitional attitude, from which SEA so signally diverges. A programme of regular scheduled meetings is far more effective at picking up early warnings and near misses as well as the equally important successes. In short, such an approach is proactive and positive rather than reactive and negative. Thirdly, the SEA process discovered a crucial link between recruitment, retention, and training and development. SEA enabled operational activity to be studied so that accurate job descriptions and person specifications could be generated, which in turn could inform selection procedures, preliminary training and assessments, and improve continuing training. Thus could the overall efficiency and effectiveness of the workforce, as well as morale, be enhanced.

Despite a tradition of examining and reviewing specific incidents and cases, sometimes even with an exploration of feelings in an appropriately supportive atmosphere, as exemplified by Balint’s work with general practitioners in the 1950s⁴, SEA was slow to be developed in British medical care. But with the evolution of training for general practice, which has benefited from Balint’s teaching, a fresh approach can be discerned. As the consultation in its own right assumed centre stage, a new way of analysing practice appeared whereby the group of learners / teachers were expected to define the good aspects of performance before they moved on to suggestions or even criticism⁵. Meanwhile, others re-discovered the Critical Incident Technique and its potential in medical education⁶ and in the development of reflective practice within the health care professions⁷. Performance review was encouraged by the Royal College of General Practitioners’ "What Sort of Doctor?"⁸ initiative with its individual case by case assessment of performance. The scene was set for the implementation of SEA across many areas of health care, from prescribing⁹. to pre-registration house officers¹⁰, culminating in a national conference on SEA in 1993.

Following this conference, Pringle and colleagues drew the threads together in their definitive study of the feasibility and potential of SEA in primary care in the mid 1990s¹¹ when they described the method, surveyed its use and compared it to conventional audit. They concluded that the technique’s strengths included the focus on outcomes, relevance to practice and practical problems, width of application and team building qualities. Amongst its drawbacks they identified possible superficiality, emotional demands on and threats to individuals, training requirements and challenges to the team. The importance, worth and relevance of SEA was indisputably established by this work.

For those who would want to build on these firm foundations, developing SEA and extending its application to as many practices as possible in the conviction that the technique addresses a substantial part of the Clinical Governance agenda, there are crucial questions. While some guidance in identifying the barriers and enablers to facilitating its optimal implementation is now available¹², original research on the benefits as perceived by the participants themselves, which will further help in both extending and sustaining it, is lacking. The present enquiry follows up the preliminary study by observing the implementation and process of SEA in practice, discovering from the participants themselves their perceptions of its impact against the background of their concerns, and the benefits gained.

Method

Study Design

Human behaviour is linked to the context in which it appears and would lose all meaning if divorced from this context. So we used a case study approach to observe a primary care practice over a period of 10 months, employing a Grounded Theory approach^13,14 to guide both data collection and analysis. Grounded theory has two meanings when associated with the seminal work of¹³. Firstly, it involves the notion of grounding theory in experiences, accounts and local contexts; it is a general methodology for developing theory that is grounded in data which has been systematically collected and analysed ¹⁵. Secondly, the term ‘grounded theory’ is used to describe a particular method which involves specific analytical strategies for handling and making sense of poorly structured qualitative data¹⁶.

All qualitative researchers acknowledge the important role of the researcher in shaping the final research product. The researcher’s contribution is not hidden, passive and impersonal: researcher skills are used to label concepts, create links and organise the data. The process recognises the human characteristics of those contributing data, and engages them as active participants in the research. In the present study, all data were collected and analysed by one of the authors (GS), who has a psychology background. The remaining authors (JS and RW) are both GPS, with interests in audit and education respectively.

Sampling

The study primary care team was recruited by identifying a team unconnected with the researchers and which, being unexposed to and untrained in SEA, had expressed an interest in being introduced to SEA. The research project was explained to, and permission granted by, members for the study to take place.

Four individuals from the core team were invited to participate in preliminary and post-SEA interviews. Sampling decisions were guided by Glaser and Strauss’ (1967) ‘theoretical sampling’ procedures, in that participants were selected (1) to represent

the opinions of different occupational groupings, including general practitioners, practice nursing, district nursing and administration, and (2) to illustrate differences between in-house staff (general practitioners, practice nurses, and administration) and community staff (district nursing). Individuals were recruited after the introductory session, and on the basis of the contribution that they made (or failed to make) to the initial meeting. It was not possible to interview the practice manager at the start of the study as her appointment was being made at the start of the study, but she was included in the post-SEA interviews.

Procedure

Data was collected in four sequential phases.

First, four members of the core team (receptionist, general practitioner, practice nurse, and district nurse) were interviewed at the beginning of the practice’s involvement in SEA, and after the initial meeting. The aim of the interview was to explore individuals’ expectations of the process of SEA, which was new to them. Interviews were conducted on a one-to-one basis, and each lasted between 30 and 40 minutes. Absolute confidentiality was assured and each interview was tape-recorded.

Second, six consecutive SEA meetings were observed over a period of 10 months, with the researcher assuming the role of non-participant observer. Field notes and memos were made both during and after the session, agendas and minutes were retained to supplement the researcher’s notes, and a list of attendees was recorded. The aims of this observation phase were to, (i) observe SEA in action, (ii) note the processes involved in reaching and agreeing solutions, (iii) keep an account of events ‘actioned’, and (iv) use the information as baseline from which to inform the post-SEA interviews.

Third, following the six observed SEA meetings, the four original core group members were re-interviewed, to examine participants’ perceptions in the light of their actual experiences. These interviews were more focused than the first, with the experience of practical issues now able to guide the line of questioning. As before the interviews were conducted on a one-to-one basis, confidentiality was assured and each interview was tape-recorded. In addition, the newly appointed practice manager was interviewed about her experiences of the process and its impact on the practice.

Finally, a ‘negotiated feedback’ session^13,14 was convened in the practice, with the aims of (i) feeding back the findings of the first three phases of the study to the research participants and to the larger group within the practice, and (ii) checking and clarifying the interpretations with the group. This session lasted for approximately one hour, with 15 minutes of this time devoted to formal feedback. The remaining 45 minutes was spent in group discussion, and this session was minute to provide further data for analysis.

Data Analysis

Analysis of the data was carried out in two separate stages.

Firstly, in line with the grounded theory framework used, each interview in the pre-SEA and post-SEA sections of the study was transcribed and then analysed separately and before the subsequent interview. Analytical strategies within grounded theory are presented to the researcher as ‘aids to analysis’ rather than as methodological straightjackets¹⁶. These strategies describe the systematic development of an open-

ended coding system, in which the analyst works rigorously through the data, in an attempt to generate codes that refer to both low-level concepts and more abstract categories. Analysis proceeds from data to outcome in only a loosely linear fashion; an important feature of grounded theory is the need to return to data sources, to check aspects of the emerging interpretations and to collect new data as and when appropriate. Different authors use slightly varying language to describe the analytical stages of the process, but all describe the process as sharing the same core features. Essentially analysis involves initial indexing (coding), developing categories, core analysis (extending categorisation, memo-writing, writing definitions, saturation of categories), and drawing theoretical outcomes. Data from the negotiated feedback session was used to verify interviewees’ perceptions, to check representativeness and to illustrate group perceptions.

Secondly, the data arising from the six observed meetings was subjected to a content analysis.

Results

We present the results of the study under three headings

(i) Preliminary interviews, (ii) Observation study, and (iii) Post-SEA interviews and feedback session.

(i) Preliminary Interviews

Three main findings emerged from these interviews to indicate that participants held a number of expectations of SEA, experienced concerns about the process, and offered suggestions for the successful implementation of the process (figure 1). The notion of SEA was novel to each of the interviewees, although the GP interviewee had been

exposed to the concept (but not the reality) of SEA before the groups commenced.

Expectations of SEA Concerns about SEA

Improved relationships Choosing issues

Problem-solving Leadership

Outcomes Structural barriers

Group dynamics

Lack of time

Poor outcomes

Suggestions for

Implementing SEA

Selection of Issues

Preparation

Outcomes

Leadership

Group Membership

Maintenance of other avenues

 

Figure 1. Preliminary Perceptions of SEA

Observation Study

The SEA meetings were consistently well attended, with between 20 and 22 members of staff attending each meeting (the team consisted of 50 individuals, all of whom had been invited). The meetings, chaired by one of the GP Partners (who put herself forward for the role), took place approximately every six weeks at lunchtime. Both surgery and community staff (health visitors, physiotherapists and district nurses)

were invited, although attendance at the meetings was voluntary. Lunch was always

provided. Agendas for the current meeting and minutes from the previous meeting were provided for everyone attending. The large size of the group caused some concern initially (both to the researcher and to group members), and in an attempt to find the most comfortable and productive group size, three of the SEA sessions split into two groups, whilst the remaining three worked as one group (see discussion).

Twenty-six separate issues were discussed over a period of 10 months at the six SEA meetings. Six of these issues had been presented jointly by individuals from more than one occupational group, one by the District Nurses as a group, and the remainder were presented by one individual. Table 1 indicates that GPs and receptionists were the most frequent presenters of events for discussion.

Discipline Frequency

General practitioners 15 (jointly or singly)

Receptionists 9 (jointly or singly)

District nurses 4 (singly or as a group)

Practice nurse 2 (jointly)

Physiotherapist 2 (singly)

Table 1. Frequency of presentation of significant events by members of different occupational groups

The focus of the significant events and therefore of the meetings was the day-to-day running of the practice and inter-disciplinary communication issues, although some events of a medical nature or of the clinical management of a case were discussed when these were perceived as having a direct bearing on the wider team and could be

used to facilitate group learning (table 2).

Significant Event Frequency

Day-to-day administration 10

Diagnosis/clinical management 4

Patients ‘hassling’ staff 4

Management of clinical emergency 3

Miscommunication 3

Patients rude to staff 1

Extra patients policy 1

Table 2. Nature and Frequency of Significant Events

As a result of discussing these 26 issues, 62 separate solutions with agreed actions were reached at the meetings. At the end of the period of data collection, 21 of these solutions had been actioned, no feedback was available on 16 of the solutions, 3 had not been actioned after all, 3 were ‘on hold’, and one was proving complicated and required further discussion. We have grouped the solutions/actions into Pringle’s 4 outcomes (table 3), but suggest a fifth category – see below.

Category Frequency

Immediate action 33

Conventional audit 3

Life’s like that 2

Congratulations 1

(Further work needed 23) (See discussion below)

Table 3. Outcomes in terms of Pringle’s categories

(iii) Post-SEA Interviews and Negotiated Feedback Session

Three findings emerged from the combined results of the post-SEA interviews and the negotiated feedback session. Figure 2 indicates that interviewees shared positive perceptions of the benefits of the process (‘The Reality of SEA’), that they still had some concerns about the process (‘Ongoing Concerns’), and that, in the light of experience, they were able to offer suggestions for the maintenance of the process.

The Reality of SEA Ongoing Concerns

(1) Improved relationships (1) Shortage of time

(2) Reassurance (2) Frequency & timing

(3) Feeling of safety (3) Appropriateness of issues/events

(4) Participation in decision-making (4) ‘Naming and shaming’

(5) Small but important changes (5) Outcomes difficult to implement & measure

(6) Reaches solutions & actions (6) Group size

(7) Feeling of working together (7) Confidentiality & privacy

(8) Shortage of congratulations

(9) Individuals dominating meeting

Maintaining the Process

(1) A healthy and varied agenda

(2) Monitor the overall process every six months

(3) Rotate the day to benefit part-timers

(4) Defer some decisions to Practice Meeting

(5) Summarise actions at end of meeting

(6) Increase number of congratulations

(7) Reconsider venue & group size

Figure 2. Perceptions & Experiences Post-SEA

Discussion of Results

The study shows that SEA can attract, and continue to attract, primary care workers from a wide variety of backgrounds. Numbers attending remained constant, the activity was well received, all groups actively contributed, and interpersonal relationships, including reassurance, feelings of safety and participation, were all improved.

Participants’ expectations of problems being solved, with definite outcomes, were met – indeed 26 issues generated 62 separate solutions with agreed actions. After practising SEA, members felt that solutions had been agreed, actions taken and important (even if sometimes small) changes made. But not all outcomes could be implemented, or measured. A substantial number (23) failed to fit comfortably into the standard 4 categories, and we suggest that a fifth category would be helpful – "further work needed". SEA has a potent capacity to identify areas for quality improvement. This category recognises that, in practice, teams sometimes need to ask one or more members to take the opportunity to think more carefully over the particular problem, and bring it back for further discussion and suggestions for improvement.

Concern about the selection of issues remained. Clearly the choice of events is critical: they need to have relevance to the team and engage participants individually, but also to respect personal boundaries and represent material that the team feels able to tackle. Thus leadership – in protecting as well as driving – continues to represent a critical part of successful SEA. Optimal leadership remains unclear. Earlier work (Pringle et al, 1995) suggested external facilitation, but this study suggests that it is possible to provide sensitive yet effective leadership from within the practice.

However, various leadership issues remained – members were still worried about individuals who dominate, confidentiality, the risk of ‘naming and shaming’ and achieving and demonstrating outcomes. Discussion at the feedback session concluded that while the leader need not necessarily be a partner, or even a ‘senior’ partner, the team chose to stay with the GP who volunteered herself for the role. With Clinical Governance contributing further challenges under this heading, the topic of leadership in primary care poses difficult questions beyond, as well as within, the immediate area of SEA. Further work is called for.

In general, hierarchical and structural barriers proved less inhibiting than feared, with evidence of receptionists making active contributions, a general awareness of the presence of a safety net of non-judgemental reassurance and a feeling of participation in all groups. The expectation of improved relationships was fulfilled, but the need to congratulate was underlined. Identifying, acknowledging and celebrating achievements represents one way of moving from traditional attitudes to unacceptable events, system failures and even personal shortcomings, towards mutual support and trust.

Some practical and administrative questions also remain.

What is the optimal size of the group? This question will become more important as the imminent roll out of SEA by Clinical Governance leads draws in the full spectrum of practice teams, ranging from single handed practices to very large groups. The present team (a large practice with three branch surgeries, 8 partners plus 1 retained doctor, 13,000 patients and a team of 50 individuals) produced over 20 participants. They experimented with splitting, but eventually decided to meet as one group, convinced that the overall identity and esprit de corps outweighed the inevitable difficulties of greater numbers.

What is the relationship between the SEA meetings and partnership and other meetings within the practice? Pressures on time, establishing the optimal frequency and timing of the meetings, determining the agenda and striking the right balance between involving different groups and spreading ownership while maintaining efficient management all call for careful planning and review.

Venue and time of the meeting also need to be widely acceptable in order to maintain the process.

While there can be no standard answer to these questions, these issues need to be decided upon and not underestimated, since they reflect relationships, priorities and underlying philosophies. The group recommended that a regular six month review of the overall process should take place, but how, by whom and when is another challenge which will need addressing, along with how much information and in what form the team will be prepared to share with clinical governance leads.

Conclusions

Assessing the expectations of members of a primary care team before undertaking SEA, observing them pursuing the activity and interviewing them with a negotiated feedback session after ten months has provided the opportunity to assess the impact of SEA, particularly in terms of the benefits achieved as seen by the participants.

Confirming earlier research, this study shows that the overall experience was found useful in team building, and generating worthwhile improvements in patient care, so that SEA was well supported.

But we have identified fears concerning personal blaming, appropriateness of issues and events, difficulty at times in implementing and measuring outcomes, and worries about confidentiality and privacy issues, as well as continuing questions on administrative aspects.

However, convinced of its benefits, the team has now adopted SEA as an integral part of its life and work. Participants made some practical suggestions for maintaining the process to include part-timers, reinforced certain aspects of SEA (particularly the need to summarise action points) and stressed the need to find cause for congratulation.

Acknowledgements

The authors are grateful to Somerset and North Devon Education Purchasing Consortium for funding this project. Wyndham House Surgery is an NHS Research and Development funded research practice.

References

1 Scally G., & Donaldson LJ. Clinical Governance and the drive for quality in the new NHS in England. British Medical Journal 1998; 317: 61-63.

2 Berwick, DM. A primer on leading the improvement of systems. British Medical Journal 1996; 312: 619-622.

3 Flanagan JC. The Critical Incident Technique. Psychological Bulletin 1954; 51: 327-58.

4 Balint M (1964) The Doctor, His Patient and the Illness. Tunbridge Wells: Pitman Medical

5 Pendleton D, Schofield T, Tate P et al. The Consultation: An Approach to Learning and Teaching. Oxford: Oxford Medical Publications, 1984.

6 Dunn and Hamilton The Critical Incident Technique: a brief guide. Medical Teacher 1986; 8: 207-215

7 Schon D A. Educating the Reflective Practitioner: Toward a New Design for Teaching and Learning in the Professions. San Francisco: Jossey-Bass Inc1987.

8 Royal College of General Practitioners. What Sort of Doctor? Assessing the Quality of Care in General Practice. Report from General Practice 23. London: Royal College of General Practitioners 1985.

9 Bradley C. Turning anecdotes into data: the critical incident technique. Family Practice 1992; 9: 98-103

10 Calman R, Donaldson M. The pre-registration officer: a critical incident study. Medical Education 1991; 25: 51-59

11 Pringle M, Bradley CP, Carmichael CM, Wallis H and Moore A. Significant Event Auditing. RCGP Occasional Paper 70. Exeter: Royal College of General Practitioners 1995.

12 Westcott, R., Sweeney, G., & Stead, J., (in press) Significant event audit in practice; A preliminary study. Family Practice, October 1999.

13 Glaser BG., & Strauss AL., (1967) The Discovery of Grounded Theory: Strategies for Qualitative Research. Chicago: Aldine.

14 Strauss AL., & Corbin J., (1990) Basics of Qualitative Research: Grounded Theory Procedures and Techniques. Newbury Park: CA: Sage Publications Ltd.

15 Strauss A & Corbin J., (1994) Grounded theory methodology: an overview. In NK Denzin & YS Lincoln (eds) Handbook of Qualitative Research. London: Sage Publications Ltd., 273-285.

16 Henwood K & Pidgeon N. Grounded theory and psychological research. The Psychologist 1995; 8(3): 115-118.