Performance Measurement: Indicators and Metrics

The final sample of 76 papers consists of 46 journal papers and 30 conference papers (Fig. 4), indicating a wide variety of outlets to reach the audience via operations and production-related journals in particular or in lower-ranked information systems journals.

Fig. 4 The distribution of the sampled papers per publication type (N = 76)

When considering the chronological distribution of the sampled papers, Fig. 5 indicates an increase in the uptake of the topic in recent years, particularly for conference papers but also for journal publications since 2005.

Fig. 5 The chronological distribution of the sampled papers per publication type (N = 76)

This uptake seems particularly situated in the Western world and Asia (Fig. 6). The countries with five or more papers in our sample are Germany (12 papers), the US (6 papers), Spain (5 papers), Croatia (5 papers) and China (5 papers). Figure 6 shows that business process performance measurement is a worldwide topic, with papers across the different continents. Nonetheless, a possible explanation for the higher coverage in the Western world could be due to its long tradition of measuring work (i.e., BSC origins).

Fig. 6 The geographical distribution of the sampled papers per continent, based on a paper's first author (N = 76)

The vast majority of the sampled papers address artifacts related to business (process) performance measurement. When looking at the research paradigm in which the papers are situated (Fig. 7), 71 % address design-science research, whereas 17 % conduct research in behavioral science and 12 % present a literature review. This could be another explanation for the increasing uptake in the Western world, as many design-science researchers are from Europe or North America.

Fig. 7 The distribution of the sampled journal papers per research paradigm (N = 76)

Figure 8 supplements Fig. 7 by specifying the research methods used in the papers. For the behavioral-science papers, case studies and surveys are equally used. The 54 papers that are situated within the design-science paradigm explicitly refer to models, meta-models, frameworks, methods and/or tools. When mapping these 54 papers to the four artifact types of March and Smith, the vast majority present (1) methods in the sense of steps to perform a task (e.g., algorithms or guidelines for performance measurement) and/or (2) models to describe solutions for the topic. The number of papers dealing with (3) constructs or a vocabulary and/or (4) instantiations or tools is much more limited, with 14 construct-related papers and 9 instantiations in our sample. We also looked at which evaluation methods, defined by Peffers et al., are typically used in the sampled design-science papers. While 7 of the 54 design-science papers do not seem to report on any evaluation effort, our sample confirms that most papers apply one or another evaluation method. Case studies and illustrative scenarios appear to be the most frequently used methods to evaluate design-science research on business (process) performance measurement.

Fig. 8 The distribution of the sampled journal papers per research method (N = 76)

The sampled design-science research papers typically build and test performance measurement frameworks, systems or models or suggest meta-models and generic templates to integrate performance indicators into the process models of an organization. Such papers can focus on the process level, organizational level or even cross-organizational level. Nonetheless, the indicators mentioned in those papers are illustrative rather than comprehensive. An all-inclusive list of generic performance indicators seems to be missing. Some authors propose a set of indicators, but those indicators are specific to a certain domain or sector instead of being generic. For instance, Table 4 shows that 36 of the 76 sampled papers are dedicated to a specific domain or sector, such as technology-related aspects or supply chain management.

Table 4 The number of sampled papers dedicated to a specific domain or sector (N = 76)

Domain or sector	Number of papers
IS/IT	7
Supply chain	5
Business network	3
Manufacturing	3
Services	3
Automobile	2
Banking/financial	2
Government	2
Health	2
Helpdesk/maintenance	2
Construction	1
HR	1
SME	1
Strategic planning	1
Telecom	1
Total	36

Furthermore, the reviewed literature was analyzed with regard to its (1) scope, (2) functionalities, (3) terminology, and (4) foundations.

Starting with scope, it is observed that nearly two-thirds of the sampled papers can be categorized as dealing with process-oriented performance measurement, whereas one-third focuses more on general performance measurement and management issues. Nonetheless, most of the studies of process performance also include general performance measurement as a supporting concept. A minor cluster of eight research papers specifically focuses on business process reengineering and measurement systems to evaluate the results of reengineering efforts. Furthermore, other researchers focus on the measurement and assessment of interoperability issues and supply chain management measurements.

Secondly, while analyzing the literature, two groups of papers were identified based on their functionalities: (1) focusing on performance measurement systems or frameworks, and (2) focusing on certain performance indicators and their categorization. Regarding the first group, it should be mentioned that while the process of building or developing a performance measurement system (PMS) or framework is well-researched, only a small number of papers explicitly address process performance measurement systems (PPMS). The papers in this first group typically suggest concrete steps or stages to be followed by particular organizations or discuss the conceptual characteristics and design of a performance measurement system. Regarding the second group of performance indicators, we can differentiate two sub-groups. Some authors focus on the process of defining performance indicators by listing requirements or quality characteristics that an indicator should meet. However, many more authors are interested in integrating performance indicators into the process models or the whole architecture of an organization, and they suggest concrete solutions to do so. Compared to the first group of papers, this second group deals more with the categorization of performance indicators into domains (financial/non-financial, lag/lead, external/internal, BSC dimensions) or levels (strategic, tactical, operational).

Thirdly, regarding terminology, different terms are used by different authors to discuss performance measurement. Performance "indicator" is the most commonly used term among the reviewed papers. For instance, it is frequently used in reference to a key performance indicator (KPI), a KPI area or a performance indicator (PI). The concept of a process performance indicator (PPI) is also used, mainly in the process-oriented literature. Performance "measure" is another prevalent term in the papers. The least-used term is performance "metric" (i.e., in only nine papers). Although the concepts of performance indicators, measures and metrics are used interchangeably throughout most of the papers, the concepts are sometimes defined in different ways. For instance, paper 17 defines a performance indicator as a metric, and paper 49 defines a performance measure as an indicator. On the other hand, paper 7 defines a performance indicator as a set of measures. Yet another perspective is taken in paper 74, which defines a performance measure as "a description of something that can be directly measured (e.g., number of reworks per day)", while defining a performance indicator as "a description of something that is calculated from performance measures (e.g., percentage reworks per day per direct employee" (p. 386). Inconsistencies exist not only in defining indicators but also in describing performance goals. For instance, some authors include a sign (e.g., minus or plus) or a verb (e.g., decrease or increase) in front of an indicator. Other authors attempt to describe performance goals in a SMART way - for instance, by including a time indication (e.g., "within a certain period") and/or target (e.g., "5 % of all orders") - whereas most of the authors are less precise. Hence, a great degree of ambiguity exists in the formulation of performance objectives among to the reviewed papers.

Fig. 9 The importance of the BSC according to the sampled papers (N = 76)
Interestingly, the BSC is also criticized by the sampled papers for not being comprehensive; for instance, due to the exclusion of environmental aspects, supply chain management aspects or cross-organizational processes. In response, some of the sampled papers also define sector-specific BSC indicators or suggest additional steps or indicators to make the process or business more sustainable (see Table 4). Nonetheless, the majority of the papers agree on the need for integrated and multidimensional measurement systems, such as the BSC, and on the importance of directly linking performance measurement to an organization's strategy. However, while these papers mention the required link with strategy, the prioritization of indicators according to their strategic importance has been studied very little thus far.