BI System Effectiveness

Developing the measurement instrument and variables

The measurement instrument comprised two parts: BISE and enterprise characteristics. Table 1 lists the variables of the measurement instrument.

Table 1 Research variables of BISE and characteristics of enterprise

Previous studies on the evaluation attributes of BI benefits were diverse. The study defined BISE as performance when implementing BI solutions, including the effectiveness with which enterprises use information to improve and optimize business processes and decision support systems to derive multiple benefits of BI. The measurement items of BISE were based on viewpoints including the arguments of Watson et al. and Işık et al. that BI comprises both technical and organizational elements and that of Laursen and Thorlund who noted that the BI system has three elements: a technological element that collects, stores and analyzes data, as well as delivering information; a human competencies element dealing with human abilities to retrieve data and generate reports, generate knowledge and make decisions; and a third element that supports specific business processes for increasing business values. The work of developing measurement items comprised two steps: (1) integrating the attributes of BISE with related studies [e.g., Gessner and Volonino; Microsoft; Bitech etc.]; (2) based on the results of step 1, interviewing and discussing with two senior BI managers in the information divisions of Taiwanese banks to determine final measurement attributes and items of BISE. Table 1 lists three measurement attributes and 16 measurement items as research variables. Responses to each variable were assessed on a 5-point Likert scale, ranging from 1, indicating "strongly disagree", to 5, indicating "strongly agree".

Table 1 also lists the characteristics of enterprises surveyed in this study on whether or not the surveyed characteristics influence BISE. Enterprise characteristics were developed based on the perspectives of Rogers and Laudon and Laudon, and the research variables included industry category, company capital, number of years the business has been established, total number of employees, number of employees in the information department and number of years of enterprise having implemented BI solutions.

Collection of raw data

A representative sample of 294 participants in the Taiwan financial services industry, including banks, insurance companies, bills finance corporations, securities firms, trust business companies, and investment companies, all listed in the Taiwan Business Directory, were selected as research participants. Respondents were general managers or executives in information divisions. Before questionnaire distribution, two chief information officers reviewed the wording of questionnaire items. One questionnaire was distributed to each firm by mail or e-mail. A total of 294 questionnaires were distributed and 77 valid questionnaires were returned for a valid return rate of 26.2 %.

Respondents were followed up after 3 weeks to increase the response rate. This study also analyzed non-response bias. Responses were divided into two groups, the initial response group, and the follow-up response group. The Chi square test was applied to both groups to identify differences in the six variables representing enterprise characteristics - industry category, company capital, number of years the business has been established, total number of employees, number of employees in the information department, and number of years of the enterprise having implemented BI solutions. The test p-values were >0.05 at 0.172–0.995, indicating no significant difference between these two data groups.

Representativeness of samples

For the six industry characteristics, banks, insurance companies, bills finance corporations, security firms, trust business companies and investment companies, respectively, the numbers of population were 73, 51, 136, 13, 3 and 18; and the numbers of the returned valid samples were 23 (31.5 %), 13 (25.5 %), 25 (18.4 %), 4 (30.8 %), 3 (100.0 %) and 9 (50.0 %). This study further adopted Chi square test to examine the representativeness of the samples. The test p value was >0.05 at 0.106, indicating that no significant differences existed between these two groups (population and the returned valid samples), and existed sufficient sample sizes to achieve adequate representation.

The sample profile was such that participants were predominantly from the banking 29.9 % (n = 23) and securities industries 32.5 % (n = 25). Roughly 24.7 % (n = 19) of companies had capital reserves of NT$10–30 billion. Firms with 101–300 employees accounted for 28.6 % (n = 22) of the sample. Approximately 58.4 % of firms (n = 45) had 1–30 employees in their information departments, and 41.6 % (n = 32) had been established for 10–20 years. Finally, 23.4 % (n = 18) of firms had implemented BI solutions for 1–3 years; while 57.2 % (n = 44) had done so for >3 years.

Reliability and validity analysis

Item analysis identified measurement items that deserved to be retained versus those that needed to be revised or discarded. This study applied the t-test to two extreme groups, namely the highest and lowest scoring groups, using the internal consistency criterion. All measurement items of BISE had p-values of 0.000 (<0.05), indicating adequate discrimination and clarity.

The Cronbach's α value for BISE was 0.955, and did not increase even after item exclusion. The item-to-total correlation of measurement items was in the range 0.650–0.831 (see Table 2), and the criterion of 0.35 was seen as an acceptable corrected item-total correlation, indicating the scale exhibited satisfactory reliability.

Table 2 Results of reliability analysis and exploratory factor analysis

Validity is the ability of a scale to measure what it is intended to measure, and to measure its accuracy in identifying key content characteristics. This study adopted exploratory factor analysis to assess the scale of BISE to verify its suitability for this investigation. Table 2 lists the analytical results. The Bartlett's test of sphericity and the Kaiser–Mayer-Olkin (KMO) measure of sampling adequacy were first conducted to confirm the appropriateness of the sample data for factor analysis. Kaiser indicated that a KMO in the range 0.80–0.89 is meritorious. Since the factors were obtained through principle component analysis, the standards for factor selection were an eigenvalue exceeding one and a factor loading larger than 0.5 following varimax rotation. Analytical results for BISE demonstrated that the KMO was 0.892, and the Bartlett's test of sphericity had a p value of 0.000 (<0.05). The cumulative variance explained was 69.224 %. Two factors were extracted, namely the business information management effectiveness of BI and the decision support effectiveness of BI. All 16 measurement items with communality were in the range 0.593–0.773 (>0.5) and factor loadings were in the range 0.585–0.842 (>0.5), indicating satisfactory construct validity.