2010 International Conference on E-Business and E-Government

A New Framework on Website Evaluation

Shanshan Qi
School of Hotel and Tourism Management The Hong Kong Polytechnic University Hung Hom, Hong Kong shan.qi@polyu.edu.hk

Crystal Ip
School of Hotel and Tourism Management The Hong Kong Polytechnic University Hung Hom, Hong Kong hmcrystal.ip@polyu.edu.hk

Rosanna Leung
School of Hotel and Tourism Management The Hong Kong Polytechnic University Hung Hom, Hong Kong rosanna.leung@polyu.edu.hk

Rob Law
School of Hotel and Tourism Management The Hong Kong Polytechnic University Hung Hom, Hong Kong hmroblaw@polyu.edu.hk

AbstractInformation Communication Technologies (ICTs) have changed how business is conducted and consumers purchase behavior. Online market has showed its importance to both academia and business circles. This paper analyses published website evaluation related studies and found that the existing ecommerce literature simply does not have any commonly agreedupon standards for evaluating overall website performance. Therefore, this research proposes a novel direction to website evaluation. A new framework has been established which suggests website assessment should be performed in three major aspects: website usefulness (technique), service quality, and physical accessibility. The expected contributions of the proposed model to academics and practitioners are discussed. Keywords: website evaluation, website usefulness, website service, , website accessibility

I.

INTRODUCTION

In the past 20 years, Information Communication Technologies (ICTs) have changed and globalized the worlds business [1, 2]. The online market shows its importance to both academia and business circles. In particular, the Internet helps practitioners to market their products and serve worldwide online customers. It also offers a convenient medium for information sharing and provides electronic communications [3]. Convenience and low cost are two major reasons for consumers to purchase online [4]. The global ecommerce income has reached to US$12.8 trillion, taking up 18% of the global trade of commodities in year 2006 [5]. Online market thus represents as a potential market on global business and it will grow larger in the near future. In order to remain competitive, many practitioners have established websites to attract customers [2]. Online market has also attracted academic researchers attention. In the existing literature, many research studies have been focused on evaluating or assessing website performance or quality that assists practitioners on enhancing their websites [6, 7, 8]. The concept of e-commerce not only performs well in technical part, but also raises the awareness of the importance of online communications and customer satisfaction as indicated by [9]. Additionally, [10] also stated that website navigation, structure and design are generally technical in nature, and website personality appearance and multimedia performance are used to test consumer perceptions
978-0-7695-3997-3/10 $26.00 2010 IEEE DOI 10.1109/ICEE.2010.27 78

and satisfactions. Consumers pay more attention on products ease-of-use. As such, their acceptability of a product largely depends on the ease-of-use, physical, mental and psychological characteristics than the technical aspect of the product [11]. Previous website evaluation studies have integrated different evaluation methods with evaluation measurements which represent website performance in different aspects. However, results of these studies were vague on indicating website performance in a certain aspect. For example, many studies integrated website design, information quality, ease-of-use, reliability, security, accessibility, and service quality together [12, 13, 14, 15, 16, 17, 18]. These studies tracked website performance in many aspects but did not specify which specific aspect/s had been tested. In other words, these studies cannot fully represent a website quality on either technique or service aspect. In view of these shortcomings, this paper proposes a new evaluation approach that suggests website evaluation should be conducted in three different aspects of: website usefulness (technique), website service quality, and website physical accessibility. These three evaluation aspects are defined after an extensive literature review. II. REVIEW OF WEBSITE EVALUATION

In the present study, three well-known academic online databases, Science Direct (http://www.sciencedirect.com), EBSCOHost (http://search.ebscohost.com), and Google Scholar (http://scholar.google.com) were selected to search for relevant studies. The searching process was based on two major principles: i) the study proposed a website evaluation model or method; and ii) the study indicated certain website evaluation attributes. The searching keywords included web site/website quality, web site/website evaluation, web site/website assessment, and web site/website measurement. At the end of the database search, more than 100 published articles were found. Among these papers, 40 were determined as the most suitable ones and were selected. These papers were then categorized into three different aspects of: website usefulness, website service quality, and website physical accessibility.

A. Website Usefulness Attracting consumers attention is the first important criterion for an online channel [19, 20, 21]. Website playfulness has a positive impact on individual attitude which affects consumer intention to use online retailing [22]. Consumers perceived website usefulness can affect online trust and continued behavior of using a website as indicated by [19]. In this study, website usefulness represents the website technical aspect which comprises functionality and usability. While functionality refers to contents of a website, usability involves the website design or ease-of-use [23]. Website usability consists of four sub-factors: website navigability which was how consumers surf and go back and forth between different pages [24]. In order to better present on the website navigability factor, website designers should consider presenting a site map and keyword search engine for visitors [3]. Ease of use reflected how consumers can easily learn to use a website. To consumers, ease of use means a clear system which is easy to be understood, skilled and controllable [22]. Website layout and design determined a websites visual appearance [25, 26]. A website should represent a simple and organized layout and structure, and be user-friendly. The design principle indicated by [19] for website interface design includes attractiveness, organization and, proper use of fonts, color and multimedia. Website playfulness was related to how a website entertains consumers [27]. Website functionality contained that factor of Content of website that is related to information quality. Website information should useful, accurate, timeliness, and reliable [22]. The content of a website should also be easy to understand, fit the website objectives and represents an appropriate detail [19, 28]. Website flexibility reflected how flexible a website is. It depends on the options of online payment channels, searching and returning products functions [17]. Many studies indicated the main concept of website accessibility is the speed which shows a web pages download speed, and opening and respond time of a website [24, 29, 30]. Website accessibility can be measured by whether the web URL which can be found on search engines as argued by [3]. Therefore, in this research website accessibility defined as how easy and fast it is for a customer to reach and access a website. B. Website Service Quality E-service quality is the extent to which a website facilitates effective shopping purchasing and delivery [25:11]. Online service is very important to a website. When consumers feel satisfied, they will be happy to stay and use the website again [31]. Furthermore, a consumer loyalty model for mobile consumers was established by [32]. Their findings indicated that consumers perceived value and satisfaction were the significant predictors of customer loyalty. Therefore, many evaluation methods have been developed on determining the quality of online service. These methods mainly based on consumer perceptions or evaluations of service quality [33]. A Multi-criteria Satisfaction Analysis (MUSA) model was adopted by [10] to measure user perceived cell phone online

service quality. [34] and [2] adopted WebQual to determine a consumers perceptions and [35] used a multiple item to examine online book store quality. A Web Assessment Index (WAI) includes accessibility, speed, navigability, and content as developed by [16]. The SERVQUAL model was modified and employed by many studies on evaluating online service, such as e-service quality on library and online recruitment websites [36, 9]. Additionally, [37] adopted the fuzzy theory to evaluate the quality of hospital websites respectively. An automatic system called Support Vector Machine (SVM) was developed by [38] to analyze consumers complaints automatically. In this study, website service quality has been categorized into two sub-factors of: e-marketing strategies and website reliability. E-marketing strategies showed websites effectiveness in winning consumers. It included different measurements such as website customization that is related to how a website attracts customer in catching their preferences, histories and ways of shopping [25, 34]. Website responsiveness was how easy a consumer can contact a websites providers and how fast the website responds to this customer [9]. Consumer satisfaction showed a websites communication ability in understanding consumers needs [30, 30, 28]. In addition, reliability factor included two measurements: trust reflected consumers acceptance on websites. That is, providing professionals and fairness services will increase consumers trust online [19]. Security showed how well a website protects consumers in the online environment especially on online payment. C. Website Physical Accessibility The website physical accessibility is W3C/WAI web content accessibility guidelines (WCAG) which is different from the discussed web accessibility. WCAG listed the online requirements of people with disabilities. There are limited studies focuses on assessing website physical accessibility. For instance, [39] established an accessibility evaluation model named MAGENTA (Multi-Analysis of Guidelines by an Enhanced Tool for Accessibility) and [40] developed an EARL (Evaluation and Report Language) model to test and enhance a websites physical accessibility. A multi-method was adopted by [41] which invited users with disabilities to evaluate e-government websites. III. PROPOSED FRAMEWORK Figure 1 shows the proposed framework for website evaluation. It indicated three main evaluation dimensions of: usefulness represents technique performance, service quality shows online service performance, and physical accessibility is related to special service to disabled people. The purpose of this evaluation is to test website quality and performance in order to advance the selected websites with certain objectives. Therefore, this research suggests website assessment should consider using this framework to identify detailed website problem in different aspects.

79

direction and suggests for further research on website evaluation which needs to specify on detailed aspects. This study made an initial attempt to propose a new website evaluation framework (or direction). Since it is still at a beginning stage, there are many unanswered research questions, which surely deserve efforts from future research. Also, the small sample size with 40 studies makes this study uneasy to generalize its findings. A future study could further expand the sample size by providing a more comprehensive review of the extent of the content of this framework. Future work can also combine different evaluation methods with this framework and involve evaluators to perform a real website evaluation. REFERENCES H. H. Bauer, T. Falk, and M. Hammerschmidt, eTransQual: A transaction process-based approach for capturing service quality in online shopping, Journal of Business Research, vol. 59, 2006, pp. 866-875. [2] H Li, and R. Suomi, Evaluating Electronic Service Quality: a Transaction Process Based Evaluation Model. The Proceedings of European Conference on Information Management and Evaluation (ECIME), Montpellier, France, 2007, Dan (editor), pp.331-340. [3] M. F. J. Gonzlez and B. T M. Palacios, Quantitative evaluation of commercial web sites: an empirical study of Spanish firms, International Journal of Information Management, vol. 24, 2004, pp. 313-328. [4] B. Yang, D. Lester and S. James, Attitude toward buying online as predictors of shopping online for British and American respondents, Cyber Psychology & Behavior, vol.10, 2007, pp.198-203. [5] Annual Report on the Development of Global ECommerce Industry, 2007; http://data.chinabyte.com/ywbgjd/342/3054842.shtml (Accessed on May 7, 2009). [6] X. Luo and M. Seyedian, Contextualmarketing and customer-orientation strategy for e-commerce: An empirical analysis, International Journal of Electronic Commerce, vol.8, 2003, pp.95118. [7] V. Moustakis L. Tsironis and C. Litos, A model of website quality assessment, The Quality Management Journal, vol.13, 2006, pp. 2237. [8] B. P. Yen, J.-H. Hu, and M. Wang, Toward an analytical approach for effective Website design: A framework for modeling, evaluation and enhancement, Electronic Commerce Research and Applications, vol.6, 2007, pp.159-170. [9] J. P. C. Tong, V. G. Duffy, G. W. Cross, F. Tsung and B. P. C. Yen, Evaluating the industrial ergonomics of service quality for online recruitment websites, International Journal of Industrial Ergonomics, vol.35, 2005, pp. 697-711. [10] E. Grioroudis, C. Litos, V. A. Moustakis, Y. Politis, and L. Tsironis, The assessment of User-perceived web quality: Application of a satisfaction benchmarking [1]

Figure 1. The framework for website evaluation

IV.

RESEARCH SIGNIFICANCE

To date, there have been very limited studies which make attempt to evaluate website performance from technical appearance. Most of the previous studies have not distinguished the measurements regarding technology and service aspects. These approaches may not be adequate to measure website performance from both users perceptions and website technical appearance. The existing e-commerce literature simply does not have any commonly agreed-upon standards for website evaluation. Thus, this research classifies evaluation factors into three aspects: technique, service, and physical accessibility. The proposed new framework is expected to benefit practitioners and researchers from adopting a novel direction on website evaluation. For practitioners, this framework can serve as an evaluation guide to help them identify the strengths and weaknesses of their websites. It also helps them improve their websites with certain objectives. For example, before a newly established website is published online, this framework could be applied to assess the website overall performance. Otherwise, a website has perfect functional design but its online transaction completion rate is very low, leading to the problem of poor usability and online service quality. The web owner can apply these two aspects to perform the evaluation. More importantly, this framework incorporates website physical accessibility into the framework. [39, 40] stated that website accessibility is especially important to public websites which serves all kinds of customers including people with disabilities. Therefore, the public websites ought to provide a high standard and qualified physical accessibility performance. For academic researchers, as previously stated, this framework is to provide a new

80

[11] [12]

[13]

[14]

[15]

[16]

[17] [18]

[19] [20]

[21] [22] [23]

[24] [25]

[26]

approach, Eurpean Journal of Operational Research, vol. 143, 2007, pp.148-170. H. Kanis, Usage centered research for everyday product design, Applied Ergonomics, vol. 29, 1998, pp. 7582. J. R. Evans and V. E. King, Business-to-business marketing and the World Wide Web: Planning, managing and assessing web sites, Industrial Marketing Management. vol. 28, 1999, pp.343358. J. Cox, and B. G. Dale, Key quality factors in Web site design and use: an examination, International Journal of Quality & Reliability Management, vol. 19, 2002, pp. 862-888. V. Hausman and J. S. Siekpe, The effect of web interface feature on consumer online purchase intentions, Journal of Business Research, vol. 62, 2009, pp. 5-13. W-J. Deng and W. Pei, Fuzzy neural based importanceperformance analysis for determining critical service attributes, Expert System with Applications. In Press, Corrected Proof. Y.Lee, and K.A.Kozar, Investigating the effect of website quality on e-business success: An analytic hierarchy process (AHP) approach, Decision Support System, vol. 42, 2006, pp.1383-1401. J. W. Palmer, Web Site Usability, Design and Performance Metrics, Information System Research, vol.13, 2002, pp.151-167. Y. Benslimane, and Z. Yang, "Linking commercial website functions to perceived usefulness: A free disposal hull approach," Mathematical and Computer Modeling, vol. 46, 2006, pp. 1191-1201. C. Liao, P. Palvia, and H-N. Lin, The roles of habit and web site quality in e-commerce, International Journal of Information Management, vol. 26, 2006, pp. 469-483. F. D. Davis, R. P. Bagozzi, and P. R. Warshaw, User Acceptance of Computer Technology: A Comparison of Two Theoretical Models, Management Science, vol. 35, 1989, pp. 982-1003. A. Bhattacherjee, Understanding information systems continuance: An expectation-confirmation model, MIS Quarterly, vol. 25, 2001, pp. 351-370. T. Ahn, S. Ryu, and I. Han, The impact of Web Quality and Playfulness on user acceptance of online retailing, Information & Management, vol. 44, 2007, pp. 263-275. M. T. Lu and W. L. Yeung, A framework for effective commercial web application development, Internet Research: Electronic Networking Applications and Policy, vol., 8, 1998, pp.166-173. K. Waite, Task Scenario Effects on Bank Web Site Expectations, Internet Research, vol.16, 2006, pp.7-22. V. A. Zeithaml, Service quality, profitability, and the economic worth of customers: what we know and what we need to learn, Journal of the Academy of Marketing Science, vol.28, 2000, pp.67-85. B. Bui and T. Vorburger, "Surface metrology algorithm testing system," Precision Engineering, vol. 30, 2007, 218-225.

[27] S. Kim, and L. Stoel, Dimensional Hierarchy of Retail Website Quality, Information and Management, vol. 4, 2004, pp.619-633. [28] S. J. Barnes, and R. T. Vidgen, Data Triangulation and Web Quality Metrics: A Case Study in E-Government, Information and Management, vol. 43, 2006, pp. 767777. [29] M. Jun and S. Cai, The key determinants of Internet banking service quality: A content analysis, International Journal of Bank Marketing, vol. 19, 2001, pp.276-291. [30] C. Jayawardhena, Measurement of Service Quality in Internet Banking: The Development of an Instrument, Journal of Marketing Management, vol. 20, 2004, pp.185-207. [31] J. E. Collier and C. C. Bienstock, Measuring Service Quality in E-Retailing, Journal of Service Research, vol. 8, 2006, pp. 260-275. [32] H. Lin and Y. Wang, An Examination of the Determinants of Customer Loyalty in Mobile Commerce Contexts, Information and Management, vol. 43, 2006, pp.271-282. [33] M. Kim, J-H, Kim, and S. J. Lennon, Online Service Attributes Available on Apparel Retail Web Sites: An ES-QUAL approach, Managing Service Quality, vol. 16, 2006, pp.51-77. [34] S. J. Barnes, and R.T. Vidgen, An Evaluation of CyberBookshops: The WebQual Method, International Journal of Electronic Commerce, vol. 6, 2001, pp.11-30. [35] H. van der Heijden and T. Verhagen, Online store image: conceptual foundations and empirical measurement, Information & Management, vol 41, 2004, pp. 609-617. [36] P. Hernon, and P. Calvert, E-Service Quality in Libraries: Exploring its Features and Dimensions, Library and Information Science Research, vol.27, 2005, pp. 377-404. [37] R. U. Bilsel, G. Bykzkan and D. G. Ruan, A fuzzy preference-ranking model for a quality evaluation of hospital web sites, International Journal of Intelligent Systems, vol. 21, 2006, pp.1181 1197. [38] S. Lo, Web service quality control based on text mining using support vector machine, Expert system with applications, vol.34, 2008, pp.603-610. [39] B. Leporini, F. Patern, and A. Scorcia, Flexible tool support for accessibility evaluation, Interacting with computers, vol.18, 2006, pp.869-890. [40] S. Abou-Zahra, A Data Model to Facilitate the Automation of Web Accessibility Evaluations, Electronic Note in Theoretical Computer Science, vol.157, 2006, pp. 3-9. [41] P.T. Jaeger, "Assessing Section 508 compliance on federal e-government Web sites: A multi-method, user-centered evaluation of accessibility for persons with disabilities," Government Information Quarterly, vol. 23, 2006, 169190.

81