Purpose – The field of website quality evaluation attracts the interest of a range of disciplines, each bringing its own particular perspective to bear. This study aims to identify the main characteristics – methods, techniques and tools – of the instruments of evaluation described in this literature, with a specific concern for the factors analysed, and based on these, a multipurpose model is proposed for the development of new comprehensive instruments.
Design/methodology/approach – Following a systematic bibliographic review, 305 publications on website quality are examined, the field’s leading authors, their disciplines of origin and the sectors to which the websites being assessed belong are identified, and the methods they employ characterised.
Findings – Evaluations of website quality tend to be conducted with one of three primary focuses: strategic, functional or experiential. The technique of expert analysis predominates over user studies and most of the instruments examined classify the characteristics to be evaluated – for example, usability and content – into factors that operate at different levels, albeit that there is little agreement on the names used in referring to them.
Originality/value – Based on the factors detected in the 50 most cited works, a model is developed that classifies these factors into 13 dimensions and more than 120 general parameters. The resulting model provides a comprehensive evaluation framework and constitutes an initial step towards a shared conceptualization of the discipline of website quality.
Keywords Website quality, Website, Website evaluation, Expert analysis, Heuristic evaluation,
User experience, Usability
Paper type Research paper
Read / download paper
Citation
Morales-Vargas, A., Pedraza-Jimenez, R. and Codina, L. (2023), «Website quality evaluation: a model for developing comprehensive assessment instruments based on key quality factors», Journal of Documentation, Vol. ahead-of-print No. ahead-of-print. https://doi.org/10.1108/JD-11-2022-0246
Links
References
Adepoju, S.A. and Shehu, I.S. (2014), “Usability evaluation of academic websites using automated tools”, 3rd International Conference on User Science and Engineering (i-USEr), IEEE, pp. 186-191.
Akincilar, A. and Dagdeviren, M. (2014), “A hybrid multi-criteria decision making model to evaluate hotel websites”, International Journal of Hospitality Management, Vol. 36, pp. 263-271.
Aladwani, A.M. and Palvia, P.C. (2002), “Developing and validating an instrument for measuring
user-perceived web quality”, Information and Management, Vol. 39 No. 6, pp. 467-476.
Al-Qeisi, K., Dennis, C., Alamanos, E. and Jayawardhena, C. (2014), “Website design quality and usage behavior: unified theory of acceptance and use of technology”, Journal of Business Research, Vol. 67 No. 11, pp. 2282-2290.
Anusha, R. (2014), “A study on website quality models”, International Journal of Scientific and
Research Publications, Vol. 4 No. 12.
Apple (2018), Human Interface Guidelines, Apple.
Bevan, N., Carter, J. and Harker, S. (2015), “ISO 9241-11 Revised: what have we learnt about usability since 1998?”, in Kurosu, M. (Ed.), Human-Computer Interaction: Design and Evaluation,
Springer International Publishing, Cham, pp. 143-151.
Booth, A., Sutton, A. and Papaioannou, D. (2016), Systematic Approaches to a Successful Literature Review, 2nd ed., SAGE Publications, London.
Cajita, M.I., Rodney, T., Xu, J., Hladek, M. and Han, H. (2017), “Quality and health literacy demand of online heart failure information”, Journal of Cardiovascular Nursing, Vol. 32 No. 2, pp. 156-164.
Cao, K. and Yang, Z. (2016), “A study of e-commerce adoption by tourism websites in China”, Journal of Destination Marketing and Management, Vol. 5 No. 3, pp. 283-289.
Chiou, W.-C., Lin, C.-C. and Perng, C. (2010), “A strategic framework for website evaluation based on a review of the literature from 1995–2006”, Information and Management, Vol. 47 Nos 5-6, pp. 282-290.
Choi, W. and Stvilia, B. (2015), “Web credibility assessment: conceptualization, operationalization, variability, and models”, Journal of the Association for Information Science and Technology, Vol. 66 No. 12, pp. 2399-2414.
Codina, L. (2018), Revisiones bibliograficas sistematizadas: procedimientos generales y framework para ciencias humanas y sociales, Universitat Pompeu Fabra, Barcelona.
Codina, L. and Pedraza-Jimenez, R. (2016), “Caracterısticas y componentes de un sistema de analisis de medios digitales: el SAAMD”, in Pedraza-Jimenez, R., Codina, L. and Guallar, J. (Eds), Calidad en sitios web: Método de análisis general, e-commerce, imágenes, hemerotecas y turismo, Editorial UOC, Barcelona, pp. 15-40.
Cristóbal Fransi, E., Hernández Soriano, F. and Marimon, F. (2017), “Critical factors in the evaluation of online media: creation and implementation of a measurement scale (e-SQ-Media)”, Universal Access in the Information Society, Vol. 16 No. 1, pp. 235-246.
Daraz, L., Morrow, A.S., Ponce, O.J., Beuschel, B., Farah, M.H., Katabi, A., Alsawas, M., Majzoub, A.M., Benkhadra, R., Seisa, M.O., Ding, J., Prokop, L. and Murad, M.H. (2019), “Can patients trust online health information? A meta-narrative systematic review addressing the quality of health
information on the internet”, Journal of General Internal Medicine, Vol. 34 No. 9, pp. 1884-1891.
Dey, A., Billinghurst, M., Lindeman, R.W. and Swan, J.E., II (2018), “A systematic review of 10 years of augmented reality usability studies: 2005 to 2014”, Frontiers in Robotics and AI, Vol. 5, p. 37, doi: 10.3389/frobt.2018.00037.
Dueppen, A.J., Bellon-Harn, M.L., Radhakrishnan, N. and Manchaiah, V. (2019), “Quality and
readability of English-language internet information for voice disorders”, Journal of Voice,
Vol. 33 No. 3, pp. 290-296.
Ecer, F. (2014), “A hybrid banking websites quality evaluation model using AHP and COPRAS-G: a Turkey case”, Technological and Economic Development of Economy, Vol. 20 No. 4, pp. 758-782.
European Commission (2016), “Europa web guide”, The EU Internet Handbook, available at: http://ec.europa.eu/ipg/ (accessed 17 June 2018).
Fernández-Cavia, J., Rovira, C., Díaz-Luque, P. and Cavaller, V. (2014), “Web Quality Index (WQI) for official tourist destination websites. Proposal for an assessment system”, Tourism Management Perspectives, Vol. 9, pp. 5-13.
Garrett, J.J. (2011), The Elements of User Experience: User-Centered Design for the Web and beyond, 2nd ed., New Riders, Indianapolis.
Gough, D., Oliver, S. and Thomas, J. (2017), An Introduction to Systematic Reviews, 2nd ed., SAGE Publications, London.
Grant, M.J. and Booth, A. (2009), “A typology of reviews: an analysis of 14 review types and
associated methodologies”, Health Information and Libraries Journal, Vol. 26 No. 2, pp. 91-108.
Gregg, D.G. and Walczak, S. (2010), “The relationship between website quality, trust and price
premiums at online auctions”, Electronic Commerce Research, Vol. 10 No. 1, pp. 1-25.
Hasan, L. (2014), “Evaluating the usability of educational websites based on students’ preferences of design characteristics”, International Arab Journal of E-Technology, Vol. 3 No. 3, pp. 179-193.
Health On the Net (2017), Principles: the HON Code of Conduct for Medical and Health Web Sites, Ginebra, Suiza.
Huang, Z. and Benyoucef, M. (2014), “Usability and credibility of e-government websites”, Government Information Quarterly, Vol. 31 No. 4, pp. 584-595.
Ismailova, R. and Inal, Y. (2017), “Web site accessibility and quality in use: a comparative study of government web sites in Kyrgyzstan, Azerbaijan, Kazakhstan and Turkey”, Universal Access in
the Information Society, Vol. 16 No. 4, pp. 987-996.
ISO (n.d), “Terms & definitions”, Online Browsing Platform (OBP), available at: https://www.iso.org/ obp/ (accessed 18 February 2021).
ISO (2008), “ISO 9241-151:2008 ergonomics of human-system interaction. Part 151: guidance on World wide web user interfaces”, International Organization for Standardization.
Jainari, M.H., Baharum, A., Deris, F.D., Mat Noor, N.A., Ismail, R. and Mat Zain, N.H. (2022), “A standard content for university websites using heuristic evaluation”, in Arai, K. (Ed.), Intelligent Computing. SAI 2022. Lecture Notes in Networks and Systems, Springer, Cham, Vol. 506, doi: 10.1007/978-3-031-10461-9_19.
Jayanthi, B. and Krishnakumari, P. (2016), “An intelligent method to assess webpage quality using extreme learning machine”, International Journal of Computer Science and Network Security, Vol. 16 No. 9, pp. 81-85.
Kamoun, F. and Almourad, M.B. (2014), “Accessibility as an integral factor in e-government web site evaluation: the case of Dubai e-government”, Information Technology and People, Vol. 27 No. 2, pp. 208-228.
Karkin, N. and Janssen, M. (2014), “Evaluating websites from a public value perspective: a review of Turkish local government websites”, International Journal of Information Management, Vol. 34 No. 3, pp. 351-363.
Kaur, S. and Gupta, S.K. (2014), “Key aspects to evaluate the performance of a commercial website”, IJCA Proceedings on International Conference on Advances in Computer Engineering and Applications, Foundation of Computer Science (FCS), pp. 7-11.
Kaushik, A. (2010), Web Analytics 2.0: the Art of Online Accountability & Science of Customer
Centricity, Wiley Publishing, Indianapolis, Indiana.
Keselman, A., Arnott Smith, C., Murcko, A.C. and Kaufman, D.R. (2019), “Evaluating the quality of health information in a changing digital ecosystem”, Journal of Medical Internet Research,
Vol. 21 No. 2, e11129.
Król, K. and Zdonek, D. (2020), “Aggregated indices in website quality assessment”, Future Internet, Vol. 12 No. 4, p. 72.
Krug, S. (2014), Don’t Make Me Think, Revisited: A Common Sense Approach to Web and Mobile Usability, 3rd ed., New Riders, Pearson Education, Berkeley, CA.
Kurosu, M. (2015), Human-computer Interaction: Design and Evaluation, in Kurosu, M. (Ed.), Springer International Publishing, Cham.
Lavrakas, P.J. (2008), Encyclopedia of Survey Research Methods, 5th ed., Sage Publications, Thousand Oaks, CA.
Law, R. (2019), “Evaluation of hotel websites: progress and future developments”, International
Journal of Hospitality Management, Vol. 76, pp. 2-9.
Law, R., Qi, S. and Buhalis, D. (2010), “Progress in tourism management: a review of website
evaluation in tourism research”, Tourism Management, Vol. 31 No. 3, pp. 297-313.
Leavitt, M.O. and Shneiderman, B. (2006), Research-based Web Design & Usability Guidelines, 2nd ed., U.S. Department of Health & Human Services, Washington, DC.
Lee-Geiller, S. and Lee, T.D. (2019), “Using government websites to enhance democratic E-governance: a conceptual model for evaluation”, Government Information Quarterly, Vol. 36 No. 2, pp. 208-225.
Leung, D., Law, R. and Lee, H.A. (2016), “A modified model for hotel website functionality evaluation”, Journal of Travel and Tourism Marketing, Vol. 33 No. 9, pp. 1268-1285.
Lopezosa, C., Codina, L. and Gonzalo-Penela, C. (2019), “Off-page SEO and link building: general strategies and authority transfer in the digital news media”, Profesional de la Información, Vol. 28 No. 1, pp. 1-12, e280107.
Maia, C.L.B. and Furtado, E.S. (2016), “A systematic review about user experience evaluation”,
in Marcus, A. (Ed.), Design, User Experience, and Usability: Design Thinking and Methods,
Springer International Publishing, Cham, pp. 445-455.
Manchaiah, V., Dockens, A.L., Flagge, A., Bellon-Harn, M., Azios, J.H., Kelly-Campbell, R.J. and
Andersson, G. (2019), “Quality and readability of English-language internet information for
tinnitus”, Journal of the American Academy of Audiology, Vol. 30 No. 01, pp. 031-040.
Martín-Martín, A., Ordu~na-Malea, E., Thelwall, M. and Delgado López-Cózar, E. (2018), “Google Scholar, Web of Science, and Scopus: a systematic comparison of citations in 252 subject categories”, Journal of Informetrics, Vol. 12 No. 4, pp. 1160-1177.
Morales-Vargas, A., Pedraza-Jimenez, R. and Codina, L. (2020), “Website quality: an analysis of
scientific production”, Profesional de la Informacion, Vol. 29 No. 5, p. e290508.
Nielsen, J. (2020), “10 usability heuristics for user interface design”, Nielsen Norman Group, available at: https://www.nngroup.com/articles/ux-research-cheat-sheet/ (accessed 3 May 2021).
Nikolic, N., Grljevic, O. and Kovacevic, A. (2020), “Aspect-based sentiment analysis of reviews in the domain of higher education”, Electronic Library, Vol. 38, pp. 44-64.
Orduña-Malea, E. and Aguillo, I.F. (2014), Cibermetrıa: Midiendo El Espacio Red, UOC, Barcelona.
Paz, F., Paz, F.A., Villanueva, D. and Pow-Sang, J.A. (2015), “Heuristic evaluation as a complement to usability testing: a case study in web domain”, 2015 12th International Conference on Information Technology – New Generations, IEEE, pp. 546-551.
Quiñones, D. and Rusu, C. (2017), “How to develop usability heuristics: a systematic literature review”, Computer Standards and Interfaces, Vol. 53, pp. 89-122.
Rekik, R., Kallel, I. and Alimi, A.M. (2015), “Quality evaluation of web sites: a comparative study of some multiple criteria decision making methods”, Intelligent Systems Design and Applications (ISDA), 2015 15th International Conference. International Conference on Intelligent Systems Design and Applications, pp. 585-590.
Rekik, R., Kallel, I., Casillas, J. and Alimi, A.M. (2018), “Assessing web sites quality: a systematic
literature review by text and association rules mining”, International Journal of Information
Management, Vol. 38 No. 1, pp. 201-216.
Rocha, A. (2012), “Framework for a global quality evaluation of a website”, Online Information Review, Vol. 36 No. 3, pp. 374-382.
Rosala, M. and Krause, R. (2020), User Experience Careers: What a Career in UX Looks Like Today, Fremont, CA.
Sanabre, C., Pedraza-Jiménez, R. and Vinyals-Mirabent, S. (2020), “Double-entry analysis system (DEAS) for comprehensive quality evaluation of websites: case study in the tourism sector”, Profesional de la Información, Vol. 29 No. 4, pp. 1-17, e290432.
Sauro, J. and Lewis, J.R. (2016), Quantifying the User Experience: Practical Statistics for User Research, 2nd ed., Elsevier/Morgan Kaufmann, Waltham, MA.
Semeradova, T. and Weinlich, P. (2020), “Looking for the definition of website quality”, in
Semeradova, T. and Weinlich, P. (Eds), Website Quality and Shopping Behavior: Quantitative
and Qualitative Evidence, SpringerBriefs in Business, Cham, pp. 5-27.
Shneiderman, B. (2016), “The eight golden rules of interface design”, Department of Computer Science, University of Maryland.
Shneiderman, B., Plaisant, C., Cohen, M.S., Jacobs, S., Elmqvist, N. and Diakopoulos, N. (2016),
Designing the User Interface: Strategies for Effective Human-Computer Interaction, 6th ed.,
Pearson Higher Education, Essex.
Sun, Y., Zhang, Y., Gwizdka, J. and Trace, C.B. (2019), “Consumer evaluation of the quality of online health information: systematic literature review of relevant criteria and indicators”, Journal of Medical Internet Research, Vol. 21 No. 5, e12522.
Thielsch, M.T. and Hirschfeld, G. (2019), “Facets of website content”, Human–Computer Interaction, Vol. 34 No. 4, pp. 279-327.
Tognazzini, B. (2014), “First principles of interaction design (revised and expanded)”, askTog.
Toxboe, A. (2018), “Design patterns”, UI Patterns: User Interface Design Patterns Library, available at: http://ui-patterns.com/patterns (accessed 17 June 2018).
Tullis, T. and Albert, W. (2013), Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics, 2nd ed., Morgan Kaufmann, Waltham, Massachusett.
W3C (2016), “Web design and applications standards”, World Wide Web Consortium.
Whitenton, K. (2021), “Triangulation: get better research results by using multiple UX methods”,
Nielsen Norman Group, available at: https://www.nngroup.com/articles/triangulationbetter-
research-results-using-multiple-ux-methods/ (accessed 4 March 2021).
Xanthidis, D., Argyrides, P. and Nicholas, D. (2009), “Web Site Evaluation Index: a systematic methodology and a metric system for the assessment of the quality of web sites”, in Demiralp, M. (Ed.),
Proceedings of the 8th WSEAS International Conference on Telecommunications and Informatics. Electrical and Computer Engineering Series, p. 194.
Yin, R.K. (2015), Qualitative Research From Start to Finish, 2nd ed., The Guilford Press,
London. Website quality
