A Systematic Review of E-government Evaluation


Abstract. This article presents a systematic literature review of ninety (90) articles (2006–2016) of e-government evaluation. The aim was to collect, summarize and integrate literature on e-government services’ evaluation of the past decade and to analyze which aspects of evaluation have received increased or less attention. Results have been synthesized and an augmented and holistic model for e-government evaluation is proposed. It was found that more emphasis has been placed on evaluation of website quality as compared to other dimensions of e-government which include customer satisfaction, technical performance and internal processes. The findings of the article suggest an agenda for future research aiming to improve and validate the proposed model through qualitative and quantitative research methodologies.

Keywords: E-government, E-service, Evaluation, E-governance, Customer Satisfaction


    There are multiple and myriad definitions of electronic government (e-government) as reported in the literature. Gil-García and Pardo (2005) define electronic government as usage of ICTs (information and communication technologies) in the sphere of public administration to improve managerial efficiency and effectiveness, encourage principles and processes entailed by democracy and develop a structure, which would provide a legal and supervisory oversight. All these steps aim to foster a more open and transparent culture where citizens/public and other societal stakeholders are able to engage in a more meaningful relationship with the government. The ultimate goal is to reduce the administrative and financial burden, and to transform the existing structure into a knowledge-based society.

    The above definition is comprehensive as it encompasses four major areas i.e. e-services, e-management, e-democracy and e-public policy. For the purpose of this study, only “e-services” area has been focused. The word ‘e-government’ is considered synonymous with “e-services”.

    In addition to clarifying as to what actually e-service comprises, it is imperative at this stage to demarcate the concepts of quality and evaluation especially in the context of e-services. As a stand-alone concept, quality has been divided into objective and subjective aspects. The objective part is more related to meeting preset criteria while the subjective part is more about user’s perception about the quality (Shewhart and Walter, 1980). Taking this concept forward, Ishikawa (1991) divided quality approaches into having true or substitute characteristics. True characteristics refer to the end-user while substitute characteristics refer to the producers. For the purpose of this review article, research studies have been categorized on the basis of ‘true’ and ‘substitute’ characteristics in order to apply traditional quality management principles on e-services.

    End-users denote the public or private organizations using e-government services while producers are the government departments and institutions engaged in providing services through electronic means.

    The other concept which needs to be defined is evaluation. Generally, evaluation involves comparing the outcomes with an already set standard in order to improve future output. There are a number of conceptions for the term ‘evaluation’. For example the Context, Inputs, Processes, Products Model (CIPP) categorizes the evaluation process in four stages (Stufflebeam, 2003). Other evaluation models consider the evolutionary stage of e-service i.e. if an e-service is evaluated at the beginning then it is more relevant to requirement engineering, feasibility, compatibility and implementation of the proposed project. Moreover, if the end stage is considered then evaluation involves public value and results’ achievement in the particular context of the project (Batini, Viscusi, and Cherubini, 2009).

    During the life of the e-project, monitoring of the service in itself demands evaluation since only proper assessment of the process can ensure its successful completion (Gouscos, Kalikakis, Legal, and Papadopoulou, 2007; Mitra and Gupta, 2008). Lastly, there is a certain evaluation which involves comparing a certain aspect of the e-service by taking a sample from countries where such initiatives have been taken. This evaluation technique provides insight into the common challenges faced by nations when they initiate similar e-service projects. Surveys conducted by United Nations usually fall under such comparability evaluations (Sandoval-Almazan and Gil-Garcia, 2012; Stowers, 2004).

    The literature related to evaluation of e-services in particular is scarce at the moment but as countries all over the world are taking e-initiatives, questions are being raised regarding their effectiveness (Irani, Kamal, Angelopoulos, Kitsios, and Papadopoulos, 2010). Complications in evaluation of e-services stems from the complicated nature of such services and the involvement of different stakeholders. Moreover, all these stakeholders may hold different views owing to their respective perspectives. Difficulties are also faced because it is burdensome to quantify true costs in implementation of such e-projects owing to traditional bureaucratic red-tapism. Moreover, due to involvement of technology, there is always a likelihood that between social and technical aspects, one would get ignored. Earlier models ignored the social aspect as citizen satisfaction was not given due importance.

    Governments were more focused on policy making and their own readiness for implementation. Later on when projects were implemented and failed to deliver the desired results, only then the focus was shifted towards citizens. In addition to this, recent evaluation models in the context of e-services are bent towards pinpointing the inefficiencies and are unable to provide strategic guidelines which could improve the services.

    Scott and Golden (2009) succinctly summarize evaluation approaches as supply side and demand side. Supply side evaluation models emphasize the delivery of services by the government while demand side models emphasize the relationship between government and people through the e-services. Bhuiyan (2011) categorizes these two approaches as internal (supply) and external (demand) and states that internal approach has been ignored over the years especially in Bangladesh where the study took place. Elsheikh and Azzeh (2014) in Jordan found that balance is lacking when two approaches to evaluation are considered. Špacek and Malý (2010) were also of similar opinion that evaluation should consider both approaches and encouraged a more integrated approach. Initial scoping study and reading of reviews such as (Scholl and Dwivedi (2014); Torres, Pina, and Royo (2005); Yildiz (2007)) also confirmed this divide between supply side or introvert and demand side or extrovert approaches.

    It is evident from the above paragraph that evaluations usually pertain to a single stage or objective. The past decade has witnessed exceptional progress regarding digitization of public services and thus, a need has arisen for evaluation models which could go beyond singular objectives. It is intended to review the approaches taken in the past decade to get a comprehensive and holistic view which could assist in coming up with an evaluation model which is able to satisfy more than one objective. Such model would take into account multiple stakeholder perspective in order to get a better picture of e-service’s performance.

    The model would help integrate quality dimensions and improve e-services and increase public satisfaction. Therefore, the research objectives of the study are:

    • To collect, summarize and integrate literature on e-government services evaluation in the past decade

    • To analyze which aspects of evaluation have been focused or ignored

    • To synthesize and come up with a holistic model for e-government evaluation, building upon the evaluation framework provided by Papadomichelaki, Magoutas et al. (2006)

    This systematic review has taken a qualitative view of the research studies as it intends to capture a broader overview of the research in the field as compared to meta-analysis, which statistically analyses and attempts to quantify the findings.



    a. Types of Study

    Here, it is prudent to distinguish e-government service and e-service. According to Papadomichelaki, Magoutas, Halaris, Apostolou, and Mentzas (2006) when one considers evaluation of government e-service, it is all about how the e-portal behaves and customer’s level of satisfaction while interacting with the portal. Customer’s prior expectations and perception do play a major role in defining their overall experience with the e-service. On the other hand, when one considers evaluation of e-service then the focus shifts towards the quality of the service itself. It becomes more about how the actual service is delivered to public through e-portal. Research studies included in this review were related to evaluation of both aspects as mentioned in the above paragraph. Studies included evaluation of the quality of the portal as well as satisfaction of the customers.

    b. Topic of E-Government Evaluation

    It was decided that records should contain the words e-government evaluation, assessment, success and/or quality in their title and/or in abstract, to make them eligible for selection for this review study. Considering the time frame, restricting to these words helped to keep the number of studies to be reviewed at a manageable level.


    a. Language

    Only research studies which were in English were made part of the review. According to Wilson, Lipsey, and Derzon (2003), this approach helps to keep the problems related to translating from another language, such as replicability of the review in check.

    b. Publication Status

    Research articles belonging to international peer...

To continue reading