Quality Assurance Frameworks

Chapter 8 Quality Assurance Frameworks and Metadata

Ensuring data quality is a core challenge of all statistical offices.  Energy data made available to users are the end product of a complex process comprising many stages, including the definition of concepts and variables, the collection of data from various sources, data processing, analysis, formatting to meet user needs and finally, data dissemination.  Achieving overall data quality is dependent upon ensuring quality in all stages of the process.
Quality assurance comprises all institutional and organizational conditions and activities that provide confidence that the product or service is adequate for its intended use by clients and stakeholders. In other words, the quality is judged by its “fitness for use.”  The pursuit of good quality means having a legal basis for the compilation of data, ensuring that the institutional environment is objective and free of political interference, ensuring the adequacy of data-sharing and coordination among data-producing agencies, assuring the confidentiality and security of information, addressing the concerns of respondents regarding reporting burden, providing adequate human, financial and technical resources for the professional operation of energy statistics, and implementing measures to ensure their efficient, cost-effective use.  All the actions that responsible agencies take to assure data quality constitute quality assurance. In the IRES, all countries were encouraged to develop their own national energy data quality assurance, to document these, to develop measures of data quality, and to make these available to users.
Managers of statistical agencies must also promote and demonstrate their support for ensuring quality throughout the organization.  This can be done in a number of ways:

• Make quality a stated goal of the organization.  Managers must highlight the need for ensuring quality, raise the profile and awareness of quality initiatives and recognize achievements.
• Establish and publicize standards of data quality.
• Track quality indicators.  Measures of quality should be monitored on an ongoing basis, issues should be flagged and corrective actions implemented, as required.
• Conduct regular quality assurance reviews.  Evaluations should be conducted on a regular basis (e.g. every five years), particularly for the most important and surveys.  These can identify issues and risks, and lead to corrective actions.
• Develop centres of expertise on dimensions of quality.  Managers should create areas that can focus on the development of knowledge, skills, standards and tools to support quality initiatives (e.g. survey methodology, questionnaire design, and automated edits).
• Deliver quality assurance training to staff.  By sending staff on quality training, managers can raise awareness, develop skills and establish a culture of quality assurance.

Most international organizations and countries have developed general definitions of data quality, outlining the various dimensions (aspects) of quality and quality measurement, and integrating them into quality assurance frameworks. Although these quality assurance frameworks may differ to some extent in their approaches to quality and in the number, name and scope of quality dimensions, they complement each other and provide comprehensive and flexible structures for the qualitative assessment of a broad range of statistics, including energy statistics.
The overall objective of these frameworks is to standardize and systematize quality practices and measurement across countries. They allow the assessment of national practices in energy statistics in terms of internationally (or regionally) accepted approaches for data quality measurement. The quality assurance frameworks can be used in a number of contexts, including for (a) guiding countries’ efforts towards strengthening and maintaining their statistical systems by providing a self-assessment tool and a means of identifying areas for improvement; (b) supporting technical development and enhancement purposes; (c) reviews of a country’s energy statistics program as performed by international organizations; and (d) assessments by other groups of data users.
National agencies responsible for energy statistics can decide to implement one of the existing frameworks for quality assurance for any type of statistics, including energy statistics, either directly or by developing, on the basis of those frameworks, a national quality assessment framework that best fits their country’s practices and circumstances.   See Box 8.1 for references to data quality frameworks from various countries and organizations.

Box 8.1  Examples of Data Quality Frameworks Eurostat Eurostat (2003). Definition of Quality in Statistics. Eurostat, Luxembourg. http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/documents/ess%20quality%20definition.pdf OECD Organisation for Economic Co-operation and Development (2011). Quality Framework and Guidelines for OECD Statistical Activities. OECD, Paris. http://search.oecd.org/officialdocuments/displaydocumentpdf/?cote=std/qfs(2011)1&doclanguage=en United Nations United Nations (2013).  Fundamental Principles of Official Statistics. Prepared by the United Nations Statistics Division, New York. http://unstats.un.org/unsd/dnss/gp/fundprincipals.aspx   United Nations (2012). National Quality Assurance Frameworks. Prepared by the United Nations Statistics Division, New York. http://unstats.un.org/unsd/dnss/QualityNQAF/ngaf.aspx United Nations (2011). International Recommendations for Energy Statistics. Chapter 9.  Prepared by the United Nations Statistics Division, New York. http://unstats.un.org/unsd/statcom/doc11/BG-IRES.pdf Australian Bureau of Statistics Australian Bureau of Statistics (2009). The ABS Data Quality Framework. ABS, Canberra. http://www.abs.gov.au/ausstats/abs@.nsf/mf/1520.0 Statistics Canada Statistics Canada (2009). Statistics Canada Quality Guidelines. Statistics Canada, Fifth Edition, Ottawa. http://www.statcan.gc.ca/pub/12-539-x/12-539-x2009001-eng.pdf Statistics Finland Statistics Finland (2007). Quality Guidelines for Official Statistics. Statistics Finland, Helsinki. http://www.stat.fi/meta/qg_2ed_en.pdf (obtain additional quality frameworks ?)

 

The following dimensions of quality reflect a broad perspective and therefore, have been incorporated in many of the existing data quality frameworks.  The dimensions of quality below should be taken into account when measuring and reporting the quality of statistics.  These dimensions can be divided into static and dynamic elements of quality.

• Relevance:   the degree to which the collected data meet the needs of clients.  Relevance is concerned with whether the available information is useful and responsive for users to address their most important issues.  As such, being relevant is an important dimension of quality and a key pillar of a statistical agency.

Quality measures/indicators: Identification of gaps between key user needs and compiled energy statistics in terms of concepts, coverage and detail. Compile through structured consultations and regular feedback. Perception from user feedback surveys. Monitor requests for information and the capacity to respond.

• Credibility:  refers to the confidence that users have in the objectivity of the data based on the reputation of the responsible agency producing the data in accordance with accepted statistical standards, and that policies and practices are transparent.  For example, data should not be manipulated, withheld or delayed, nor should their release be influenced by political pressure.  Data must be kept confidential and secure. 

Quality measure/indicator: Perceptions from user feedback survey.

• Accuracy:  the extent to which the information correctly describes the phenomena it was designed to measure.   This is usually characterized in terms of the error in statistical estimates and is traditionally broken down into bias (systematic error) and variance (random error) components.

Quality measures/indicators:  Sampling errors (standard errors).  Non-sampling errors (overall and item response rate).  Quantity response rate (e.g., percentage of total energy production reported, weighted response rate).  Number, frequency and size of revisions to energy data. 

 

• Timeliness:  the elapsed time (i.e. delay) between the end of the reference period to which the information pertains and the date on which the information becomes available.   Achieving data timeliness is often viewed as a trade-off against ensuring accuracy. The timeliness of information will influence its relevance and utility for users.

Quality measures/indicators:  Time lag between the end of the reference period and the date of the first release (or the release of final results) of energy data.

• Coherence:  the degree to which the data can be successfully brought together with other statistical information within a broad analytic framework and over time. The use of standard concepts, classifications and target populations promotes coherence, as does the use of common methodology across surveys.

Quality measures/indicators:  Comparison and joint use of related energy data from different sources.  Number and rates of divergences from the relevant international statistical standards in concepts and measurement procedures used in the collection/compilation of energy statistics.

 

• Accessibility:  the ease with which data can be obtained from the statistical agency.  This includes awareness of the availability of the information, the ease with which the existence of information can be found and understood, as well as the suitability of the form or medium through which the information can be accessed.   Barriers to access must also be minimized.  Some examples of potential barriers could include the cost of the information, technological limitations or complexity.

Quality measures/indicators:  Number of announcements of release of energy data.  Number and types of methods used for dissemination of energy statistics.  Number of energy statistics data sets made available by mode of dissemination, as a percentage of total energy statistics data sets produced.  The number of requests for information.

 

• Non-response:  respresents a challenge in maintaining quality and ensuring a good response rate.  To maintain the cooperation of respondents, statistical agencies must be responsive to their needs and issues such as response burden.  Strategies to ensure a good response rate could include electronic reporting, making greater use of administrative data sources, imputation or to adjust for non-response at the aggregate level.

Quality measures/indicators:  Non-response rate and imputation rate. 

• Coverage:  is determined by the quality of survey frames.   Businesses are constantly forming, merging or exiting industries, and adding or dropping products and services.  The use of administrative data sources to establish frames can place surveys at risk since there is often a time lag in detecting these changes from administrative data.  Agencies must be prepared to invest in the maintenance of survey frames.

Quality measure/indicator:  Proportion of population covered by data collected.

• Sampling:  is the data used to stratify and select units to be surveyed.  Overtime samples deteriorate as units become out of date or demand for data on specific subpopulations may emerge that the sample was not designed to support.  Sample redesign is an opportunity to keep up with new techniques, changes in the business universe, and to spread respondent burden more evenly.

Quality measure/indicator:  Deterioration of sample.

In addition to the above quality dimensions, interpretability is another important criterion of quality in regards to metadata.

• Interpretability:   the availability of the supplementary information and metadata necessary to understand and utilize the data appropriately. This information normally covers the underlying concepts, variables and classifications used, the methodology of data collection and processing, and indications of the accuracy of the statistical information.

The above dimensions of quality were also incorporated into the country practice template that was developed by the Oslo Group and the UN Statistics Division.  This template enables countries to report and share their practices.  Some of these practices are presented below in Box 8.2 to demonstrate how the dimensions of quality are applied. 

Box 8.2  Examples of Country Practices on the Quality Dimensions in Energy Statistics  Electricity Sustainable Energy Authority of Ireland (October 2012).  Electricity Production and Supply.  Prepared by the Sustainable Energy Authority of Ireland. http://unstats.un.org/unsd/energy/template/Ireland_electricity_supply.pdf Energy Balances Statistics Austria (October 2012).  Energy Balances for Austria and the Laender of Austria.  Prepared by Statistics Austria. http://unstats.un.org/unsd/energy/template/Austria_energy_balance.pdf Central Statistical Bureau of Latvia (April 2012).  Energy Balance.  Prepared by the Central Statistical Bureau of Latvia. http://unstats.un.org/unsd/energy/template/Latvia_energy_balance.pdf Statistics Mauritius (March 2012).  Energy Balance Compilation.  Prepared by Statistics Mauritius. http://unstats.un.org/unsd/energy/template/Mauritius_energy_balance.pdf Consumption Australian Bureau of Statistics (April 2012).  Energy, Water, and Environment Survey.  Prepared by the Australian Bureau of Statistics. http://unstats.un.org/unsd/energy/template/Australia_energy_survey.pdf   Statistics Canada (August 2012).  Industrial Consumption of Energy Survey.  Prepared by Statistics Canada. http://unstats.un.org/unsd/energy/template/Canada_energy_consumption.pdf Czech Statistical Office (March 2012).  Energy Consumption and Fuel by Year.  Prepared by the Czech Statistical Office. http://unstats.un.org/unsd/energy/template/Czech_consumption_fuel.pdf Other energy topics Statistics Austria (April 2012).  Fuel Input and District Heat Output of Biomass Heating Plants.  Prepared by Statistics Austria. http://unstats.un.org/unsd/energy/template/Austria_biomass.pdf Sustainable Energy Authority of Ireland (October 2012).  Combined Heat and Power.  Prepared by the Sustainable Energy Authority of Ireland. http://unstats.un.org/unsd/energy/template/Ireland_CHP.pdf ISTAT (April 2012).  Urban Environment Indicators on Energy.  Prepared by ISTAT, Italy. http://unstats.un.org/unsd/energy/template/Italy_urban.pdf

 

 

 

 

Ensuring data quality is an important function of any statistical organization, whether it be centralized or decentralized.  Below is the example of Sweden’s decentralized statistical system and quality of official statistics.

Box 8.3  Example of Data Quality in a Decentralized System The Swedish Official Statistical System The Swedish Official Statistics System has been decentralised since the Statistics Reform was implemented in 1 July 1994. Statistics Sweden is responsible for cross-sectoral statistics such as economic statistics, national accounts, while 26 sector specific agencies are responsible for official statistics in each of their areas. The Swedish Energy Agency (SEA) is responsible for Official Energy Statistics.  Quality of official statistics The experience review of Sweden Official Statistical System accomplished by an Inquiry – through contact with users and foreign agencies – indicated that, current official statistics are of good quality. This does not mean that there are no problems, or that quality can’t be improved in certain respects. But the measures that may be required to improve quality essentially involve making a basic well-functioning system even better. Most important quality requirements for Sweden’ official statistics should be stated in the Official Statistics Act. The review of Sweden Official Statistical System proposes that the wording of the quality requirements in the Act should be modelled on the quality criteria in the EU’s statistics regulation. Much of the content of the European Statistics Code of Practice is already met in Swedish’ law. However, additional principles contained in the Code of Practice may need to be regulated by law. In the review is proposed that the principle of a non-excessive burden to respondents is introduced in the Sweden Official Statistics Ordinance. The suggestion is that most of the principles would be more suited as regulations from Statistics Sweden rather than, as it are today, general guidelines. The Statistical Agencies such as SEA are working on quality issues in a professional manner, both individually and in joint forums, within the framework of the Council for Official Statistics in Sweden. Since good quality is crucial for the reliability and credibility of official statistics, it is essential that the agencies continue to conduct active quality efforts. The Council for Official Statistics has established certain guidelines and criteria to promote sufficient quality in official statistics. Based on this, statistical agencies can take an ‘official statistics pledge’, which means that they promise to operate in accordance with the criteria. At present, two agencies, the National Board of Health and Welfare and the Swedish Board of Agriculture, have taken the pledge. It would send an important signal to the users of statistics if more of the statistical agencies, and particularly Statistics Sweden, were to take the pledge. Unfortunately, errors occasionally occur in the statistics, but these do not seem to be due to fundamental system errors. They more likely arise because, in practice, it is impossible to completely avoid errors in such a complex system as the statistics system. When errors happen, it is important that the statistical agencies have procedures and routines to identify, correct and learn from them. It is also important that the agencies openly report the errors, so commissioning organisations, users and others will be aware of the circumstances. In Sweden and also abroad, response rates for statistical surveys are tending to decline. This development is a real problem. The Statistical Agencies are seriously aware of this problem and are trying to find different methods of dealing with it. The problem of declining response rates seems to be a fundamentally trend depending of structural causes. It is now more difficult to reach people using traditional methods than it was in the past. Statistics producers may need to develop better methods of managing the continuing decline and also find other ways of accessing the information. If response rates continue to decline sharply, there may be reason to consider the introduction of an obligation on private individuals to provide information, as is the case in many other countries. According to some estimates, more than 95 per cent of official statistics are based on administrative data collected for purposes other than statistics. Problems with administrative data can arise if the agency responsible chooses to change or terminate the collection of the data in question. This could be solved if the agencies responsible for registers had to consult Statistics Sweden in such a situation. Statistics Sweden, in its role of coordinating the statistics system, could be given the task of safeguarding the interests of statistical agencies. The Council for Official Statistics could probably be responsible for coordinating this. Furthermore, the consultation obligation should only concern registers whose data is reasonably likely to be used for the production of statistics. However, the consultation obligation should not be introduced before final design of the EU’s statistics regulation has been established.

 

 
Ensuring Data Quality in a Statistical Survey Process
To ensure data quality, strategies must be implemented at every stage of a statistical survey process, from start to finish.  Chapter 4 looks at quality measures related to each stage of the survey process.  The main stages of a statistical survey process are:  specify needs, design, build, collect, process, analyze, disseminate, archive, and evaluate.  These represent the nine stages of the Generic Statistical Business Process Model (GSBPM) which are described in detail in Chapter 4.

 

Metadata on statistics
The term metadata defines all information used to describe other data. A very short definition of metadata is “data about data.” Metadata descriptions go beyond the pure form and content of data to encompass administrative facts about the data (e.g., who has created them and when), and how data were collected and processed before they were disseminated or stored in a database. In addition, metadata facilitate the efficient search for and location of data. Documentation on data quality and methodology is an integral component of statistical data and analytical results based on these data. Such documentation provides the means of assessing fitness for use and contributes directly to their interpretability.
Statistical metadata describe or document microdata, macrodata or other metadata and facilitate the sharing, querying and understanding of data. Statistical metadata also refer to any methodological descriptions on how data are collected and manipulated. For energy statistics, for example, metadata include the name of the data variable, the statistical unit from which the information has been collected, data sources, information about classifications of energy products used, and series breaks, and definitions of energy products, and methodologies used in their compilation. Metadata are essential for the interpretation of statistical data. Without appropriate metadata, it would not be possible to fully understand energy statistics or to conduct international comparisons.
There is a bidirectional relationship between metadata and quality. On the one hand, metadata describe the quality of statistics. On the other hand, metadata are a quality component which improves the availability and accessibility of statistical data. There are many types of users and uses for any given set of data. The wide range of possible users and uses means that a broad spectrum of metadata requirements has to be addressed. In particular, the responsible agencies as data suppliers must make sufficient metadata available to enable both the least and the most sophisticated users to readily assess the data and their quality.  The following Box 8.4 presents the type of information that should be available to data users when disseminating data.  Next, in Box 8.5, are examples of metadata or survey documentation published by some countries to assist users with the interpretation of the statistics.

Box 8.4   Information that Should Accompany Statistical Releases (Metadata)

· Survey/Product name

· Objectives of survey o Why are the data collected? o Who are the intended users?

· Timeframe o Frequency of collection (e.g., monthly) o Reference period (e.g., month) o Collection period (e.g., 1-15 days following end of reference period)

· Concepts and definitions o Definitions of key variables and concepts

· Target population o Survey universe/sampling frame o Classifications used (e.g., ISIC, NACE, NAICS)

· Collection method o Direct survey of respondents (sample or census; paper survey or electronic; mandatory or voluntary) o Administrative data sources

· For sample surveys: o Sample size o Desired/achieved sample error o Response rates o Imputation rates

· Error detection o Processes to identify errors (e.g., missing data, data entry errors, assessing validity of reported data, macro level edits, reconciliation with other data sources)

· Imputation of missing data o Methods used

· Disclosure control o Explanation of rules of confidentiality and confidentiality analysis

· Revisions o Description of revisions policy o Explanation of revisions, if any

· Description of analytical methods used o Seasonal adjustment o Rounding

· Other explanatory notes o Breaks in times series due to changes in concepts, coverage, collection, methodology, frame refreshment

· Links to other information or documents o Questionnaire and reporting guide o Quality control and editing procedures

Source: United Nations, 2011, International Recommendations for Energy Statistics, Working Group “Oslo Working Group on Energy Statistics”, 42nd meeting, New York, February 201

 

Box 8.5  Country Examples of Metadata on Statistics Statistics Canada Statistics Canada.  Annual Industrial Consumption of Energy Survey (ICE).  Record number 5047.  Prepared by Statistics Canada.  Date modified: 2013-10-30.  The survey provides estimates of energy consumption by manufacturing establishments in Canada. http://www23.statcan.gc.ca/imdb/p2SV.pl?Function=getSurvey&SDDS=5047 Statistics Canada.  Oil and Gas Extraction.  Record number 2178.  Prepared by Statistics Canada.  Date modified: 2013-09-12.  This annual survey collects information on Canadian companies involved in the oil and gas exploration, development and production industry. http://www23.statcan.gc.ca/imdb/p2SV.pl?Function=getSurvey&SDDS=2178 Other examples of statistical releases by Statistics Canada are available at: http://www23.statcan.gc.ca/imdb-bmdi/pub/indexA-eng.htm Finland Statistics Finland.  Metadata.  Statistics Finland, Helsinki. Last updated 21.1.2014.    http://www.stat.fi/meta/index_en.html The Netherlands Statistics Netherlands.  Methods.  Statistics Netherlands, The Hague.  © Statistics Netherlands, 2014 http://www.cbs.nl/en-GB/menu/methoden/default.htm (ask countries for additional examples of metadata/documentation)

 

The Future of Metadata
As statistical processes evolve, there is a push to make metadata a driver of statistical business process design and to standardize the collection of metadata within and across different international, statistical organisations.  While these efforts to streamline rules and procedures for metadata are still in development, the goal is to create an integrated approach to producing and recording metadata.  
Examples of international metadata and exchange protocols are the Data Documentation Initiative (DDI) and the Statistical Data and Metadata Exchange (SDMX) being developed by Eurostat, on behalf of seven international organizations (Eurostat, OECD, UNSD, IMF, WB, ECB, and IBS). With this increased international cooperation, it is hoped that a standardized framework can be established that will increase the comparability of statistical data across countries and international organisations.
In the future, it is expected that these metadata can be used on a proactive basis to prescribe definitions, concepts, variables and standards to surveys that are being designed or redeveloped.  This will help to ensure consistency and comparability of the data to be collected.  Refer to Box 8.6 provides for a list of metadata resources and guidelines.

Box 8.6   Metadata Resources and Guidelines OECD Organisation for Economic Co-operation and Development (2007). Data and metadata reporting and presentation handbook. OECD, Paris. http://www.oecd.org/std/37671574.pdf United Nations United Nations (2012). Metis Wiki. UN, Last updated: March 21, 2013 http://www1.unece.org/stat/platform/display/metis/METIS-wiki United Nations (2011). International Recommendations for Energy Statistics. Chapter 9B, Prepared by the United Nations Statistics Division, New York. http://unstats.un.org/unsd/statcom/doc11/BG-IRES.pdf United Nations (2012). United Nations Common Metadata Framework. UN, Last updated: March 21, 2013 http://www1.unece.org/stat/platform/display/metis/The+Common+Metadata+Framework United Nations (2009). Statistical Metadata in a Corporate Context. UN, New York. http://www1.unece.org/stat/platform/display/metis/Part+A+-+Statistical+Metadata+in+a+Corporate+Context Australian Bureau of Statistics Australian Bureau of Statistics (2012). What is Metadata? ABS, Last updated: February 2, 2012. http://www.abs.gov.au/websitedbs/a3121120.nsf/home/statistical+language+-+what+is+metadata Statistics Finland Statistics Finland (2005). Metadata Guidelines. Statistics Finland, Last updated: June 6, 2005. http://www.stat.fi/meta/index_en.html