Data Quality Guidelines

Section 515 - Data Quality Guidelines

The General Services Administration (GSA) provides information to the public about the current experience of GSA programs, the projected scope and impact of GSA programs in the future, and the effect of proposed changes to regulations involving GSA. Information products describe the impact of GSA programs on our economy, beneficiary populations, other government agencies and demographic and economic information on client agencies. GSA’s information products are used by government planners, vendors, policymakers, economists, the media and the public to analyze GSA’s product lines and services and their impact on the United States.

Information released by GSA is developed from reliable data sources using accepted data collection methods and is based on thoroughly reviewed analyses and models.

Pre-dissemination review procedures. The guidelines, below, describe procedures that GSA employs to assure the quality of its information products, including their utility, objectivity, integrity, transparency and reproducibility prior to disseminating information to the public.

1. Utility
Utility involves the usefulness of the information to its intended users. Utility is achieved by staying informed of information needs and developing new data, models, and information products.

Based on internal analyses of information requirements, convening and attending conferences, working with advisory committees, and sponsoring outreach activities, GSA keeps abreast of information needs with respect to the analyses of GSA programs. We base our analyses on the Office of Management and Budget (OMB) generic standard (Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies; Republication - February 22, 2002), which GSA has adopted as a preferred standard.

We also employ GSA administrative data, and surveys by other federal agencies or established survey organizations. When major needs for data related to GSA populations are identified, GSA conducts special purpose surveys to address these needs. GSA’s ongoing publication series and other information products are reviewed to ensure that they remain relevant and address current information needs. Based on internal product reviews, consultation with users, and in response to changing needs and emphases, content of ongoing information products is changed, new products are introduced and others discontinued. GSA prepares special reports and topical studies that address emerging information needs stemming from proposed changes in the law and related policy debates.

GSA also identifies requirements for simulation models based on generally accepted statistical, financial and scientific standards to support the preparation of analytic reports and policy studies, and modifies current models or develops new models accordingly. Where appropriate, contact information is available on each publication (and in some cases on each table of a publication) to allow feedback and questions by users.

2. Objectivity
Objectivity involves a focus on ensuring that information is accurate, reliable and unbiased and that information products are presented in an accurate, clear, complete and unbiased manner. Objectivity is achieved by using reliable data sources and sound analytical techniques, and preparing information products that use proven methods by qualified people who carefully reviewed those products.

Use Reliable Data Sources

Much of the information disseminated by GSA is based on GSA administrative data files. These files contain information used to manage GSA programs, including data to determine product and service pricing and to compute lease payments. GSA’s individual Service and Staff Offices conduct ongoing reviews of GSA data systems to ensure accuracy. In support of these activities, GSA employs outside contractors to review GSA’s quality control methodology and processes to confirm the validity of its review processes. GSA administrative data is also covered under GSA’s Financial Management Systems and conforms to the high standards of financial accountability demanded by these systems. These financial management systems are mandated by the Office of Management and Budget and are designed to provide complete, reliable, consistent, timely and useful management information to enable agencies to carry out their fiduciary responsibilities.

GSA-sponsored surveys are conducted using methodologies that are consistent with generally accepted statistical, financial, and scientific standards for all aspects of survey development, including sample frame development, statistical design of the survey sample, questionnaire design and testing, data collection, sampling and coverage errors, non-response analysis, imputation of missing data, weights and variance estimates. GSA surveys follow guidelines and policies set forth in the Paperwork Reduction Act and other regulations related to the conduct of government surveys. GSA also prepares information products using data produced or maintained by other federal agencies and established survey organizations. All such external survey and administrative data used by GSA are produced using generally accepted scientific and statistical methodologies.

Where samples from administrative data files are employed for analysis, sound statistical methods are employed to develop samples. Staff producing statistical publications are knowledgeable about the content, structure and limitations of the administrative data files and maintain working relations with staff who create, update and maintain these files. Where administrative files are linked to surveys for analysis, sound procedures for extracting and linking external data files are developed and tested by technically qualified staff. Samples from administrative files are evaluated to ensure that samples are representative of the underlying administrative data files. Estimates prepared from external data sources are reviewed to ensure that the data extraction and linkage processes were implemented correctly.

Use Sound Analytic Techniques

Analytical reports are prepared using a variety of analytical techniques from simple tabulations and descriptive summary statistics to multivariate statistical methods and econometric models. Analytical techniques are reviewed for their appropriateness to the data and the analysis being conducted and are clearly identified in reports.

GSA utilizes several simulation models, based on accepted statistical, financial, and scientific standards, to make estimates of the effects of economic trends and legislative and policy options on GSA products and programs now and in the future. Some simulation models have been or are being developed within GSA. Others are being developed under contracts. All contracts to develop simulation models provide for detailed documentation which describes the goals and objectives of the model, the data sources being used and the methodologies and assumptions employed. Contract reports are being made available on the Internet. Documentation is available for some simulation models that have been developed within GSA and will be prepared for others.

All simulation models are extensively tested and reviewed within GSA to verify that the computer programs that were developed to implement the model conform to the stated objectives. Where appropriate, historical simulations are developed to evaluate the success of a model in producing reasonable projections. Periodically, technical advisory committees are convened to review a model’s performance to help evaluate whether it meets its objectives.

Simulation models are based on GSA’s and its contractor’s best judgments of current and future business relationships and methods of projecting key program outcomes. These models are periodically updated to reflect input from internal and external reviews and research findings on economic and product relationships.

Preparation of analytical reports and policy studies

Information contained in analytical reports and policy studies is based on estimates derived from reliable administrative data files and external data sources. Analysts apply sound statistical, financial, scientific, and analytical techniques and are knowledgeable about the data sources and models being used.

All data sources are identified. When analyses are based on simulation model projections, the assumptions used to produce the projections are also identified as well as the rationale for the assumptions used and the impact of using alternative assumptions.

All analytic reports and policy studies are reviewed by technically qualified staff to ensure that analysis is valid, complete, unbiased, objective, and relevant. Analytic reports and policy studies considered to be more technically complex are also reviewed by subject matter experts outside of the originating component, to provide additional perspective and expertise.

Editorial review for accuracy and clarity of information in publications

All information products are edited and proofread before release to ensure clarity and coherence of the final report. Text is edited to ensure that the report is easy to read and grammatically correct, thoughts and arguments flow logically, and information is worded concisely and lucidly. Tables and charts are edited to ensure that they clearly and accurately illustrate and support points made in the text, and include concise but descriptive titles. Tables and charts clearly indicate the unit of measure and the universe being examined and all internal labels (column heads, row stubs, and panel headings) should accurately describe the information they contain. All changes made to a manuscript during the editing process are checked by a proofreader and reviewed and approved by the author.

Policy for correcting errors

If an error is detected before an initial mailing, GSA includes an errata notice with the mailing. If the mailing has been sent out, GSA issues an errata sheet with all subsequent publications, and, as appropriate, sends the errata sheet to all those who received the initial notice. Errata notices are put on the first page of the web version to inform both new and repeat site visitors about the mistake, and the corrected version of the document is posted on the web.

3. Integrity
Integrity refers to the security of information from unauthorized access or revision to ensure that the information is not compromised through corruption or falsification. To ensure the integrity of its administrative information, GSA has in place rigorous controls that have been identified as representing sound security practices.

GSA is highly protective of the confidentiality of information it holds through its policies and practices. GSA administers financial and product programs that touch the operations of almost every government agency and many vendors. GSA has in place programs and policies for securing GSA resources as required by the Government Information Security Reform Act (P.L. 106-387, Title X, Subtitle G). These security procedures address all major components of information security and apply to all GSA operating components.

GSA is subject to statutory requirements to protect the sensitive information it gathers and maintains on companies, products, and services. These requirements are contained in the following documents:

• Privacy Act of 1974
• Computer Security Act of 1987
• Office of Management and Budget (OMB) Circulars A-123, A-127 and A-130
• Government Information Security Reform Act
• Federal Managers’ Financial Integrity Act (FMFIA) of 1982
• Paperwork Reduction Act

It is important that GSA organizational elements make use of OMB’s Paperwork Reduction Act (PRA) clearance process to help improve the quality of information that the GSA organizational elements collect and disseminate to the public. For all proposed collections of information which will be disseminated to the public, GSA organizational elements should demonstrate in their PRA clearance submissions to OMB that the proposed collection of information will result in information that will be collected, maintained, and used in a way consistent with the OMB and GSA information quality guidelines.

4. Transparency and Reproducibility
If an agency is responsible for disseminating “influential” information, guidelines for dissemination should include a high degree of transparency about data and methods to facilitate its reproducibility by qualified third parties. Information is considered influential if it will have a substantial impact on important public policies or important private sector decisions. Since much of GSA’s statistical and analytical information products potentially have an impact on important public policies, GSA’s information that is subject to Section 515 should be highly transparent and capable of being reproduced by qualified persons.

GSA’s guidelines call for identification and documentation of data sets used in producing estimates and projections and clear descriptions of methods used to produce estimates and to develop model projections to make its results as transparent as possible. Many estimates and projections included in certain information products are not directly reproducible by the public because the underlying data sets used to produce them are confidential. However, some statistical publications that are based on publicly available data and whose programs are made available on request are fully reproducible by the public. And GSA is in the process of developing public-use versions of several data files, which will increase the reproducibility of estimates and projections to the extent possible while still protecting confidentiality. Some estimates and projections may not be easily reproduced by third parties due to the complexity and detail of the methods and data. In these cases greater emphasis is place on periodic review by outside panels of technical experts.

GSA also achieves transparency through wide dissemination of its information. Most reports and other data products are available both as printed and electronic documents. They are announced on the GSA website and most electronic versions can be accessed and downloaded directly from the GSA website. All documents posted on our website since June 21, 2001, are Section 508 compliant, making information available to an audience that includes persons who have a visual impairment and read online using assistive technology.

5.Influential Information
Government information that is particularly influential needs to meet higher quality standards, and in particular must be reproducible. Per the OMB guidelines, information is designated as influential if the the Agency determines that the information is reasonably likely to have a clear and substantial impact on public policies or private sector decisions if disseminated. Scientific, financial, and statistical information all may be considered influential. Individual programs within the General Services Administration may designate certain classes of scientific, financial, and statistical information as influential.

An example of what GSA would consider to be influential data is GSA’s consolidation of federal workspace as a result of teleworking practices that result in an overall and gradual reduction of leased workspace needed from the private sector. Or, with another example, if GSA policies and requirements regarding contractor practices would have an effect on a broad set of vendors, based on services offered or on broad geographic areas of the country.

As specified in the OMB guidelines, influential information must be accompanied by supporting documentation that allows an external user to understand clearly the information and be able to reproduce it, or understand the steps involved in producing it. With respect to original and supporting data related thereto, GSA will assure reproducibility for such data according to commonly accepted scientific, financial, or statistical standards for that type of data, taking into account any ethical and confidentiality constraints. In the case of influential analytic results, the mathematical and statistical processes used to produce the report must be described in sufficient detail to allow an independent analyst to substantially reproduce the findings using the original data and identical methods. In situations where the public cannot access the data and methods due to other compelling interests such as privacy, intellectual property or other confidentiality protections, GSA will apply especially rigorous robustness checks to analytic results and document what checks were undertaken.

6. GSA Reporting Requirements
On an annual fiscal-year basis, the GSA CIO will report to the Director of the Office of Management and Budget concerning requests for correction received under these Guidelines. GSA organizational elements must designate a reporting official, except as agreed otherwise between the GSA organizational element and the CIO, for example, where the CIO might compile the data for the GSA organizational element. Where a GSA organizational element reporting official has been designated, that official must report to the CIO no later than December 15 every year concerning requests received during the previous fiscal year and their resolutions, including requests with regard to information subject to public comment. The CIO compiled the GSA consolidated report and submitted the first report to OMB January 1, 2004. GSA organizational element reports contain the number of complaints received, nature of complaints (e.g., request for deletion or correction) and how they were resolved (e.g., number corrected, denied, or pending review). The report also includes a compilation of the number of staff hours devoted to handling and resolving such complaints and preparing reports.