Finding Goverment Information - a Challenging Quest

OMB Watch and the Center for Democracy and Technology have released an 18 page report entitled Hiding in Plain Sight: Why Important Government Information Cannot Be Found Through Commercial Search Engines. The Web sites of many federal agencies block web crawlers from commonly-used search engines such as Google, Yahoo, Microsoft Live, and Ask. Even searches done on USA.gov can miss crucial information that Government Information Librarians know to find directly on agency Web sites, but the public might miss without expert help. To remedy the situation, the report makes the following recommendations:

• Congress should pass the E-Government reauthorization act, which would require the Office of Management and Budget (OMB) to create best practices to encourage searchability of federal Web sites.
• OMB should officially recognize the importance of commercial search engines to Internet users and work with the CIO Council to adopt policies to help users find information.
• Agencies should adopt an information policy that makes public accessibility of online content and resources a priority.
• Agencies should create Sitemaps of content on their sites, with special attention given to materials stored in databases and accessible only through drop-down menus. For example, many agencies have FAQ databases that are not accessible to search crawlers but contain very succinct and useful answers to common questions.
• Agencies should review their use of robots.txt files in order to ensure they are used in the least restrictive way possible. Every effort should be made to include, rather than exclude, materials from the website, whether materials were excluded purposefully or accidentally in the past.