Security and Web Search Engines

Date: Mon, 18 Mar 1996 18:49:51 -0800 (PST)
Originator: ciac-bulletin@cheetah.llnl.gov
From: fisher@bill.llnl.gov (John M. Fisher)
Subject: CIAC Notes 96-01

[...text cut by dan...]

=========================================================================
2) Security and Web Search Engines
=========================================================================

By John Fisher

A variety of very powerful search engines are available on the
Internet, including Netscape
(http://home.netscape.com/home/internet-search.html), Yahoo
(http://www.yahoo.com), Alta Vista (http://altavista.digital.com),
Lycos (http://www.lycos.com), and others. These engines, known as "web
crawlers," provide a means for users to find URLs matching
keywords. Their databases are populated by gathering up all of the
available information provided by any Web server that can be
found. The Alta Vista database, for example, contains an index over 30
gigabytes in size.

But, perhaps, these search engines have become a little too
powerful. Some Web sites have provided a few too many links, including
information related to system configuration. For example, doing a
search for keywords such as "root", "daemon:", "passwd", etc, will
return back a few stray /etc/passwd or /etc/group files, located on
systems that have very poor Web configurations. This is a way to
quickly find those Web sites that are not maintained very well and are
most vulnerable to attack.

So, the lesson here, if you maintain a Web site, be very sure of what
information you are making available. Make sure no URLs are available
which give configuration information about your system. With today's
advanced search engines, you may be opening yourself to unnecessary
problems.