Web server directory index explained

When an HTTP client (generally a web browser) requests a URL that points to a directory structure instead of an actual web page within the directory structure, the web server will generally serve a default page, which is often referred to as a main or "index" page.

A common filename for such a page is '''index.[[HTML|html]]''', but most modern HTTP servers offer a configurable list of filenames that the server can use as an index. If a server is configured to support server-side scripting, the list will usually include entries allowing dynamic content to be used as the index page (e.g. index.[[Common Gateway Interface|cgi]], index.[[Perl|pl]], index.[[PHP|php]], index.[[Server Side Includes|shtml]], index.[[JavaServer Pages|jsp]], default.[[Active Server Pages|asp]]) even though it may be more appropriate to still specify the HTML output (index.html.php or index.html.aspx), as this should not be taken for granted. An example is the popular open source web server Apache, where the list of filenames is controlled by the DirectoryIndex[1] directive in the main server configuration file or in the configuration file for that directory. It is possible to not use file extensions at all, and be neutral to content delivery methods, and set the server to automatically pick the best file through content negotiation.

If the server is unable to find a file with any of the names listed in its configuration, it may either return an error (usually 403 Index Listing Forbidden or 404 Not Found) or generate its own index page listing the files in the directory. Usually this option, often named autoindex, is also configurable.[2]

History

A scheme where web server serves a default file on per-subdirectory basis has been supported as early as NCSA HTTPd 0.3beta (22 April 1993),[3] which defaults to serve index.html file in the directory.[3] [4] This scheme has been then adopted by CERN HTTPd since at least 2.17beta (5 April 1994), whose default supports Welcome.html and welcome.html in addition to the NCSA-originated index.html.[5]

Later web servers typically support this default file scheme in one form or another; this is usually configurable, with index.html being one of the default file names.[6] [7] [8]

Implementation

In some cases, the home page of a website can be a menu of language options for large sites that use geotargeting. It is also possible to avoid this step, for example, by using content negotiation.

In cases where no known index.* file exists within a given directory, the web server may be configured to provide an automatically generated listing of the files within the directory instead. With the Apache web server, for example, this behavior is provided by the mod_autoindex module[9] and controlled by the Options +Indexes directive[10] in the web server configuration files. These automated directory listings are sometimes a security risk because they enumerate sensitive files which may not be intended for public access, in a process known as a directory indexing attack.[11] Such a security misconfiguration[12] may also assist in other attacks, such as a path or directory traversal attack.[13]

Performances

When accessing a directory, the various available index methods may also have a different impact on usage of OS resources (RAM, CPU time, etc.) and thus on web server performances.

Proceeding from fastest to slowest method, here is the list:

Notes and References

  1. Web site: mod_dir - Apache HTTP Server . httpd.apache.org . 2014-05-30.
  2. Web site: Directory listings. ASF Infrabot. Apache foundation: HTTPd server project. 2019-05-22. 2021-11-16. en.
  3. Web site: WWW-Talk Apr-Jun 1993: NCSA httpd version 0.3. 1997.webhistory.org.
  4. Web site: NCSA HTTPd DirectoryIndex. https://web.archive.org/web/20090131074833/http://hoohoo.ncsa.uiuc.edu/docs/setup/srm/DirectoryIndex.html. dead. January 31, 2009. January 31, 2009.
  5. Web site: Change History of W3C httpd. https://web.archive.org/web/19970605110106/http://www.w3.org/pub/WWW/Daemon/Features.html. dead. June 5, 1997. June 5, 1997.
  6. Web site: mod_dir - Apache HTTP Server Version 2.4 § DirectoryIndex Directive. live. https://web.archive.org/web/20201112002816/https://httpd.apache.org/docs/2.4/mod/mod_dir.html. 2020-11-12. 2021-01-13. httpd.apache.org.
  7. Web site: NGINX Docs Serving Static Content. live. https://web.archive.org/web/20201111202956/https://docs.nginx.com/nginx/admin-guide/web-server/serving-static-content/. 2020-11-11. 2021-01-13. docs.nginx.com.
  8. Web site: Default Document Microsoft Docs. live. https://web.archive.org/web/20201208215905/https://docs.microsoft.com/en-us/iis/configuration/system.webserver/defaultdocument/. 2020-12-08. 2021-01-13. docs.microsoft.com.
  9. Web site: mod_autoindex - Apache HTTP Server Version 2.4. 2021-01-13. httpd.apache.org.
  10. Web site: core - Apache HTTP Server Version 2.4 § Options Directive. 2021-01-13. httpd.apache.org.
  11. Web site: IBM Docs . IBM . 2021-03-08 . 2021-05-07.
  12. Web site: A6:2017-Security Misconfiguration . OWASP . 2021-05-07.
  13. Web site: Path Traversal . OWASP . 2021-05-07.