Skip to main content.
home | support | download

Back to List Archive

Reading the spider report

From: Justin Tang <justin.tang(at)>
Date: Wed Jun 02 2004 - 17:33:23 GMT
  After running, it produced a summary of the site it had just
spidered.  I was wondering if there is a guide to reading the summary, as to
what each field means.  This is what the report looks like:

Summary for:

    Duplicates: 796  (1.5/sec)
MD5 Duplicates:   1  (0.0/sec)
Off-site links: 164  (0.3/sec)
       Skipped: 114  (0.2/sec)
   Unique URLs: 108  (0.2/sec)
    robots.txt:   3  (0.0/sec)


Received on Wed Jun 2 10:33:24 2004