We don't want protected pages to show up in search results. So I have
excluded the spider from looking at those pages using the robots.txt file in
the server document root.
Is the session ID set as a cookie? You might be able to hack the SWISH
spider to send a valid session cookie in its request headers.
> -----Original Message-----
> From: Nathaniel Meyers [mailto:Nathaniel.Meyers@segasoft.com]
> Sent: Wednesday, March 15, 2000 5:07 PM
> To: 'Halasz, Steve'
> Subject: RE: [SWISH-E] indexing templates and data files for dynamic
> It might. We'll give that a try. I wonder though ... We
> have many pages
> that are accessable only when the user has logged on and been given a
> session id (else the script defaults to a login screen). I
> wonder if we run
> swish without a session ID if it will attempt to index these
> pages on the
> content of the log in screen. Did you ever have a problem like this?
> Thanks for the response.
> >-----Original Message-----
> >From: Halasz, Steve [mailto:firstname.lastname@example.org]
> >Sent: Wednesday, March 15, 2000 1:57 PM
> >To: 'Nathaniel.Meyers@segasoft.com'
> >Subject: RE: [SWISH-E] indexing templates and data files for dynamic
> >Will running SWISH in http mode solve your problem? I do this
> >for a site
> >that is entirely dynamic pages and it works fine. Although it
> >takes a little
> >longer and probably hits the server harder too.
> >put -S http on the command line and make IndexDir in the
> >config file the URL
> >of where you want to start spidering.
> >> -----Original Message-----
> >> From: Nathaniel Meyers [mailto:Nathaniel.Meyers@segasoft.com]
> >> Sent: Wednesday, March 15, 2000 4:37 PM
> >> To: Multiple recipients of list
> >> Subject: [SWISH-E] indexing templates and data files for
> >dynamic pages
> >> I'm wondering if anyone has had any experience indexing pages
> >> on a site that
> >> pretty much has no static pages (nearly all pages are
> >> dynamically generated
> >> from template files). It looks like it shouldn't be too
> much work to
> >> develope something that will do this but I wonder if anyone
> >> has already
> >> attempted it.
> >> -Nathaniel Meyers
> >> Heat.Net
Received on Wed Mar 15 17:49:55 2000