Swish-e

From: Bill Moseley <moseley(at)not-real.hank.org> Date: Thu Mar 27 2003 - 07:15:59 GMT

Hi Brian,

Here's some testing on my machine -- didn't see any problems like you are
describing. Maybe you can try this code with your index and see if you
have problems.  I still suspect a problem with the size or specific data
in your index.  You might try indexing a smaller set of docs and see if
you can repeat the problem.

I only indexed about 2000 documents in my /usr/share/doc directory, but I
stored a fair amount in the property file - 40,000 bytes max in the
description property.

My search returns 183 hits, and I "display" the first 20 records.

Here's ab -n 1000 -c 20:

Finished 1000 requests
Server Software:        Apache/1.3.26                                      
Server Hostname:        localhost
Server Port:            4000

Document Path:          /
Document Length:        791595 bytes

Concurrency Level:      20
Time taken for tests:   49.883 seconds
Complete requests:      1000
Failed requests:        832
   (Connect: 0, Length: 832, Exceptions: 0)
Broken pipe errors:     0
Total transferred:      796312584 bytes
HTML transferred:       796147668 bytes
Requests per second:    20.05 [#/sec] (mean)
Time per request:       997.66 [ms] (mean)
Time per request:       49.88 [ms] (mean, across all concurrent requests)
Transfer rate:          15963.61 [Kbytes/sec] received

Note the size of the doc -- about 79,000 bytes.  Still getting 20 request
per second.

I had no errors in my error_log and all requests completed (the "Failed
requests" above is because the page I return has the PID number and
request count, so 832 of the 1000 files were not exactly 791597 bytes).

So if I reduce the "description" down to 200 bytes, then I get:

Finished 1000 requests
Server Software:        Apache/1.3.26                                      
Server Hostname:        localhost
Server Port:            4000

Document Path:          /
Document Length:        6066 bytes

Concurrency Level:      20
Time taken for tests:   6.952 seconds
Complete requests:      1000
Failed requests:        835
   (Connect: 0, Length: 835, Exceptions: 0)
Broken pipe errors:     0
Total transferred:      6237203 bytes
HTML transferred:       6074879 bytes
Requests per second:    143.84 [#/sec] (mean)
Time per request:       139.04 [ms] (mean)
Time per request:       6.95 [ms] (mean, across all concurrent requests)
Transfer rate:          897.18 [Kbytes/sec] received

143 request per second for a very simple test.  Not too bad.

Note if you don't cache the $swish handle then it drops down to about 27
requests per second.  Also note if you do cache the $swish handle (like I
do below) then you will want to hup the server after reindexing, or add
some code to stat the index file to see when it changes and the reopen.

Here's my complete test setup:

moseley@bumby:~/swish-e/example$ cat c
DefaultContents HTML2
StoreDescription HTML2 <body> 200
Indexonly .html .htm

moseley@bumby:~/swish-e/example$ cat httpd.conf

LoadModule mime_module /usr/lib/apache/1.3/mod_mime.so

ServerRoot /home/moseley/swish-e/example
ErrorLog error_log
PidFile httpd.pid
Listen 4000
ServerName bumby

<perl>
use lib '/home/moseley/swish-e/example';
</perl>

PerlModule Swishtest

<location />
    SetHandler perl-script
    PerlHandler Swishtest
</location>

I start Apache like:

 /usr/sbin/apache-perl -f /home/moseley/swish-e/example/httpd.conf

This is:

Apache/1.3.26 (Unix) Debian GNU/Linux mod_perl/1.26

moseley@bumby:~/swish-e/example$ cat Swishtest.pm 
package Swishtest;
use strict;
use SWISH::API;
use Apache::Constants;

my $index = '/home/moseley/swish-e/example/index.swish-e';

my @props = qw/
    swishtitle
    swishdocpath
    swishlastmodified
    swishdocsize
    swishrank
    swishdescription
    swishreccount
/;

my $swish;
my $count;

sub handler {
    my $r = shift;

    $count++;

    # Cache the swish object!

    $swish ||= SWISH::API->new( $index );
    $swish->AbortLastError # just killed the Server
               if $swish->Error;

    my $results = $swish->Query( "linux" );

    if ( $swish->Error ) {
         $r->log_error("[$$:$count] Query failed: " . $swish->ErrorString . ":" . $swish->LastErrorMsg );
         return SERVER_ERROR;
    }

    my $hits = $results->Hits;

    if ( !$hits ) {
         $r->log_error("[$$:$count] No hits" );
         return SERVER_ERROR;
    }

    #$r->log_error("[$$:$count] $hits hits" );

    $r->send_http_header('text/plain');
    $r->print("[$$:$count] Found $hits hits\n");

    for ( 1..20 ) {
        my $result = $results->NextResult;
        last unless $result;
        $r->print(  join( ":", map {  $result->Property($_) || '' } @props) . "\n");
    }
    return OK;
}

1;

Note that I also run apache in single process mode (-X) and ran 5000
request:

Requests per second:    161.48 [#/sec] (mean)

And didn't see any memory leaks -- stayed just like this:

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND
32228 moseley   15   0  3828 3828  1828 R    93.4  0.7   1:08 apache-perl

-- 
Bill Moseley moseley@hank.org

Swish-e

Re: Strange error message in logs