PerlのWAFのベンチ大会 - とほほのN88-BASIC日記

仕事の都合上、素のCGIとCGI::ApplicationとCatalystと今話題のMojoでmod_perlなどを使わない素のcgiでベンチとった。

結論から言うとCGI下では

-	Requests per second:	バージョン
素のCGI	60.54 [#/sec] (mean)	-
CGI.pm	19.56 [#/sec] (mean)	3.42
CGI::Application	16.20 [#/sec] (mean)	4.11
Mojo	6.40 [#/sec] (mean)	0.8009
Catalyst	3.29 [#/sec] (mean)	5.7015

って感じです。

見るとわかるとおりモジュールを使い始めた途端にパフォーマンスが劣化します。
Devel::Profilerでプロファイリングしてみると

Total Elapsed Time = 0.020035 Seconds
  User+System Time = 0.020035 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c  Name
 99.8   0.020  0.020      5   0.0040 0.0040  CGI::_compile
 0.00   0.000  0.020      1   0.0000 0.0200  CGI::init
 0.00   0.000  0.000      5   0.0000 0.0000  CGI::param
 0.00   0.000  0.000     12   0.0000 0.0000  CGI::self_or_default
 0.00   0.000  0.000      3   0.0000 0.0000  CGI::Util::rearrange
 0.00   0.000  0.000      3   0.0000 0.0000  CGI::charset
 0.00   0.000  0.000      3   0.0000 0.0000  CGI::Util::_rearrange_params
 0.00   0.000  0.000      1   0.0000 0.0000  CGI::save_request
 0.00   0.000  0.000      3   0.0000 0.0000  CGI::all_parameters
 0.00   0.000  0.020      1   0.0000 0.0200  CGI::new
 0.00   0.000  0.000      1   0.0000 0.0000  CGI::add_parameter
 0.00   0.000  0.000      1   0.0000 0.0000  CGI::Util::unescape

コンパイルが実行時間のほぼ全てに費やされていることがわかります。
ということは、mod_perlなどを使って最初にインスタンスを生成してしまい、再利用すれば高速化されるはずです。
特にCatalystは

Total Elapsed Time =  0.01923 Seconds
  User+System Time =  0.02023 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c  Name
 49.4   0.010  0.010      2   0.0050 0.0050  Catalyst::Response::content_length
 49.4   0.010  0.010      1   0.0100 0.0100  Catalyst::Engine::CGI::prepare_pat
                                             h
 0.00   0.000  0.000      6   0.0000 0.0000  Catalyst::execute
 0.00   0.000  0.010      1   0.0000 0.0101  Catalyst::prepare
 0.00   0.000  0.000      6   0.0000 0.0000  Catalyst::component
 0.00   0.000  0.000      5   0.0000 0.0000  Catalyst::Dispatcher::forward
 0.00   0.000  0.000      5   0.0000 0.0000  Catalyst::Dispatcher::_invoke_as_p
                                             ath
 0.00   0.000  0.000      6   0.0000 0.0000  Catalyst::Action::execute
 0.00   0.000  0.000     14   0.0000 0.0000  Catalyst::use_stats
 0.00   0.000  0.000      4   0.0000 0.0000  Catalyst::config
 0.00   0.000  0.000      6   0.0000 0.0000  Catalyst::get_action
 0.00   0.000  0.000      6   0.0000 0.0000  Catalyst::Action::dispatch
 0.00   0.000  0.000      5   0.0000 0.0000  Catalyst::forward
 0.00   0.000  0.000      3   0.0000 0.0000  Catalyst::Dispatcher::get_actions
 0.00   0.000  0.000      5   0.0000 0.0000  Catalyst::Request::path

となっているので更に高速化が見込めます。
というわけで、次回はこの辺を探ってみたいと思います。

以下使ったベンチマークソースと結果です。

素のCGI

[hide@local] $ ab -n 2000 -c 2 http://127.0.0.1/~hide/bench/cgi/index.cgi
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 200 requests
Completed 400 requests
Completed 600 requests
Completed 800 requests
Completed 1000 requests
Completed 1200 requests
Completed 1400 requests
Completed 1600 requests
Completed 1800 requests
Finished 2000 requests


Server Software:        Apache
Server Hostname:        127.0.0.1
Server Port:            80

Document Path:          /~hide/bench/cgi/index.cgi
Document Length:        10 bytes

Concurrency Level:      2
Time taken for tests:   33.35551 seconds
Complete requests:      2000
Failed requests:        0
Write errors:           0
Total transferred:      255660 bytes
HTML transferred:       20000 bytes
Requests per second:    60.54 [#/sec] (mean)
Time per request:       33.036 [ms] (mean)
Time per request:       16.518 [ms] (mean, across all concurrent requests)
Transfer rate:          7.54 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:    28   32   5.2     31      94
Waiting:       27   31   4.9     31      94
Total:         28   32   5.2     31      94

Percentage of the requests served within a certain time (ms)
  50%     31
  66%     32
  75%     33
  80%     33
  90%     34
  95%     36
  98%     50
  99%     61
 100%     94 (longest request

CGI.pm

[hide@local] $ ab -n 2000 -c 2 http://127.0.0.1/~hide/bench/cgi-pm/index.cgi
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 200 requests
Completed 400 requests
Completed 600 requests
Completed 800 requests
Completed 1000 requests
Completed 1200 requests
Completed 1400 requests
Completed 1600 requests
Completed 1800 requests
Finished 2000 requests


Server Software:        Apache
Server Hostname:        127.0.0.1
Server Port:            80

Document Path:          /~hide/bench/cgi-pm/index.cgi
Document Length:        10 bytes

Concurrency Level:      2
Time taken for tests:   102.233771 seconds
Complete requests:      2000
Failed requests:        0
Write errors:           0
Total transferred:      294400 bytes
HTML transferred:       20000 bytes
Requests per second:    19.56 [#/sec] (mean)
Time per request:       102.234 [ms] (mean)
Time per request:       51.117 [ms] (mean, across all concurrent requests)
Transfer rate:          2.81 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:    64  101  12.9    100     308
Waiting:       64  100  12.6     99     307
Total:         64  101  12.9    100     308

Percentage of the requests served within a certain time (ms)
  50%    100
  66%    101
  75%    102
  80%    102
  90%    105
  95%    108
  98%    126
  99%    180
 100%    308 (longest request)

CGI::Application

[hide@local] $ ab -n 2000 -c 2 http://127.0.0.1/~hide/bench/cgi-app/index.cgi
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 200 requests
Completed 400 requests
Completed 600 requests
Completed 800 requests
Completed 1000 requests
Completed 1200 requests
Completed 1400 requests
Completed 1600 requests
Completed 1800 requests
Finished 2000 requests


Server Software:        Apache
Server Hostname:        127.0.0.1
Server Port:            80

Document Path:          /~hide/bench/cgi-app/index.cgi
Document Length:        10 bytes

Concurrency Level:      2
Time taken for tests:   123.494355 seconds
Complete requests:      2000
Failed requests:        0
Write errors:           0
Total transferred:      292560 bytes
HTML transferred:       20000 bytes
Requests per second:    16.20 [#/sec] (mean)
Time per request:       123.494 [ms] (mean)
Time per request:       61.747 [ms] (mean, across all concurrent requests)
Transfer rate:          2.31 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:    78  122  14.1    120     360
Waiting:       77  121  14.0    119     359
Total:         78  122  14.1    120     360

Percentage of the requests served within a certain time (ms)
  50%    120
  66%    122
  75%    123
  80%    124
  90%    128
  95%    130
  98%    147
  99%    180
 100%    360 (longest request)

Mojo

[hide@local] $ ab -n 2000 -c 2 http://127.0.0.1/~hide/bench/mojo/index.cgi
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 200 requests
Completed 400 requests
Completed 600 requests
Completed 800 requests
Completed 1000 requests
Completed 1200 requests
Completed 1400 requests
Completed 1600 requests
Completed 1800 requests
Finished 2000 requests


Server Software:        Apache
Server Hostname:        127.0.0.1
Server Port:            80

Document Path:          /~hide/bench/mojo/index.cgi
Document Length:        10 bytes

Concurrency Level:      2
Time taken for tests:   312.562704 seconds
Complete requests:      2000
Failed requests:        0
Write errors:           0
Total transferred:      294000 bytes
HTML transferred:       20000 bytes
Requests per second:    6.40 [#/sec] (mean)
Time per request:       312.563 [ms] (mean)
Time per request:       156.281 [ms] (mean, across all concurrent requests)
Transfer rate:          0.92 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:   270  311  40.6    304    1026
Waiting:      268  309  40.6    302    1023
Total:        270  311  40.6    304    1026

Percentage of the requests served within a certain time (ms)
  50%    304
  66%    307
  75%    309
  80%    311
  90%    322
  95%    327
  98%    385
  99%    486
 100%   1026 (longest request)

Catalyst

[hide@ns.hide-k.local] $ ab -n 2000 -c 2 http://127.0.0.1/~hide/bench/catalyst/index.cgi
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 200 requests
Completed 400 requests
Completed 600 requests
Completed 800 requests
Completed 1000 requests
Completed 1200 requests
Completed 1400 requests
Completed 1600 requests
Completed 1800 requests
Finished 2000 requests


Server Software:        Apache
Server Hostname:        127.0.0.1
Server Port:            80

Document Path:          /~hide/bench/catalyst/index.cgi
Document Length:        10 bytes

Concurrency Level:      2
Time taken for tests:   607.841381 seconds
Complete requests:      2000
Failed requests:        0
Write errors:           0
Total transferred:      294000 bytes
HTML transferred:       20000 bytes
Requests per second:    3.29 [#/sec] (mean)
Time per request:       607.841 [ms] (mean)
Time per request:       303.921 [ms] (mean, across all concurrent requests)
Transfer rate:          0.47 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:   385  607  45.9    594    1193
Waiting:      381  600  45.9    588    1186
Total:        385  607  45.9    594    1193

Percentage of the requests served within a certain time (ms)
  50%    594
  66%    599
  75%    604
  80%    608
  90%    632
  95%    651
  98%    764
  99%    855
 100%   1193 (longest request)