This was on a T2000 box with 1 thread pinned at around 180,000 syscalls per second:

# vmstat 1

kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr 1m 1m 1m 1m in sy cs us sy id
0 0 0 45446528 21367256 0 0 0 0 0 0 0 0 0 0 0 739 180876 776 1 2 96
0 0 0 45446528 21367256 0 0 0 0 0 0 0 0 0 0 0 735 181411 777 2 2 96
0 0 0 45446528 21367256 0 1 0 0 0 0 0 13 13 0 0 720 180982 740 1 2 97
0 0 0 45446528 21367256 0 0 0 0 0 0 0 0 0 0 0 738 180826 755 1 2 96
0 0 0 45446528 21367256 0 0 0 0 0 0 0 0 0 0 0 831 180869 810 1 2 96
0 0 0 45446528 21367256 0 0 0 0 0 0 0 0 0 0 0 745 180902 794 1 2 96
0 0 0 45446528 21367256 0 1 0 0 0 0 0 0 0 0 0 708 181005 719 1 2 96
0 0 0 45446528 21367256 0 0 0 0 0 0 0 0 0 0 0 706 180835 717 2 2 97

Because this was on a T2000, and the most likely the application isn’t multithreaded, the calls were only pinned to one thread only affecting that thread and that core (4 threads share a core). mpstat showed this thread at around 180,000 syscalls per second and 0 idle time. I wonder if that is the max syscalls a thread can handle. Putting this process on another machine I bet the calls would of went way higher causing a huge performance hit. Unfortunately I didn’t get to investigate further.

.

.

.

.

Someone looking to buy domain names can get it from any web hosting company. Especially the companies like startlogic offer free domain name with their dedicated server. Additionally anyone looking for internet software can get it from such companies along with hosting services. But to handle these network software people need to have basic IT knowledge in the form of 310-055 and 156-215. Other than this people having 642-552 certification gets handy advantage over the others.

An update to my previous post, here are some results for Apache/PHP/MySQL. I used Apache 2.2.3 (prefork), PHP 5.2.1, MySQL 5.0.27, and APC 3.0.13. These were custom compiled pretty much the same options the CoolStack uses, with Sun Studio 11.

APC is the PHP caching extension for PHP, and the apc.stat setting in the php.ini tells it to check if the php file has changed and if it has, recompile it and cache it. Using apc.stat on or off didn’t seem to make much of a difference. These tests were done with apc.stat on. For Apache I have mem_cache turned on as well.

5 rows and 1 column of MySQL returning:

# ./ab -n 50000 -c 500 http://x.x.x.x:30000/test.php
This is ApacheBench, Version 2.0.41-dev < $Revision: 1.121.2.12 $> apache-2.0
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 2006 The Apache Software Foundation, http://www.apache.org/

Server Software: Apache/2.2.3
Server Hostname: x.x.x.x
Server Port: 30000

Document Path: /test.php
Document Length: 67 bytes

Concurrency Level: 500
Time taken for tests: 4.793971 seconds
Complete requests: 50000
Failed requests: 48583
(Connect: 0, Length: 48583, Exceptions: 0)
Write errors: 0
Total transferred: 11475208 bytes
HTML transferred: 823930 bytes
Requests per second: 10429.77 [#/sec] (mean)
Time per request: 47.940 [ms] (mean)
Time per request: 0.096 [ms] (mean, across all concurrent requests)
Transfer rate: 2337.52 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 4 3.8 4 30
Processing: 15 41 59.6 29 1036
Waiting: 14 40 59.1 28 1032
Total: 23 46 59.7 33 1042

36 rows and 6 columns of MySQL returning:

Requests per second: 10448.16 [#/sec] (mean)

134 rows and 7 columns of MySQL returning:

Requests per second: 801.05 [#/sec] (mean)

Performance really degrades on the 134 row test and I think MySQL is the bottleneck here. More to come on this when I get MySQL recompiled.

UPDATE: This is the results without APC cache turned on at all, with 36 rows and 6 columns:

Requests per second: 2242.18 [#/sec] (mean)

As you can see APC cache helps a lot, it went from 10,000 req/sec to 2,000 req/sec.

Â

These tests were done with 2.2.3 prefork and 2.2.4 with worker. Both of them were tested with mem_cache and without. mem_cache seemed to really improve the performance of prefork but it didn’t do anything for worker, it actually hurt the performance by a little bit. Worker also seemed to drop a core file when it got too much traffic. The index.html is a small 175 byte file that loads a 5 of the apache powered by gif files and the httpd.conf were both stripped with minimal modules loaded. From the transfer rate apache bench shows I think the bottleneck here is the network.

The mem_cache settings:

CacheEnable mem /
MCacheSize 320000
MCacheMaxObjectCount 1000
MCacheMinObjectSize 1
MCacheMaxObjectSize 4048

Apache 2.2.3, prefork with mem_cache (-n 100000 -c 1000):

Requests per second: 11566.21 [#/sec] (mean)

Apache 2.2.3, prefork without mem_cache (-n 100000 -c 1000):

Requests per second: 9492.40 [#/sec] (mean)

Apache 2.2.4, worker with mem_cache (-n 100000 -c 1000):

Requests per second: 12118.41 [#/sec] (mean)

Apache 2.2.4, worker without mem_cache (-n 100000 -c 1000):

Requests per second: 12413.07 [#/sec] (mean)

I only had one machine to use for benchmark testing and apache bench seemed to be the only benchmark to hold the amount of requests. Siege would report around 2-3,000 req/sec but would bum out after that. This was Apache 2.2.3 compiled with Sun Studio 11 on Solaris 10 1/06:

export CFLAGS=’-DSSL_EXPERIMENTAL -DSSL_ENGINE -xO4 -xtarget=generic’
export LDFLAGS=”-R/usr/ucblib”

# ./configure –prefix=/path/apache2 –enable-mods-shared=all –enable-cache –enable-mem-cache –enable-ssl –with-mpm=prefork –enable-so –enable-rule=SSL_EXPERIMENTAL –with-ssl=/usr/sfw

Amount of time it took to build Apache 2.2.3 with 24 threads:

# time dmake -j 24
real 6m4.925s
user 18m12.183s
sys 13m37.746s

Results:

# ab -n 40000 -c 30 http://x.x.x.x:30000/index.html
This is ApacheBench, Version 1.3d < $Revision: 1.73 $> apache-1.3
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 2006 The Apache Software Foundation, http://www.apache.org/

Benchmarking x.x.x.x (be patient)
Completed 4000 requests
Completed 8000 requests
Completed 12000 requests
Completed 16000 requests
Completed 20000 requests
Completed 24000 requests
Completed 28000 requests
Completed 32000 requests
Completed 36000 requests
Finished 40000 requests
Server Software: Apache/2.2.3
Server Hostname: x.x.x.x
Server Port: 30000

Document Path: /index.html
Document Length: 125 bytes

Concurrency Level: 30
Time taken for tests: 4.114 seconds
Complete requests: 40000
Failed requests: 0
Broken pipe errors: 0
Total transferred: 16001200 bytes
HTML transferred: 5000375 bytes
Requests per second: 9722.90 [#/sec] (mean)
Time per request: 3.09 [ms] (mean)
Time per request: 0.10 [ms] (mean, across all concurrent requests)
Transfer rate: 3889.45 [Kbytes/sec] received

Connnection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.9 0 16
Processing: 0 2 0.8 2 14
Waiting: 0 2 0.7 1 14
Total: 0 3 1.3 2 18

Percentage of the requests served within a certain time (ms)
50% 2
66% 3
75% 3
80% 3
90% 4
95% 6
98% 7
99% 9
100% 18 (last request)

httpd-mpm.conf prefork settings:

ServerLimit 8192
MaxClients 2048

StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 2048
MaxRequestsPerChild 0
ListenBacklog 8192
ServerLimit 2048

I couldn’t get it to go much over 9000-10000 req/sec with any more tweaking to the apache config. From checking other people’s results it looks like I should be able to get around double that using Nevada and certain OS tunings that I don’t have here. More to come later with PHP/MySQL/SSL.

I’ve been doing some testing at work with a T2000 and MySQL and I am getting outstanding numbers with complex queries (by default sysbench uses complex queries to test):

$ ./sysbench –num-threads=24 –test=oltp –mysql-host=x.x.x.x –mysql-table-engine=innodb –mysql-port=3300 –oltp-table-size=1000000 run
sysbench v0.4.8: multi-threaded system evaluation benchmark

OLTP test statistics:
queries performed:
read: 140000
write: 50000
other: 20000
total: 210000
transactions: 10000 (795.16 per sec.)
deadlocks: 0 (0.00 per sec.)
read/write requests: 190000 (15108.11 per sec.)
other operations: 20000 (1590.33 per sec.)

Test execution summary:
total time: 12.5760s
total number of events: 10000
total time taken by event execution: 301.3571
per-request statistics:
min: 0.0204s
avg: 0.0301s
max: 0.1999s
approx. 95 percentile: 0.0372s

Threads fairness:
events (avg/stddev): 416.6667/2.67
execution time (avg/stddev): 12.5565/0.01

This is on Solaris 10 1/06 using the CoolStack AMP Cluster 64-bit MySQL. For MySQL to be tuned you need to use the libfasttime, mtmalloc libararies, all the latest and greatest OS tunings from http://www.sun.com/servers/coolthreads/tnb/parameters.jsp, and a proper configuration file. Here is my MySQL config file:

#
# MySQL configure
#
# The MySQL server
[mysqld]
port=3300
socket=/srcnew/usr/local/mysql/mysql.sock
user=root
datadir=/srcnew/usr/local/mysql/var
basedir=/srcnew/usr/local/mysql
max_connections=1600
max_connect_errors=10
table_cache=2048
max_allowed_packet=1048576
binlog_cache_size=1048576
key_buffer_size=16777216
max_heap_table_size=67108864
sort_buffer_size=65536
join_buffer_size=1048576
query_cache_size=26214400
thread_cache=16
thread_concurrency=32
thread_stack=64K
ft_min_word_len=4
default_table_type=MYISAM
transaction_isolation=REPEATABLE-READ
tmp_table_size=64M
skip-locking
skip-concurrent-insert
server-id=1
#######below is for innodb############
innodb_status_file=0
innodb_data_file_path=ibdata1:100M:autoextend
innodb_buffer_pool_size=1800M
innodb_additional_mem_pool_size=20M
innodb_log_file_size=200M
innodb_log_files_in_group=2
innodb_log_buffer_size=80M
innodb_flush_log_at_trx_commit=1
innodb_lock_wait_timeout=300
innodb_max_dirty_pages_pct=90
innodb_thread_concurrency=32
innodb_sync_spin_loops=200

The compile options for this are:

CC=cc
CFLAGS=”-xO4 -mt -fsimple=1 -ftrap=%none -xbuiltin=%all -xlibmil -xlibmopt -xstrconst -xarch=v9″
CXX=CC
CXXFLAGS=”-xO4 -noex -mt -fsimple=1 -ftrap=%none -xbuiltin=%all -xlibmil -xlibmopt -xarch=v9″
./configure –prefix=/usr/local/mysql –libexecdir=/usr/local/mysql/bin –with-extra-charsets=complex –with-server-suffix=-standard –enable-thread-saf
e-client –enable-local-infile –with-named-z-libs=no –with-big-tables –disable-shared –with-readline –with-archive-storage-engine –with-innodb

And here’s my results for simple queries:

$ ./sysbench –num-threads=24 –test=oltp –mysql-host=x.x.x.x –mysql-table-engine=innodb –mysql-port=3300 –oltp-test-mode=simple –oltp-table-size=1000000 run

OLTP test statistics:
queries performed:
read: 10000
write: 0
other: 0
total: 10000
transactions: 10000 (40223.97 per sec.)
deadlocks: 0 (0.00 per sec.)
read/write requests: 10000 (40223.97 per sec.)
other operations: 0 (0.00 per sec.)

Test execution summary:
total time: 0.2486s
total number of events: 10000
total time taken by event execution: 5.9262
per-request statistics:
min: 0.0004s
avg: 0.0006s
max: 0.0086s
approx. 95 percentile: 0.0009s

Threads fairness:
events (avg/stddev): 416.6667/17.53
execution time (avg/stddev): 0.2469/0.00

Here are good resources for tuning MySQL:

http://developers.sun.com/solaris/articles/mysql_perf_tune.html
http://blogs.digitar.com/media/2/T2000_Experience.pdf