MooseFS 2.x 简单性能测试
2015-05-19 by dongnan
环境
操作系统: CentOS 6.6 amd64
配置
ChunkServer: 物理机 4Core CPU/16GB Mem/单磁盘 1TB/1GB NIC
Master: VM 4Core CPU/4GB Mem/100GB Disk/1GB NIC
Client: VM 4Core CPU/4GB Mem/50GB Disk/1GB NIC
IP
master: 172.27.244.69
client: 172.27.244.99
chunk1: 172.27.244.31
chunk2: 172.27.244.32
chunk3: 172.27.244.33
步骤
大文件
# block=1M byte
dd if=/dev/zero of=folder3/1.img bs=1M count=5000
5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB) copied, 81.5635 s, 64.3 MB/s
小文件
du -sh html
100K html
# 执行脚本生成100万个小文件
time bash t-file.sh
real 396m39.467s
user 19m50.861s
sys 86m34.964s
# 脚本文件
#!/bin/bash
for ((d=1;d<=100;d++));do
mkdir ${d}
sleep 1
for ((f=0;f<=10000;f++));do
cp /mnt/html ${d}/${f}.html
done
sleep 1
done
小结
- 测试结果不理想，特别是小文件。
- 由于硬件条件有限，操作系统与
chunk存储使用同一块磁盘,导致系统iowait值非常高,所以不能下结论mfs性能低下。 - 可以肯定的是测试期间，
Master的CPU消耗不高，推论整个mfs集群吞吐量取决于chunk server磁盘数量和性能、以及网络性能。
Sysbench 测试
MFS 副本数为2：
mfsgetgoal /mnt/mfs/sysbench/
/mnt/mfs/sysbench/: 2
准备工作
# 输入命令
sysbench --test=fileio --file-total-size=10G prepare
sysbench 0.4.12: multi-threaded system evaluation benchmark
128 files, 81920Kb each, 10240Mb total
Creating files for the test...
顺序写
# 输入命令
sysbench --test=fileio --num-threads=1 --file-total-size=10G --file-test-mode=seqwr --max-time=300 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 1
Extra file open flags: 0
128 files, 80Mb each
10Gb total file size
Block size 16Kb
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing sequential write (creation) test
Threads started!
Done.
Operations performed: 0 Read, 655360 Write, 128 Other = 655488 Total
Read 0b Written 10Gb Total transferred 10Gb (58.582Mb/sec)
3749.22 Requests/sec executed
Test execution summary:
total time: 174.7988s
total number of events: 655360
total time taken by event execution: 170.7423
per-request statistics:
min: 0.13ms
avg: 0.26ms
max: 815.31ms
approx. 95 percentile: 0.47ms
Threads fairness:
events (avg/stddev): 655360.0000/0.00
execution time (avg/stddev): 170.7423/0.00
顺序读
# 输入命令
sysbench --test=fileio --num-threads=1 --file-total-size=10G --file-test-mode=seqrd --max-time=300 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 1
Extra file open flags: 0
128 files, 80Mb each
10Gb total file size
Block size 16Kb
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing sequential read test
Threads started!
Done.
Operations performed: 655360 Read, 0 Write, 0 Other = 655360 Total
Read 10Gb Written 0b Total transferred 10Gb (85.261Mb/sec)
5456.73 Requests/sec executed
Test execution summary:
total time: 120.1012s
total number of events: 655360
total time taken by event execution: 116.1584
per-request statistics:
min: 0.01ms
avg: 0.18ms
max: 296.27ms
approx. 95 percentile: 0.25ms
Threads fairness:
events (avg/stddev): 655360.0000/0.00
execution time (avg/stddev): 116.1584/0.00
随机写
# 输入命令
sysbench --test=fileio --num-threads=1 --file-total-size=10G --file-test-mode=rndwr --max-time=300 --max-requests=0 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 1
Extra file open flags: 0
128 files, 80Mb each
10Gb total file size
Block size 16Kb
Number of random requests for random IO: 0
Read/Write ratio for combined random IO test: 1.50
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random write test
Threads started!
Time limit exceeded, exiting...
Done.
Operations performed: 0 Read, 109000 Write, 139404 Other = 248404 Total
Read 0b Written 1.6632Gb Total transferred 1.6632Gb (5.6771Mb/sec)
363.33 Requests/sec executed
Test execution summary:
total time: 300.0005s
total number of events: 109000
total time taken by event execution: 49.0880
per-request statistics:
min: 0.15ms
avg: 0.45ms
max: 16.31ms
approx. 95 percentile: 0.70ms
Threads fairness:
events (avg/stddev): 109000.0000/0.00
execution time (avg/stddev): 49.0880/0.00
随机读
# 输入命令
sysbench --test=fileio --num-threads=1 --file-total-size=10G --file-test-mode=rndrd --max-time=300 --max-requests=0 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 1
Extra file open flags: 0
128 files, 80Mb each
10Gb total file size
Block size 16Kb
Number of random requests for random IO: 0
Read/Write ratio for combined random IO test: 1.50
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random read test
Threads started!
Time limit exceeded, exiting...
Done.
Operations performed: 280713 Read, 0 Write, 0 Other = 280713 Total
Read 4.2833Gb Written 0b Total transferred 4.2833Gb (14.62Mb/sec)
935.71 Requests/sec executed
Test execution summary:
total time: 300.0008s
total number of events: 280713
total time taken by event execution: 298.4950
per-request statistics:
min: 0.01ms
avg: 1.06ms
max: 250.08ms
approx. 95 percentile: 2.35ms
Threads fairness:
events (avg/stddev): 280713.0000/0.00
execution time (avg/stddev): 298.4950/0.00
不同环境测试结果不同,测试结果仅供参考.
系统负载测试
单客户端
客户端系统负载
Master系统负载
ChunkServer负载
其它ChunkServer负载
双客户端
Master系统负载
ChunkServer负载
其它ChunkServer负载
小结
Moosefs 并发写入，在相同的时间内写入双倍数据，观察得出结论，Master 与 客户端资源开销较小，写入瓶颈取决于 ChunkServer 磁盘数量与性能。