跳转至

MooseFS 2.x 简单性能测试


2015-05-19 by dongnan

环境

操作系统: CentOS 6.6 amd64

配置

ChunkServer: 物理机 4Core CPU/16GB Mem/单磁盘 1TB/1GB NIC
Master: VM 4Core CPU/4GB Mem/100GB Disk/1GB NIC
Client: VM 4Core CPU/4GB Mem/50GB Disk/1GB NIC

IP

master: 172.27.244.69
client: 172.27.244.99
chunk1: 172.27.244.31
chunk2: 172.27.244.32
chunk3: 172.27.244.33

步骤

大文件

# block=1M byte
dd if=/dev/zero of=folder3/1.img bs=1M count=5000
5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB) copied, 81.5635 s, 64.3 MB/s

小文件

du -sh html
100K    html

# 执行脚本生成100万个小文件
time bash t-file.sh
real   396m39.467s
user    19m50.861s
sys    86m34.964s


# 脚本文件
#!/bin/bash
for ((d=1;d<=100;d++));do
    mkdir ${d}
    sleep 1

    for ((f=0;f<=10000;f++));do
        cp /mnt/html ${d}/${f}.html
    done
    sleep 1
done

小结

  • 测试结果不理想,特别是小文件。
  • 由于硬件条件有限,操作系统与chunk存储使用同一块磁盘,导致系统 iowait 值非常高,所以不能下结论 mfs 性能低下。
  • 可以肯定的是测试期间,Master的CPU消耗不高,推论整个mfs集群吞吐量取决于chunk server磁盘数量和性能、以及网络性能。

Sysbench 测试

MFS 副本数为2

mfsgetgoal /mnt/mfs/sysbench/
/mnt/mfs/sysbench/: 2

准备工作

# 输入命令
sysbench --test=fileio --file-total-size=10G prepare

sysbench 0.4.12:  multi-threaded system evaluation benchmark
128 files, 81920Kb each, 10240Mb total
Creating files for the test...

顺序写

# 输入命令
sysbench --test=fileio --num-threads=1 --file-total-size=10G --file-test-mode=seqwr --max-time=300 run

sysbench 0.4.12:  multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 1
Extra file open flags: 0
128 files, 80Mb each
10Gb total file size
Block size 16Kb
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing sequential write (creation) test
Threads started!
Done.

Operations performed: 0 Read, 655360 Write, 128 Other = 655488 Total
Read 0b  Written 10Gb  Total transferred 10Gb  (58.582Mb/sec)
3749.22 Requests/sec executed

Test execution summary:
    total time:                          174.7988s
    total number of events:              655360
    total time taken by event execution: 170.7423
    per-request statistics:
         min:                                  0.13ms
         avg:                                  0.26ms
         max:                                815.31ms
         approx.  95 percentile:               0.47ms

Threads fairness:
    events (avg/stddev):           655360.0000/0.00
    execution time (avg/stddev):   170.7423/0.00

顺序读

# 输入命令

sysbench --test=fileio --num-threads=1 --file-total-size=10G --file-test-mode=seqrd --max-time=300 run

sysbench 0.4.12:  multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 1
Extra file open flags: 0
128 files, 80Mb each
10Gb total file size
Block size 16Kb
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing sequential read test
Threads started!
Done.

Operations performed:  655360 Read, 0 Write, 0 Other = 655360 Total
Read 10Gb  Written 0b  Total transferred 10Gb  (85.261Mb/sec)
5456.73 Requests/sec executed

Test execution summary:
    total time:                          120.1012s
    total number of events:              655360
    total time taken by event execution: 116.1584
    per-request statistics:
         min:                                  0.01ms
         avg:                                  0.18ms
         max:                                296.27ms
         approx.  95 percentile:               0.25ms

Threads fairness:
    events (avg/stddev):           655360.0000/0.00
    execution time (avg/stddev):   116.1584/0.00

随机写

# 输入命令
sysbench --test=fileio --num-threads=1 --file-total-size=10G --file-test-mode=rndwr --max-time=300 --max-requests=0 run

sysbench 0.4.12:  multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 1
Extra file open flags: 0
128 files, 80Mb each
10Gb total file size
Block size 16Kb
Number of random requests for random IO: 0
Read/Write ratio for combined random IO test: 1.50
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random write test
Threads started!
Time limit exceeded, exiting...
Done.

Operations performed:  0 Read, 109000 Write, 139404 Other = 248404 Total
Read 0b  Written 1.6632Gb  Total transferred 1.6632Gb  (5.6771Mb/sec)
  363.33 Requests/sec executed

Test execution summary:
    total time:                          300.0005s
    total number of events:              109000
    total time taken by event execution: 49.0880
    per-request statistics:
         min:                                  0.15ms
         avg:                                  0.45ms
         max:                                 16.31ms
         approx.  95 percentile:              0.70ms

Threads fairness:
    events (avg/stddev):           109000.0000/0.00
    execution time (avg/stddev):   49.0880/0.00

随机读

# 输入命令
sysbench --test=fileio --num-threads=1 --file-total-size=10G --file-test-mode=rndrd --max-time=300 --max-requests=0 run

sysbench 0.4.12:  multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 1
Extra file open flags: 0
128 files, 80Mb each
10Gb total file size
Block size 16Kb
Number of random requests for random IO: 0
Read/Write ratio for combined random IO test: 1.50
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random read test
Threads started!
Time limit exceeded, exiting...
Done.

Operations performed:  280713 Read, 0 Write, 0 Other = 280713 Total
Read 4.2833Gb  Written 0b  Total transferred 4.2833Gb  (14.62Mb/sec)
  935.71 Requests/sec executed

Test execution summary:
    total time:                          300.0008s
    total number of events:              280713
    total time taken by event execution: 298.4950
    per-request statistics:
         min:                                  0.01ms
         avg:                                  1.06ms
         max:                                250.08ms
         approx.  95 percentile:               2.35ms

Threads fairness:
    events (avg/stddev):           280713.0000/0.00
    execution time (avg/stddev):   298.4950/0.00

不同环境测试结果不同,测试结果仅供参考.

系统负载测试

单客户端

客户端系统负载

Master系统负载

ChunkServer负载

其它ChunkServer负载

双客户端

Master系统负载

ChunkServer负载

其它ChunkServer负载

小结

Moosefs 并发写入,在相同的时间内写入双倍数据,观察得出结论,Master客户端资源开销较小,写入瓶颈取决于 ChunkServer 磁盘数量与性能。

欢迎关注微信公众号: 运维录

Back to top