Привет, коллеги-проксмоксы! Застрял с очень медленным бэкапом некоторых наших KVM-машин.
Окружение: pve-manager/3.4-6/102d4547 (ядро: 2.6.32-39-pve)
Intel® Xeon® CPU E5-2620 v2, 64GB RAM
NFS-хранилище на Synology RS3614RPxs, подключённое через 10Gbit
Патч для дифференциального бэкапа от ayufan.eu
Результат pveperf на NFS шэре:
Code:
root@vmhost3:~# pveperf /mnt/pve/storage2
CPU BOGOMIPS: 100548.12
REGEX/SECOND: 1030793
HD SIZE: 7374.94 GB (10.20.30.247:/volume2/vms2)
FSYNCS/SECOND: 2357.50
DNS EXT: 7.76 ms
DNS INT: 14.09 ms
(..........)
Конфигурация затронутых ВМ:
debian 7, диск virtio qcow2 на NFS, кеш = none
150-200 ГБ данных внутри виртуального диска, в основном базы mysql и сайты (php/html/css файлы)
Сеть: только virtio
Пример лога бэкапа одной ВМ:
Code:
501: Jul 31 22:00:02 INFO: Starting Backup of VM 501 (qemu)
501: Jul 31 22:00:02 INFO: status = running
501: Jul 31 22:00:02 INFO: update VM 501: -lock backup
501: Jul 31 22:00:03 INFO: backup mode: snapshot
501: Jul 31 22:00:03 INFO: bandwidth limit: 50000 KB/s
501: Jul 31 22:00:03 INFO: ionice priority: 7
501: Jul 31 22:00:03 INFO: creating archive '/mnt/pve/storage2/dump/vzdump-qemu-501-2015_07_30-22_00_03.vma.lzo--differential-2015_07_31-22_00_02.vcdiff.lzo'
501: Jul 31 22:00:03 INFO: started backup task 'b9753c83-966e-4582-b1f7-2b4397b1d15d'
501: Jul 31 22:00:06 INFO: status: 0% (102170624/429496729600), sparse 0% (33157120), duration 3, 34/23 MB/s
501: Jul 31 22:13:23 INFO: status: 1% (4299030528/429496729600), sparse 0% (104890368), duration 800, 5/5 MB/s
501: Jul 31 22:29:10 INFO: status: 2% (8594259968/429496729600), sparse 0% (178905088), duration 1747, 4/4 MB/s
501: Jul 31 22:31:38 INFO: status: 3% (12897091584/429496729600), sparse 0% (246472704), duration 1895, 29/28 MB/s
501: Jul 31 22:39:39 INFO: status: 4% (17180917760/429496729600), sparse 0% (318775296), duration 2376, 8/8 MB/s
501: Jul 31 22:49:57 INFO: status: 5% (21478768640/429496729600), sparse 0% (418586624), duration 2994, 6/6 MB/s
501: Jul 31 23:02:53 INFO: status: 6% (25771376640/429496729600), sparse 0% (480432128), duration 3770, 5/5 MB/s
501: Jul 31 23:18:17 INFO: status: 7% (30070407168/429496729600), sparse 0% (559599616), duration 4694, 4/4 MB/s
501: Jul 31 23:28:31 INFO: status: 8% (34361835520/429496729600), sparse 0% (643051520), duration 5308, 6/6 MB/s
501: Jul 31 23:35:32 INFO: status: 9% (38663815168/429496729600), sparse 0% (711585792), duration 5729, 10/10 MB/s
501: Jul 31 23:40:29 INFO: status: 10% (42963697664/429496729600), sparse 0% (782962688), duration 6026, 14/14 MB/s
501: Jul 31 23:44:41 INFO: status: 11% (47258927104/429496729600), sparse 0% (848728064), duration 6278, 17/16 MB/s
501: Jul 31 23:53:23 INFO: status: 12% (51542753280/429496729600), sparse 0% (921100288), duration 6800, 8/8 MB/s
501: Jul 31 23:57:59 INFO: status: 13% (55854563328/429496729600), sparse 0% (984903680), duration 7076, 15/15 MB/s
501: Aug 01 00:04:06 INFO: status: 14% (60140814336/429496729600), sparse 0% (1057447936), duration 7443, 11/11 MB/s
501: Aug 01 00:11:56 INFO: status: 15% (64428507136/429496729600), sparse 0% (1136947200), duration 7913, 9/8 MB/s
501: Aug 01 00:21:00 INFO: status: 16% (68726816768/429496729600), sparse 0% (1207209984), duration 8457, 7/7 MB/s
501: Aug 01 00:31:14 INFO: status: 17% (73018900480/429496729600), sparse 0% (1254965248), duration 9071, 6/6 MB/s
501: Aug 01 00:50:38 INFO: status: 18% (77323108352/429496729600), sparse 0% (1299681280), duration 10235, 3/3 MB/s
..... обрезано ...
501: Aug 01 10:19:34 INFO: status: 46% (197576753152/429496729600), sparse 0% (2959114240), duration 44371, 3/3 MB/s
501: Aug 01 10:41:44 INFO: status: 47% (201864380416/429496729600), sparse 0% (3021684736), duration 45701, 3/3 MB/s
501: Aug 01 10:43:55 ERROR: interrupted by signal
501: Aug 01 10:43:55 INFO: aborting backup job
С другой стороны, начальный полный бэкап ВСЕХ ВМ на этом хосте хоть как-то завершился за 11:39:26 на 746.47 ГБ
Code:
501: Jul 30 22:00:04 INFO: Starting Backup of VM 501 (qemu)
501: Jul 30 22:00:04 INFO: status = running
501: Jul 30 22:00:04 INFO: update VM 501: -lock backup
501: Jul 30 22:00:05 INFO: backup mode: snapshot
501: Jul 30 22:00:05 INFO: bandwidth limit: 50000 KB/s
501: Jul 30 22:00:05 INFO: ionice priority: 7
501: Jul 30 22:00:05 INFO: creating archive '/mnt/pve/storage2/dump/vzdump-qemu-501-2015_07_30-22_00_03.vma.lzo'
501: Jul 30 22:00:05 INFO: started backup task '30d33d70-b88b-4a25-98df-95bef7f5b766'
501: Jul 30 22:00:08 INFO: status: 0% (154533888/429496729600), sparse 0% (36929536), duration 3, 51/39 MB/s
501: Jul 30 22:01:41 INFO: status: 1% (4332978176/429496729600), sparse 0% (102297600), duration 96, 44/44 MB/s
501: Jul 30 22:03:16 INFO: status: 2% (8637775872/429496729600), sparse 0% (170512384), duration 191, 45/44 MB/s
501: Jul 30 22:04:44 INFO: status: 3% (12923305984/429496729600), sparse 0% (237907968), duration 279, 48/47 MB/s
...
501: Jul 31 00:30:34 INFO: status: 98% (420907253760/429496729600), sparse 3% (15786156032), duration 9029, 49/46 MB/s
501: Jul 31 00:32:00 INFO: status: 99% (425203662848/429496729600), sparse 4% (20001759232), duration 9115, 49/0 MB/s
501: Jul 31 00:33:25 INFO: status: 100% (429496729600/429496729600), sparse 5% (24294825984), duration 9200, 50/0 MB/s
501: Jul 31 00:33:25 INFO: transferred 429496 MB in 9200 seconds (46 MB/s)
501: Jul 31 00:33:25 INFO: archive file size: 184.21GB
501: Jul 31 00:33:26 INFO: Finished Backup of VM 501 (02:33:23)
Есть идеи, почему дифференциальный бэкап идёт так чертовски медленно? Какие есть альтернативы или рекомендации по настройке надёжного бэкапа без постоянного хранения полного образа машины?
Окружение: pve-manager/3.4-6/102d4547 (ядро: 2.6.32-39-pve)
Intel® Xeon® CPU E5-2620 v2, 64GB RAM
NFS-хранилище на Synology RS3614RPxs, подключённое через 10Gbit
Патч для дифференциального бэкапа от ayufan.eu
Результат pveperf на NFS шэре:
Code:
root@vmhost3:~# pveperf /mnt/pve/storage2
CPU BOGOMIPS: 100548.12
REGEX/SECOND: 1030793
HD SIZE: 7374.94 GB (10.20.30.247:/volume2/vms2)
FSYNCS/SECOND: 2357.50
DNS EXT: 7.76 ms
DNS INT: 14.09 ms
(..........)
Конфигурация затронутых ВМ:
debian 7, диск virtio qcow2 на NFS, кеш = none
150-200 ГБ данных внутри виртуального диска, в основном базы mysql и сайты (php/html/css файлы)
Сеть: только virtio
Пример лога бэкапа одной ВМ:
Code:
501: Jul 31 22:00:02 INFO: Starting Backup of VM 501 (qemu)
501: Jul 31 22:00:02 INFO: status = running
501: Jul 31 22:00:02 INFO: update VM 501: -lock backup
501: Jul 31 22:00:03 INFO: backup mode: snapshot
501: Jul 31 22:00:03 INFO: bandwidth limit: 50000 KB/s
501: Jul 31 22:00:03 INFO: ionice priority: 7
501: Jul 31 22:00:03 INFO: creating archive '/mnt/pve/storage2/dump/vzdump-qemu-501-2015_07_30-22_00_03.vma.lzo--differential-2015_07_31-22_00_02.vcdiff.lzo'
501: Jul 31 22:00:03 INFO: started backup task 'b9753c83-966e-4582-b1f7-2b4397b1d15d'
501: Jul 31 22:00:06 INFO: status: 0% (102170624/429496729600), sparse 0% (33157120), duration 3, 34/23 MB/s
501: Jul 31 22:13:23 INFO: status: 1% (4299030528/429496729600), sparse 0% (104890368), duration 800, 5/5 MB/s
501: Jul 31 22:29:10 INFO: status: 2% (8594259968/429496729600), sparse 0% (178905088), duration 1747, 4/4 MB/s
501: Jul 31 22:31:38 INFO: status: 3% (12897091584/429496729600), sparse 0% (246472704), duration 1895, 29/28 MB/s
501: Jul 31 22:39:39 INFO: status: 4% (17180917760/429496729600), sparse 0% (318775296), duration 2376, 8/8 MB/s
501: Jul 31 22:49:57 INFO: status: 5% (21478768640/429496729600), sparse 0% (418586624), duration 2994, 6/6 MB/s
501: Jul 31 23:02:53 INFO: status: 6% (25771376640/429496729600), sparse 0% (480432128), duration 3770, 5/5 MB/s
501: Jul 31 23:18:17 INFO: status: 7% (30070407168/429496729600), sparse 0% (559599616), duration 4694, 4/4 MB/s
501: Jul 31 23:28:31 INFO: status: 8% (34361835520/429496729600), sparse 0% (643051520), duration 5308, 6/6 MB/s
501: Jul 31 23:35:32 INFO: status: 9% (38663815168/429496729600), sparse 0% (711585792), duration 5729, 10/10 MB/s
501: Jul 31 23:40:29 INFO: status: 10% (42963697664/429496729600), sparse 0% (782962688), duration 6026, 14/14 MB/s
501: Jul 31 23:44:41 INFO: status: 11% (47258927104/429496729600), sparse 0% (848728064), duration 6278, 17/16 MB/s
501: Jul 31 23:53:23 INFO: status: 12% (51542753280/429496729600), sparse 0% (921100288), duration 6800, 8/8 MB/s
501: Jul 31 23:57:59 INFO: status: 13% (55854563328/429496729600), sparse 0% (984903680), duration 7076, 15/15 MB/s
501: Aug 01 00:04:06 INFO: status: 14% (60140814336/429496729600), sparse 0% (1057447936), duration 7443, 11/11 MB/s
501: Aug 01 00:11:56 INFO: status: 15% (64428507136/429496729600), sparse 0% (1136947200), duration 7913, 9/8 MB/s
501: Aug 01 00:21:00 INFO: status: 16% (68726816768/429496729600), sparse 0% (1207209984), duration 8457, 7/7 MB/s
501: Aug 01 00:31:14 INFO: status: 17% (73018900480/429496729600), sparse 0% (1254965248), duration 9071, 6/6 MB/s
501: Aug 01 00:50:38 INFO: status: 18% (77323108352/429496729600), sparse 0% (1299681280), duration 10235, 3/3 MB/s
..... обрезано ...
501: Aug 01 10:19:34 INFO: status: 46% (197576753152/429496729600), sparse 0% (2959114240), duration 44371, 3/3 MB/s
501: Aug 01 10:41:44 INFO: status: 47% (201864380416/429496729600), sparse 0% (3021684736), duration 45701, 3/3 MB/s
501: Aug 01 10:43:55 ERROR: interrupted by signal
501: Aug 01 10:43:55 INFO: aborting backup job
С другой стороны, начальный полный бэкап ВСЕХ ВМ на этом хосте хоть как-то завершился за 11:39:26 на 746.47 ГБ
Code:
501: Jul 30 22:00:04 INFO: Starting Backup of VM 501 (qemu)
501: Jul 30 22:00:04 INFO: status = running
501: Jul 30 22:00:04 INFO: update VM 501: -lock backup
501: Jul 30 22:00:05 INFO: backup mode: snapshot
501: Jul 30 22:00:05 INFO: bandwidth limit: 50000 KB/s
501: Jul 30 22:00:05 INFO: ionice priority: 7
501: Jul 30 22:00:05 INFO: creating archive '/mnt/pve/storage2/dump/vzdump-qemu-501-2015_07_30-22_00_03.vma.lzo'
501: Jul 30 22:00:05 INFO: started backup task '30d33d70-b88b-4a25-98df-95bef7f5b766'
501: Jul 30 22:00:08 INFO: status: 0% (154533888/429496729600), sparse 0% (36929536), duration 3, 51/39 MB/s
501: Jul 30 22:01:41 INFO: status: 1% (4332978176/429496729600), sparse 0% (102297600), duration 96, 44/44 MB/s
501: Jul 30 22:03:16 INFO: status: 2% (8637775872/429496729600), sparse 0% (170512384), duration 191, 45/44 MB/s
501: Jul 30 22:04:44 INFO: status: 3% (12923305984/429496729600), sparse 0% (237907968), duration 279, 48/47 MB/s
...
501: Jul 31 00:30:34 INFO: status: 98% (420907253760/429496729600), sparse 3% (15786156032), duration 9029, 49/46 MB/s
501: Jul 31 00:32:00 INFO: status: 99% (425203662848/429496729600), sparse 4% (20001759232), duration 9115, 49/0 MB/s
501: Jul 31 00:33:25 INFO: status: 100% (429496729600/429496729600), sparse 5% (24294825984), duration 9200, 50/0 MB/s
501: Jul 31 00:33:25 INFO: transferred 429496 MB in 9200 seconds (46 MB/s)
501: Jul 31 00:33:25 INFO: archive file size: 184.21GB
501: Jul 31 00:33:26 INFO: Finished Backup of VM 501 (02:33:23)
Есть идеи, почему дифференциальный бэкап идёт так чертовски медленно? Какие есть альтернативы или рекомендации по настройке надёжного бэкапа без постоянного хранения полного образа машины?