Making free space on a Full Data Domain
Have a Data Domain that's full? Need to free up space so your backups continue to run? Check for old snapshots, expire them, then free up space with a file system cleaning.
Had a customer who's replication Data Domain was significantly (20% more!) more utilized than the primary site, and also was completely full. Which was causing replications to fail. When I checked out the snapshots on one of the Mtrees, I saw that there were 10 snapshots from almost a year ago. These snapshots together were using around 16Tb of the available 67Tb of space on the Data Domain.
SSH into the Data Domain as sysadmin. Use the mtree list command to show the Data Domain Mtree structure.
sysadmin@DataDomain# mtree list Name Pre-Comp (GiB) Status ---------------------------- -------------- ------ /data/col1/avamar-13896542221487697.5 RW /data/col1/avamar-1477523534 561707.6 RW /data/col1/backup 0.0 RW ---------------------------- -------------- ------
Checking the smaller of the Mtrees shows what we expect to see
sysadmin@DataDom1# snapshot list mtree /data/col1/avamar-1477523534 Snapshot Information for MTree: /data/col1/avamar-1477523534 ---------------------------------------------- NamePre-Comp (GiB) Create Date Retain UntilStatus ----------------- -------------- ----------------- ----------------- ------- cp.20170622161010 637792.0 Jun 22 2017 09:11 Jun 28 2017 08:50 expired cp.20170622164953 637792.0 Jun 22 2017 09:51 Jun 28 2017 08:51 expired cp.20170628151023 656374.0 Jun 28 2017 08:11 cp.20170628155016 661756.7 Jun 28 2017 08:51 ----------------- -------------- ----------------- ----------------- ------- Snapshot Summary ------------------- Total:4 Not expired:2 Expired:2
However, when we take a look at the larger of the Mtrees, we see where the problem lies.
sysadmin@DataDom1# snapshot list mtree /data/col1/avamar-1389654222 Snapshot Information for MTree: /data/col1/avamar-1389654222 ---------------------------------------------- NamePre-Comp (GiB) Create Date Retain Until Status ----------------- -------------- ----------------- ------------ ------ cp.201609101621301076444.1 Sep 10 2016 09:21 cp.201609111532331100654.4 Sep 11 2016 08:32 cp.201609131705211161823.8 Sep 13 2016 10:05 cp.201609171657211257219.9 Sep 17 2016 09:57 cp.201609181543091284386.1 Sep 18 2016 08:43 cp.201609191705561309786.6 Sep 19 2016 10:05 cp.201609201542051339382.0 Sep 20 2016 08:42 cp.201609211541431359362.4 Sep 21 2016 08:41 cp.201609221542311377744.1 Sep 22 2016 08:42 cp.201609281624221432382.5 Sep 28 2016 09:24 ----------------- -------------- ----------------- ------------ ------ Snapshot Summary ------------------- Total: 10 Not expired: 10 Expired:0
Ten snapshots from almost a year ago, taking up so much space is bad. So lets manually expire each snapshot.
sysadmin@DataDom1# snapshot expire cp.20160910162130 mtree /data/col1/avamar-1389654222 Snapshot "cp.20160910162130" for mtree "/data/col1/avamar-1389654222" will be retained until Jul3 2017 08:16.
Do this nine more times to mark all of the old snapshots for deletion.
Checking the snapshots now shows them all expired.
sysadmin@DataDom1# snapshot list mtree /data/col1/avamar-1389654222 Snapshot Information for MTree: /data/col1/avamar-1389654222 ---------------------------------------------- NamePre-Comp (GiB) Create Date Retain UntilStatus ----------------- -------------- ----------------- ----------------- ------- cp.201609101621301076444.1 Sep 10 2016 09:21 Jul3 2017 08:16 expired cp.201609111532331100654.4 Sep 11 2016 08:32 Jul3 2017 08:17 expired cp.201609131705211161823.8 Sep 13 2016 10:05 Jul3 2017 08:17 expired cp.201609171657211257219.9 Sep 17 2016 09:57 Jul3 2017 08:17 expired cp.201609181543091284386.1 Sep 18 2016 08:43 Jul3 2017 08:18 expired cp.201609191705561309786.6 Sep 19 2016 10:05 Jul3 2017 08:18 expired cp.201609201542051339382.0 Sep 20 2016 08:42 Jul3 2017 08:18 expired cp.201609211541431359362.4 Sep 21 2016 08:41 Jul3 2017 08:19 expired cp.201609221542311377744.1 Sep 22 2016 08:42 Jul3 2017 08:19 expired cp.201609281624221432382.5 Sep 28 2016 09:24 Jul3 2017 08:19 expired ----------------- -------------- ----------------- ----------------- ------- Snapshot Summary ------------------- Total: 10 Not expired:0 Expired: 10
And now we manually kick off the file system cleaning.
sysadmin@DataDom1# filesys clean start nowait Cleaning started.Use 'filesys clean watch' to monitor progress.
When the cleaning is finished we should be good to go. In this case we cleared the Avamar logs, kicked off an Avamar checkpoint, and resumed the scheduler.