… but when I do, it’s always on a Friday afternoon.
During a Friday meeting, some friendly Oracle ExCite types directed my attention to this GEM of a bug:
Bug 16057129 – Exadata cell optimized incremental backup can skip some blocks to backup
If your incremental backups are “cell-optimized” – which they are by default in Exadata – they can skip blocks if a database file grows in size during the backup.
If you’re taking incremental backups with RMAN of your Exadata databases, have not applied the one-off patch for this bug or the “hidden” parameter workaround and you’re running:
- 22.214.171.124 GIPSU1 or earlier
- 126.96.36.199 BP21 or earlier
- 188.8.131.52 (any)
- 184.108.40.206 (any)
You should assume that all your incremental backups – both cumulative and differential – are invalid and unusable.
Unfortunately, there is no way to know whether your incremental backup has been affected until you use it for recovery and see an ORA-00600  error message.
Only incremental backups are possibly affected by this problem – you can still use archive logs to recover from a Level 0 backup and other recovery options are not impacted.
To squash this bug, you have three options:
- Apply the workaround to disable cell optimized backups.
- Apply the one-off bug fix for your version.
- Upgrade to either 220.127.116.11.2, 18.104.22.168 or 22.214.171.124.22.
The Quarterly Full Stack Patch for January 2014 includes Bundle Patch 22 for 126.96.36.199, so you may want to review the QFSP January 2014 upgrade process. It looks like you will have to apply a one-off patch 17599908 on top of 188.8.131.52.22, so review it closely!
If you haven’t done so already, bookmark the MOS list of Exadata Critical Issues.