Lots of shops dipped their toes in the Exadata water with a quarter-rack first of all.
(For those who are new to the Exadata party and don’t know of a world without elastic configurations, a quarter-rack is a machine with two compute nodes and three storage cells).
If you are / were one of those customers, you’ll probably have winced at the difference between the “raw” storage capacity and the “usable” storage capacity when you got to play with it for the first time.
While you could choose to configure your DATA and RECO diskgroups with HIGH redundancy in ASM, did you notice that you couldn’t do the same with the DBFS_DG / SYSTEM_DG?
Check out page 5 in this document about best practices for consolidation on Exadata.
“A slight HA disadvantage of an Oracle Exadata Database Machine X3-2 quarter or eighth rack is that there are insufficient Exadata cells for the voting disks to reside in any high redundancy disk group which can be worked around by expanding with 2 more Exadata cells. Voting disks require 5 failure groups or 5 Exadata cells; this is one of the main reasons why an Exadata half rack is the recommended minimum size.”
Basically, you need at least 5 storage cells for each Exadata environment if you want to have true “high availability” with your Exadata machine.
While quarter-rack machines have 3 storage cells, half-rack machines have 7 or 8 storage cells, depending on the model.
Let’s say that you have the model with 8 storage cells: if you split a half-rack machine equally, you’ll have 2x quarter-rack machines with 4 storage cells, so you would need one more storage cell per machine to provide HA for the SYSTEMDG / DATA_DG diskgroup.
For some reason, this nugget escaped my attention until recently. Even more reason to have a standby Exadata machine at your DR site …