Exadata Critical Issue DB27

Oracle announced a new Exadata Critical Issue yesterday (DB27) as per MOS 2004572.1.

11.2.0.4 databases running with Grid Infrastructure 12.1 (either 12.1.0.1 or 12.1.0.2) will crash whenever a health update is received (such as when a cell disk is marked “predictive failure”).

The database ASMB process terminates causing the database instance to crash.  The following errors are reported in the database alert.log:

ORA-15064: communication failure with ASM instance
ORA-03115: unsupported network datatype or representation
ASMB: terminating the instance due to error 15064

Perform one of the following actions to prevent bug 20361671:

1. Upgrade the Grid Infrastructure home to 12.1.0.2.7 (Database Patch for Engineered Systems and DB In-Memory 12.1.0.2.7) or later.
2. Apply patch 20361671 to the Grid Infrastructure home.

At the time of writing, the patch README incorrectly omits the rootcrs.pl commands required to unlock and lock the Grid Infrastructure home before and after patching, respectively.

Prior to running the opatch command to apply the patch run the following rootcrs.pl command as the root user to unlock the Grid Infrastructure home:

$GI_HOME/crs/install/rootcrs.pl -unlock

After applying the patch run the following rootcrs.pl command as the root user to lock the Grid Infrastructure home:

$GI_HOME/crs/install/rootcrs.pl -patch

My Collaborate IOUG 2015 Abstract

I will be presenting DBA 3.0 or “How to Become a Real-World Exadata DBA” at Collaborate 2015 – IOUG’s annual user conference – from April 12th to 16th at the Mandalay Bay Resort and Casino in Las Vegas. I submitted this as my abstract:

“DBA resources are more scarce than ever before and it can be very difficult to allocate time on anything but keeping the lights on – even when an organization has made a (substantial) hardware investment in Exadata.

However, if Exadata is treated like any other Oracle database, the promised “extreme performance” will likely be very underwhelming to developers, users and managers and can become unwieldy for DBAs to support.

On the other hand, when an organization configures and supports Exadata properly, they can realize exponential performance improvements in key IT infrastructure, can facilitate better business decisions and may actually reduce infrastructure costs.

The customer has bought a sports car – but might not realize that they haven’t taken it out of second gear (yet).

I will talk about the evolution of Exadata and then get into the “nuts and bolts” of how to support a high-performance Exadata environment as a Production DBA.

I will discuss how to get performance improvements of up to 20x, what NOT to do as an Exadata DBA and how Exadata can become the foundation of your organization’s high-performance enterprise infrastructure.”

I hope to see you in Las Vegas!

UKOUG 2014 – Dan Norris – Exadata Security Best Practices

Dan Norris of the Maximum Availability Architecture team gave what sounded like a very interesting presentation at UKOUG 2014. There seemed to be a lot of really cool stuff at this year’s event, which is to be expected as I no longer reside in the UK!

I encourage you to take a look at the slides, but also at the interesting links he provided:

• Exadata consolidated databases and roles.
• Enterprise security assessment.

Naturally, he also quoted a plethora of My Oracle Support notes – some of the greatest hits and some which you might not have seen before:

• Responses to common Exadata security scan findings (Doc ID 1405320.1)
• Oracle Sun Database Machine X2-2/X2-8, X3-2/X3-8 and X4-2 Security Best Practices (Doc ID 1071314.1)
• How to change OS user password for Cell Node, Database Node , ILOM, KVM , Infiniband Switch , GigaBit Ethernet Switch and PDU on Exadata Database Machine (Doc ID 1291766.1)
• Exadata Database Machine and Exadata Storage Server Supported Versions (Doc ID 888828.1)
• Information Center: Oracle Exadata Database Machine (Doc ID 1306791.2)

Happy reading!

ORAchk and Collection Manager

Quite by chance, I noticed today that Oracle are now offering an exachk-like health check for non-Exadata systems: ORAchk.

This includes some of the exachk functionality and replaces the RACcheck tool for Oracle databases (both clustered and single-instance).

One of the components of ORAchk is the Collection Manager, a ApEx web app, which provides a unified dashboard view of collections (ORAchk, RACcheck and exachk) across your environment.

The Collection Manager uses ApEx 4.2 and can be run against all editions of the database (XE, SE1, SE, EE) 10.2.0.4 or higher. It is supported as part of your support contract, with the exception of the XE edition – you’ll have to visit the OTN forums for help with that.

There are two features in particular which interest me: the ability to compare different health check runs and the creation of incidents for tracking of issues.

Oddly enough, I had just started to create my own system to do just this today – which would have only provided a fraction of what this does, of course. I just need to find somewhere to run ApEx, a spare database and to brush up on my ApEx knowledge.

As a slight tangent, I think it’s a little strange that Oracle are running this on ApEx. I presume this is because it’s an early version and I would imagine that this – along with the OCM/ASR functionality – might end up making its way into a future release of OEM pretty soon as Oracle continue to mature the management of Exadata and its cohorts.

Useful links:

• MOS 1602329.1 – Collection Manager for ORAchk, RACcheck and Exachk.
• MOS 1268927.2 – ORAchk – Oracle Configuration Audit Tool.
• ApEx download page.
• ApEx documentation page (including Installation Guide).
• ORAchk Collection User Guide.