Today’s Nugget: Oracle GoldenGate on E-Business Suite Databases

Throughout my work week, I often learn something new and unexpected about my travels. Instead of writing a full blog post about the subject, I figure I’ll occasionally post these as “nuggets” – bite-size chunks which are easily digested!

Here is today’s “nugget”:

Oracle GoldenGate is certified to capture data from E-Business Suite databases, but it cannot be used to apply data to E-Business Suite databases.

This includes writing back to the source database in an active-active configuration.



Tagged , , , , ,

Dude … Where’s My GUI?

Are you running Exadata Storage Server

Have you tried to get a graphical tool, such as DBCA, DBUA or even VNC to run on your Exadata lately?

If, like me, you had quite the struggle until a friendly sysadmin installed a bunch of packages for you, you might be interested in reading this MOS note:

1969308.1 – Unable to run graphical tools (gui) on Exadata or later.

I understand that Oracle have been increasingly hardening Exadata since the birth of their X3-2 machines, but you’d think that you wouldn’t need to add extra packages to a system that’s meant to be “ready to use” once your choice of consultant has finished with the Deployment Assistant.

After all, aren’t DBCA / DBUA Oracle’s tools of choice? Do they really want DBAs to spend their time creating response files and running these tools from the command line?


Tagged , , , , , ,

Hamlet – Prince of DBAs

While searching through my exhaustive collection of notes, scribbles and documentation, I found this particular GEM of mine that I wrote to Oracle Support after they had ignored my Priority Service Request for a month.  After finding that various degrees of pleading didn’t work, I went on the offensive and THREATENED them!

*I* found it very funny, anyway …

A last-ditch attempt to resolve this peacefully from ME:

I’m not kidding – the next update you’re going to be getting from me is poem. And I’m a very bad poet.

You have been warned.

They gave me no choice by ignoring me again.  I did what I had to do.  I’m not proud of it, but I’m not sorry either:

You were warned. You’re not only getting a poem, but you’re getting a Shakespeare soliloquy. And a TERRIBLE one at that.

If there’s no update tomorrow, I’m going to write and perform a song. Trust me when I say I can’t even speak in tune, never mind sing…

I hope you recognize this fabulous piece of literature…

To reply to the SR or not to reply to the SR, that is the question:
Whether there are nobler items in your in-queue
So to deal with other, less annoying customers
Or to at least *look* at this priority handled SR.
To update, to ignore
No more; and by ignoring, to say
The ignorance of a month-old priority SR
That a tortured DBA is heir to?
‘Tis a consummation devoutly to be wished,
To escalate or wait
To update, perchance to persuade
Aye, there’s the rub
For in that persuading, what solutions may come,
When you have stopped ignoring this request,
Must give us pause. There’s the respect
That makes calamity of ‘PRIORITY SR’
For who would bear the whips and scorns of having to deal with Metalink
The analysts’ wrong, the DBA’s contumely,
The pangs of paying for this, no response and delay,
The insolence of writing you a poem
To get your attention after too many winter nights
When he himself might his answer make
By putting this down to a freak of Exadata
That will be fixed in a future release.
Be all my sins remembered.

Seriously – is there any sign of an update – I don’t want to call in and ask why a Priority SR has been ignored for over a month. At least tell me you’ve not forgotten it…


Acknowledging your last update..
I will make sure this SR gets adequate attention and is progressed
Let me review the SR and get back to you…
(BTW, your poem is good- but I am sorry you had to write it)

Tagged , ,

New Exadata Critical Issues – EX23 and EX24

Over the weekend, Oracle announced two new Critical Issues for Exadata Storage Server (EX23 and EX24), both impacting version

Both the new Critical Issues can be patched by either applying the one-off patch 21251493 or by upgrading the Exadata Storage Server software to


Critical Issue EX23
Affects Exadata Storage Server and

Bug 21174310 – wrong results, ORA-1438 errors or other internal errors (ORA-00600 and ORA-07445) are possible from smart scan offloaded queries against HCC or OLTP compressed tables stored on Exadata storage if:

  • Exadata Storage Server is version or AND
  • Oracle Database was upgraded from 11.2 to 12.1 AND
  • A smart scan offloaded query is issued against an OLTP compressed table or an HCC table containing OLTP compressed blocks

The workaround is to recreate the table.

The recommended action is to upgrade to Exadata Storage Server software version (or higher).

Alternatively, apply patch 21251493 to Exadata Storage Servers running version  Note that patch 21251493 contains additional fixes required to resolve other critical issues.

MOS 2032464.1 has additional details.


Critical Issue EX24
Affects Exadata Storage Server

After replacing a failed system disk (disk 0 or disk 1), the new disk is not correctly configured leaving the system vulnerable to the other system disk failing. The likelihood of occurrence is high when running Exadata version and a failed system disk is replaced.

The workaround is to follow the instructions in MOS 2003674.1.

The recommended action is to upgrade to Exadata Storage Server software version (or higher).

Alternatively, apply Patch 21251493 to Exadata Storage Servers running version Note that patch 21251493 contains additional fixes required to resolve other critical issues.

MOS 2032402.1 has additional details.



  • Bug 21174310 – Wrong results or ORA-1438 errors possible from smart scan offloaded queries against HCC or OLTP compressed tables stored on Exadata storage (Doc ID 2032464.1)
  • Important Fixes Required for System Disk Replacement on Exadata Storage Servers Running Version (Doc ID 2032402.1)
  • Exadata Storage Software and System Disk Replacement Issues (Doc ID 2003674.1)
  • Exadata Critical Issues (Doc ID 1270094.1)
  • Patch 21251493
Tagged , , , , , , , , , , ,

Oracle Critical Patch Update for July 2015

Oracle’s Critical Patch Update is out for July 2015:

Affected are database versions,,, and

This is the final patch for both the and releases. The final patch for will be released in January 2016.

The most prominent bug on the risk matrix is CVE-2015-2629 whereby a remote authenticated user can exploit a flaw in the Java VM component to gain elevated privileges.

For the patches, you can apply one of the following: SPU for UNIX: patch 20803583 PSU for UNIX: patch 20760982 Quarterly Database Patch for Exadata (July 2015): patch 21142006
July 2015 Quarterly Full-Stack Patch for Exadata: patch 21186703

Don’t forget your Grid Infrastructure patching: PSU for UNIX: patch 20996923

And, of course, ever since those Java bugs were discovered, we should also patch the JVM: Database PSU for UNIX: patch 21068539

Happy patching!

Tagged , , ,

Friday Links

I’m on-call and it’s a Friday – here are some random links for your perusal. Happy weekend!

Sadly, even Disney isn’t immune from the insanity or from ignoring immigration law (H1-B visas are for “exceptional” candidates and must only fill positions which CANNOT be filled by US-based employees).

If you buy into the trade mag hype, us DBAs are living on borrowed time. According to NPR, however, we have less chance of being automated than a police officer.  We also have less chance of being automated than “IT management”, which is far more amusing!

Hadoop isn’t as prevalent as trade mags would have you believe, apparently.  This chimes with what I noticed from Collaborate IOUG earlier this year.

Oracle’s potentially neat Real Application Security feature, which could make deducing those roles and privileges a bit easier.

An interesting article from 2011 which relates to my “DBA 3.0” presentation (in some way!)

Don’t forget not to anger the IT Gods today!

Tagged , ,

Exadata Critical Issue EX21

Oracle announced a new Exadata Critical Issue this morning (EX21) which applies to the ESS software versions and

“This issue is encountered only when a disk media error occurs while synchronous I/O is performed. Because the majority of I/O operations issued with Exadata storage are done asynchronously, and this problem is possible only when disk media errors are experienced while synchronous I/O is performed, the likelihood of experiencing this problem is low. However, the impact of hitting this problem can potentially be high.

This problem affects Exadata Storage Server software versions and

Disk corruption symptoms are varied. Some corruptions will be resolved automatically by Oracle Database, while other corruptions will lead to unexpected process shutdown due to internal errors.”

ESS DOES have a patch available, but does not at the moment (the patch is “pending”).  I’m sure it will become available soon.

I have MOS email me whenever the Exadata Critical Issues document (1270094.1) is updated so I’m quickly aware of the latest important bugs. It’s pretty neat and I’d advise other Exadata types to make use of it as well.

Tagged , , , ,

Leap Second 2015

L’Observatoire de Paris has decided that there will be a “leap second” on June 30th, 2015.  At 23:59:60 on this date, an additional second will be “inserted” into UTC (Coordinated Universal Time) to take into account the slightly irregular rotation of our planet.

The last “leap second” was on June 30th, 2012, when a bunch of servers running Linux had problems (including, and not limited to, Qantas Airways, reddit and anything running Hadoop).

This year, Google and Amazon both plan to implement a “leap smear” whereby they will add the “leap second” over an extended period on June 30th.

Be aware that a number of AWS services are affected and resolving issues with your EC2 instances is your responsibility.

The “Leap Second” and Oracle
The Oracle database requires no patches and has no problem with the “leap second” changes on the O/S level.

No action is required for Exadata servers which are NOT running  If you ARE running this version, you will need to follow MOS note 1986986.1 to update your NTP configuration.

Linux Servers
However, any derivative of Red Hat Enterprise Linux (including Oracle Enterprise Linux, Oracle Unbreakable Enterprise Kernel and Asianux) versions 4.4 through 6.2, using kernel versions 2.4 to 2.6.39, may be affected.  This applies to both baremetal or virtualized environments.

In MOS 1472421.1, Oracle state that impacted servers may become unresponsive sometime before the “leap second” on June 30th, with the following seen in various logs (system, console, netconsole, etc):

INFO: task kjournald:1119 blocked for more than 120 seconds.
“echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
kjournald D ffff880028087f00 0 1119 2 0x00000000
ffff8807ac15dc40 0000000000000246 ffffffff8100e6a1 ffffffffb053069f
ffff8807ac22e140 ffff8807ada96080 ffff8807ac22e510 ffff880028073000
ffff8807ac15dcd0 ffff88002802ea60 ffff8807ac15dc20 ffff8807ac22e140

Alternatively, Java applications may suddenly start to use 100% of the CPU with the event “Leap second insertion causes futex to repeatedly timeout“.

The primary workaround is to stop the NTP service, reset the system clock and restart the NTP service:

/etc/init.d/ntpd stop
date -s “`date`”
/etc/init.d/ntpd start


An additional workaround is to reboot the server.

Oracle Enterprise Manager
Per MOS 1472651.1, any version of OEM from to 12c running on Linux may see the OEM agent or the OMS service consume excessive CPU on or around “leap seconds”.

Suggested workarounds are identical to the Linux servers (reset the system clock or reboot the server).

Oracle Clusterware on Solaris Servers
Per MOS 759143.1, servers running Solaris 5.8 to 5.10 and running Oracle Clusterware 10.1 to 11.1 may suffer a node reboot unless they have the required patches.

The workaround for this issue is to configure the local xntpd daemon to disable PLL mode and enable skewing or apply Oracle Clusterware patch bundles / MLRs and increase the oprocd daemon timeout margin appropriately.


  • Leap seconds (extra second in a year) and impact on the Oracle database. (Doc ID 730795.1)
  • Leap Second Time Adjustment (e.g. on June 30, 2015 at 23:59:59 UTC) and Its Impact on Exadata Database Machine (Doc ID 1986986.1)
  • Enterprise Manager Management Agent or OMS CPU Use Is Excessive near Leap Second Additions on Linux (Doc ID 1472651.1)
  • NTP leap second event causing Oracle Clusterware node reboot (Doc ID 759143.1)
  • Leap Second Hang – CPU Can Be Seen at 100% (Doc ID 1472421.1)


Tagged , , , , , , ,

Exadata Critical Issue DB20

A new Exadata Critical Issue – EX20 – has been announced on MOS note 1270094.1 and applies to Exadata Storage Server versions and

The issue is caused by bug 19211091:

CELLSRV Internal Error ORA-600 [DiskIOSched::GetCatIndex:2]

Further details can be found in MOS 1967985.1

You might hit this bug if your database resource manager plan contains sub-plans and OTHER_GROUPS is present in a sub-plan instead of the top plan.

The CELLSRV trace file will contain one or more entries indicating CELLSRV process failure similar to the following:

ORA-00600: internal error code, arguments: [DiskIOSched::GetCatIndex:2], [4294967295], [], [], [], [], [], [], [], [], [], []

CELLSRV encountered a fatal signal 11. LWPID: 28000 userId: 80 kernelId: 80 pthreadID: 139785595115840
Ignoring fatal signal encountered during Cellsrv state dump LWPID: 28000 userId: 80 kernelId: 80 pthreadID: 139785595115840

If CELLSRV fails on multiple cells simultaneously, then the ASM disk groups may dismount or ASM instances may crash, potentially causing databases to crash.

Typically, the Restart Server (RS) process will restart CELLSRV after it fails.  However, too many CELLSRV failures will trigger “flood control” and prevent further CELLSRV restarts.  Flood control is indicated in the trace file with entries similar to the following:

[RS] monitoring process /opt/oracle/cell/cellsrv/bin/cellrsomt (pid: 26763) returned with error: 126
[RS] Monitoring process for service CELLSRV detected a flood of restarts. Disable monitoring process.
RS-7445 [CELLSRV monitor disabled] [Detected a flood of restarts] [] [] [] [] [] [] [] [] [] []

The recommended action is to upgrade to Exadata Storage Server software version (or higher) or (or higher).

Alternately, you can apply patch 19211091.

As a temporary workaround, you can disable the Resource Manager on the affected databases, modify the appropriate plan so that the OTHER_GROUPS directive is in the top plan (and not any sub-plan) and re-enable the Resource Manager:

ALTER SYSTEM SET resource_manager_plan=” SCOPE=both SID=’*’;

SELECT unique name
FROM v$rsrc_plan_history
FROM dba_rsrc_plan_directives
WHERE plan IN (
SELECT unique name
FROM v$rsrc_plan_history)
AND group_or_subplan = ‘OTHER_GROUPS’);

plan => ‘MY_PLAN’,
group_or_subplan => ‘OTHER_GROUPS’,
mgmt_p2 => 80,
switch_estimate => FALSE,
comment => NULL);

ALTER SYSTEM SET resource_manager_plan=’MY_PLAN’ SCOPE=both SID=’*’;

Tagged , , ,

VENOM – A Potentially Serious Virtual Machine Exploit

VENOM is a virtual machine bug which exploits the floppy disk controller used in some hypervisors and allows an attacker to break out of a guest O/S and escape into a host O/S.

“VENOM, CVE-2015-3456, is a security vulnerability in the virtual floppy drive code used by many computer virtualization platforms. This vulnerability may allow an attacker to escape from the confines of an affected virtual machine (VM) guest and potentially obtain code-execution access to the host. Absent mitigation, this VM escape could open access to the host system and all other VMs running on that host, potentially giving adversaries significant elevated access to the host’s local network and adjacent systems.”

Affected Oracle products:

  • VirtualBox prior to 4.3.28
  • Oracle VM 2.2 to 3.3
  • Oracle Linux 5 to 7

Hypervisors affected:

  • Xen (VirtualBox), KVM, QEMU, possibly others

Hypervisors not affected:

  • VMWare, Microsoft Hyper-V, Bochs

Patches available:


  • VirtualBox users should disable the floppy controller in their VM configuration.


Tagged , , , , , , , ,
%d bloggers like this: