L’Observatoire de Paris has decided that there will be a “leap second” on June 30th, 2015. At 23:59:60 on this date, an additional second will be “inserted” into UTC (Coordinated Universal Time) to take into account the slightly irregular rotation of our planet.
The last “leap second” was on June 30th, 2012, when a bunch of servers running Linux had problems (including, and not limited to, Qantas Airways, reddit and anything running Hadoop).
This year, Google and Amazon both plan to implement a “leap smear” whereby they will add the “leap second” over an extended period on June 30th.
Be aware that a number of AWS services are affected and resolving issues with your EC2 instances is your responsibility.
The “Leap Second” and Oracle
The Oracle database requires no patches and has no problem with the “leap second” changes on the O/S level.
No action is required for Exadata servers which are NOT running 220.127.116.11.0. If you ARE running this version, you will need to follow MOS note 1986986.1 to update your NTP configuration.
However, any derivative of Red Hat Enterprise Linux (including Oracle Enterprise Linux, Oracle Unbreakable Enterprise Kernel and Asianux) versions 4.4 through 6.2, using kernel versions 2.4 to 2.6.39, may be affected. This applies to both baremetal or virtualized environments.
In MOS 1472421.1, Oracle state that impacted servers may become unresponsive sometime before the “leap second” on June 30th, with the following seen in various logs (system, console, netconsole, etc):
INFO: task kjournald:1119 blocked for more than 120 seconds.
“echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
kjournald D ffff880028087f00 0 1119 2 0x00000000
ffff8807ac15dc40 0000000000000246 ffffffff8100e6a1 ffffffffb053069f
ffff8807ac22e140 ffff8807ada96080 ffff8807ac22e510 ffff880028073000
ffff8807ac15dcd0 ffff88002802ea60 ffff8807ac15dc20 ffff8807ac22e140
Alternatively, Java applications may suddenly start to use 100% of the CPU with the event “Leap second insertion causes futex to repeatedly timeout“.
The primary workaround is to stop the NTP service, reset the system clock and restart the NTP service:
date -s “`date`”
An additional workaround is to reboot the server.
Oracle Enterprise Manager
Per MOS 1472651.1, any version of OEM from 10.2.0.5 to 12c running on Linux may see the OEM agent or the OMS service consume excessive CPU on or around “leap seconds”.
Suggested workarounds are identical to the Linux servers (reset the system clock or reboot the server).
Oracle Clusterware on Solaris Servers
Per MOS 759143.1, servers running Solaris 5.8 to 5.10 and running Oracle Clusterware 10.1 to 11.1 may suffer a node reboot unless they have the required patches.
The workaround for this issue is to configure the local xntpd daemon to disable PLL mode and enable skewing or apply Oracle Clusterware patch bundles / MLRs and increase the oprocd daemon timeout margin appropriately.
- Leap seconds (extra second in a year) and impact on the Oracle database. (Doc ID 730795.1)
- Leap Second Time Adjustment (e.g. on June 30, 2015 at 23:59:59 UTC) and Its Impact on Exadata Database Machine (Doc ID 1986986.1)
- Enterprise Manager Management Agent or OMS CPU Use Is Excessive near Leap Second Additions on Linux (Doc ID 1472651.1)
- NTP leap second event causing Oracle Clusterware node reboot (Doc ID 759143.1)
- Leap Second Hang – CPU Can Be Seen at 100% (Doc ID 1472421.1)