Category Archives: Amazon Web Services

Leap Second 2015

L’Observatoire de Paris has decided that there will be a “leap second” on June 30th, 2015.  At 23:59:60 on this date, an additional second will be “inserted” into UTC (Coordinated Universal Time) to take into account the slightly irregular rotation of our planet.

The last “leap second” was on June 30th, 2012, when a bunch of servers running Linux had problems (including, and not limited to, Qantas Airways, reddit and anything running Hadoop).

This year, Google and Amazon both plan to implement a “leap smear” whereby they will add the “leap second” over an extended period on June 30th.

Be aware that a number of AWS services are affected and resolving issues with your EC2 instances is your responsibility.
 

The “Leap Second” and Oracle
The Oracle database requires no patches and has no problem with the “leap second” changes on the O/S level.

No action is required for Exadata servers which are NOT running 12.1.2.1.0.  If you ARE running this version, you will need to follow MOS note 1986986.1 to update your NTP configuration.
 

Linux Servers
However, any derivative of Red Hat Enterprise Linux (including Oracle Enterprise Linux, Oracle Unbreakable Enterprise Kernel and Asianux) versions 4.4 through 6.2, using kernel versions 2.4 to 2.6.39, may be affected.  This applies to both baremetal or virtualized environments.

In MOS 1472421.1, Oracle state that impacted servers may become unresponsive sometime before the “leap second” on June 30th, with the following seen in various logs (system, console, netconsole, etc):
 

INFO: task kjournald:1119 blocked for more than 120 seconds.
“echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
kjournald D ffff880028087f00 0 1119 2 0x00000000
ffff8807ac15dc40 0000000000000246 ffffffff8100e6a1 ffffffffb053069f
ffff8807ac22e140 ffff8807ada96080 ffff8807ac22e510 ffff880028073000
ffff8807ac15dcd0 ffff88002802ea60 ffff8807ac15dc20 ffff8807ac22e140

 
Alternatively, Java applications may suddenly start to use 100% of the CPU with the event “Leap second insertion causes futex to repeatedly timeout“.

The primary workaround is to stop the NTP service, reset the system clock and restart the NTP service:
 

/etc/init.d/ntpd stop
date -s “`date`”
/etc/init.d/ntpd start

 

An additional workaround is to reboot the server.
 

Oracle Enterprise Manager
Per MOS 1472651.1, any version of OEM from 10.2.0.5 to 12c running on Linux may see the OEM agent or the OMS service consume excessive CPU on or around “leap seconds”.

Suggested workarounds are identical to the Linux servers (reset the system clock or reboot the server).
 

Oracle Clusterware on Solaris Servers
Per MOS 759143.1, servers running Solaris 5.8 to 5.10 and running Oracle Clusterware 10.1 to 11.1 may suffer a node reboot unless they have the required patches.

The workaround for this issue is to configure the local xntpd daemon to disable PLL mode and enable skewing or apply Oracle Clusterware patch bundles / MLRs and increase the oprocd daemon timeout margin appropriately.
 

References

  • Leap seconds (extra second in a year) and impact on the Oracle database. (Doc ID 730795.1)
  • Leap Second Time Adjustment (e.g. on June 30, 2015 at 23:59:59 UTC) and Its Impact on Exadata Database Machine (Doc ID 1986986.1)
  • Enterprise Manager Management Agent or OMS CPU Use Is Excessive near Leap Second Additions on Linux (Doc ID 1472651.1)
  • NTP leap second event causing Oracle Clusterware node reboot (Doc ID 759143.1)
  • Leap Second Hang – CPU Can Be Seen at 100% (Doc ID 1472421.1)

 

Tagged , , , , , , ,

VENOM – A Potentially Serious Virtual Machine Exploit

VENOM is a virtual machine bug which exploits the floppy disk controller used in some hypervisors and allows an attacker to break out of a guest O/S and escape into a host O/S.

“VENOM, CVE-2015-3456, is a security vulnerability in the virtual floppy drive code used by many computer virtualization platforms. This vulnerability may allow an attacker to escape from the confines of an affected virtual machine (VM) guest and potentially obtain code-execution access to the host. Absent mitigation, this VM escape could open access to the host system and all other VMs running on that host, potentially giving adversaries significant elevated access to the host’s local network and adjacent systems.”

Affected Oracle products:

  • VirtualBox prior to 4.3.28
  • Oracle VM 2.2 to 3.3
  • Oracle Linux 5 to 7

Hypervisors affected:

  • Xen (VirtualBox), KVM, QEMU, possibly others

Hypervisors not affected:

  • VMWare, Microsoft Hyper-V, Bochs

Patches available:

Workarounds:

  • VirtualBox users should disable the floppy controller in their VM configuration.

Reference:

Tagged , , , , , , , ,

Complete Cloud Confusion

Version X5 of Oracle’s engineered systems – presumably Exadata, Exalogic and Exalytics with a garnishing of a ZFS/ZDLR appliance or two – will be finally unveiled tomorrow.

No doubt more of everything will be involved (Flash, memory, CPU, cupcakes), making DBA geeks drool and widening the performance chasm between Oracle’s engineered systems and a lot of the “industry trends” we read so much about right now. Hopefully, those who have been on the waiting list since they stopped shipping the X4s will feel it’s been worth the wait.  Enjoy your new gadgets!

As a technologist, it’s difficult not to be impressed with exponentially-improving kit, especially when it feels like the industry is collectively yearning for 1990s technology.

Imagine, if you will, my surprise, when I learned of another webcast a week later, followed by a serious PR push from Larry Ellison and Mark Hurd about THE CLOUD(TM).

Huh? Isn’t pushing a new class of engineered systems (lots of lovely CapEx … mmm-hmm!) and then pushing CLOUDCLOUDCLOUD (CapEx, be GONE!) a week later a juxtaposition?

And what about this quote:

” … on-premises software sales grew 6% in constant currency. I continue to expect this business to grow nicely while our cloud business continues to maintain hypergrowth … “

Really?

Oracle believes CIOs are going to maintain spending in “traditional” infrastructure AND invest big in THE CLOUD(TM) at the same time? Hmm.

And isn’t THE CLOUD(TM) fantastic and magical and revolutionary because organizations plan to eliminate spending on support groups and hardware and transfer their budgets to OpEx instead, saving tons of cash? (We’ll put the many and varied issues of doing this to one side for the moment).

Am I the only one confused by this?

That being said …

Unlike most THE CLOUD(TM) vendors, Oracle’s cloud offering includes Platform-as-a-Service, which provides the first “real” managed database service in THE CLOUD(TM) including Exadata and all the performance and security cost options you can buy for the Oracle database “on-site”.

Even as someone who isn’t exactly a strong advocate of THE CLOUD(TM), it’s difficult to dispute that this addresses some – though by no means all – of the problems associated with cloud computing.

Up until now, most providers have been offering more of an Infrastructure-as-a-Service solution, which is geared almost entirely towards cost savings. With PaaS, a viable argument can be made that functionality and performance can be as good, if not better, than internally managed systems.

I’ll admit that this all had my curiosity, but now has my attention.

Maybe Oracle is one of two companies (IBM, perhaps?) who can afford to invest the massive sums needed to cover both bases well enough, though it should be noted that Amazon STILL hasn’t made a profit on AWS yet. And how will they avoid their sales pitches becoming confusing muddles of uncertainty involving DOUBLE the salespeople (one set for engineered systems, one set for THE CLOUD(TM))?

I’ll be honest, this still doesn’t make sense to me – I just don’t get it.

I have no doubt Oracle will be pushing Exadata’s suitability for THE CLOUD(TM) tomorrow by introducing new elastic/scalable/on-demand features, but engineered systems and THE CLOUD(TM) seem so diametrically opposed that they’re all but mutually exclusive.

We’ll know soon enough, I guess!  The cloud is coming, whether we agrees with it or not!

Tagged , ,

Using THE CLOUD(TM) For Off-Site Backup Media Storage

Recently, a client of mine asked me an excellent question about whether they could use AWS as an off-site storage location for their backup media files.

As I may have previously suggested, I have quite a long list of reasons why I think that running important databases from the cloud is a terrible idea, though if a client does really want to pursue this route, I will naturally oblige and make it work as best as I can.

This particular client’s suggestion, however – which, surprisingly, isn’t something that I’d heard made before – was to move at-rest backup media files to Amazon’s S3 storage instead of putting them on tape and shipping them off-site.

I asked around and did some digging but I couldn’t find anyone who was using THE CLOUD(TM) in this manner. Others are migrating their live database to THE CLOUD(TM) – or, at least, attempting to and becoming very frustrated with it – while others have stood up standby databases in AWS as a “DR-of-last-resort”.

Continue reading

Tagged , , ,

AWS Customer Service

A couple of weeks ago, I explained why I was not impressed with the cloud as a game-changing, magical, revolutionary, outstanding, groud-breaking panacea for IT expenditure after I’d spent some time with Amazon Web Services.

I contacted their support about a technical issue I was having and, to their credit, their customer service seems to be just as good for AWS as it is for their .com site.

I made a(n attempt at a) joke about time travel and how they meter usage for their RDS service and, to my surprise, they played along:

Me: Thanks for the resolution. I’ll keep you updated with my time travel travails. Do you happen to have a spare DeLorean car that can hit 88mph?

CSR: I’m glad to hear there was a resolution to your case. Currently, our DeLorean, as well as our Tardis, our Phone Booth, and our HG Well’s Special are all being used to try to prevent bad Fantasy Football drafts, as well as preventing from eating too much for Thanksgiving.

Got to love it when customer service has a sense of humor, especially considering the dog’s abuse they often get.

Tagged , , , ,

The Road to Damascus?

So, last month I decided to bow to peer (well, “industry”) pressure and check out Amazon Web Services for myself.

That’s right, before the year 2014 was out, I finally started my own personal “journey” to THE CLOUD(TM).

I wish I could say that I experienced a “Road to Damascus” moment and that all the major (i.e. “showstopper”) concerns I had with actually having to migrate databases to THE CLOUD(TM) magically disappeared once I had actually used it myself. Maybe I had been wrong this whole time?

“Just try it and you’ll see”…

Unfortunately, like a cigarette to a teenager, it was exactly what I had expected. No more, no less.

There is no real benefit to using it aside from looking cool.

Continue reading

Tagged , , , , ,
%d bloggers like this: