Our primary NTP server of some years’ standing has recently begun to slow down and not respond as well as it once did to the cheery “HI, THERE!” from my Exadata storage cells while they validate their configuration every 12 hours.
I have mail alerts set up, so I’ve been receiving a number of these errors lately:
20_1 2014-04-08T14:51:51-04:00 critical “Cell configuration check discovered the following problems: Check Exadata configuration via ipconf utility Verifying of Exadata configuration file /opt/oracle.cellos/cell.conf
Checking NTP server on 10.100.5.255 : FAILED
Error. Overall status of verification of Exadata configuration file: FAILED
[INFO] The ipconf check may generate a failure for temporary inability to reach NTP or DNS server. You may ignore this alert, if the NTP or DNS servers are valid and available.
[INFO] You may ignore this alert, if the NTP or DNS servers are valid and available.
[INFO] As root user run /usr/local/bin/ipconf -verify -semantic to verify consistent network configurations.”20_2 2014-04-13T14:51:51-04:00 clear “The cell configuration check was successful.”
After prying the IP address of our newer, faster, not-as-deaf NTP server out of our UNIX team’s iron grasp, I then had to update the NTP and storage cell configuration on all of my storage cells.
In this example, we’re changing the storage cell’s primary NTP server from 10.100.5.255 to 10.100.4.100. We will leave the secondary NTP server alone as it seems to be doing fine.
N.B. I edited the post to use “ipconf” interactively to change the NTP configuration, instead of having to change the ntp.conf file (Thanks to Dan Norris!)
First, check that your storage cell isn’t synchronizing with the other storage cells:
cellcli -e “list griddisk attributes name,asmmodestatus,asmdeactivationoutcome”
DATA_CD_00_markdwcel01 ONLINE YES
The “asmmodestatus” should be either ONLINE or OFFLINE and the “asmdeactivationoutcome” should be YES.
Do not attempt to reboot the storage cell if its disks are in SYNCING mode. If any disks are, re-run the cellcli command until they have finished.
Offline the storage cell’s grid disks and then check they are all inactive:
cellcli -e “alter griddisk all inactive”
cellcli -e “list griddisk”
We have to make sure that the cell services are shutdown and that the ocrvottargetd service has been stopped before we invoke the ipconf tool.
First, shutdown the cell services:
cellcli -e alter cell shutdown services all
Next, stop the ocrvottargetd service (this may not exist on your storage cell)
service ocrvottargetd stop
Run the ipconf utility to update the NTP servers:
/usr/local/bin/ipconf
…
The current NTP server(s): 10.100.5.255
Do you want to change it (y/n) [n]:y
You will be prompted to make the changes and then to review the settings. Confirm that they are correct.
Now, for the ILOM configuration:
ILOM Use NTP Servers (enabled/disabled) [enabled]: enabled
ILOM First NTP server. Fully qualified hostname or ip address or none [10.100.5.255]: 10.100.4.100
ILOM Second NTP server. Fully qualified hostname or ip address or none [10.100.3.100]: 10.100.3.100
Confirm that these changes are correct.
At this point, you can either reboot the cell and allow the storage cell to start up the services or you can restart cell services, ocrvottargetd and online the grid disks.
To reboot the cell:
shutdown -r now
If you prefer, restart the services and online the grid disks manually instead:
service ocrvottargetd start
cellcli -e “alter cell start services all”
cellcli -e “alter griddisk all active”
When the storage cell is back up or you have restarted the services, make sure both the O/S and the storage cell NTP configuration files are updated – they should reference 10.100.4.100 and not 10.100.5.255
more /etc/ntp.conf
more /opt/oracle.cellos/cell.conf
Check that the cell services are up and running:
cellcli -e “list cell detail” | tail -3
cellsrvStatus: running
msStatus: running
rsStatus: running
Finally, check that the griddisks are ONLINE. After the reboot, your DATA griddisks are likely to be SYNCING, so keep running this until they are all ONLINE.
cellcli -e “list griddisk attributes name,asmmodestatus,asmdeactivationoutcome”
If they’re still OFFLINE, try to bring them all up again:
cellcli -e “alter griddisk all active”
Move onto the next storage cell. Rinse, wash and repeat.
It Was Like That When I Got Here 
This is not a good method to use for something as simple as changing the NTP server. The documentation has a section in the maintenance chapter on changing the NTP server using ipconf interactively. That is much less error prone than editing configuration files. One should never edit the ntp.conf files on a storage cell.
LikeLike
Thanks for raising this, Dan! I’ll update the post presently with the ‘ipconf’ command.
LikeLike
Thanks for sharing such a informative article !
LikeLike
By following their preference, they just lose the possibility of getting more rewards and higher complimentary cash. Fundamental technique is the optimal way to play a hand of blackjack. The quality of the chips can vary greatly.
LikeLike