OCR Repair..

We are in the middle of a test cycle trying to implement FAN between BEA weblogic and Oracle 10g R2 3 node RAC database on OEL 5.   As part of the configuration and setup, after adding the remote application servers to the ONS configuration.   The clusterware did not restart on reboot.

1. Checking the demons using ps -ef | grep cssd, crsd,evmd . all demons where up. however crs_stat – all or crsstat did not give any output.

2. Checking the CSSD log files I noticed the following message.. in the cssd log file

$ORA_CRS_HOME/log/prddb3/cssd/cssd.log

[CSSD]2009-12-24 19:30:36.042 [1274124608] >TRACE:   clssnmRcfgMgrThread: Local Join
[CSSD]2009-12-24 19:30:36.042 [1274124608] >WARNING: clssnmLocalJoinEvent: takeover aborted due to ALIVE node on Disk

Note:  This basically indicated that a node was locking the disk not allowing other nodes to join the cluster.  The node (prddb4) was trying to read the OCR file  (please note OCR is the first file that is accessed by the clusterware during startup) and was not able to.  This potentially indicates a bad OCR file. 

In a similar situation before, a reboot of all servers fixed the locking and the clusterware started without any hiccups.

There may have been other reasons on why this could have happened however due to the urgent nature of the problem and the time it could take to debug and or troubleshoot the situation we decided to repair the OCR file.

3. Nodes prddb3 and prddb5 was repeatable attempting to start the CRS. which generated lots of log entries.  To avoid the logs filling up the disks we requested system admins to shut down the cluster.

Now to fix the problem only one node was started prddb3.  We disabled the autostart of crs using  (this requires ROOT access)
/etc/init.d/init.crs stop  (to stop the crs stack)
/etc/init.d/init.crs disable. (to disable aiuto start on reboot)

4. Based on analysis in step 2 above, the next step was  to repair the OCR file using the following steps also as user root.

[root@prddb3 bin]# ./ocrconfig -repair ocr /dev/raw/ocr1
[root@prddb3 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          2
         Total space (kbytes)     :     306968
         Used space (kbytes)      :      12852
         Available space (kbytes) :     294116
         ID                       :  658275539
         Device/File Name         : /dev/raw/ocr1
         Device/File integrity check succeeded

         Device/File not configured

Cluster registry integrity check succeeded

5.  ‘Device/File not configured’?  Then what is the check succeeded message. Isn’t it a bit confusing. We had configured two OCR files and then why is the second file missing?  Realized that the repair command will only repair one OCR file at a time, besides we had only repaired the primary copy.  Next step was to repair the mirror copy

[root@prddb3bin]#./ocrconfig -repair ocrmirror  /dev/raw/ocr2
[root@prddb3 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          2
         Total space (kbytes)     :     306968
         Used space (kbytes)      :      12852
         Available space (kbytes) :     294116
         ID                       :  658275539
         Device/File Name         : /dev/raw/ocr1
         Device/File integrity check succeeded
         Device/File Name         : /dev/raw/ocr2
         Device/File integrity check succeeded

Cluster registry integrity check succeeded

6.  Now that both the OCR files are fixed we started the clusterware stack using the /etc/init.d/init.crs start command. This brought up the cluterware and the complete stack without any issues.  

7. Now we had to reset the clusterware sstartup process to auto restart on reboot. . (recollect we disabled reboot in Step 3 above).

/etc/init.d/init.crs enable
/etc/init.d/init.crs start

OOW 2009

OOW was a great success …gone are the days when this event is focused towards just marketing.  Normally this conference had more VP, CIO, etc attending to get overview of marketing fluff from various presentations.  These days presentations are filled with good technical information useful for database administrators in their day to day activities. In this open world I had the opportunity to present three papers..

1. RAC for developers:  It was a pretty to see that basic knowledge of how Oracle works lacking amongst the developers.. they did not know their query was executed by oracle and the result sets returned back to them.  this was a hard presentation as I was dragged into a rat hole on what an instance was and how the database is common against many instances in a RAC environment.  I did not do much justice on what the presentation was focused on  namely discussing FAN, FCF, load balancing..how to code and troubleshoot them.

2. Looking under the hood at the Oracle Clusterware. I had done this presentation in OOW 2008 and received good response/feedback.  This year also the response was very good, had a full room and lots of good questions at the end of the session.

3.  Tools and Utilities for RAC available on Metalink.  Most popular of all the presentations, its surprising how many of us really know the various tools and utilities available on Metalink and OTN that could be used for troubleshooting and monitoring a RAC.  Yet we spend all the money we have to third party tool vendors and developers when most of what is provided by them and essential for day to day administration and troubleshooting of the RAC environment are available free of cost from Oracle.

Looking forward to presenting at future OOW’s  and conferences around the world.

_____________

16th Jan ‘10  –  this has been sitting in my draft folder for such a long time.. I understand this is old material.. but since its already done.. i decided to post it.

______________