DGB::ORA-16608:one or more sites have warnings

I am at a customer site trying to help them with a physical standby implementation in a RAC environment. The primary database is a three node RAC cluster but the standby environment is a single instance RAC database.

During setup and configuration of the dataguard broker, we had the following error . ” ORA-16608: one or more sites have warnings”. Like it says one or more errors? Even if there is only one error with the configuration or setup the misleading message can make you wondering.. Lets take at analyzing one such scenario..

1. From the command line interface of the broker when you check the configuration status using show configuration, the broker would give you the current setup at the end of the show configuration output, the broker tries to query the current status from all servers involved, this includes all instances in the RAC configuration on both the primary and physical standby side.

DGMGRL> show configuration verbose

Configuration
Name: AT_DGB
Enabled: YES
Protection Mode: MaxPerformance
Fast-Start Failover: DISABLED

Databases:
ATDB – Primary database
ATSB – Physical standby database

Current status for “AT_DGB”:
Warning: ORA-16608: one or more databases have warnings

2. Which database in the DGB configuration has this error? SHOW DATABASE VERBOSE <database name> would tell us which database has this error. The error above really does not tell which database has the error. So we may have to check the details against both the databases. Checking the status of the Primary database we observed “ORA-16809: multiple warnings detected for the database” another error? again not one specific error but potentially many.. that what the error description tells us..

DGMGRL> SHOW DATABASE VERBOSE ‘ATDB’;

Current status for “ATDB”:
Warning: ORA-16809: multiple warnings detected for the database

Properties:
InitialConnectIdentifier = ‘atdb’
ObserverConnectIdentifier = ”
LogXptMode = ‘SYNC’
Dependency = ”
DelayMins = ‘0′
Binding = ‘OPTIONAL’
MaxFailure = ‘0′
MaxConnections = ‘1′
ReopenSecs = ‘300′
NetTimeout = ‘180′
LogShipping = ‘ON’
PreferredApplyInstance = ”
ApplyInstanceTimeout = ‘0′
ApplyParallel = ‘AUTO’
StandbyFileManagement = ‘auto’
ArchiveLagTarget = ‘0′
LogArchiveMaxProcesses = ‘30′
LogArchiveMinSucceedDest = ‘1′
DbFileNameConvert = ”
LogFileNameConvert = ”
FastStartFailoverTarget = ”
StatusReport = ‘(monitor)’
InconsistentProperties = ‘(monitor)’
InconsistentLogXptProps = ‘(monitor)’
SendQEntries = ‘(monitor)’
LogXptStatus = ‘(monitor)’
RecvQEntries = ‘(monitor)’
HostName(*)
SidName(*)
LocalListenerAddress(*)
StandbyArchiveLocation(*)
AlternateLocation(*)
LogArchiveTrace(*)
LogArchiveFormat(*)
LatestLog(*)
TopWaitEvents(*)

Checking the details of the other database in the configuration reveled no errors.

3. Time to drill down further to get more details. Checking the DMON process log files under ‘bdump’ destination directory we noticed the following.. (during initial configuration it easier if we tail -f on the DMON log file on both the primary and standby servers to monitor its activity)..

for example: /oracle/admin/atdb/bdump/$ tail -f drcatdb1.log

DG 2009-09-30-14:21:34 0 2 0 RSM Warning: Property ‘StandbyFileManagement’ has inconsistent values:METADATA=’auto’, SPFILE=”, DATABASE=’auto’
DG 2009-09-30-14:21:34 0 2 0 RSM0: HEALTH CHECK WARNING: ORA-16714: the value of property StandbyFileManagement is inconsistent with the database setting
DG 2009-09-30-14:21:34 0 2 0 RSM Warning: Property ‘ArchiveLagTarget’ has inconsistent values:METADATA=’0′, SPFILE=”, DATABASE=’0′
DG 2009-09-30-14:21:34 0 2 0 RSM0: HEALTH CHECK WARNING: ORA-16714: the value of property ArchiveLagTarget is inconsistent with the database setting
DG 2009-09-30-14:21:34 0 2 0 RSM Warning: Property ‘LogArchiveMinSucceedDest’ has inconsistent values:METADATA=’1′, SPFILE=”, DATABASE=’1′
DG 2009-09-30-14:21:34 0 2 0 RSM0: HEALTH CHECK WARNING: ORA-16714: the value of property LogArchiveMinSucceedDest is inconsistent with the database setting
DG 2009-09-30-14:21:35 0 2 698930610 Operation MON_VERIFY continuing with warning, status = ORA-1679

4. There are definitely so many errors in the DMON log file all related to property mismatch between dataguard broker and the database. “HEALTH CHECK WARNING: ORA-16714: the value of property ArchiveLagTarget is inconsistent with the database setting” and all these properties have a mismatch?

Checking the values on both the DGB and the instance really could not find any mismatch. After considerable searching on metalink and google had no luck with any specific reasons why this could be wrong or showing as a mismatch.

5. If you recollect one of the requirements for the DGB is to have an SPFILE configuration, the customer did have an SPFILE however instead of keeping the file on shared location, maintained three copies (one on each instance) of the SPFILE in the $ORACLE_HOME/dbs directory..

6. We created a new pfile from the spfile, disabled the init<SID>.ora file and created an spfile file on ASM.

SQL>create spfile=’+data_01_dg/atdb/spfileatdb.ora’ from pfile=’/home/oracle/pfileATB.ora

This did not solve the problem. Apparently initatdb1.ora located in $ORACLE_HOME/dbs directory was not rightly pointing to the ASM spfile. We changed this to look like the output below..

$ORACLE_HOME/dbs/cat initatdb1.ora
SPFILE=’+DATA_01_DG/atdb/spfileatdb.ora’

7. Starting the instance either using SQLplus or SRVCTL utility brought the database instance up successfully however the instance did not use the right spfile that was created on ASM but used the spfile located in the $ORACLE_HOME/dbs directory.

8. We had to add the spfile using the srvctl utility to the OCR file. Where is the SPFILE information stored in the OCR file? Every database managed by on the RAC cluster could keep the SPFILE file (optionally) in the OCR file. Lets analyze the OCR file, to read the contents of the OCR file, we take a dump of the contents into ASCII readable format.

Not seen such a behavior before.  We are continuing to investigate why this is happening?

$ORA_CRS_HOME/bin/ocrdump will create an output called OCRDUMP in the default directory where the command was executed from. Edit the OCRDUMP file (vi OCRDUMP) and search for SPFILE, against the database ATDB database the output was blank. You would think that if the database was created using DBCA that wrote the database and instance related entries to the OCR file

[DATABASE.DATABASES.atdb.SPFILE]
ORATEXT :
SECURITY:{USER_PERMISSION:PROCR_ALL_ACCESS, GROUP_PERMISSION:PROCR_WRITE,OTHER_PERMISSION: PROCR_READ, USER_NAME : oracle, GROUP_NAME : dba}

Using SRVCTL utility we added the SPFILE information to the OCR file and this fixed the startup issue to use the right SPFILE.

$ /oracle/crs/bin/srvctl modify database -d atdb -p +DATA_01_DG/ATDB/spfileatdb.ora

To make sure that the entry was in the OCR we did another dump of the OCR file and checked the entry for the SPFILE for the ATDB database. Yes it was there..

[DATABASE.DATABASES.atdb.SPFILE]
ORATEXT : +SHARED_DATA_01_DG/ARM00TDB/spfilearm00tdb.ora
SECURITY:{USER_PERMISSION:PROCR_ALL_ACCESS, GROUP_PERMISSION:PROCR_WRITE,OTHER_PERMISSION: PROCR_READ, USER_NAME : oracle, GROUP_NAME : dba}

9. This really fixed the issue, it was the DGB not able to synchronize the value of these various parameters/properties between the various instances in the cluster. BTW, in a RAC environment its a best practice to maintain the SPFILE in a shared location such as ASM storage.

The SHOW CONFIGURATION command from the DGMGRL returned error free.

DGMGRL> SHOW CONFIGURATION VERBOSE

Configuration
Name: AT_DGB
Enabled: YES
Protection Mode: MaxPerformance
Fast-Start Failover: DISABLED
Databases:

ATDB – Primary database
ATSB – Physical standby database

Current status for “AT_DGB”:
SUCCESS

Rundown of the last 1 odd years….

I had just started and after two entries disappeared?  hmmm..  yes apologize for staying away…

Lets do a rundown of the last 1 odd years..

1.  ASM book launch was good, the book is doing good in the  market its interesting to see how well ASM is being adopted in the Oracle space. Is time to update this book for the next release?   with 11gR2 released maybe something to consider.. need to get in touch with the other authors.  maybe soon..

2.  Did do several presentations at the OOW 2008:
1. Looking under the hood at the  Oracle clusterware (2). Understanding Oracle RAC   (3). Tools and utilities available on metalink . All presentations where well attended, several good questions and while sharing we also learn.  Will I be a the OOW09?  ;OOW is a great place to meet up with several of the Oracle users from all over the world.  Unlike the earlier open world shows, OOW  has become more technical in the recent years and there is a lot we learn from some the excellent topics presented in the RAC space.  Its the lack of time or session conflicts that prevent many of us from attending all of them.

3. RAC SIG..
The RAC SIG had opened elections for its officer positions.  As mentioned before, I wanted a smaller role, besides I could not stay away from not working with user groups. Stood for the ‘International Chair’ position and was elected.  This is a great role, because it allows me to helping take the RAC SIG to the next level.  We are encouraging starting smaller SIGS in the various regions of the world.  Actually China, Singapore and India have adopted this already, with several meetings already scheduled and organized we hope other regions will also do the same..

4. AIOUG
Great start, and 2008 was a great year. we had two technology nights at Bangalore and Hyderabad.  Great start, great interest from Oracle users in India.  In 2009 we have already done our first quarterly meeting in Mumbai, instead of a technight we organized a techday.  We wanted to try this based on the feedback received from the other two tech events.
We just completed our next major event ‘3 day Oracle user conference Sangam’09.  With Tom Kyte being the key speaker for the event, we also had other speakers from inside and outside Oracle.  More about it in a separate blog.

Sangam09 just completed, OOW just around the corner. . lot to write and share.. not to mention more technical content also on RAC and ASM

Back from UKOUG 2007

UKOUG is really a very well organized conference. A conference I look forward to attend every year, either as speaker or as an attendee. This years conference was no different as usual great. Unbelievable set of presenters on great topics. It was interesting to see some excellent RAC related topics this year. I did give a presentation on “Exploiting Parellel Queries in a RAC Environment” just focusing on RAC related areas. Apart from this I also had the opportunity to chair a RAC roundtable on the final day.

Julian Dykes presentation was different compared to last year. Those who attended last years UKOUG presentation, Julian had done a presentation where he requested some attendees (including myself) to act as Oracle instances to demonstrate Cache Fusion. Here are some photographs from this session last year (courtesy Connor McDonald) Picture 1 and picture 2. This year his presentation focused on Interconnects. Was excellent as usual. Here is another view from Alex’s blog on this presentation..

There were 4-5 papers from CERN on RAC topics, two of the ones I had attended are 1) Real Life experiences of RAC Scalability with a 6 Node Cluster, 2) Installing and Managing Hundreds of RAC Servers. Its interesting to see more and more companies undertaking challenging projects using Oracle RAC.

Another outstanding presentation was on Oracle Clusterware By Bernhard de Cock Buning. He explained the clusterware details with such great detail. Showing all the components of the clusterware and then drilling down into each component at a greater detail. The new component piece that I learnt from this presentation was the oclsomon, which was introduced in Oracle 10.2.0.2 and I have not covered this in my Oracle 10g RAC Grid Services and Clustering book. Great job Bernhard..

I will discuss about oclsmon later on this blog.

I hope these presenters can do a webcast on these topics for the Oracle RAC-SIG.

Back from OOW 2007

Just returned from OOW’2007. As usual this is one of the events that tend to be pretty hectic with several sessions and events that get scheduled in parallel and you end up confused on which one would be better to pick. Though several times, several of us think that OOW is Oracle’s show case to show off what is coming or some kind of marketing talk. Over the past couple of years, I have noticed that this is not completely true. There are presentations that contains product and new release overview, if you carefully select the presentation titles, you can get some good technical presentation.

Some of the good technical sessions I had attended were the performance tuning for data warehouses, performance tuning roundtable, performance tuning RAC and several of the IOUG and SIG events.

OOW started on Sunday with various SIGs presenting in their respective areas. I had presented a topic on Parallel queries in Oracle RAC for the RAC-SIG on Sunday.

At OOW, we also released a book on Oracle Automatic Storage Management (ASM) published by Oracle Press. This book is a combined effort of Nitin, Rich Long and myself. I am happy that the contents are pretty well covered from the basics of storage management to some internals and also to topics such as how to change the paradigm of the current of storage management implementation.

.

This week had another great importance, a dream had come true, and a dream I had been working on for several years had become a reality. While every country had an Oracle Users Group that provided knowledge and information sharing to Oracle users, one such group was long past due in India. I had been working on getting one started over the past several years. I am happy to say we officially went live with this group. Those interested should check the user group website at www.aioug.org