Home > Oracle > ORA-00600: internal error code, arguments: [2662],

ORA-00600: internal error code, arguments: [2662],


Wed Jul  7 03:59:44 2010
Errors in file /opt/oracle/admin/xxx/bdump/xxx_smon_7875.trc:
ORA-00600: internal error code, arguments: [2662], [1696], [1478225386], [1696], [1956780490], [8388610], [], []
replication_dependency_tracking turned off (no async multimaster replication found)
Completed: alter database open
Wed Jul  7 03:59:45 2010
Fatal internal error happened while SMON was doing active transaction recovery.
Wed Jul  7 03:59:45 2010
Errors in file /opt/oracle/admin/xxx/bdump/xxx_smon_7875.trc:
ORA-00600: internal error code, arguments: [2662], [1696], [1478225386], [1696], [1956780490], [8388610], [], []

That was an interesting problem.  Infact database was UP in READONLY mode, what the DBA did, just brought it down.  When he re-started (sql> startup;) the database started throwing   ORA-00600 error and instance terminated by SMON.
ORA-00600: internal error code, arguments: [2662], [1696], [1478225386], [1696], [1956780490], [8388610], [], []

That was very interesting why a re-start caused an ORA-00600 and SMON terminated the instance.

We search through Metalink for this specific ORa-00600 error and found the first argument [2662] points a SCN mismatch, or block corruption etc. But, since the [b] and [d] are same, shows there is no SCN mismatch.  

we tried to open the database in READ ONLY mode, and failed with database need recovery.

Wed Jul  7 01:21:30 2010
alter database open read only
Wed Jul  7 01:21:30 2010
ORA-16005 signalled during: alter database open read only…

But intestingly, it was clear from the alert log, the recovery was completed! 

Beginning crash recovery of 1 threads
Wed Jul  7 01:18:08 2010
Started redo scan
Wed Jul  7 01:18:08 2010
Completed redo scan
 282 redo blocks read, 96 data blocks need recovery
Wed Jul  7 01:18:08 2010
Started recovery at
 Thread 1: logseq 1, block 3871, scn 0.0
Recovery of Online Redo Log: Thread 1 Group 5 Seq 1 Reading mem 0
  Mem# 0 errs 0: /oradata1/xxx/redo5.log
Wed Jul  7 01:18:08 2010
Completed redo application
Wed Jul  7 01:18:08 2010
Ended recovery at
 Thread 1: logseq 1, block 4153, scn 1696.1477902943
 96 data blocks read, 96 data blocks written, 282 redo blocks read
Crash recovery completed successfully

But still we try to do a recovery

Wed Jul  7 01:21:40 2010
ALTER DATABASE RECOVER  database
Media Recovery Start
Starting datafile 1 recovery in thread 1 sequence 2
………………………………………………..
Media Recovery Complete
Completed: ALTER DATABASE RECOVER  database
Wed Jul  7 01:21:47 2010
alter database open
Wed Jul  7 01:21:47 2010
Beginning crash recovery of 1 threads
Wed Jul  7 01:21:47 2010
Started redo scan
Wed Jul  7 01:21:47 2010
Completed redo scan
 288 redo blocks read, 0 data blocks need recovery
Wed Jul  7 01:21:47 2010
Started recovery at
 Thread 1: logseq 2, block 3, scn 0.0
Recovery of Online Redo Log: Thread 1 Group 1 Seq 2 Reading mem 0
  Mem# 0 errs 0: /xxx/xxx/redo01.log
Wed Jul  7 01:21:47 2010
Completed redo application
Wed Jul  7 01:21:47 2010
Ended recovery at
 Thread 1: logseq 2, block 291, scn 1696.1477923082
 0 data blocks read, 0 data blocks written, 288 redo blocks read
Crash recovery completed successfully

So, both crash recovery and Media recovery completed.  Tried to open the database, end up in ORA-00600 error again!

Errors in file /opt/oracle/admin/xxx/bdump/xxx_smon_2644.trc:
ORA-00600: internal error code, arguments: [2662], [1696], [1477923142], [1696], [1956780490], [8388610], [], []
replication_dependency_tracking turned off (no async multimaster replication found)
Completed: alter database open
Wed Jul  7 01:21:49 2010
Fatal internal error happened while SMON was doing active transaction recovery.
Wed Jul  7 01:21:49 2010
Errors in file /opt/oracle/admin/xxx/bdump/xxx_smon_2644.trc:
ORA-00600: internal error code, arguments: [2662], [1696], [1477923142], [1696], [1956780490], [8388610], [], []
SMON: terminating instance due to error 600
Instance terminated by SMON, pid = 2644

So, it was clear that first part of instance recovery (rollforward) and Media recovery is fine, we have a problem with transaction level rollback. But, how come there is a transaction when the database was in READ ONLY mode.  we never bought the database in READ WRITE mode after the restore…! 
 
The DBA knows the database was built reccently from a hot backup and he just tried his luck – just because it was using a backup control file during the recovery.  So when the database asked for a recovery he just recoverd using the BACKUP CONTROLFILE.

Wed Jul  7 05:10:10 2010
ALTER DATABASE RECOVER  database using backup controlfile
Wed Jul  7 05:10:10 2010
Media Recovery Start
……………………………………………………
Media Recovery Complete
Completed: ALTER DATABASE RECOVER    LOGFILE ‘/xxxxx
Wed Jul  7 05:10:35 2010
alter database open read only
Wed Jul  7 05:10:36 2010
SMON: enabling cache recovery
Wed Jul  7 05:10:36 2010
Database Characterset is WE8ISO8859P1
replication_dependency_tracking turned off (no async multimaster replication found)
Completed: alter database open read only

Interesting, database got opened in READ ONLY mode with out any issues. Then we understood the the original restore did a recovery using BACKUP CONTTROLFILE and open in READ ONLY mode. and never opened in RESETLOGS.  So, after restarting the database, got messed up..  There you go..

He again opened the database in RESETLOGS and the database opened normally

Wed Jul  7 05:15:46 2010
alter database open resetlogs
Wed Jul  7 05:15:46 2010
RESETLOGS after complete recovery through change 7285742799758
Resetting resetlogs activation ID 1050409165 (0x3e9bf8cd)
Wed Jul  7 05:15:54 2010
LGWR: Primary database is in CLUSTER CONSISTENT mode
Assigning activation ID 1065243956 (0x3f7e5534)
 
woops.. end of ORA-00600: internal error code, arguments: [2662], [1696], [1477923142], [1696], [1956780490], [8388610], [], [].   And the impact, I named my Blog with ORA600Tom 🙂

Advertisements
  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: