Emergency DBA Support JAM-Digital Edge Triumphs

On Sunday, November 1, 2009, Digital Edge’s Oracle DBA team received a complaint that one of the Oracle PL/SQL batch jobs was causing an ORA-00600 error. The team performed immediate troubleshooting to see what was going on and detected that their client’s custom program crashed the RAC secondary node. Fortunately, Digital Edge always provisions databases with high availability, which in this case, insured that the entire database did not go down.

PL/SQL batch was running against:

· 2 Node Oracle RAC

· Linux version 2.6.18-92.el5 (Red Hat 4.1.2-41)

· Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit

· Accessing EMC san as a shared storage resource.

· RAC also implements multiple services for load balancing and failover.

Once Digital Edge found the problem and tried to recover, they noticed there was an issue, the instances that were supposed to be running on the node did not start. The attempt to start service on the node produced an error:

· $ srvctl start service -d DB_NAME -s SERVICE_NAME -i INSTANCE_NAME

· PRKP-1030 : Failed to start the service S01.

· CRS-0215: Could not start resource 'ora.DB_NAME.SERVICE_NAME.INSTANCE_NAME.srv'

However, after reviewing the RAC log file, we figured out a reason for the failure:

· 2009-11-01 06:59:58.350: [ RACG][1287227712] [6710][1287227712][ora.SDPRD_SI1

· .SDPRD2.inst]: ORA-44305: service SDPRD_SI1 is running

· ORA-06512: at "SYS.DBMS_SYS_ERROR", line 86

· ORA-06512: at "SYS.DBMS_SERVICE", line 444

· ORA-06512: at "SYS.DBMS_SERVICE", line 365

· ORA-06512: at line 1

So we figured out that RAC was thinking that the service was already started. So to force a service shutdown we used:

· exec DBMS_SERVICE.STOP_SERVICE('SERVICE_NAME', 'INSTANCE_NAME')

After forcing the shutdown, the service started working normal.

Updated November 2026: the original post is kept above; the sections below add context for readers today.

What this incident teaches

The fix took one command, but finding it took experience: the cluster believed the service was still running on the failed node, so every normal start attempt was refused. Reading the RAC logs instead of retrying is what shortened the outage.

• High availability works: the second node kept the database online while the first was repaired.
• Logs answer questions faster than guesses. Keep them, and know where they are before an incident.
• Document the recovery steps in the run book so the next person needs minutes, not hours.

Oracle 10g is long out of support, but the pattern is the same on current Oracle, SQL Server and PostgreSQL clusters. Digital Edge runs 24×7 monitoring and database support with engineers on call, and keeps centralized logs so the evidence is there when it is needed.