Oracle DBA – A lifelong learning experience

Bugs with rm -rf from within asmcmd

Posted by John Hallas on July 8, 2014

Performing an rm -rf operation is normally a simple operation albeit risky if you are in the wrong folder. However within ASMCMD there are a couple of bugs associated with it.

Here are two examples and workrounds

FRA full - warning in the alert log  - action needs to be taken - this is an 11.2.0.4 database on OEL (5.8)
ORA-19815: WARNING: db_recovery_file_dest_size of 10484711424 bytes is 100.00% used, and has 0 remaining bytes available.
 ARCH: Error 19809 Creating archive log file to '+FRA'
Errors in file /app/oracle/diag/rdbms/tst11204/TST11204/trace/TST11204_ora_13209.trc:
ORA-16038: log 1 sequence# 148 cannot be archived
ORA-19809: limit exceeded for recovery files
ORA-00312: online log 1 thread 1: '+DATA/tst11204/redo01.log'
USER (ospid: 13209): terminating the instance due to error 16038

 

Let’s look at the disk from asmcmd and we see a lot of old folders that we do not need

ASMCMD [+FRA/TST11204/ARCHIVELOG] > ls
2013_12_08/
2013_12_09/
2013_12_10/
2013_12_11/
2013_12_12/
2013_12_13/
2013_12_14/
2013_12_15/
2013_12_16/
2013_12_17/
2013_12_18/
2013_12_19/
2013_12_20/
2013_12_21/
2013_12_22/
2013_12_23/
2013_12_24/

Lets remove them all

 rm -r 2013*

You may delete multiple files and/or directories.
Are you sure? (y/n) y
ORA-29261: bad argument
ORA-00600: internal error code, arguments: [ORA_NPI_ERROR], [600], [ORA-00600: internal error code, arguments: [KFCHK_ERRRET], [kfa.c], [3025], [], [], [], [], [], [], [], [], []
], [], [], [], [], [], [], [], [], []
ORA-06512: at line 4 (DBD ERROR: OCIStmtExecute)

rm -r 2013_12_08
You may delete multiple files and/or directories.
Are you sure? (y/n) y
ORA-29261: bad argument
ORA-00600: internal error code, arguments: [ORA_NPI_ERROR], [600], [ORA-00600: internal error code, arguments: [KFCHK_ERRRET], [kfa.c], [3025], [], [], [], [], [], [], [], [], []
], [], [], [], [], [], [], [], [], []
ORA-06512: at line 4 (DBD ERROR: OCIStmtExecute)

We have hit a known bug – only applies to 11.2.0.4 but I am sure I have seen manifestations of this issue on previous version of 11GR2

 

Bug 16892396  ORA-600 [KFCHK_ERRRET] [kfa.c] from ASMCMD “rm -rf” operation

11.2.0.4

fixed:

The fix for 16892396 is first included in

Interim patches may be available for earlier versions – click here to check.

Symptoms:

Related To:

Description

This bug is only relevant when using Automatic Storage Management (ASM)

Running the ASMCMD rm -rf command interleaved with the RDBMS file activities can trigger an ORA-600 [KFCHK_ERRRET] [kfa.c] error.Note:  If you run ‘rm -rf’ on a directory not accessed by an RDBMS instance, you  should not hit the issue.

I got round it using probably the best way of deleting  archive logs, through rman

Startup mount

rman target / catalog

 

 DELETE ARCHIVELOG UNTIL TIME 'sysdate -10';

DELETE EXPIRED ARCHIVELOG ALL;

CROSSCHECK ARCHIVELOG ALL;

The second bug is

Bug 15994107  ORA-600 [kfcDeadlockAvoid01] from “asmcmd rm -rf” if fix for bug 15900104 present

 This note gives a brief overview of bug 15994107. 
  The content was last updated on: 28-JUN-2013
  Click here for details of each of the sections below.

Affects:

Product (Component) Oracle Server (Rdbms)
Range of versions believed to be affected Versions BELOW 12.2
Versions confirmed as being affected
  • (None Specified)
Platforms affected Generic (all / most platforms affected)

Fixed:

The fix for 15994107 is first included in

Interim patches may be available for earlier versions – click here to check.

Symptoms:

Related To:

Description

This bug is only relevant when using Automatic Storage Management (ASM)ORA 600 [kfcDeadlockAvoid01] can occur during “asmcmd rm -rf” if the fix for bug 15900104 is present. Rediscovery Notes If ORA 600 [kfcDeadlockAvoid01] is encountered during asmcmd rm -rf AND the fix of bug 15900104 is present then you are likely hitting this issue. Workaround None Note: For interim patches please use this fix instead of 15900104 to address both issues.

This fix has been superseded – please see the fixed version information for Bug:15994107 . The box below only shows versions where the code change/s for 15900104 are first included – those versions may not contain the later improved fix.

The fix for 15900104 referred to is actually then superceded by the original bug I started with.  So the moral of the story is not to use rm -rf wherever possible and try and remove files from rman – assuming they are database related files of course

The fix for 15900104 is first included in
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: