UKOUG Database SIG Leeds- review

Last week we held the UKOUG Database SIG in Leeds as I described in a recent post. It was a well attended event in a  central location with excellent refreshments available. – Thank you to UKOUG’s  Anna Crellin for organising the day so well.

The talks were all very good and I think the event was enjoyed by all. Neil Johnson wrote a blog entry about his experience of the day and of presenting for the first time. He has now got the bit between his teeth and wants to present a 2 hour masterclass at the December conference in Manchester – (perhaps I am exaggerating slightly)

I ran the ‘BitsandPieces’ session where audience participation was the key idea and this was achieved. The theme was small pieces of Oracle functionality that are not well known or used.

We discussed :-

The dbms_workload_repository.add_colored_sql procedure which allows you to capture specific pices of sql that individually would not appear as a top resource in AWR data. Using the package to add a sql_id and the data does then appear and performance can be fully explored. A good introductory  blog entry is available from Dion Cho at http://dioncho.wordpress.com/tag/colored-sql/

• The benefits of using restore points and  flashback database  were covered. Examples were given of flashing back through a 10g to 11g upgrade (if you might want to), using a restore point before making a major table or code change that might need to be rolled back (better than rolling forward from a previous backup) and using the same restore point repeatedly as part of performance testing. The point was made that anything that might include an OS command (such as dropping or resizing a datafile probably would not be recovered through so be careful with the command. My advice is that I consider it as the belt that goes with the braces of a full backup.
• Invisible indexes – how easy they were to use and how they allowed the data_dictionary to be aware of the index but the optimizer ignores it. There was also a initialization parameter OPTIMIZER_USE_INVISIBLE_INDEXES  that allows the index to become visible at both a session and system level to allow full testing. Beware that if you rebuild an index it then becomes visible again.
• One tip that came out was that in pre-packaged environments such as EBS when an index is not wanted, rather than drop it, mark it as invisible. Then when any upgrades or patches are applied the  index still exists in the dictionary and is not recreated as it might be if it had been dropped.
• Another idea was the use of the opatch auto command. This came out in 11GR2 but not until PSU2. It manages a full GI and RDBMS patchset application and can stop all dependant databases, stop crs resources, stop crs, patch and restart everything. It can save a serious amount of time and is well worth investigating.

I did pick up another few interesting thoughts which I want to do a bit of research on and I will blog about those shortly.

So overall it  as a good day with a lot of community interaction, networking and a good laugh afterwards when about 15 of us went for a few beers.

Migrating an Oracle Home – dead easy?

Changing a database to use a new oracle home is easy isn’t it? A simple interview question that you would never get wrong. Everyone knows that you shutdown the database , edit the oratab file and restart the database again.  So then the interviewer presses and is obviously looking for more.

You confidently follow-up with ‘of course I would have copied the passwd and the init.ora file to the new ORACLE_HOME/dbs folder during the outage.’ The interviewer says “go on, what else will you do?”.

It is getting a bit more awkward now, but you rise to the occasion with a comment about editing the listener to show the new ORACLE_HOME and then reloading the listener – but he still wants more and you begin to dry up a bit.

After a bit of hinting you recognise that you might need to go into OEM and edit the configuration but you have given your all, you’re are a spent force, wishing you could move onto the next easy question.

The above is from an interview question I asked only last week and unless a candidate has recently been migrating databases around then I think that would be representative of many people’s answers.  Therefore the idea of this post is just to put together a few things to form a checklist of things that may be required when migrating to a new ORACLE_HOME. I am sure all are not relevant and the OWB comment might well be replaced with some other tool you might be using.

So just a bullet point list follows and if I have failed my own interview question then I will be really pleased to accept suggestions and add them to the list

Speeding up datapump exports in 11.1.0.7

I noticed that a small datapump export  involving a few tables was taking more than 30 minutes. I tried enabling parallelism, which was a mistake as for small tables parallelism is not utilised even if the parameter is used in the parameter file – that is probably worth a blog in itself. I then assumed it was a disk performance issue until I tried it on other systems and realised it was quite a general thing.

I then came across bug 10153617  which is quite old (Sept 2010) but the 11.1.0.7  release is old now as well.

=========================================================================
Interim Patch for Base Bug: 10153617
=========================================================================

Date: Wed Sep 29 06:12:33 2010
————————————————————————-
Platform Patch for          : Generic
Product Patched             : ORACLE DATABASE
Product Version #           : 11.1.0.7.0
RAC Rolling Installable     : YES

Bugs Fixed by this patch:
————————-
6460304: EXPDP TAKES MORE TIME
7362589: GSIST12: EXPDP TABLE POOR PERFORMANCE OVERHEAD COMPARED TO ORIGINAL EXP
7710931: DATAPUMP EXPORT IS EXTREMELY SLOW WHEN EXTRACTING SCHEMA
7722575: DATAPUMP VIEW KU$_NTABLE_DATA_VIEW CAUSES POOR PLAN / SLOW EXPDP

I applied this as part of a general upgrade to PSU13 (it is not dependant upon any specific PSU) and the impact is very significant.

A simple test demonstrates the difference

create user test identified by xxxxxx default tablespace users;

grant create session, create table to test;

create table test.test as select * from sys.dba_objects

exdp schemas=test directory=data_pump_dir content=ALL dumpfile=test.dmp logfile=test.log

The first datapump export took 20 minutes before I killed the job and the second finished in less than 2 minutes and I have seen this repeated on a number of systems. I applied this on top of PSU 13 with no problems and I would recommend reviewing any 11.1.0.7 databases where  you might be exporting schemas or running a lot of datapump jobs. The test above or something similar shoudl provide an easy way of seeing if you are hitting the problem.

In the patch details above you will notice a bug 7710931 is also mentioned as being fixed by bug/patch 10153617. The detail of that suggests it is still valid in 11.2.0.1 but the same test above on a couple of systems did not indicate any poor performance

Rebuild of standby using incremental backup of primary

I have a long to-do list of things I want to test out and one is rebuilding a standby by using an incremental backup from primary. Then along comes a note from my ex-colleague Vitaly Kaminsky  who had  recently been faced with the problem when a customer relocated two Primary 2-node RACs and a single node standby databases to a new location and just happened to start the standby databases in read-only mode. Vitaly tells the story :-

As you may know, the read-only mode will prevent any redo logs being applied to standby database, but on the surface everything looks OK – no errors and MRP0 process is running and showing “applying log” in  v$managed_standby.

The only problem is – the recovery is “stuck” on the last log the database was trying to apply before it was opened in read-only mode.

Unfortunately, the customer did not notice the omission for over 2 weeks and by the time I have had a chance to look at the environments, there were about 50G of redo logs accumulated for each and some of them were missing and/or corrupt which excluded the possibility of SCPing the logs over to the standby server and registering them with standby databases.

One of another factors which caused a lack of  attention to the standby databases falling behind is the absence of any error messages in the alert logs – every single log was shown as shipped and received.

In a case like this, the only option is to rebuild the standby database and in the past I did it using the traditional RMAN duplicate for standby routine.  However, in this particular case I had 2 databases to rebuild – one is small and another is large. The network between primary cluster and standby was slow as well.

For the small database I decided to use Grid Control GUI based wizard for creating the standby database and this process is quite straightforward and described in the documentation. For the large one, however, duplicating the database using RMAN would be too slow,there may have been a  performance degradation during the run and the maintenance window was too short for out-of-hours run.

This was a perfect case to try “incremental backup” approach. This method is described in a number of sources (if you Google it) but none of the “vanilla” cases worked for me.

I will not be listing the names and detailed output due to the production nature of the work – just the list of steps.

So, this is what I did at the end of the day:

PRE-REQUISITES:

Primary database can be single node or RAC and running OK.
No downtime of Primary is required.
All Dataguard settings are intact

Step-by-step:

1. 1.       Get the latest SCN from standby:

 select to_char(current_scn) from v$database;

10615562421

1. 2.       Create incremental backup on Primary for all the changes since SCN on standby:

[oracle@primary backup]$ rman target /</pre>
connected to target database: PRIMARY (DBID=720063942)

RMAN> run

2> {

3> allocate channel d1 type disk;

4> allocate channel d2 type disk;

5> backup incremental from scn 10615562421 database format

6> '/tmp/backup/primary_%U';

7> release channel d1;

8> release channel d2;
<pre>9> }

1. 3.       Create copy of control file on Primary:

 alter database create standby controlfile as ‘/tmp/backup/stby.ctl’;

1. 4.       SCP the backup files and standby control file to the standby server. A little tip: if you copy the backup files to the directory with the same name (like /tmp/backup here), your controlfile will know about them and you can bypass the registration bit later on.

1. 5.       The next step is to replace the standby control file with the new one. May sound simple, but this proved to be the trickiest part due to the fact that standby controlfile is OMF and in ASM.  You will need to use RMAN for the restore operation:

-          Switch the database to nomount, then:

restore controlfile from ‘/tmp/backup/stby.ctl';

-          Mount the database.

At this point you have the controlfile with the information about the files as they are on the Primary side, so, the next step is to register everything we have on Standby side:

catalog start with '+data/standby/';

Check the output and YES to register any reported standby files.

-          Shutdown immediate your standby instance.

 RMAN> switch database to copy;

RMAN> report schema;

On this stage you should have a nice and clean list of actual standby files.

 

Now we are ready to apply our differential backup to bring the standby in line with Primary:

RMAN> recover database noredo;

 Because the online redo logs are lost, you must specify the NOREDO option in the RECOVER command.

You must also specify NOREDO if the online logs are available but the redo cannot be applied to the incrementals.

If you do not specify NOREDO, then RMAN searches for redo logs after applying the incremental backup, and issues an error message when it does not find them.

When the recovery completes, you may start the managed recovery process again:

SQL> alter database recover managed standby database using current logfile disconnect;

Provided, all FAL settings are correct, your managed recovery will pick-up all logs generated on primary since the incremental backup and you will have fully synchronised standby again.

database user xs$null

When we put a new system into production we get the whole set of infrastructure penetration tested. Reading a recent review I saw the following recommendation as part of the database section.

XS$null Recommendation:

Ensure the XS$NULL account is not set to the default password even though it is locked.

That brought two things to mind.

1. What does the xs$null account actually do
2.  If an account is locked why is the default password such an issue

The focus of this article is to share some information about the xs$null account and then to ponder the changing of a default password on a locked account.

The oracle documentation for that user states

An internal account that represents the absence of a user in a session. Because XS$NULL is not a user, this account can only be accessed by the Oracle Database instance. XS$NULL has no privileges and no one can authenticate as XS$NULL, nor can authentication credentials ever be assigned to XS$NULL.

Which is enough to tell me that changing the password probably will not make a difference to any internal usage of that account that is supported by a MoS note which says  (in my simplified words) – don’t try to do anything with this account

When you try to alter user XS$NULL (for example to switch to another profile or to unlock it) you get the following error:

SQL> alter user xs$null profile default;
alter user xs$null profile default
*
ERROR at line 1:
ORA-01031: insufficient privileges

When you dump a callstack for this error you will discover it is raised from kzuial():

alter session set events ’1031 trace name errorstack level 3′;
alter user xs$null profile default;

……

full trace dump details can be viewed in MoS note  – Cannot Alter User XS$NULL [ID 1325766.1]

Changes

You upgraded to 11gR2

Cause

In 11.2 development explicitly added a check in the code to make sure that nobody (not even user SYS) can alter user XS$NULL. This is the expected behaviour.

Solution

You cannot alter user XS$NULL. This is the expected behaviour. User XS$NULL is internal and it may not be altered in any way. To guarantee the stability of the system, this was explicitly forbidden.

Even if this may work in earlier releases it is strongly recommended to not alter user XS$NULL.

If you try to do this because of some security auditing requirements (which for example demand that  all users should be in a certain profile) explain to the auditors that this is impossible.

Another reason why you may want this is to change the default tablespace of the user, while you cannnot alter this user specifically you can change the default tablespace at the database level and when you do this it will change the XS$NULL user temp tablespace as well.

In version 11.2.0.4 an additional fix to bug 12822989 was introduced that prevents the change of the password for internal user XS$NULL using the sqlplus ‘password’ command (or using the OCIPasswordchange call in general), please do not use the ‘password’ command in earlier releases to bypass the check, since it is unsupported to do so.

PS I do like the words ‘in 11.2.0.4 an additional fix was introduced’ , in the past tense, which is interesting because 11.2.0.4 is reputed to be the 11GR2 terminal release and is not scheduled for release until at least late into Q3 2013 as I understand the situation.

The release we are on for this database is 11.2.0.1 (supportability issues with the application is the main/only reason for that). So I will be going back with the response that the account is locked and we will not be changing it on Oracle’s advice.

PPS a little test shows that you can change account in 11.1.0.7 but not do anything with it in 11.2.0.3

11.1.0.7

SQL>select  username, profile from dba_users where username like 'XS%';

USERNAME                       PROFILE
------------------------------ ------------------------------
XS$NULL                        DEFAULT

SQL>alter user xs$null profile m_std_profile identified by  Changepass123;

User altered.

connect xs$null/Changepass123
ERROR:
ORA-28000: the account is locked
SQL>alter user xs$null account unlock;

User altered.

Connected.
SYS@MOMPTC2A SQL>connect xs$null/Changepass123
ERROR:
ORA-01017: invalid username/password; logon denied

11.2.0.3

SQL> select  username, profile from dba_users where username like 'XS%';

USERNAME                       PROFILE
------------------------------ ------------------------------
XS$NULL                        DEFAULT

SQL> alter user xs$null profile m_std_profile identified by  Changepass123;
alter user xs$null profile m_std_profile identified by  Changepass123
*
ERROR at line 1:
ORA-01031: insufficient privileges

Onto point 2. Now I am sure that there is a good reason to change a default password as well as locking the account but I fail to see it. My thought process would be that if someone has got into your database with enough privileges to unlock an account then changing a password is not going to be a big obstacle to them. Happy to have some explanation of that requirement. As the title of my  blog states – a lifelong learning experience

UKOUG Database Server SIG in Leeds, May 9th 2013

Last time a database server SIG was held in Leeds we had a very good attendance and hopefully this will be repeated on Thursday May 9th 2013 when the Metropole Hotel will be the host. This 4 star hotel is very conveniently placed not more than a couple of hundred yards from the station and should be a very good venue.

As always we are looking for good presentations from UKOUG members so that we can have a really strong database focused agenda.

If anyone wishes to present for the first time this would be a good opportunity but experienced presenters are always welcome. I am responsible for pulling the agenda together this time and hope I am inundated with offers.

The UKOUG link is here if anybody wishes to get more details or offer their support.

As a side note for Yorkshire based Oracle DBAs, Eter Pani has been organising meet-ups at the Adelphi for the last couple of months and the week before last we had 10 people attend for a night of chat about football, beer, and many other topics, as well as small measure of database stuff. It was a really good chance to meet others in the same job in the same area.  You never know it might make the next interview you go for easier if you have had a few beers previously with one of the interviewing panel. There again if you did not get a round in it might not have been a good move.

When I know the date of the next one I will post a note on this blog.

John

Adding new ASM disks – what is best practise?

According to My Oracle Support note – “How To Add a New Disk(s) to An Existing Diskgroup on RAC (Best Practices). [ID 557348.1]” you should create a test diskgroup using new storage before adding it to an existing diskgroup. That seems eminently sensible, although it is not something I normally do. It proves you can access the disk, and if there is a conflict (i.e the disk is already mapped and in use elsewhere) you are not risking your production DATA diskgroup. I have pasted the note info at the bottom of this post but basically you just create  a new diskgroup and add the new disk to it. If all is OK then drop the diskgroup and add the new disk to your existing diskgroup.

However the downside of that is that you can hit non-published bug:12398300 which is a duplicate of bug:12356910 (also non-published). Diskgroup Mount Hangs with RBAL Waiting on ‘GPnP Get Item’ and ‘enq: DD – contention’ [ID 1375505.1].  Note: This issue so far has been reported on RAC 11.2.0.2.3 and 11.2.0.3 environments which is where we saw it (RAC – 11.2.0.3 clusterware – 11.2.0.1 rdbms)

Simply the ALTER DISKGROUP MOUNT just hangs and has to be interrupted (CTRL-C). No errors in the ASM alert log

Killing the ora.gpnpd on the node when ASM is blocked in the gpnp wait, permits not having  to stop the ASM instance. For details, please see Note:1392934.1. Otherwise, restart the ASM instance that is causing the lock condition.

The fix will be included in future 11.2.0.3.x Patch Set Updates (PSUs) but no patches yet exists (at the time this article was written – Nov.9.2011). Also likely there will be patch requests for the fix to be included on top of existing 11.2.0.2 PSU, but none yet exist.

So I think I will stick with what I have always done, best practise or not.

--From Node 1

. oraenv

-- specify ASM instance from node 1

+ASM1

-- sudo -u oracle sqlplus may not work when run the first time so run:

sudo -u oracle ls

sudo -u oracle sqlplus / as sysasm

CREATE DISKGROUP TEST EXTERNAL REDUNDANCY DISK '' [DISK ''];

SELECT STATE, NAME FROM V$ASM_DISKGROUP;

-- from node 2

. oraenv

+ASM2

sudo -u oracle sqlplus / as sysasm

 

ALTER DISKGROUP TEST MOUNT;

SELECT STATE, NAME FROM V$ASM_DISKGROUP;

 

-- if all ok then

-- from node 2

alter diskgroup test dismount;

 

-- from node 1

DROP DISKGROUP TEST;

Now we can add the disk to the desired diskgroup safely.

 

-- From node 1

. oraenv

+ASM1

sudo -u oracle sqlplus / as sysasm

-- check disks visible in v$asm_disk

-- header_status should be CANDIDATE or FORMER

set lines 120 pages 100

column path format a20

SELECT name, path, mode_status, state, header_status, os_mb, free_mb

FROM v$asm_disk ORDER BY name, path;

-- check diskgroups

select GROUP_NUMBER,NAME,STATE,TOTAL_MB,FREE_MB from v$asm_diskgroup;

-- add disks to appropriate diskgroups

alter diskgroup x add disk '/dev/hdiskX';

-- monitor rebalance

set lines 170

select * from v$asm_operation;

select GROUP_NUMBER,NAME,STATE,TOTAL_MB,FREE_MB from v$asm_diskgroup;

How to identify if a patch is available within a PSU

Within MoS there are a set of notes which list all the patches of each PSU and show in which PSU they came in (PSUs are cumulative). The READme.html file with each PSU does not contain this information

11.1.0.7 Patch Set Updates – List Of Fixes In Each PSU [ID 1337836.1]
11.2.0.1 Patch Set Updates – List Of Fixes In Each PSU [ID 1340010.1]
11.2.0.2 Patch Set Updates – List Of Fixes In Each PSU [ID 1340011.1]
11.2.0.3 Patch Set Updates – List Of Fixes In Each PSU [ID 1449750.1] 

However beware if you are looking for a patch that is called Merged or something similar as you have to search the MoS notes for the included patches, not the merged patch itself

 As an example Patch 12640090 is an ASM fix but it is not found in note 1337836.1

But if you look at patch 12640090 in MoS you find the information about the included patches

 (4) Bugs Fixed by This Patch

———————————

The following are the bugs fixed by this patch:

  9007102: KFNMCLIENTSREGISTER12 WHEN HUNG UFG RECOVERS AND REGS FREED GPN

  9029156: ADDITIONAL TRACING FOR ASMB AND ASMB CONNECTIONS

 From note 1337836.1 I can see that 9007102 was included in PSU 8 but 9029156 is not quite as clear but it suggests it is included in PSU 13

Patch Details

 Patch 9029156: ADDITIONAL TRACING FOR ASMB AND ASMB CONNECTIONS 

Last Updated 21-Oct-2012 21:42 (2+ months ago)
Product Oracle Database Family Release Oracle 11.1.0.7.13

Size 104.5 KB
Download Access Extended Support
Classification General
Patch Tag 

Prerequisite Patches
14275623
DATABASE PATCH SET UPDATE 11.1.0.7.13 (INCLUDES CPUOCT2012)

 I understand Oracle have an easier tool for this but it is not generally available. I am sure there is a good reason for that but it escapes me

Me and my Exadata Xmas present

Actually it is a cardboard cutout – not me, the server. We have real ones downstairs in the data centre but my cardboard one doesn’t cost as much to support or keep air-conditioned.

Merry Xmas to all

Managing OCR and voting disks

This is basically a set of notes I wrote for myself about adding new voting disks and OCR disks to a sandpit RAC cluster as part of testing for migration between HP XP disk array and HP 3PAR disk array. The o/s was HPUX with 11.1.0.7 database.

 View status of OCR disks and Voting disks

sudo /app/oracle/product/crs/bin/ocrcheck

Status of Oracle Cluster Registry is as follows :

         Version                  :          2

         Total space (kbytes)     :     306972

         Used space (kbytes)      :       5880

         Available space (kbytes) :     301092

         ID                       :  746041401

         Device/File Name         : /dev/oracle/disk500

                                    Device/File integrity check succeeded

         Device/File Name         : /dev/oracle/disk501

                                    Device/File integrity check succeeded

          Cluster registry integrity check succeeded

          Logical corruption check succeeded

crsctl query css votedisk

0.     0    /dev/oracle/disk502

1.     0    /dev/oracle/disk503

2.     0    /dev/oracle/disk504

Located 3 voting disk(s).

Add a new OCR disk

Backup first  (the 10GR1 command format still works)

sudo /app/oracle/product/crs/bin/ocrconfig -export /home/oracle/ocr_backup -s online

Owned by root

-rw——-   1 root       sys         136140 Nov 27 08:15 /home/oracle/ocr_backup

As this is an 11GR1 cluster we will use the 11GR1 format

sudo /app/oracle/product/crs/bin/ocrconfig -manualbackup

dhpor43     2012/11/27 08:23:00     /app/oracle/product/crs/cdata/SANDPITR1/backup_20121127_082300.ocr

Listing the backups shows the recent backups

sudo /app/oracle/product/crs/bin/ocrconfig -showbackup

dhpor43     2012/11/27 06:36:27     /app/oracle/product/crs/cdata/SANDPITR1/backup00.ocr

dhpor43     2012/11/27 02:36:27     /app/oracle/product/crs/cdata/SANDPITR1/backup01.ocr

dhpor43     2012/11/26 22:36:27     /app/oracle/product/crs/cdata/SANDPITR1/backup02.ocr

dhpor43     2012/11/25 02:36:27     /app/oracle/product/crs/cdata/SANDPITR1/day.ocr

dhpor43     2012/11/24 06:36:27     /app/oracle/product/crs/cdata/SANDPITR1/week.ocr

dhpor43     2012/11/27 08:23:00     /app/oracle/product/crs/cdata/SANDPITR1/backup_20121127_082300.ocr

I have 3 disks available (all at 1Gb,  which is easily enough for either a voting or OCR disk)

Free ASM disks and their paths

==============================

Header    Mode     Path                      Disk Size

——— ——– ————————- ———

CANDIDATE ONLINE   /dev/oracle/disk507             1Gb

CANDIDATE ONLINE   /dev/oracle/disk508             1Gb

CANDIDATE ONLINE   /dev/oracle/disk509             1Gb

sudo /app/oracle/product/crs/bin/ocrconfig -replace ocrmirror /dev/oracle/disk507

ocrcheck

Status of Oracle Cluster Registry is as follows :

         Version                  :          2

         Total space (kbytes)     :     306972

         Used space (kbytes)      :       5908

         Available space (kbytes) :     301064

         ID                       :  746041401

         Device/File Name         : /dev/oracle/disk500

                                    Device/File integrity check succeeded

         Device/File Name         : /dev/oracle/disk507 disk replaced (was disk501)

                                    Device/File integrity check succeeded

         Cluster registry integrity check succeeded

However the replaced disk is not available although 507 has been removed from the list of candidate disks

Free ASM disks and their paths

==============================

Header    Mode     Path                      Disk Size

——— ——– ————————- ———

CANDIDATE ONLINE   /dev/oracle/disk508             1Gb

CANDIDATE ONLINE   /dev/oracle/disk509             1Gb

 

sudo /app/oracle/product/crs/bin/ocrconfig -replace ocr /dev/oracle/disk508

app/oracle/product/11.1.0/asm/bin $ocrcheck

Status of Oracle Cluster Registry is as follows :

         Version                  :          2

         Total space (kbytes)     :     306972

         Used space (kbytes)      :       5908

         Available space (kbytes) :     301064

         ID                       :  746041401

         Device/File Name         : /dev/oracle/disk508

                                    Device/File integrity check succeeded

         Device/File Name         : /dev/oracle/disk507

                                    Device/File integrity check succeeded

Add a new voting disk

sudo /app/oracle/product/crs/bin/crsctl add css votedisk /dev/oracle/disk500 –force

/app/oracle/product/crs/bin/crsctl add css votedisk /dev/oracle/disk509 -force

 Now formatting voting disk: /dev/oracle/disk500.

Successful addition of voting disk /dev/oracle/disk500.

app/oracle/product/11.1.0/asm/bin $sudo /app/oracle/product/crs/bin/crsctl query css  votedisk

 0.     0    /dev/oracle/disk502

 1.     0    /dev/oracle/disk503

 2.     0    /dev/oracle/disk504

 3.     0    /dev/oracle/disk509

 4.     0    /dev/oracle/disk500

Located 5 voting disk(s).

crsctl delete css votedisk  /dev/oracle/disk509 

Successful deletion of voting disk /dev/oracle/disk509.

crsctl delete css votedisk  /dev/oracle/disk500 

Successful deletion of voting disk /dev/oracle/disk500.

sudo /app/oracle/product/crs/bin/crsctl query css  votedisk

 0.     0    /dev/oracle/disk502

 1.     0    /dev/oracle/disk503

 2.     0    /dev/oracle/disk504

Located 3 voting disk(s).

My follow up actions are to see if Linux performs in the same manner and what the difference is on a 11GR2 cluster. Finally I want to understand why the released disk retained their header and if there is any way of avoiding  having to dd the header. I expect that asmlib on Linux will prove different