Oracle DBA – A lifelong learning experience

Curing unevenly balanced ASM diskgroups to reduce poor file distribution

Posted by John Hallas on May 2, 2012

Back in Nov 2011 I posted a question on the Oracle-L mailing group about my perception that ASM disk rebalances seemed to be required on DATA diskgroups in ASM  (never FRA, presumably because the FRA had lots of similar sized objects (flashlogs archivelogs  etc)) and even after rebalancing there seemed to be a permanent imbalance between disks of the same size. I was querying the effectiveness of the rebalancing operation.

There was some good responses including a script that Dave Herring had created, based on  key MOS articles: 818171.1 (Identifying Files with Imbalances), 351117.1 (Troubleshooting ASM Space Issues), 367445.1 (Advanced Balance and Space Report on ASM). Manual rebalancing  would normally not be required, because ASM automatically rebalances disk groups when their configuration changes. You might want to do a manual rebalance operation if you want to control the speed of what would otherwise be an automatic rebalance operation.

I have been chasing this down and have now come across Bug 7699985: UNBALANCED DISTRIBUTION OF FILES ACROSS DISKS.

It appears to be a problem in 11.1.0.7 and I know a number of sites have reported it. Despite  repeated manual rebalance operations there can be a variance of up to 10% between different disks of the same diskgroup, even if they are the same size. It is fixed in 11.2.0.1.  The workaround is to set the _asm_imbalance_tolerance parameter to be 0 rather than the default of 3.This controls the hundredths of a percentage of inter-disk imbalance to tolerate. Then rebalance the disks manually and reset the parameter back again to 3 as you don’t need to leave it  balancing all the time.

SQL> @asm_imbalance   (from Report the Percentage of Imbalance in all Mounted Diskgroups (Doc ID 367445.1)

@asm_imbalance.sql

 Columns Described in Script               Percent Minimum
                                 Percent Disk Size Percent  Disk Diskgroup
Diskgroup Name                 Imbalance  Varience    Free Count Redundancy
------------------------------ --------- --------- ------- ----- ----------
DATA                                 9.0        .5    15.2    84 EXTERN

SYS@+ASM SQL>l
    Select dg.name,dg.allocation_unit_size/1024/1024 "AU(Mb)",min(d.free_mb) Min,
    max(d.free_mb) Max, avg(d.free_mb) Avg
    from v$asm_disk d, v$asm_diskgroup dg
    where d.group_number = dg.group_number
    group by dg.name, dg.allocation_unit_size/1024/1024
SY@+ASM SQL>/

NAME                               AU(Mb)        MIN        MAX    AVG
------------------------------ ---------- ---------- ---------- ------
DATA                                    1       7728      11599   8026

alter diskgroup data rebalance power 4;

NAME                               AU(Mb)        MIN        MAX    AVG
------------------------------ ---------- ---------- ---------- ------
DATA                                    1       7734      11483   8026 – no difference

alter system set "_asm_imbalance_tolerance"=0;
alter diskgroup data rebalance power 4;
NAME                               AU(Mb)        MIN        MAX        AVG
------------------------------ ---------- ---------- ---------- ----------
DATA                                    1       8020       8062   8026  - Now nicely rebalanced
About these ads

4 Responses to “Curing unevenly balanced ASM diskgroups to reduce poor file distribution”

  1. Setting _asm_imbalance_tolerance = 0 and scheduling weekly rebalance has worked out well for me.

    • John Hallas said

      Thanks Brian, it is always good to get validation of a technique. I have had no problems in setting the parameter across a number of databases and forcing a rebalance. Weekly may be good practise but I think I will leave it to a more ad-hoc basis

      John

  2. This is very nice research and good information to have if you have a significant issue where some actual process is taking longer than it should due to the imbalance. Showing that an imbalance actually causes slower file i/o in a particular case is not trivial. Balance is overall a statistical advantage, but burning cycles to get balance is questionable. So I agree with John’s thought to leave it ad hoc. And I in fact would only do it to address an actual response time or throughput problem tracked back to a particular file imbalance. Even then there is a real question of whether moving a particular object to a different tablespace is more effective in balancing the problematic object. Of course if you have a really hot object that needs specific and protected iops, then you build an unconflicted disk group for it (presuming you cannot implement a design that avoids the heat without damaging the logic of your data model.) Especially since this appears to not happen after 11.2+, I would really avoid institutionalizing maintenance windows to fix a meta-problem that is going away.

    • John Hallas said

      Thanks for the feedback Mark.
      I held back for a couple of weeks on this post because ideally I would like to have been able to answer the questions – what impact does this have and how do you prove the impact?

      My original post on Oracle-L was about my theory that it consumed storage as all disks in a diskgroup can only use the same level of storage therefore if the maximum available on some disks is 13Gb and others is 7Gb then 6Gb wil be unuable on those disks with 13Gb free. I would have liked to spend some time analysing that but dont really have the will to do it.

      It is interesting that oracle have it down as a bug as that does imply that it causes a problem, either via storge or through performance

      John

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
Follow

Get every new post delivered to your Inbox.

Join 190 other followers

%d bloggers like this: