Oracle DBA – A lifelong learning experience

What does ‘extended’ mean when referring to RAC clusters

Posted by John Hallas on March 11, 2010

Yesterday I attended and spoke at the Scotland DBA SIG. It was an enjoyable day with not one but two presentation from Julian Dyke. The second one was an overview of HA options and it in Julian mentioned extended RAC. Now I have worked at several sites which have used or wanted to use extended RAC and I have heard a number of views on what exactly the term extended means.

From MoS Note 220970.1 we get the definitive statement on the maximum distance for extended RAC

The high impact of latency create practical limitations as to where this architecture can be deployed. While there is not fixed distance limitation, the additional latency on round trip on I/O and a one way cache fusion will have an affect on performance as distance increases. For example tests at 100km showed a 3-4 ms impact on I/O and 1 ms impact on cache fusion, thus the farther distance is the greater the impact on performance. This architecture fits best where the 2 datacenters are relatively close (<~25km) and the impact is negligible. Most customers implement under this distance w/ only a handful above and the farthest known example is at 100km. Largest distances than the commonly implemented may want to estimate or measure the performance hit on their application before implementing. Do ensure a proper setup of SAN buffer credits to limit the impact of distance at the I/O layer.

So we now have figures that 100Km is technically possible but probably won’t work well and most sites use less than 25Km. Co-incidentally in the paragraph above that quotation it refers to SE RAC (10g) as being limited to having all nodes located in the same room.  So in two lines of text we have just gone from a 100ft to 100KMs.

Yesterday Julian’s answer when I asked him the question was that he thought extended was defined as a couple of kilometres or above which is reasonable. However I am thinking that a better definition of extended is not to focus on distance of nodes apart but on what connects the nodes. My thoughts are that that if fibre channel or DWDM is involved and the nodes are not in the same building  then that would count as an extended RAC cluster despite potentially only being a 100 yards apart. Another view would be that if some form of array based mirroring is in place and RAC technology is in place then that again would define the system as ‘extended’ .

Now that all fits with a view I have heard from a well-known RAC consultant from Oracle who defines any RAC system where nodes are in different rooms as extended.  I have never subscribed to that theory but I can see that the definition I provided in the previous paragraph would lead to that conclusion.

Now we just need to consolidate on the word ‘extended’ rather than the use of  ‘stretched’ which I see occasionally.

I really do hope I get some comments on this piece as I am sure it is worthy of debate and comment and anything that helps to create a shared view on the matter can only be good.

7 Responses to “What does ‘extended’ mean when referring to RAC clusters”

  1. I think the text only makes it more difficult.

    I don’t think the distance of the nodes in a RAC cluster makes *any* impact at all. It’s the latency of I/O and the interconnect that impacts performance. Of course the distance and latency of both I/O and the interconnect are related (latency is likely to go up when the distance increases), but different implementation strategies will have different latencies…the text makes a nice statement when latency is too high (3-4ms at 100km, but works best when datacentres are relative close <= 25km, so it basically is saying here 3-4ms is too high)

    My interpretation of both 'extended' and 'stretched' would be 'not in the same datacentre'. But I think this strongly depends on personal interpretation.

  2. 100km sounds like science fiction. With <10m, you can add performance by adding nodes to the cluster. With >10km, you are (mis)using RAC as a disaster solution.

    Everything between is a compromise between ha and performance, right?

    • John Hallas said

      Thanks to both of you for your inputs. 100Km does not seem realistic although I did work at a site that wanted to do RAC over a 80km link but were deterred because Oracle did not give their full support to the idea. I don’t blame Oracle really as the system was non active/active RAC at the time so we had no idea how much traffic would be moving between the nodes


  3. Yoon said

    I really want to know reference sites that implemented this solution(oracle rac extended over 100Km)
    Is there any site(company)??

  4. Ever said

    Across two data centres we have around 5 ms ping response time with dual redudancy line and not sure type of link. Is it fine implementig the extended rac? Is there max thershold, where it will do node evicitions? pls advise…

    • John Hallas said

      Dark fibre would be good but 5ms sounds pretty good. Does that latency hold up all teh time, even when the netwrk might be busy.
      I would still be tempted to put the main RAC cluster at the primary site, closest to any systems that it connects to and have a DG standby at the remote site but I say that with no knowledge whatsoever of your site.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: