Apart from the space usage that Robin refers to, another worrying aspect is the time taken when gathering stats on a partitioned table and most of that time is taken by running a delete statement

DELETE
FROM SYS.WRI$_OPTSTAT_SYNOPSIS$
WHERE SYNOPSIS# IN
(SELECT H.SYNOPSIS#
FROM SYS.WRI$_OPTSTAT_SYNOPSIS_HEAD$ H
WHERE H.BO# = :B1
AND H.GROUP# NOT IN
(SELECT T.OBJ# * 2
FROM SYS.TABPART$ T
WHERE T.BO# = :B1
UNION ALL
SELECT T.OBJ# * 2
FROM SYS.TABCOMPART$ T
WHERE T.BO# = :B1))

I will demonstrate the problem and a simple solution and you will be able to see the significant performance improvements achieved.

I first of all enable tracing against a gather statsjob using the standard database configuration.

SQL ID: 7q4x1r46xgq0a
Plan Hash: 0
BEGIN dbms_stats.gather_table_stats('BIA_RTL','DWB_RTL_SL_RETRN_LINE_ITEM',
  granularity=>'AUTO',cascade=>false,degree=>dbms_stats.auto_degree); END;

VERALL TOTALS FOR ALL RECURSIVE STATEMENTS

call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse      162      0.11       0.05          0         27         50           0
Execute   1054     27.84    1023.64     254884     553622       1562         512   ***************************
Fetch      868      3.81       4.18         28      97431          0         928
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total     2084     31.76    1027.88     254912     651080       1612        1440

Misses in library cache during parse: 84
Misses in library cache during execute: 58

Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  db file sequential read                    254912        1.01        995.82
  unspecified wait event                     254912        0.04          7.09
SQL ID: 1zdkwckmyutrd
Plan Hash: 413388401
DELETE FROM SYS.WRI$_OPTSTAT_SYNOPSIS$
WHERE
 SYNOPSIS# IN (SELECT H.SYNOPSIS# FROM SYS.WRI$_OPTSTAT_SYNOPSIS_HEAD$ H
  WHERE H.BO# = :B1 AND H.GROUP# NOT IN (SELECT T.OBJ# * 2 FROM SYS.TABPART$
  T WHERE T.BO# = :B1 UNION ALL SELECT T.OBJ# * 2 FROM SYS.TABCOMPART$ T
  WHERE T.BO# = :B1 ))
call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.01       0.00          0          0          0           0
Execute      1     26.50    1022.20     254877     462319          0           0
Fetch        0      0.00       0.00          0          0          0           0
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total        2     26.51    1022.20     254877     462319          0           0

Misses in library cache during parse: 1
Misses in library cache during execute: 1
Optimizer mode: ALL_ROWS
Parsing user id: SYS   (recursive depth: 1)

Rows     Row Source Operation
-------  ---------------------------------------------------
      0  DELETE  WRI$_OPTSTAT_SYNOPSIS$ (cr=462319 pr=254877 pw=0 time=0 us)
      0   FILTER  (cr=462319 pr=254877 pw=0 time=0 us)
      0    HASH JOIN  (cr=462319 pr=254877 pw=0 time=0 us cost=31972 size=519892260 card=17329742)
  49312     TABLE ACCESS FULL WRI$_OPTSTAT_SYNOPSIS_HEAD$ (cr=819 pr=0 pw=0 time=0 us cost=311 size=1180632 card=49193)
16953751     INDEX FULL SCAN I_WRI$_OPTSTAT_SYNOPSIS (cr=461500 pr=254877 pw=0 time=86847 us cost=31546 size=103979388 card=17329898)(object id 478)
      0    UNION-ALL  (cr=0 pr=0 pw=0 time=0 us)
      0     TABLE ACCESS BY INDEX ROWID TABPART$ (cr=0 pr=0 pw=0 time=0 us cost=3 size=12 card=1)
      0      INDEX RANGE SCAN I_TABPART_BOPART$ (cr=0 pr=0 pw=0 time=0 us cost=2 size=0 card=1)(object id 566)
      0     TABLE ACCESS BY INDEX ROWID TABCOMPART$ (cr=0 pr=0 pw=0 time=0 us cost=9 size=48 card=4)
      0      INDEX RANGE SCAN I_TABCOMPART_BOPART$ (cr=0 pr=0 pw=0 time=0 us cost=2 size=0 card=368)(object id 588)
Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  db file sequential read                    254877        1.01        995.53
  unspecified wait event                     254877        0.01          7.05
********************************************************************************

Th main highlights of  this trace are the 1022 seconds run the delete  in sql_id 1zdkwckmyutrd and the big wait event is on db file sequential read   -  995 seconds

We now alter the WRI$_OPTSTAT_SYNOPSIS$ table so that it uses parallelism and note the difference.

SQL ID: 7q4x1r46xgq0a
Plan Hash: 0
BEGIN dbms_stats.gather_table_stats('BIA_RTL','DWB_RTL_SL_RETRN_LINE_ITEM',
  granularity=>'AUTO',cascade=>false,degree=>dbms_stats.auto_degree); END;
 
  OVERALL TOTALS FOR ALL RECURSIVE STATEMENTS

call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.00       0.00          0          0          0           0
Execute    368      9.32      38.21          1    1055398          0           0   ***************************</span>
Fetch        0      0.00       0.00          0          0          0           0
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total      369      9.32      38.21          1    1055398          0           0

Misses in library cache during parse: 0

Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  PX Deq: Msg Fragment                            1        0.00          0.00
  cursor: pin S wait on X                         5        0.01          0.07
  PX Deq: Execution Msg                        9272        0.03         20.49
  db file sequential read                         1        0.00          0.00
  unspecified wait event                          1        0.07          0.07
  resmgr:cpu quantum                             23        0.00          0.05
  PX qref latch                                   8        0.00          0.00
 
  SQL ID: 1zdkwckmyutrd
Plan Hash: 2668798483
DELETE FROM SYS.WRI$_OPTSTAT_SYNOPSIS$
WHERE
 SYNOPSIS# IN (SELECT H.SYNOPSIS# FROM SYS.WRI$_OPTSTAT_SYNOPSIS_HEAD$ H
  WHERE H.BO# = :B1 AND H.GROUP# NOT IN (SELECT T.OBJ# * 2 FROM SYS.TABPART$
  T WHERE T.BO# = :B1 UNION ALL SELECT T.OBJ# * 2 FROM SYS.TABCOMPART$ T
  WHERE T.BO# = :B1 ))
call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.00       0.00          0          0          0           0
Execute      1      0.13       8.63          0         11          0           0
Fetch        0      0.00       0.00          0          0          0           0
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total        2      0.13       8.63          0         11          0           0

Misses in library cache during parse: 1
Misses in library cache during execute: 1
Optimizer mode: ALL_ROWS
Parsing user id: SYS   (recursive depth: 1)

Rows     Row Source Operation
-------  ---------------------------------------------------
      0  DELETE  WRI$_OPTSTAT_SYNOPSIS$ (cr=11 pr=0 pw=0 time=0 us)
      0   FILTER  (cr=11 pr=0 pw=0 time=0 us)
      0    PX COORDINATOR  (cr=11 pr=0 pw=0 time=0 us)
      0     PX SEND QC (RANDOM) :TQ10001 (cr=0 pr=0 pw=0 time=0 us cost=4809 size=519892260 card=17329742)
      0      HASH JOIN  (cr=0 pr=0 pw=0 time=0 us cost=4809 size=519892260 card=17329742)
      0       PX RECEIVE  (cr=0 pr=0 pw=0 time=0 us cost=11 size=1180632 card=49193)
      0        PX SEND BROADCAST :TQ10000 (cr=0 pr=0 pw=0 time=0 us cost=11 size=1180632 card=49193)
      0         PX BLOCK ITERATOR (cr=0 pr=0 pw=0 time=0 us cost=11 size=1180632 card=49193)
      0          TABLE ACCESS FULL WRI$_OPTSTAT_SYNOPSIS_HEAD$ (cr=0 pr=0 pw=0 time=0 us cost=11 size=1180632 card=49193)
      0       PX BLOCK ITERATOR (cr=0 pr=0 pw=0 time=0 us cost=4795 size=103979388 card=17329898)
      0        TABLE ACCESS FULL WRI$_OPTSTAT_SYNOPSIS$ (cr=0 pr=0 pw=0 time=0 us cost=4795 size=103979388 card=17329898)
      0    UNION-ALL  (cr=0 pr=0 pw=0 time=0 us)
      0     TABLE ACCESS BY INDEX ROWID TABPART$ (cr=0 pr=0 pw=0 time=0 us cost=3 size=12 card=1)
      0      INDEX RANGE SCAN I_TABPART_BOPART$ (cr=0 pr=0 pw=0 time=0 us cost=2 size=0 card=1)(object id 566)
      0     TABLE ACCESS BY INDEX ROWID TABCOMPART$ (cr=0 pr=0 pw=0 time=0 us cost=9 size=48 card=4)
      0      INDEX RANGE SCAN I_TABCOMPART_BOPART$ (cr=0 pr=0 pw=0 time=0 us cost=2 size=0 card=368)(object id 588)
Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  reliable message                                2        0.00          0.00
  enq: KO - fast object checkpoint                1        0.00          0.00
  PX Deq Credit: send blkd                      239        0.00          0.04
  PX Deq: Join ACK                                8        0.00          0.00
  os thread startup                              32        0.06          1.87
  PX Deq: Parse Reply                            33        0.00          0.04
  PX qref latch                                   8        0.00          0.00
  PX Deq Credit: need buffer                      7        0.00          0.00
  PX Deq: Execute Reply                         696        0.19          6.40
  resmgr:cpu quantum                              1        0.05          0.05
  PX Deq: Table Q Normal                         11        0.00          0.00
  PX Deq: Signal ACK RSG                         40        0.00          0.00
  PX Deq: Signal ACK EXT                         21        0.00          0.00
  latch: session allocation                      12        0.00          0.00
********************************************************************************

I think 1023 seconds down to 38 seconds is a benefit worth having for the overhead of running an alter table command. In production where there is much more data the difference will be even more significant.

There is even more to come once this database can be upgraded to 11.2.0.2 (currently 11.1.0.7) because at  that release the table  WRI$_OPTSTAT_SYNOPSIS$ becomes partitioned. 

The WRI$_OPTSTAT_SYNOPSIS$ table also contains two extra columns to enable the partitioning. Does this mean that an upgrade from 11.1.0.7 to 11.2.0.2+ would require all tables that have incremental stats to be subject to a full re-gathering of statistics, or is there some method to migrate/convert the statistics between versions? That is my next piece of research

These are the posts that got the most views in 2011.

• 1 Purging statistics from the SYSAUX tablespace 15 comments May 2009
• 2 RMAN backup script – example – logging output 3 comments March 2008
• 3 ORA-19809: limit exceeded for recovery files – db_recovery_file_dest_size and archiver error 6 comments July 2008
• 4 11g – library cache mutex X – known bug 14 comments March 2008
• 5 Where has consistent=y gone when using Datapump export 3 comments December 2008

The second surprise of the WordPress summary is that I only created 17 new posts. That has really caught me out as I thought I had been more prolific than that. I will certainly try harder in 2012.

In the idle time that generally happens between Xmas and New Year I was catching up with various mails that I  had marked as ‘follow-up – today’, which is quite a misnomer for me as I rarely follow-up and never on the same day. I came across a post from Karl Arao on Oracle-L in which he mentioned a  script stored on a site he runs.

Karl describes it as “Just another web notebook about Oracle,Linux,Troubleshooting,Performance,etc..etc” and that is a good summary. It just contains excerpts of notes and lots of links to other web articles. It just seems a different format to most blogs and I found it interesting to browse down the r/h menu list and look at a few interesting notelets. I suspect mots of us have something similar, mine is a Word document called vodafone_notes.do although it is 5 years and 6 sites since I worked there. The difference is that this is an online shared utility and that is what attracted it to me.

Karl no doubt will be sending me as hamper as a thank you for directing thousands of hits to his site.

My next 2 posts will cover  :-

• altering WRI$_OPTSTAT_SYNOPSIS  tables to speed up the gathering of stats
• adding a trigger to detect when a standby database is open, in which case it will  to turn off the dataguard broker.

There I have committed myself  – but note that I am using the nowait option of commit so it might be a while

Best wishes to my readers in 2012.

 My site is just going down the road with Goldengate so the talk by Pythian’s Marc Fielding on a real life Goldengate migration was very useful. This was a large financial institution where the system was crucial to business continuity and GG was to be used to provide a rapid fallback facility if things went wrong. The main thing I took away from the talk was how small-minded they must be not to provide adequate testing facilities for such a large project. Not being able to use full data sets and similar sized hardware (OK it was a 14TB database) does add a lot of risk and no small matter of frustration to the technicians involved in the migration. Some of the diagnostics that Marc talked about will be very useful to use and I was interested in the alternatives to supplementary logging which may be required if there is no primary key and it is difficult to identify a row specifically.

I did start to listen to another talk but after around 10 people had left I plucked up courage and made a hasty exit myself. It was just not for me.

The best presentation I saw at the conference was Connor Macdonald on a fresh approach to the optimizer statistics. Connor is a real showman and his easy on-stage manner belies the degree of effort he must spending preparing his numerous slides. The set of slides associated with the ITIL process deserved a round of applause by itself and indeed it received one.  This was the second session I went  to that mentioned the value of index key compression and the way it can be calculated by using ‘analyze index validate structure’. A very good presentation that provided food for thought.

My final session was Mike Swing talking about database tuning for packaged apps. He had way too much content and rushed through it much too fast. As several people said to me afterwards, all he really recommended was getting faster disk and more memory.  I liked his presentation style and easy manner but it was a bit light on useful content.

So here endeth day 3. I think this was the conference I have enjoyed the most and got the most from. The presentations were of a top standard and even though I was only interested in the RDBMS stream I had plenty of choice for most time slots. I know that cancellations and changes are hard to avoid but there did seem to be a lot and that made planning harder than it should have been. I think my only constructive critique would be that there were a number of presentations repeated from last year (and some from other SIGs almost 2 years ago). I fully understand that a good presentation is  still a good presentation a year later and not everyone has the chance to have seen it but personally  I am not in favour of too much repeated material.

First up was Thomas Presslie talking about Dataguard fast start failover. How he managed to demonstrate transactions and network connectivity using whisky and toilet paper could not be done full justice in a blog – it had to be seen to be believed.

It did make me want to do more with FSFO, especially noting how easy the setup was using OEM. However my belief that the database is only part of the end solution and failing that over to a second datacentre after a network flicker may leave the application stack in a mess does still concern me. Co-incidentally I have a requirement to set up a second standby configuration cascaded from a physical standby but keeping the 3^rd database perhaps one hour behind whilst the standby is in real time apply mode with no lag. That might give us a chance to determine the status of the data before a logical corruption (user error) had occurred. Much more likely to be of value is flashback query but we are going to look at both avenues. It is highly unlikely we would ever be in a position to flashback the database.

Julian Dyke then talked for an hour about RAC trouble-shooting (mostly 11.2.0.2) and the time flew past. I made quite a few notes of things to think about. The pros and cons of putting the scan addresses in /etc/hosts (HPUX) to be used in the event of a DNS failure was one thought. Looking at the exectask function and the scripts used to call various function was another action I took for myself. Another was a big list of asmcmd commands, some of which I did not recognise. I think they must have come in with 11GR2 which I have not really used myself although we are using it on site.

Tanel Poder’s biggest ever problem was next up. I had seen this presentation last year and knew the answer but how he got there was still interesting. The use of the HPUX command kitrace (similar to dtrace on Solaris – see reply below for more details) reminded me that I was going to look at that in some detail but have never got around to it. As my site is likely to be moving away from HPUX sooner rather than later perhaps there is not much point now.

After lunch John Beresniewicz was talking about ASH outliers. Quite mathematically based, which is always a challenge for me but he will be posting a script (possibly via Doug Burn’s blog) which he has developed as another means of dissecting and analysing ASH data.

Michael Salt’s talk on indexes was full of real world examples and there were lots of nice little hints and tips, none of which were earth-shattering but all of which were good practise and I found it a useful reminder of what I should be doing when looking at code. On the same theme two slots later Tony Hasler was presenting a beginners guide to SQL tuning.   I have never seen Tony present before but I really liked both his style and the content. A lot of information thrown in and good explanations of various autotrace outputs. I will definitely be downloading his presentation to run through it and see what I can put to further use. Whilst I do not think I am expert in the field of SQL tuning, indeed far from it, I do like to think I know what to look for. Sometimes listening to others you realise in the same lecture both how much you already know and how little you actually follow best practises. There is no real substitute from looking at code and trying to improve performance. For a lot of us who have a very wide-ranging DBA role then that opportunity to practise odes not appear often enough which is why it is good to review and refresh your approach now and then.

At every conference I like to try and hear something new or touch on an area that is outside my day job. John King’s talk on Edition Based Redefinition was just that. I am not really in a position to take advantage of the ability to let users run differing sets of code and then migrate them across to a new release in a seamless manner, all without any outages or interruption to service. However I could see how useful it could be, especially in the world of the Apps DBA, say for EBS. Apparently no less a person than Tom Kyte referred to EBR as the ‘killer feature’ within 11GR2.  John had an easy, comfortable manner  and the time flew past, so much so that he had to be dragged kicking and screaming from the stage by the next presenter.

All in all another good day, rounded off with a couple of beers with work colleagues and a few presenters, all with plenty of Oracle chat included.

Mariusz Piorkowsk talked about Real Application Testing and in particular using it for upgrades from 10.2 to 11.2. I don’t want to be too critical of the presentation but having tested RAT for a similar purpose (11GR1) I thought it was a bit light on facts and figures. How many plans changed, how many had to be tuned, what were the big gotchas in his upgrades etc. However if I knew nothing of RAT beforehand I would have been eager to test it out after listening to Mariusz.

Onto my presentation which was straight after lunch. I was slightly stressed beforehand as my eldest daughter had gone into labour overnight with our first grandchild and I was on tenterhooks waiting for news. Having to turn my phone off did not help!!. The presentation was well-attended and I got through my material without any major hitches, albeit rather quickly. I had given the presentation twice before. Once at work and once at a SIG and it had taken an hour each time. Having fifteen minutes less this time I removed a couple of slides and moved through in a speedy manner. It was good to see many of the audience scribbling notes, unless they were all busy marking me down on the feedback form. I answered plenty of questions afterwards and the feeling I got was that the content was new to many.

I came out the presentation to see that I had three missed calls. After ringing home I was ecstatic to discover I now have a granddaughter, 8lb 14 ounces who was born just before I started talking.  Needless to say that I was that drained that I missed the next lecture on my agenda.

I did make it to Alex Gorbachev talking about measuring I/O. I am sure the first half was a fantastic talk but I am ashamed to say I had a bit of the nodding dog syndrome, (OK, I fell asleep). Nothing to do with Alex  but I had been awake most of the night followed by a long drive and then the pressure of doing a presentation. The second half was quite interesting. The output (graphs) were very good but I would like to have seen more on how to get the data into them. The key takeaway for me was that visualisation is everything and that there is no point in posting reams of data when a single well presented chart can explain so much more in an easy to assimilate manner.

The final talk of the day for me was Doug Burns on SQL plan management. Having never met Doug but having been in an email conversation with him recently it was good to put a face to the name. It was an interesting talk put across in a lively manner. The main  aspect for me was to use plan management to capture the hints from an existing database (maybe 10g) and then copy them across to the new database (11g) and retain the current plan without making wholesale database changes. Then you can fix the code to be more efficient in the 11G database in your own timescales.

The UKOUG conference produces a real wealth of talent and just within the RDBMS stream, which is all I really look at, there are several alternatives for every time slot, most of which I could attend and be interested in. Having been involved in the background with some of the work in both selecting speakers and the putting them into a schedule to suit everyone I am aware of how much effort goes in, both by the UKOUG staff and volunteers. If you add the additional complexity of late changes due to speaker unavailability it is a massive piece of organisation which appears to be so seamless to the majority of attendees.

We start of with the executive summary for those that like to cut to the chase.

Instance 1 of a 2 node RAC cluster (11GR1 HPUX) crashed and the evidence indicates this was related to excessive AMM resize operations. The resize operations left the instance with an undersized buffer cache at instance start-up and this has steadily grown since. However, the _small_table_threshold value appears to have been set too low and does not appear to have been dynamically sized. As such the method for buffer cache or direct file read has changed and was the cause for a change in disk read activity on instance 1.  This would be exacerbated by multiple high disk IO operations such as RMAN running.

End of summary – however I will point out that this is not the first time we have been bitten by the automatic memory management killing an instance because it decides that it will consume excessive CPU by constantly resizing the SGA. In fact it is my view that AMM is an unnecessary overhead on any systems which are have been in production for a while and realistic SGA values are known. AMM can be very flakey/buggy and can sporadically lead to a high number of resize operations (if Oracle can’t get it right). In fact, the high number of operations actually occurred in the final 3 minutes of the final snap period before the crash indicating 1324 resize operations in a 3 minute period 

 So we had an instance crash and restart on a Sunday. The next day, when main usage was starting up, very poor performance of sql_id  ‘2cahqnpt4cq23’ was noticed on instance 1. The sql was generating intensive ‘direct file read’, later it reverted to ‘ksfd: async disk IO’ However, on instance 2 the same query continued to use buffer gets rather than direct disk reads and was proving to be far more efficient. It was noted that from about 08:10 (after the RMAN backup was complete) the response time for the problem SQL was much improved and observations were that it had reverted to buffer gets rather than ‘direct file read’. 

Scott  performed research to understand why Oracle would select to use a direct file read rather than read records into or from the buffer cache.  This led to the parameter _small_table_threshold which is used to determine whether a buffer cache or disk read will be more efficient (MOS Note 787373.1 gives a full description and introduced me to the term smallness logic). This value was set to 418 on Instance 1 and 57802 on instance 2.  We noted that this value (if not set) takes it’s value from the size of the buffer cache at instance start-up.  When using AMM the buffer cache at instance start-up is initiated from the setting at instance shutdown.  This was be the  key to why we were seeing high disk reads on Instance 1 and not on instance 2. The last entries we can see in the DBA_HIST_MEMORY_RESIZE_OPS view indicates that db_cache_size had gone from 8.5Gb to 180M after the database restart

One side aspect was that db_file_multiblock_read_count was noticed (after the event) as having changed from 128 to 10 after the instance crash and reported as changing from 10 to 128 when the database was restarted with new AMM settings This is left to be determined automatically by Oracle in our database so I’m guessing the small value was chosen due to the undersized buffer cache and will have contributed significantly to worsened I/O experienced on that instance

So we now have an explanation as to what caused the difference in performance between the nodes.

My focus is really about what processes we can put in place to stop a re-occurrence. My thoughts are :-

• Why use AMM when we know we have had problems before. What value does it really provide on a stable system.
• If using AMM then why not fix key values such as buffer cache so they have no opportunity to fluctuate. One aspect I see regularly with AMM resize operations  is that it makes minute changes such as amending a 15Gb buffer cache to a 14.9Gb buffer cache and then reverts that change a few minutes (even seconds) later. I appreciate that the shared pool is the likely beneficiary of this change but when that is 4Gb for example then 0.1Gb is not going to make a significant difference anyway.
• We already have  a daily ‘morning check’ that reports difference in spfile parameter changes  and that did pick up the db_file_multiblock_read_count change but missed the _small_table_threshold change, probably because it was an underscore parameter. The script is shown below.

Implement processes on major systems that report differences between parameters on startup. That should be relatively easy to do as we just need to create a pfile from spfile before a shutdown and repeat after startup and do a diff command  on the two files. If we have a crash the pfile from the previous startup should be in place so we have a constant comparison available. The more I think about the more I think we should have done that a long-time ago and indeed we have already implemented the reporting capability later on but not the discipline of checking after a restart.

So I have offered a problem we had, shown how the solution was identified, no doubt generated some thoughts on researching _small_table_threshold  and consequently developed some ideas, if only for myself for ways we can improve management of critical systems.

 PS ‘ksfd: async disk IO’ deservers a blog entry of it’s own – currently hoping PSU 7 will fix it. But not tested as yet

deltatime "Date",
target_name "Database",
hostname "Host",
operation "Operation",
key1 "Parameter",
attribute "Attribute",
oldvalue "Old Vlue",
newvalue "New Value"
from MGMT$ECM_CONFIG_HISTORY_KEY1
where target_name like '%PRD%'
and deltatime >= sysdate -1
and collectiontype='MGMT_DB_INIT_PARAMS_ECM' and key1 !='resource_manager_plan 

LGWR: Insufficient standby redo logfiles to archive

The first is an error I see on a few databases and had assumed that it was to do with insufficient standby log files. The recommendation is that there should be at least one more than the normal redo log files so that applying the redo will not be delayed on standby, which would be critical in maximum protection mode when the primary would wait until log was applied remotely.

Wed Oct 19 11:58:13 2011
LGWR: Insufficient standby redo logfiles to archive thread 1 sequence 47505
LGWR: Standby redo logfile selected for thread 1 sequence 47505 for destination LOG_ARCHIVE_DEST_2
Thread 1 advanced to log sequence 47505 (LGWR switch)
Current log# 20 seq# 47505 mem# 0: +DATA/SID/onlinelog/group_20.2083.735986363
Current log# 20 seq# 47505 mem# 1: +FRA/SID/onlinelog/group_20.1668.735986365

MyOracleSupport document  798361.1 “LGWR: Insufficient standby redo logfiles to archive” Messages in Primary Database

This states:- Bug 8358103 – will be fixed in 10.2.0.5 and 11.1.0.8. “Insufficient standby redo logfiles” message is misleading as it is followed by “Standby redo logfile selected”, which indicates the standby redo logs is selected and being used. You can safely ignore these messages.  Note that 11.1.0.8 is unlikely to happen as 11.1.0.7 is the terminal release

Lag apply alert on standby database

The second is something I came across yesterday after we had performed a switchover from primary to standby. Normally we set the lag apply metric in OEM to alert us if standby gets more than 900 seconds behind). This is set in OEM for each database (unless you want to issue a modified template)

Database > Metric and Policy Settings > then first metric is Apply Lag (seconds) In order to see graph, you can go to all metrics > Data Guard Performance > Apply Lag (Seconds).

The alert is set on standby and yet when a switchover is invoked (I am assuming either by DG broker or manually as we did it, then we would expect that alert to be still in place. However it is removed and has to be re-applied.

Block change tracking on a standby requires active dataguard license

From 11GR1 onwards running a standby database with the block change tracking file enabled (which helps make incremental backups much faster) requires an active dataguard license to be in place. Whilst I think this policy is very mean spirited, especially when a license was not required in a similar situation on 10G it is a requirement and should not be ignored.

Hope these heads-up are useful

The fix for that is to set the parameter resource_manager =’FORCE:DW_PLAN’ and the DW_PLAN is retained. The reason why the default plan is started off is so that  scheduler knows it has sufficient resources to get the job done and it will not be artificially constrained. If your plan does not limit the scheduler required resources then there is no harm in making your normal plan the plan for 24*7

I had noticed that  the plan was defaulting from a couple of entries in the alert log but wanted to get an exact listing of what was happening. I used the xml logs that came with the ADR package and the X$DBGALERTEXT view.

The syntax was easy to work out and produced a much better report than that delivered by using grep against the alert log (text version)

set lines 125 pages 0
col when form a20
col message_text form a95
select substr(originating_timestamp,1,15) when , message_text from x$dbgalertext where originating_timestamp > (sysdate - 10) and  message_text like '%Resource%’; 

07-OCT-11 23.00      Setting Resource Manager plan SCHEDULER[0x2C57]:DEFAULT_MAINTENANCE_PLAN via scheduler window
07-OCT-11 23.00      Setting Resource Manager plan DEFAULT_MAINTENANCE_PLAN via parameter
08-OCT-11 03.00      Setting Resource Manager plan DW_PLAN via parameter
08-OCT-11 07.00      Setting Resource Manager plan SCHEDULER[0x2C58]:DEFAULT_MAINTENANCE_PLAN via scheduler window
08-OCT-11 07.00      Setting Resource Manager plan DEFAULT_MAINTENANCE_PLAN via parameter
09-OCT-11 03.00      Setting Resource Manager plan DW_PLAN via parameter
09-OCT-11 07.00      Setting Resource Manager plan SCHEDULER[0x2C59]:DEFAULT_MAINTENANCE_PLAN via scheduler window
09-OCT-11 07.00      Setting Resource Manager plan DEFAULT_MAINTENANCE_PLAN via parameter
10-OCT-11 03.00      Setting Resource Manager plan DW_PLAN via parameter
10-OCT-11 23.00      Setting Resource Manager plan SCHEDULER[0x2C53]:DEFAULT_MAINTENANCE_PLAN via scheduler window
10-OCT-11 23.00      Setting Resource Manager plan DEFAULT_MAINTENANCE_PLAN via parameter
11-OCT-11 03.00      Setting Resource Manager plan DW_PLAN via parameter
11-OCT-11 23.00      Setting Resource Manager plan SCHEDULER[0x2C54]:DEFAULT_MAINTENANCE_PLAN via scheduler window
11-OCT-11 23.00      Setting Resource Manager plan DEFAULT_MAINTENANCE_PLAN via parameter
12-OCT-11 03.00      Setting Resource Manager plan DW_PLAN via parameter
12-OCT-11 23.00      Setting Resource Manager plan SCHEDULER[0x2C55]:DEFAULT_MAINTENANCE_PLAN via scheduler window
12-OCT-11 23.00      Setting Resource Manager plan DEFAULT_MAINTENANCE_PLAN via parameter
13-OCT-11 03.00      Setting Resource Manager plan DW_PLAN via parameter
13-OCT-11 23.00      Setting Resource Manager plan SCHEDULER[0x2C56]:DEFAULT_MAINTENANCE_PLAN via scheduler window
13-OCT-11 23.00      Setting Resource Manager plan DEFAULT_MAINTENANCE_PLAN via parameter

I knew I had resolved this on the pre-production environment by submitting some scheduler attribute commands but I could not remember the format until I came across MoS note 786364.1 which reminded me how I did it last time.

Before

First of all change the scheduler windows to use the DW_PLAN (or other nonrestrictive plan) using:

execute dbms_scheduler.set_attribute('WEEKNIGHT_WINDOW','RESOURCE_PLAN','DW_PLAN');
execute dbms_scheduler.set_attribute('WEEKEND_WINDOW','RESOURCE_PLAN','DW_PLAN');

--Then, for each window_name (WINDOW_NAME from DBA_SCHEDULER_WINDOWS), run:

execute dbms_scheduler.set_attribute('<window name>',' DW_PLAN ','');
execute dbms_scheduler.set_attribute('MONDAY_WINDOW','RESOURCE_PLAN','DW_PLAN');
execute dbms_scheduler.set_attribute('TUESDAY_WINDOW','RESOURCE_PLAN','DW_PLAN');
execute dbms_scheduler.set_attribute('WEDNESDAY_WINDOW','RESOURCE_PLAN','DW_PLAN');
execute dbms_scheduler.set_attribute('THURSDAY_WINDOW','RESOURCE_PLAN','DW_PLAN');
execute dbms_scheduler.set_attribute('FRIDAY_WINDOW','RESOURCE_PLAN','DW_PLAN');
execute dbms_scheduler.set_attribute('SATURDAY_WINDOW','RESOURCE_PLAN','DW_PLAN');
execute dbms_scheduler.set_attribute('SUNDAY_WINDOW','RESOURCE_PLAN','DW_PLAN');

After

 The proof is in the pudding  –  the change was made on the 18th October and the required plan stays in place

17-OCT-11 08.47      Setting Resource Manager plan DW_PLAN via parameter
17-OCT-11 23.00      Setting Resource Manager plan SCHEDULER[0x2C53]:DEFAULT_MAINTENANCE_PLAN via scheduler window
17-OCT-11 23.00      Setting Resource Manager plan DEFAULT_MAINTENANCE_PLAN via parameter
18-OCT-11 03.00      Setting Resource Manager plan DW_PLAN via parameter
18-OCT-11 23.00      Setting Resource Manager plan SCHEDULER[0x2C54]:DW_PLAN via scheduler window
18-OCT-11 23.00      Setting Resource Manager plan DW_PLAN via parameter
19-OCT-11 03.00      Setting Resource Manager plan DW_PLAN via parameterset

I think self-development is an important part of presenting. Yes I know the altruistic believe it should all be for the audience but I am happy to take away confidence and self-belief from my experiences. I set myself a target originally of doing a SIG presentation and then a conference presentation. My next step was to talk on a technical subject which has now been achieved and my final objective is to try and do an interactive technical demonstration using a laptop. At the moment I have no idea on what that will be about and I have no timescale so I can consider it a work in progress at the moment.