PSU for 11.1.0.7.1

One of the most read entries on this site is about the PSU release for 10.2.0.4 http://jhdba.wordpress.com/2009/07/28/applying-the-10-2-0-4-1-patch-set-update-psu/

On 20th October, 2009 the first PSU for 11.1.0.7 was released. This also incorporates the Oct 2009 CPU as well as a Dataguard Broker patchset bundle. Bug 7628357 – 11.1.0.7 Data Guard Recommended Patch Bundle #1.

 Oracle have maintained consistency with the 10.2.0.4 PSU by retaining the misinformation in the patch instructions. I did originally think that there was no need to update opatch as there was in 10g but I was wrong. Opatch should be upgraded but the PSU  does apply even if it is not upgraded.

Ensure that the latest Opatch version is installed

 

opatch version

Invoking OPatch 11.1.0.6.2

 

OPatch Version: 11.1.0.6.2

 

If it is not 11.1.0.6.7 or later then upgrade

 

cd $ORACLE_HOME

cp /shared/oracle/rdbms_patches/PSU_patches/11.1.0.7/p6880880_111000_HPUX-IA64.zip .

unzip p6880880_111000_HPUX-IA64.zip

rm p6880880_111000_HPUX-IA64.zip

 

opatch version

Invoking OPatch 11.1.0.6.8

 

OPatch Version: 11.1.0.6.8

 

 

Basic steps for a new build server are as follows

1) Install 11.1.0.6 and upgrade to 11.1.0.7

2) ensure opatch is in the path, an Oracle Home is set and the patcheset (8833297) is unzipped

3) Run the pre-check

Following the README notes I would use this command

$cd /shared/oracle/rdbms_patches/PSU_patches/11.1.0.7

$opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir ./8833297

which then fails

Oracle Interim Patch Installer version 11.1.0.6.2
Copyright (c) 2007, Oracle Corporation.  All rights reserved.

PREREQ session

Oracle Home       : /app/oracle/product/11.1.0/db_1
Central Inventory : /app/oraInventory
   from           : /var/opt/oracle/oraInst.loc
OPatch version    : 11.1.0.6.2
OUI version       : 11.1.0.7.0
OUI location      : /app/oracle/product/11.1.0/db_1/oui
Log file location : /app/oracle/product/11.1.0/db_1/cfgtoollogs/opatch/opatch2009-10-28_14-16-33PM.log

Invoking prereq "checkconflictagainstohwithdetail"
<span style="color: #ff0000;">The location "./8833297/README.html" is not a directory or a valid patch zip file.
Prereq "checkConflictAgainstOHWithDetail" not executed
PrereqSession failed: Invalid patch location.</span>

<span style="color: #ff0000;">OPatch failed with error code 73

Then use the corrected syntax

opatch prereq CheckConflictAgainstOHWithDetail -ph ./8833297

Invoking OPatch 11.1.0.6.2

Oracle Interim Patch Installer version 11.1.0.6.2
Copyright (c) 2007, Oracle Corporation.  All rights reserved.

PREREQ session

Oracle Home       : /app/oracle/product/11.1.0/db_1
Central Inventory : /app/oraInventory
   from           : /var/opt/oracle/oraInst.loc
OPatch version    : 11.1.0.6.2
OUI version       : 11.1.0.7.0
OUI location      : /app/oracle/product/11.1.0/db_1/oui
Log file location : /app/oracle/product/11.1.0/db_1/cfgtoollogs/opatch/opatch2009-10-28_14-16-57PM.log

Invoking prereq "checkconflictagainstohwithdetail"

Prereq "checkConflictAgainstOHWithDetail" passed.

OPatch succeeded.  

This is very quick, less than one minute

4) Now apply the PSU

$cd 8833297
$opatch apply

The patchset took 28 minutes to apply

Afterwards an opatch lsinventory command shows

Oracle Home       : /app/oracle/product/11.1.0/db_1
Central Inventory : /app/oraInventory
   from           : /var/opt/oracle/oraInst.loc
OPatch version    : 11.1.0.6.2
OUI version       : 11.1.0.7.0
OUI location      : /app/oracle/product/11.1.0/db_1/oui
Log file location : /app/oracle/product/11.1.0/db_1/cfgtoollogs/opatch/opatch2009-10-28_15-55-41PM.log

Lsinventory Output file location : /app/oracle/product/11.1.0/db_1/cfgtoollogs/opatch/lsinv/lsinventory2009-10-28_15-55-41PM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (2):

Oracle Database 11g                                                  11.1.0.6.0
Oracle Database 11g Patch Set 1                                      11.1.0.7.0
There are 2 products installed in this Oracle Home.
Interim patches (1) :

Patch  8833297      : applied on Wed Oct 28 14:54:39 GMT 2009
   Created on 7 Oct 2009, 23:52:06 hrs PST8PDT
   Bugs fixed:
     6870937, 7627743, 7652888, 7299153, 8242410, 6059178, 8563946, 6955744
     7497788, 6840740, 8244217, 8702276, 6981690, 8450529, 7432556, 7719143
     7523787, 8251486, 8367827, 7613481, 8341623, 7515145, 7348847, 8416414
     8250643, 8284633, 8230457, 8563948, 6900214, 7044551, 7318049, 8940197
     7013124, 7432514, 7393258, 7553884, 7639121, 8563944, 8860821, 7606362
     7426959, 7330434, 7708340, 7352414, 6452375, 7356443, 7341598, 8213302
     7196532, 7446163, 8409848, 8236851, 8342506, 7593835, 7340448, 7309458
     8290478, 8391256, 7462112, 7013817, 8499600, 7411865, 7331867, 7527650
     6977167, 8855565, 6501490, 6598432, 7524944, 7706138, 6941717, 8402555
     7494333, 7586451, 8402551, 7499353, 8408887, 7496908, 7511040, 7719148
     7311601, 7497640, 7373196, 7424804, 7452373, 7597354, 6882739, 7366290
     8543737, 6851669, 7318276, 8284438, 8324760, 7420394, 7834195, 7350127
     7475055, 8563942, 7122161, 8361398, 7451927, 7705669, 7676737, 8301559
     8224083, 8658581, 8211920, 8462173, 7206858, 8563945, 8855553, 8402637
     8257122, 8199266, 7454752, 7516536, 7345904, 8352304, 7416901, 7426336
     8352309, 6812439, 7219752, 8534338, 8542307, 8413059, 7572069, 7436280
     7432601, 7311909, 7506785, 7460818, 7276960, 8855559, 7013835, 7378322
     8402548, 7189645, 8563947, 8306933, 7477246, 7263842, 7480809, 8855575
     8433270, 7556778, 8836375, 7330611, 8339352, 7225720, 8563943, 7036453
     7628387, 8419383, 6970731, 7719668, 7203349, 8402562, 7680907, 7438445
     8855577, 8243648, 7630416, 6851110, 6980597, 6618461, 7357609, 8855570
     8369094, 8318050, 8306934, 7434194, 8833297, 7486595, 7716219, 8362693
     6599920, 7628866, 7183523, 7412296, 7135702, 7720494, 7436152, 7175513
     7602341, 6679303, 8339404, 7393804, 8856696, 7650993, 6980601, 7830065
     7462709, 8563941, 8226397, 7515779

Note that if you already have databases built then you need to run a catbundle script and check for invalid objects. Check the documentation for full details that but a quick summary is to run
cd $ORACLE_HOME/rdbms/admin
sqlplus /nolog
SQL CONNECT / AS SYSDBA
SQL STARTUP
SQL @catbundle.sql psu apply
SQL QUIT

Oracle Performance Management with Gaja Vaidyanatha

I attended this 2 day workshop ,held in London, last week.

I already owned both of Gaja’s book, Tales of the Oak Table and 101 Performance Tuning so I was reasonably familiar with  his background.

The course started of badly as Gaja was late arriving due to travel delays and so we did not kick off until 11:30 which was either 1.5 or 2 hours depending on whom you asked. He hit the ground running with a quick introduction and then a round the room ‘what do you want to get from this’ session.

The sessions covered everything set out in the advertising blurb, namely architecture, tracing, use of the OWI interface and a planned methodology for dealing with problems. One key point , apart from the availability of RAT in 11g was that the newer release did not add much to what Oracle 10g already provided with regard to performance tuning. 

The key methodology thoughts were to clearly define a goal, run a simple diagnostic routine and then repeat. With regard to SQL optimisation the 2 things to remeber were

1. Reducing logical I/O is primary in any performance tuning engagement
2. Hand in hand reducing elapsed time is equally important

The repeatable process is to find ‘interesting’ PIDS from glance or top, map them to a SPID (SID and SERIAL#) to find a session then run a 10046 trace and find the most expensive step (often the  most indented call).

Fix the code (one change at a time) and repeat.  If a 10046 trace does not help then get down to the OS level and look at how resources are being used.

That all sounds pretty simple and indeed it is.  I could write more about the methodology but I am concious that it is someone’s livelyhood and not to say too much.

Without sounding too knowledgeable or clever I was aware of most of the content that was discussed in the 2 days. Where I gained real benefits was in the storage aspects that Gaja covered very well and offered real insights into CPU, hyper threading, HBA usage and loading and suchlike. Various RAID levels were well covered and we had  a very good discussion on the value of placing redo logs on RAID 1+0 and suchlike.

Most of the second day was taken up with running through some real world examples, some where the problem was obvious, others where I had no clue at all. Several people had brought in 10046 traces and AWR snapshots and we all gained real insight as Gaja looked at them for the first time and showed us the key points worthy of investigation.

All in all a very worthwhile 2 days. The target audience needs to have a reasonable Oracle knowledge beforehand as a couple of students were saying that they had found it very difficult to follow. Oh and be prepared for long days as on day one Gaja was still talking strongly upto 18:00 despite being deadbeat from travelling and the second day went on up to 17:00.

Executive summary. Highly recommended

Configuring Dataguard Broker in 11G

DataGuard Broken?

A guest blog from colleague Simon Ryan

Last year we did some sandpit work on implementing DataGuard.  This was 10g on a Windows platform.  When it came to the broker we didn’t have much luck.  At the UKOUG Michael Moller referred to the Broker as DataGuard Broken (DataGuard 11g New Features).  From the audience reaction this seemed to be a popular perception.

Since then we’ve successfully implemented DataGuard on most of our Production databases (both 10g and 11g) on HP-UX.  However, we’ve still not embraced the Broker!  The ‘broken’ tag still taints its reputation.  A recent visit by Oracle attempted to assure us that the Broker with 11g is much improved.

The following is a basic guide to implementing the broker and our findings:

Our Setup

o       Primary – Two node RAC Cluster (XXXPRD1A)

o       Physical Standby – Single node (XXXPRD1B)

o       HP-UX 11.31 Itanium

o       Database using ASM (both 11.1.0.7).

o       DataGuard running in Max Availability with Real Time Apply.

Configuration

Add the following entries to tnsnames.ora on all instances:

 

XXXPRD1A_DGMGRL =

   (DESCRIPTION=

     (ADDRESS=(PROTOCOL=tcp)(HOST=xxxem03-vip)(PORT=1526))

     (CONNECT_DATA=

       (SERVICE_NAME=XXXPRD1A_DGMGRL)

     )

    )

 

XXXPRD2A_DGMGRL =

   (DESCRIPTION=

     (ADDRESS=(PROTOCOL=tcp)(HOST=xxxem04-vip)(PORT=1526))

     (CONNECT_DATA=

       (SERVICE_NAME=XXXPRD2A_DGMGRL)

     )

    )

 

XXXPRD1B_DGMGRL =

   (DESCRIPTION=

     (ADDRESS=(PROTOCOL=tcp)(HOST=xxxem01-vip)(PORT=1526))

     (CONNECT_DATA=

       (SERVICE_NAME=XXXPRD1B_DGMGRL)

     )

    )

 

Add the following entry to SID_LIST of your listener in listener.ora on all instances:

 

 (SID_DESC = 

      (ORACLE_HOME = /app/oracle/product/11.1.0/db_1)

      (SID_NAME = XXXPRDxx)

      (GLOBAL_DBNAME = XXXPRDxx_DGMGRL)

    )

 Stop/start listener.

 Test you can now connect:

 

 sqlplus sys@XXXPRD2A_dgmgrl as sysdba

  

Make the following parameter changes:

 

 XXXPRD1A:
<pre>
 

alter system set dg_broker_start=true scope=spfile;

alter system set dg_broker_config_file1='+DATA/XXXPRD1A/broker1.dat' scope=spfile;

alter system set dg_broker_config_file2='+FRA/XXXPRD1A/broker2.dat' scope=spfile;

 XXXPRD1B:

alter system set dg_broker_start=true scope=spfile;

alter system set dg_broker_config_file1='+DATA/XXXPRD1B/broker1.dat' scope=spfile;

alter system set dg_broker_config_file2='+FRA/XXXPRD1B/broker2.dat' scope=spfile; 

 Stop/Start the databases:

 

 Stop XXXPRD1A:
<pre>
 

srvctl stop database -d XXXPRD1A -o immediate

 Stop XXXPRD1B:

shutdown immediate

 Start XXXPRD1B:

startup mount

 Start XXXPRD1A:

srvctl start database -d XXXPRD1A

 

 Check for the broker process:

 

 XXXPRD1A:
<pre>
 

ps –ef | grep ora_dmon_XXXPRD1A

XXXPRD1B:

ps –ef | grep ora_dmon_XXXPRD1B

 

Create DG Broker Configuration:

 

[code lang='xml'] Primary:

dgmgrl

connect sys

<em>enter sys password</em>

 

create configuration XXXPRD1A_DGCONFIG as primary database is “XXXPRD1A” connect identifier is XXXPRD1A;

Configuration "XXXPRD1A_dgconfig" created with primary database "XXXPRD1A"

 

add database “XXXPRD1B” as connect identifier is XXXPRD1B maintained as physical;

Database "XXXPRD1B" added

 

enable configuration;

Enabled.

 

show configuration;

 

Configuration

  Name:                XXXPRD1A_dgconfig

  Enabled:             YES

  Protection Mode:     MaxAvailability

  Databases:

    XXXPRD1A - Primary database

    XXXPRD1B - Physical standby database

 

Fast-Start Failover: DISABLED

 

Current status for "XXXPRD1A_dgconfig":

SUCCESS

 

 Check Broker Files Exist:

Primary and Standby ASM:

Check they are there.

 

Show Database:

 Primary: 

show database verbose “XXXPRD1A”;

 

Database

  Name:            XXXPRD1A

  Role:            PRIMARY

  Enabled:         YES

  Intended State:  TRANSPORT-ON

  Instance(s):

    XXXPRD1A

    XXXPRD2A

 

  Properties:

    DGConnectIdentifier             = 'XXXPRD1A'

    ObserverConnectIdentifier       = ''

    LogXptMode                      = 'SYNC'

    DelayMins                       = '0'

    Binding                         = 'OPTIONAL'

    MaxFailure                      = '0'

    MaxConnections                  = '1'

    ReopenSecs                      = '300'

    NetTimeout                      = '30'

    RedoCompression                 = 'DISABLE'

    LogShipping                     = 'ON'

    PreferredApplyInstance          = ''

    ApplyInstanceTimeout            = '0'

    ApplyParallel                   = 'AUTO'

    StandbyFileManagement           = 'AUTO'

    ArchiveLagTarget                = '0'

    LogArchiveMaxProcesses          = '10'

    LogArchiveMinSucceedDest        = '1'

    DbFileNameConvert               = 'XXXPRD1B, XXXPRD1A'

    LogFileNameConvert              = 'XXXPRD1B, XXXPRD1A'

    FastStartFailoverTarget         = ''

    StatusReport                    = '(monitor)'

    InconsistentProperties          = '(monitor)'

    InconsistentLogXptProps         = '(monitor)'

    SendQEntries                    = '(monitor)'

    LogXptStatus                    = '(monitor)'

    RecvQEntries                    = '(monitor)'

    HostName(*)

    SidName(*)

    StaticConnectIdentifier(*)

    StandbyArchiveLocation(*)

    AlternateLocation(*)

    LogArchiveTrace(*)

    LogArchiveFormat(*)

    LatestLog(*)

    TopWaitEvents(*)

    (*) - Please check specific instance for the property value

 

Current status for "XXXPRD1A":

SUCCESS

 

Standby:

show database verbose “XXXPRD1B”;

 

Database

  Name:            XXXPRD1B

  Role:            PHYSICAL STANDBY

  Enabled:         YES

  Intended State:  APPLY-ON

  Instance(s):

    XXXPRD1B

 

  Properties:

    DGConnectIdentifier             = 'XXXPRD1B'

    ObserverConnectIdentifier       = ''

    LogXptMode                      = 'SYNC'

    DelayMins                       = '0'

    Binding                         = 'OPTIONAL'

    MaxFailure                      = '0'

    MaxConnections                  = '1'

    ReopenSecs                      = '60'

    NetTimeout                      = '30'

    RedoCompression                 = 'DISABLE'

    LogShipping                     = 'ON'

    PreferredApplyInstance          = ''

    ApplyInstanceTimeout            = '0'

    ApplyParallel                   = 'AUTO'

    StandbyFileManagement           = 'AUTO'

    ArchiveLagTarget                = '0'

    LogArchiveMaxProcesses          = '10'

    LogArchiveMinSucceedDest        = '1'

    DbFileNameConvert               = 'XXXPRD1A, XXXPRD1B'

    LogFileNameConvert              = 'XXXPRD1A, XXXPRD1B'

    FastStartFailoverTarget         = ''

    StatusReport                    = '(monitor)'

    InconsistentProperties          = '(monitor)'

    InconsistentLogXptProps         = '(monitor)'

    SendQEntries                    = '(monitor)'

    LogXptStatus                    = '(monitor)'

    RecvQEntries                    = '(monitor)'

    HostName                        = 'xxxem01'

    SidName                         = 'XXXPRD1B'

    StaticConnectIdentifier         = '(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=xxxem01)(PORT=1526))(CONNECT_DATA=(SERVICE_NAME=XXXPRD1B_DGMGRL)(INSTANCE_NAME=XXXPRD1B)(SERVER=DEDICATED)))'

    StandbyArchiveLocation          = 'USE_DB_RECOVERY_FILE_DEST'

    AlternateLocation               = ''

    LogArchiveTrace                 = '0'

    LogArchiveFormat                = '%t_%s_%r.dbf'

    LatestLog                       = '(monitor)'

    TopWaitEvents                   = '(monitor)'

 

Current status for "XXXPRD1B":

SUCCESS

 Ensure logs are still being transported and applied

All still works!

 

What does it give us?

• Improved monitoring in OEM.
• Simpler administration via DGMGRL or OEM i.e. changing Protection Mode.
• Easy switchover/failover
  • DGMGRL: switchover to “XXXPRD1A”
  • Click of a button in OEM.
• Fast-Start Failover (with an observer).

 

Conclusions

The broker configuration was relatively painless and I have successfully performed a switchover via both DGMGRL and OEM.  I’ve also used the various OEM functions without issue too.  I’m uneasy about performing such tasks via OEM but then most DBA’s prefer to be at the helm via the command line rather than clicking a button (that’s too easy!). 

If you plan to use DataGuard Broker then embrace it and make sure you don’t alter parameters behind the covers.  It does that for you and it doesn’t like you meddling!  I also think it’s a good idea to setup DataGuard initially without the broker and then configure it once you’re happy that DataGuard is working.

So what’s made the difference since our experience last year?  UNIX v Windows (maybe), 10g v 11g (probably).  A big factor would also have to be our experience level.  In the last year we’ve increased our DataGuard count from zero to ten so we are therefore much more comfortable in the DataGuard world.

When a sand-pit database becomes available I’d like to try and setup the observer and test Fast-Start Failover.  I’m inclined to remove the ‘broken’ tag but I’ll reserve judgment until I’ve tested the remaining functions.

Simon Ryan

Much faster performance with ASM – a real world example

We have a  Peoplesoft database that  manages HR for ~80,000 staff.  Oracle 10.2.03.  Peoplesoft Tools 8.49 PS HR 9 HPUX 11.31 Itanium

There has been long running problems caused by I/O bottlenecks writing down to Hitachi XP SAN disks. We were using filesystem storage and disk_asynch_io was set to false

The root cause of our problem was that the Time and Labour batches contend when they need to acquire exclusive access to write dirty blocks to data files during truncate operations. 

David Kurtz has written about a specific Peoplesoft feature around truncate http://www.go-faster.co.uk/docs.htm#local_write_wait

 We migrated to using ASM disk and achieved significant benefits.

 The testing stages were

1)       Prepare a 2 hour Time and labour batch run which would generate significant throughput

2)       Capture a baseline at 10.2.0.3

3)       Upgrade to 10.2.0.4 and retest

4)       Upgrade to 10.2.0.4 and migrate to ASM (11.1.0.7) and retest

5)       Upgrade to 11.1.0.7 and migrate to ASM (11.1.0.7) and retest

 

Test 3 did not show any difference at all. We were still blocked on disk, which was expected. We probably would have remained at 10.2.0.3 but gone to ASM but there are problems with seeing ASM disk when using 10.2.0.3 (Metalink Note 434500, ORA-15059) so we had the option of applying a fix to get ASM working with 10.2.0.3 or upgrading to 10.2.0.4. We decided that the benefits of getting onto a terminal release and suggestions from Oracle that small changes in the truncate process had been made between 10.2.0.3 and 10.2.0.4 out-weighed the overhead of any application testing that we needed to do.

Test 4 gave us big performance benefits. Test 5 was not as clear cut but we knew some code had to be optimised and we did not have the luxury of time to address the issues. Remaining on 10g minimised the application testing, certainly compared to the testing require to upgrade to 11g.

The benefits can be seen quite clearly in the graph below

This shows a 2 hour period where a constant throughput of T&L jobs were being processed. In parallel a constant online load was being simulated

The black line shows throughput and elapsed time. The run consists of a number of jobs which take 2 hours to complete and batch time is around the 450 - 600 second mark

The same processing is then run with 10.2.0.4 database and ASM. disk_aysnch_io is set to FALSE. The jobs complete 40 minutes faster and elapsed time is around the 300-400 second mark but showing spikes and inconsistent performance

Finally the orange line shows 10.2.0.4 database and ASM. disk_aysnch_io =TRUE. The processing time has now been reduced to just over an hour and the elapsed time is significantly better at an average of 250 seconds and with a very consistent profile.

As with all things in life we have now moved onto the next bottleneck which is the wait flashback buf free caused because we have enabled flashback logging and as the redo log buffer fills up we wait for flashback to be written out. However this is a miniscule issue compared to the original problems.  We have taken flashback off some tablespaces primarily used for PS temporary tables (aka working storage tables) although that does give us an issue when when rebuilding the primary database after a DataGuard failover using flashback logs

I hope this real world example is of use

Managing control files in ASM

Just a couple of notes on how to create multiple ASM control files and moving control files toa different diskgroup.

Assuming we already have a ASM control file in the DATA diskgroup and we wish to add one to the FRA diskgroup

 SQL>show parameter control_files
+DATA/SID/controlfile/current.730.698514673
alter system set control_files='+DATA','+FRA' scope=spfile;
shutdown immediate;
startup nomount;
exit
rman target
/restore controlfile from '+DATAHR/MHRPRD1A/CONTROLFILE/current.730.698514673';
alter database mount;
alter database open;
SQL>show parameter control_files
+DATA/SID/controlfile/current.730.698514673, +FRAHR/SID/controlfile/current.766.698514673

 

                                                                           

Here is an example of how to migrate from one ASM diskgroup to another

Add the new disk discovery path in (if needed), create a diskgroup and add disks

alter system set asm_diskstring='/dev/oracle','/dev/oracle_hr/r*';

create diskgroup DATANEW  external redundancy disk '/dev/oracle_hr/rdisk1','/dev/oracle_hr/rdisk2';

SQL>show parameter control_files

control_files                        string      +DATA10/SID/controlfile/c urrent.796.695384211
alter system set CONTROL_FILES = '+DATANEW' scope=spfile

shutdown
rman target /
startup nomount
restore controlfile  from  '/+DATA/SID/controlfile/current.796.695384211';
alter database mount;
alter database open;

Regressing an Opatch version

According to Metalink the only supported  way to regress to the version of Opatch that was installed is to re-install. However Opatch is a self-contained directory and the contents can be copied from another base installation.

Why would I want to regress the Opatch version. Let me explain with the commands I used.

Database version 10.2.0.4 (which gives opatch version 10.2.0.4.3)  

I wanted to install the PSU patch which requires a leter version of opatch (10.2.0.4.7). What I should have done was copy the opatch folder and save ready for a restore. However I didn’t and when the PSU patch failed due to issues with previously applied patches I had trouble applying the April 2009 CPU patch.

I followed these steps to resolve the problem

Secure existing folder and remove contents

cd $ORACLE_HOME

tar cvf OPatch.20090917_local.tar ./OPatch

rm -R OPatch

Logon onto the remote server and tar the OPatch folder up and ftp to the local server. Note use a different name for the tar file so as not to overwrite the tar files that you have just created.

tar cvf OPatch.20090917_remote.tar ./OPatch

Back on the local server unzip the tar file and check the version of opatch

tar xvf OPatch.20090917_remote.tar 

a./OPatch/docs/FAQ 31 blocks
a ./OPatch/docs/Users_Guide.txt 38 blocks
a ./OPatch/docs/Prereq_Users_Guide.txt 25 blocks
a ./OPatch/jlib/opatch.jar 1415 blocks
a ./OPatch/jlib/opatchutil.jar 270 blocks
a ./OPatch/jlib/opatchprereq.jar 242 blocks
a ./OPatch/jlib/opatchactions.jar 142 blocks
a ./OPatch/opatch 14 blocks
a ./OPatch/opatch.pl 6 blocks
a ./OPatch/opatch.ini 1 blocks
a ./OPatch/emdpatch.pl 38 blocks
a ./OPatch/opatchprereqs/oui/knowledgesrc.xml 2 blocks
a ./OPatch/opatchprereqs/opatch/opatch_prereq.xml 60 blocks
a ./OPatch/opatchprereqs/opatch/rulemap.xml 5 blocks
a ./OPatch/opatchprereqs/opatch/runtime_prereq.xml 1 blocks
a ./OPatch/opatchprereqs/prerequisite.properties 1 blocks

opatch version
Invoking OPatch 10.2.0.4.3

OPatch Version: 10.2.0.4.3

Moving from ASM storage back to filesystem

Whilst it is not very common to move from ASM to filesystem I needed to prepare a regression plan for the migration of a 450 datafile database in the event of problems during the migration or shortly afterwards.

The example below is from a small database called TEST and I used filestore in /app/oracle whereas normally I would have a /oradata/SID/ filesystem

The details below should be enough to get someone through the migration and the sql scripts can be modified to be more generic if necessary but it is not too hard to use them as they are now. 

List the datafiles and we will add a second datafile to the users tablespace to provide an example of an issue that can occur.

SYS@TEST SQL>select file_name from dba_data_files;  +DATA/test/datafile/undotbs1.301.697649965
+DATA/test/datafile/sysaux.300.697649963
+DATA/test/datafile/system.299.697649963
+DATA/test/datafile/users.302.697649965
+DATA/test/datafile/users.309.697650601

 mkdir /app/oracle/TEST

Some tablespaces may have multiple datafiles which would end up with the same name using the code above. However as a quick circumvention I manually identified those tablespaces using this sql and then edited the new data file names

select tablespace_name,count(file_name) from dba_data_files group by tablespace_name ORDER BY 2 

In my example it only applied to the USERS tablespace and I edited the second ASM datafile to be mapped to users02.dbf not users01.dbf as the script will output. This action would have to be repeated for all tablespaces with multiple datafiles

sqlplus / as sysdba

set lines 120 pages 0

select 'copy datafile '||file_id||' to '||'''/app/oracle/TEST/'||substr(file_name,21,instr(file_name,'.')-21)||'_01.dbf'''||';' from dba_data_files order by 1;

copy datafile 1 to '/app/oracle/TEST/system_01.dbf';

copy datafile 2 to '/app/oracle/TEST/sysaux_01.dbf';

copy datafile 3 to '/app/oracle/TEST/undotbs1_01.dbf';

copy datafile 4 to '/app/oracle/TEST/users_01.dbf';

copy datafile 5 to '/app/oracle/TEST/users_02.dbf';

shutdown immediate;

startup mount;

exit

rman target /

copy datafile 1 to '/app/oracle/TEST/system_01.dbf';

copy datafile 2 to '/app/oracle/TEST/sysaux_01.dbf';

copy datafile 3 to '/app/oracle/TEST/undotbs1_01.dbf';

copy datafile 4 to '/app/oracle/TEST/users_01.dbf';

copy datafile 5 to '/app/oracle/TEST/users_02.dbf';

exit

sysdba

select 'alter database rename file '''||file_name||''' to '||'''/app/oracle/TEST/'||substr(file_name,21,instr(file_name,'.')-21)||'_01.dbf'''||';' from dba_data_files order by 1;

alter database rename file '+DATA/test/datafile/sysaux.300.697649963' to '/app/oracle/TEST/sysaux_01.dbf';
alter database rename file '+DATA/test/datafile/system.299.697649963' to '/app/oracle/TEST/system_01.dbf';
alter database rename file '+DATA/test/datafile/undotbs1.301.697649965' to '/app/oracle/TEST/undotbs1_01.dbf';
alter database rename file '+DATA/test/datafile/users.302.697649965' to '/app/oracle/TEST/users_01.dbf';
alter database rename file '+DATA/test/datafile/users.309.697650601' to '/app/oracle/TEST/users_02.dbf';

open the database and check for files still using ASM

open database

select name from v$controlfile
union
select name from v$tempfile
union
select member from v$logfile
union
select filename from v$block_change_tracking;

 

NAME
--------------------------------------------------------------------------------
+DATA/test/controlfile/current.303.697650047
+DATA/test/onlinelog/group_1.304.697650049
+DATA/test/onlinelog/group_2.305.697650049
+DATA/test/onlinelog/group_3.306.697650049
+DATA/test/tempfile/temp.307.697650095
+FRA/test/controlfile/current.260.697650047
+FRA/test/onlinelog/group_1.261.697650049
+FRA/test/onlinelog/group_2.262.697650049
+FRA/test/onlinelog/group_3.263.697650051

alter database rename file '+DATA/test/datafile/sysaux.300.697649963' to '/app/oracle/TEST/sysaux_01.dbf';

alter database rename file '+DATA/test/datafile/system.299.697649963' to '/app/oracle/TEST/system_01.dbf';

alter database rename file '+DATA/test/datafile/undotbs1.301.697649965' to '/app/oracle/TEST/undotbs1_01.dbf';

alter database rename file '+DATA/test/datafile/users.302.697649965' to '/app/oracle/TEST/users_01.dbf';

alter database rename file '+DATA/test/datafile/users.309.697650601' to '/app/oracle/TEST/users_02.dbf';

ALTER DATABASE ADD LOGFILE GROUP 4 ('/app/oracle/TEST/redo_04a.dbf','/app/oracle/TEST/redo_04b.dbf') size 50M;

ALTER DATABASE ADD LOGFILE GROUP 5 ('/app/oracle/TEST/redo_05a.dbf','/app/oracle/TEST/redo_05b.dbf') size 50M;

ALTER DATABASE ADD LOGFILE GROUP 6 ('/app/oracle/TEST/redo_06a.dbf','/app/oracle/TEST/redo_06b.dbf') size 50M; 

You will need to perform logfile switches or checkpoint before the log groups using ASM are available for removal. 

SQL>

show parameter control

 control_files +DATA/test/controlfile/current .303.697650047, +FRA/test/cont rolfile/current.260.697650047

create a pfile from spfile and edit the pfile to show new placement for the controlfiles

*.control_files='/app/oracle/TEST/control01.ctl','/app/oracle/TEST/control02.ctl'

startup nomount

$rman target / 

Recovery Manager: Release 11.1.0.7.0 - Production on Wed Sep 16 16:36:15 2009 connected to target database:

TEST (not mounted) using target database control file instead of recovery catalog

 restore controlfile from '+DATA/test/controlfile/current.303.697650047';

Starting restore at 16-SEP-09 allocated channel: ORA_DISK_1 channel ORA_DISK_1: SID=70 device type=DISK channel ORA_DISK_1: copied control file copy output file name=/app/oracle/TEST/control01.ctl output file name=/app/oracle/TEST/control02.ctl

Finished restore at 16-SEP-09

 exit

Recovery Manager complete.

 

create spfile='/app/oracle/product/11.1.0/db_1/dbs/spfileTEST.ora' from pfile='/home/oracle/john.ora';

File created.
shutdown immediate; 
startup

show parameter control

control_files  /app/oracle/TEST/control01.ctl , /app/oracle/TEST/control02.ctl 

All that is left is to recreate the temp tablespace to use filesystem storage.

startup mount;

RMAN> run { set newname for tempfile 1 to '/app/oracle/TEST/temp01.dbf'; switch tempfile all;}

executing command:

SET NEWNAME renamed tempfile 1 to /app/oracle/TEST/temp01.dbf in control file  rman target /

 TEST (not mounted) using target database control file instead of recovery catalog RMAN> restore controlfile from '+DATA/test/controlfile/current.303.697650047';

Starting restore at 16-SEP-09 allocated channel: ORA_DISK_1 channel ORA_DISK_1: SID=69 device type=DISK channel ORA_DISK_1:

copied control file copy output file

name=/app/oracle/TEST/control01.ctl output file name=/app/oracle/TEST/control02.ctl

Finished restore at 16-SEP-09

All should be complete now. As I said, this is just a hint on what actions to take and the sql scripts should be useful if you have lots of datafiles.

All that is left is to create a new  block_change_tracking file and the ASM storage can be released.

Performing calculations in unix

Normally if I working on a database and want perform simple calculations I use the Oracle database rather than starting up the windows calculator.

select 6*1024*1024 from dual;

6*1024*1024
-----------
    6291456
 

However I recently found out a method of using unix itself (I am sure there are many more options other than this one but I find this quite easy)

man bc gives

bc is an interactive processor for a language that resembles C but provides unlimited-precision arithmetic.  It takes input from any  files given, then reads the standard input.

In usage

/home/oracle $bc
6*1024*1024
6291456
quit

Arithmetic operators are shown in the normal manner + – * / % ^ and there are lots of functions available such as sine, cosine and log, although I would be likely to use a calculator if I wanted to use these.

Procedure to kill a session

On the Oracle-L mailing list a user requested a procedure to allow a developer to kill his own process. This contributed quite a few comments which I will list below and I posted a method that I use when I have the same requirement.

To create the stored procedure logon as a sysdba through sqlplus and explicitly grant the required rights to system and then create the procedure.

grant alter system to system;
grant select on sys.v_$session to system;

create or replace procedure system.killsession (killsid IN VARCHAR2)
as
killserial   varchar2(20);

begin
killserial:='none';
select serial#
into killserial
from sys.v_$session
where sid = killsid
and username = 'WES';

execute immediate 'alter system kill session ''' || to_char (killsid) || ', ' || to_char (killserial) || '''' ;

EXCEPTION
WHEN NO_DATA_FOUND THEN
dbms_output.put_line('The SID ' || killsid || ' does not exist or cannot be killed');

end ;

Grant execute on the procedure to the user

For example, userxxx needs access to kill sessions so you would type the following:-

grant execute on system.killsession to userxxx;

The user needs to know the sid of the session being killed. For example, if a user needs to kill a session with a SID of 140 they would type:-

set serveroutput on
exec system.killsession ('140');

If they get the following error then there is no SID with a username of WES currently connected:-

The SID 140 does not exist or cannot be killed

  The various thoughts on allowing such a procedure centred on two strands, security and practical considerations

 

Security issues

• Why do they need to kill jobs, what is causing the issue
  What will developers want next
  Sarbanes-Oxley considerations

Practical

• How do you know the developer will kill one of his own processes and not a co-worker (think apps or sysadm job)
  Can you use resource profiles to limit CPU usage
  Use a standby to allow unrestricted resource usage

My view is that in certain development environments, for certain users it is reasonable to allow them to kill a process that is running rogue and as such I am happy to provide such a facility. This would not be allowed either unrestricted or in any sort of controlled environment such as production and pre-production.

ASM space marked as internal – use check all repair

The problem was that the FRA diskgroup seemed to be using a lot of space and and yet there was hardly  files on  disk as far as I could tell. HP Itanium – 11.1.0.7

ASM Disk Groups
===============

     Group Group Name                State           Type      Total GB Free GB
---------- ------------------------- --------------- ------- ---------- -------
         1 DATA                      MOUNTED         EXTERN        2581     109
         2 FRA                       MOUNTED         EXTERN        1602     127

 
Group Disk Header    Mode     State           Redundancy   Total MB    Free MB Disk Name                      Failure Gr
----- ---- --------- -------- --------------- ---------- ---------- ---------- ------------------------------ ----------
    2    1 MEMBER    ONLINE   NORMAL          UNKNOWN         91138          0 FRA_0001                       FRA_0001
    2    2 MEMBER    ONLINE   NORMAL          UNKNOWN         91138          0 FRA_0002                       FRA_0002
    2    4 MEMBER    ONLINE   NORMAL          UNKNOWN         91138          0 FRA_0004                       FRA_0004
    2    5 MEMBER    ONLINE   NORMAL          UNKNOWN         91138          0 FRA_0005                       FRA_0005
    2    6 MEMBER    ONLINE   NORMAL          UNKNOWN         91138          0 FRA_0006                       FRA_0006
    2    7 MEMBER    ONLINE   NORMAL          UNKNOWN         91138          0 FRA_0007                       FRA_0007
    2    8 MEMBER    ONLINE   NORMAL          UNKNOWN         91138          0 FRA_0008                       FRA_0008
    2    9 MEMBER    ONLINE   NORMAL          UNKNOWN         91138          0 FRA_0009                       FRA_0009
    2   10 MEMBER    ONLINE   NORMAL          UNKNOWN         91138          0 FRA_0010                       FRA_0010
    2   11 MEMBER    ONLINE   NORMAL          UNKNOWN         91138          0 FRA_0011                       FRA_0011
    2   12 MEMBER    ONLINE   NORMAL          UNKNOWN         91138          0 FRA_0012                       FRA_0012
    2   13 MEMBER    ONLINE   NORMAL          UNKNOWN         91138          0 FRA_0013                       FRA_0013
    2   14 MEMBER    ONLINE   NORMAL          UNKNOWN         91138          0 FRA_0014                       FRA_0014
    2   15 MEMBER    ONLINE   NORMAL          UNKNOWN         91138          0 FRA_0015                       FRA_0015
    2   16 MEMBER    ONLINE   NORMAL          UNKNOWN         91138          0 FRA_0016                       FRA_0016
    2   17 MEMBER    ONLINE   NORMAL          UNKNOWN         91138          0 FRA_0017                       FRA_0017
    2   18 MEMBER    ONLINE   NORMAL          UNKNOWN         91138      65242 FRA_0018                       FRA_0018
    2   19 MEMBER    ONLINE   NORMAL          UNKNOWN         91138      65112 FRA_0019                       FRA_0019

 I had removed one disk from FRA and forced a rebalance but that still did not release space. I had checked the FRA disks using asmcmd but that only showed 2 online logs which were 2Gb each (we were not in archive log mode).

I had checked the diskgroup using the EM Gui which issued the command  

ALTER DISKGROUP FRA CHECK ALL:

but still no success. Looking at the documentation the check command taking NOREPAIR as the default should have reported any issues into the ASM alert log.  Which indeed it did (and I would have seen them if I had bothered to look)

Wed Jul 29 16:37:55 2009
SQL> ALTER DISKGROUP FRA CHECK DISK FRA_0001,FRA_0002,FRA_0004,FRA_0005,FRA_0006,FRA_0007,FRA_0008,FRA_0009,FRA_0010,FRA_0011,FRA_0012,FRA_0013,FRA_0014,FRA_0015,FRA_0016,FRA_0017,FRA_0018,FRA_0019
WARNING: Deprecated privilege SYSDBA for command 'ALTER DISKGROUP CHECK'
kfdp_checkDsk(): 33
NOTE: disk FRA_0001, used AU total mismatch: DD=4294965892, AT=2283
kfdp_checkDsk(): 34
NOTE: disk FRA_0002, used AU total mismatch: DD=4294966012, AT=2283
kfdp_checkDsk(): 35
NOTE: disk FRA_0004, used AU total mismatch: DD=4294966210, AT=2283
kfdp_checkDsk(): 36
NOTE: disk FRA_0005, used AU total mismatch: DD=4294966625, AT=2283
kfdp_checkDsk(): 37
NOTE: disk FRA_0006, used AU total mismatch: DD=4294965615, AT=2280
kfdp_checkDsk(): 38
NOTE: disk FRA_0007, used AU total mismatch: DD=4294964292, AT=2283
kfdp_checkDsk(): 39
NOTE: disk FRA_0008, used AU total mismatch: DD=4294967237, AT=2284
kfdp_checkDsk(): 40
NOTE: disk FRA_0009, used AU total mismatch: DD=4294965595, AT=2280
kfdp_checkDsk(): 41
NOTE: disk FRA_0010, used AU total mismatch: DD=4294966758, AT=2279
kfdp_checkDsk(): 42
NOTE: disk FRA_0011, used AU total mismatch: DD=4294966324, AT=2279
kfdp_checkDsk(): 43
NOTE: disk FRA_0012, used AU total mismatch: DD=4294966743, AT=2279
kfdp_checkDsk(): 44
NOTE: disk FRA_0013, used AU total mismatch: DD=4294964514, AT=2278
kfdp_checkDsk(): 45
NOTE: disk FRA_0014, used AU total mismatch: DD=4294965734, AT=2278
kfdp_checkDsk(): 46
NOTE: disk FRA_0015, used AU total mismatch: DD=4294965204, AT=2279
kfdp_checkDsk(): 47
NOTE: disk FRA_0016, used AU total mismatch: DD=4294965599, AT=2279
kfdp_checkDsk(): 48
NOTE: disk FRA_0017, used AU total mismatch: DD=4294964950, AT=2280
kfdp_checkDsk(): 49
NOTE: disk FRA_0018, used AU total mismatch: DD=25896, AT=2278
kfdp_checkDsk(): 50
NOTE: disk FRA_0019, used AU total mismatch: DD=26026, AT=2280
WARNING: deprecated use of ALTER DISKGROUP CHECK arguments
SUCCESS: ALTER DISKGROUP FRA CHECK DISK FRA_0001,FRA_0002,FRA_0004,FRA_0005,FRA_0006,FRA_0007,FRA_0008,FRA_0009,FRA_0010,FRA_0011,FRA_0012,FRA_0013,FRA_0014,FRA_0015,FRA_0016,FRA_0017,FRA_0018,FRA_0019

From the command line I then ran the comand

ALTER DISKGROUP FRA CHECK ALL REPAIR;

and I had immediate success. It took about 1 minute and corrected the issues and then released the space that had been grabbed internally.

     Group Group Name                State           Type      Total GB Free GB
---------- ------------------------- --------------- ------- ---------- -------
         1 DATA                      MOUNTED         EXTERN        2581     109
         2 FRA                       MOUNTED         EXTERN  1602   1,562

Group Disk Header    Mode     State           Redundancy   Total MB    Free MB Disk Name                      Failure Gr
----- ---- --------- -------- --------------- ---------- ---------- ---------- ------------------------------ ----------
    2    1 MEMBER    ONLINE   NORMAL          UNKNOWN         91138      88855 FRA_0001                       FRA_0001
    2    2 MEMBER    ONLINE   NORMAL          UNKNOWN         91138      88855 FRA_0002                       FRA_0002
    2    4 MEMBER    ONLINE   NORMAL          UNKNOWN         91138      88855 FRA_0004                       FRA_0004
    2    5 MEMBER    ONLINE   NORMAL          UNKNOWN         91138      88855 FRA_0005                       FRA_0005
    2    6 MEMBER    ONLINE   NORMAL          UNKNOWN         91138      88858 FRA_0006                       FRA_0006
    2    7 MEMBER    ONLINE   NORMAL          UNKNOWN         91138      88855 FRA_0007                       FRA_0007
    2    8 MEMBER    ONLINE   NORMAL          UNKNOWN         91138      88854 FRA_0008                       FRA_0008
    2    9 MEMBER    ONLINE   NORMAL          UNKNOWN         91138      88858 FRA_0009                       FRA_0009
    2   10 MEMBER    ONLINE   NORMAL          UNKNOWN         91138      88859 FRA_0010                       FRA_0010
    2   11 MEMBER    ONLINE   NORMAL          UNKNOWN         91138      88859 FRA_0011                       FRA_0011
    2   12 MEMBER    ONLINE   NORMAL          UNKNOWN         91138      88859 FRA_0012                       FRA_0012
    2   13 MEMBER    ONLINE   NORMAL          UNKNOWN         91138      88860 FRA_0013                       FRA_0013
    2   14 MEMBER    ONLINE   NORMAL          UNKNOWN         91138      88860 FRA_0014                       FRA_0014
    2   15 MEMBER    ONLINE   NORMAL          UNKNOWN         91138      88859 FRA_0015                       FRA_0015
    2   16 MEMBER    ONLINE   NORMAL          UNKNOWN         91138      88859 FRA_0016                       FRA_0016
    2   17 MEMBER    ONLINE   NORMAL          UNKNOWN         91138      88858 FRA_0017                       FRA_0017
    2   18 MEMBER    ONLINE   NORMAL          UNKNOWN         91138      88860 FRA_0018                       FRA_0018
    2   19 MEMBER    ONLINE   NORMAL          UNKNOWN         91138      88858 FRA_0019                       FRA_0019

 

SQL> alter diskgroup fra check all repair
NOTE: starting check of diskgroup FRA
kfdp_checkDsk(): 51
WARNING: disk FRA_0001, changing DD used AUs from 4294965892 to 2283
kfdp_checkDsk(): 52
WARNING: disk FRA_0002, changing DD used AUs from 4294966012 to 2283
kfdp_checkDsk(): 53
WARNING: disk FRA_0004, changing DD used AUs from 4294966210 to 2283
kfdp_checkDsk(): 54
WARNING: disk FRA_0005, changing DD used AUs from 4294966625 to 2283
kfdp_checkDsk(): 55
WARNING: disk FRA_0006, changing DD used AUs from 4294965615 to 2280
kfdp_checkDsk(): 56
WARNING: disk FRA_0007, changing DD used AUs from 4294964292 to 2283
kfdp_checkDsk(): 57
WARNING: disk FRA_0008, changing DD used AUs from 4294967237 to 2284
kfdp_checkDsk(): 58
WARNING: disk FRA_0009, changing DD used AUs from 4294965595 to 2280
kfdp_checkDsk(): 59
WARNING: disk FRA_0010, changing DD used AUs from 4294966758 to 2279
kfdp_checkDsk(): 60
WARNING: disk FRA_0011, changing DD used AUs from 4294966324 to 2279
kfdp_checkDsk(): 61
WARNING: disk FRA_0012, changing DD used AUs from 4294966743 to 2279
kfdp_checkDsk(): 62
WARNING: disk FRA_0013, changing DD used AUs from 4294964514 to 2278
kfdp_checkDsk(): 63
WARNING: disk FRA_0014, changing DD used AUs from 4294965734 to 2278
kfdp_checkDsk(): 64
WARNING: disk FRA_0015, changing DD used AUs from 4294965204 to 2279
kfdp_checkDsk(): 65
WARNING: disk FRA_0016, changing DD used AUs from 4294965599 to 2279
kfdp_checkDsk(): 66
WARNING: disk FRA_0017, changing DD used AUs from 4294964950 to 2280
kfdp_checkDsk(): 67
WARNING: disk FRA_0018, changing DD used AUs from 25896 to 2278
kfdp_checkDsk(): 68
WARNING: disk FRA_0019, changing DD used AUs from 26026 to 2280
SUCCESS: check of diskgroup FRA found no errors
WARNING: deprecated use of ALTER DISKGROUP CHECK arguments
SUCCESS: alter diskgroup fra check all repair
Thu Jul 30 10:31:33 2009

 

I would have expected the gui to have reported errors but it didn’t. Equally I can see no good reason  for not fixing errors it identified when a check is run. That was the default in 10G  so  there must be a good reason why it was changed. Perhaps it had a performance impact which I would not have noticed on a pretty empty diskgroup. The other reason is that it might make the assumption that the DBA would want a good backup before repairing.

As we had nothing in there and we backup ASM metadata every day then I was happy to go ahead. http://jhdba.wordpress.com/2009/06/11/script-to-backup-asm-metadata/

 So the lessons learned are

1. Use the check all feature
2. Check the alert log afterwards
3. Then use the check all repair statement from command line