The talks were all very good and I think the event was enjoyed by all. Neil Johnson wrote a blog entry about his experience of the day and of presenting for the first time. He has now got the bit between his teeth and wants to present a 2 hour masterclass at the December conference in Manchester – (perhaps I am exaggerating slightly)

I ran the ‘BitsandPieces’ session where audience participation was the key idea and this was achieved. The theme was small pieces of Oracle functionality that are not well known or used.

We discussed :-

The dbms_workload_repository.add_colored_sql procedure which allows you to capture specific pices of sql that individually would not appear as a top resource in AWR data. Using the package to add a sql_id and the data does then appear and performance can be fully explored. A good introductory  blog entry is available from Dion Cho at http://dioncho.wordpress.com/tag/colored-sql/

• The benefits of using restore points and  flashback database  were covered. Examples were given of flashing back through a 10g to 11g upgrade (if you might want to), using a restore point before making a major table or code change that might need to be rolled back (better than rolling forward from a previous backup) and using the same restore point repeatedly as part of performance testing. The point was made that anything that might include an OS command (such as dropping or resizing a datafile probably would not be recovered through so be careful with the command. My advice is that I consider it as the belt that goes with the braces of a full backup.
• Invisible indexes – how easy they were to use and how they allowed the data_dictionary to be aware of the index but the optimizer ignores it. There was also a initialization parameter OPTIMIZER_USE_INVISIBLE_INDEXES  that allows the index to become visible at both a session and system level to allow full testing. Beware that if you rebuild an index it then becomes visible again.
• One tip that came out was that in pre-packaged environments such as EBS when an index is not wanted, rather than drop it, mark it as invisible. Then when any upgrades or patches are applied the  index still exists in the dictionary and is not recreated as it might be if it had been dropped.
• Another idea was the use of the opatch auto command. This came out in 11GR2 but not until PSU2. It manages a full GI and RDBMS patchset application and can stop all dependant databases, stop crs resources, stop crs, patch and restart everything. It can save a serious amount of time and is well worth investigating.

I did pick up another few interesting thoughts which I want to do a bit of research on and I will blog about those shortly.

So overall it  as a good day with a lot of community interaction, networking and a good laugh afterwards when about 15 of us went for a few beers.

The location is the Met Hotel in Leeds, right by the station and the last meeting in Leeds several years ago, was well attended so I am sure this one will be as well.

The agenda is below

UKOUG Database Server SIG Meeting

Date: Thursday 9th May 2013

Time: 09:00 – 17:00

Location: Leeds/ West Yorkshire

Venue: The Met, Leeds

You confidently follow-up with ‘of course I would have copied the passwd and the init.ora file to the new ORACLE_HOME/dbs folder during the outage.’ The interviewer says “go on, what else will you do?”.

It is getting a bit more awkward now, but you rise to the occasion with a comment about editing the listener to show the new ORACLE_HOME and then reloading the listener – but he still wants more and you begin to dry up a bit.

After a bit of hinting you recognise that you might need to go into OEM and edit the configuration but you have given your all, you’re are a spent force, wishing you could move onto the next easy question.

The above is from an interview question I asked only last week and unless a candidate has recently been migrating databases around then I think that would be representative of many people’s answers.  Therefore the idea of this post is just to put together a few things to form a checklist of things that may be required when migrating to a new ORACLE_HOME. I am sure all are not relevant and the OWB comment might well be replaced with some other tool you might be using.

So just a bullet point list follows and if I have failed my own interview question then I will be really pleased to accept suggestions and add them to the list

• Edit  the oratab entry to update the database to reflect the new ORACLE_HOME
• Copy the $ORACLE_HOME/dbs/orapwSID.ora and initSID.ora file to the new $ORACLE_HOME/dbs. You may hold these somewhere different  in which case the same principle applies. If  a RAC system ensure you do this on all nodes.
• Edit the listener.ora file and restart or reload the listener
• If you use Oracle Warehouse Builder then follow the note further down
• Update  Oracle Enterprise Manager to reflect the new home
• Review any ONAMES or OID or even LDAP config to see if they are in use or need to be changed
• If your oraenv equivalent or login script does not set the path then check that you are now using the correct path. If it does then all should be OK
• If you use a single listener and tnsnames file  for many homes on a server then you might want to them link them to the new ORACLE_HOME

cd new $ ORACLE_HOME

ln -s /app/oracle/product/11.1.0/asm/network/admin/listener.ora listener.ora

ln -s /app/oracle/product/11.1.0/asm/network/admin/tnsnames.ora tnsnames.ora

• When changing an Oracle Home with either RAC or stand alone 11gr2 Oracle Restart, Oracle Clusterware needs to be modified for the new oracle home

 srvctl modify database –d ORCL –o /app/oracle/product/11.2.0.3/dbhome_AWR1_1

Alternatively the resource can be deleted and re-created with

 srvctl stop service -d ORCL2A -s ORCL2

srvctl remove service -d ORCL2A -s ORCL2

srvctl remove database –d ORCL2A

 srvctl add database -d ORCL2A -o /app/oracle/product/11.2.0.3/dbhome_AWR1_1 -r PRIMARY

or

srvctl add database -d ORCL2B -o /app/oracle/product/11.2.0.1/dbhome_AWR1_1 -r PHYSICAL_STANDBY -s mount

Extra step for RAC only

{

srvctl add instance -d ORCL2A -i ORCL2A1 -n node1

srvctl add instance -d ORCL2A -i ORCL2A2 -n node2

}

 crsctl status resource ora.ORCL2a.db -f | grep DEPENDENCIES

srvctl modify database -d ORCL2A -a “DATA,FRA”

crsctl status resource ora.ORCL2a.db -f | grep DEPENDENCIES

srvctl add service -d ORCL2A -r ORCL2A1,ORCL2A2 -s ORCL2 -l PRIMARY -e SELECT -m BASIC

srvctl modify service -d ORCL2A -s ORCL2 -z 180 -w 1

• Instructions to change OWB Home on the same server - 11GR1

Install the new Oracle Home

Stop the OWB service

Here I am assuming OWB_HOME is the same place as ORACLE_HOME

$ORACLE_HOME/owb/rtp/sql/stop_service

SQL> select key, value from OWBSYS.OWBRTPS order by key;
SQL> Update OWBSYS.OWBRTPS set VALUE='<owb_new_home>' where key='key_from_previous_SQL';
SQL> commit;

$ORACLE_HOME/owb/rtp/sql/start_service

 Connect via Control Centre and ensure everything works

I then came across bug 10153617  which is quite old (Sept 2010) but the 11.1.0.7  release is old now as well.

=========================================================================
Interim Patch for Base Bug: 10153617
=========================================================================

Date: Wed Sep 29 06:12:33 2010
————————————————————————-
Platform Patch for          : Generic
Product Patched             : ORACLE DATABASE
Product Version #           : 11.1.0.7.0
RAC Rolling Installable     : YES

Bugs Fixed by this patch:
————————-
6460304: EXPDP TAKES MORE TIME
7362589: GSIST12: EXPDP TABLE POOR PERFORMANCE OVERHEAD COMPARED TO ORIGINAL EXP
7710931: DATAPUMP EXPORT IS EXTREMELY SLOW WHEN EXTRACTING SCHEMA
7722575: DATAPUMP VIEW KU$_NTABLE_DATA_VIEW CAUSES POOR PLAN / SLOW EXPDP

I applied this as part of a general upgrade to PSU13 (it is not dependant upon any specific PSU) and the impact is very significant.

A simple test demonstrates the difference

create user test identified by xxxxxx default tablespace users;

grant create session, create table to test;

create table test.test as select * from sys.dba_objects

exdp schemas=test directory=data_pump_dir content=ALL dumpfile=test.dmp logfile=test.log

The first datapump export took 20 minutes before I killed the job and the second finished in less than 2 minutes and I have seen this repeated on a number of systems. I applied this on top of PSU 13 with no problems and I would recommend reviewing any 11.1.0.7 databases where  you might be exporting schemas or running a lot of datapump jobs. The test above or something similar shoudl provide an easy way of seeing if you are hitting the problem.

In the patch details above you will notice a bug 7710931 is also mentioned as being fixed by bug/patch 10153617. The detail of that suggests it is still valid in 11.2.0.1 but the same test above on a couple of systems did not indicate any poor performance

As you may know, the read-only mode will prevent any redo logs being applied to standby database, but on the surface everything looks OK – no errors and MRP0 process is running and showing “applying log” in  v$managed_standby.

The only problem is – the recovery is “stuck” on the last log the database was trying to apply before it was opened in read-only mode.

Unfortunately, the customer did not notice the omission for over 2 weeks and by the time I have had a chance to look at the environments, there were about 50G of redo logs accumulated for each and some of them were missing and/or corrupt which excluded the possibility of SCPing the logs over to the standby server and registering them with standby databases.

One of another factors which caused a lack of  attention to the standby databases falling behind is the absence of any error messages in the alert logs – every single log was shown as shipped and received.

In a case like this, the only option is to rebuild the standby database and in the past I did it using the traditional RMAN duplicate for standby routine.  However, in this particular case I had 2 databases to rebuild – one is small and another is large. The network between primary cluster and standby was slow as well.

For the small database I decided to use Grid Control GUI based wizard for creating the standby database and this process is quite straightforward and described in the documentation. For the large one, however, duplicating the database using RMAN would be too slow,there may have been a  performance degradation during the run and the maintenance window was too short for out-of-hours run.

This was a perfect case to try “incremental backup” approach. This method is described in a number of sources (if you Google it) but none of the “vanilla” cases worked for me.

I will not be listing the names and detailed output due to the production nature of the work – just the list of steps.

So, this is what I did at the end of the day:

PRE-REQUISITES:

Primary database can be single node or RAC and running OK.
No downtime of Primary is required.
All Dataguard settings are intact

Step-by-step:

1. 1.       Get the latest SCN from standby:

 select to_char(current_scn) from v$database;

10615562421

1. 2.       Create incremental backup on Primary for all the changes since SCN on standby:

[oracle@primary backup]$ rman target /</pre>
connected to target database: PRIMARY (DBID=720063942)

RMAN> run

2> {

3> allocate channel d1 type disk;

4> allocate channel d2 type disk;

5> backup incremental from scn 10615562421 database format

6> '/tmp/backup/primary_%U';

7> release channel d1;

8> release channel d2;
<pre>9> }

1. 3.       Create copy of control file on Primary:

 alter database create standby controlfile as ‘/tmp/backup/stby.ctl’;

1. 4.       SCP the backup files and standby control file to the standby server. A little tip: if you copy the backup files to the directory with the same name (like /tmp/backup here), your controlfile will know about them and you can bypass the registration bit later on.

1. 5.       The next step is to replace the standby control file with the new one. May sound simple, but this proved to be the trickiest part due to the fact that standby controlfile is OMF and in ASM.  You will need to use RMAN for the restore operation:

-          Switch the database to nomount, then:

restore controlfile from ‘/tmp/backup/stby.ctl';

-          Mount the database.

At this point you have the controlfile with the information about the files as they are on the Primary side, so, the next step is to register everything we have on Standby side:

catalog start with '+data/standby/';

Check the output and YES to register any reported standby files.

-          Shutdown immediate your standby instance.

 RMAN> switch database to copy;

RMAN> report schema;

On this stage you should have a nice and clean list of actual standby files.

Now we are ready to apply our differential backup to bring the standby in line with Primary:

RMAN> recover database noredo;

 Because the online redo logs are lost, you must specify the NOREDO option in the RECOVER command.

You must also specify NOREDO if the online logs are available but the redo cannot be applied to the incrementals.

If you do not specify NOREDO, then RMAN searches for redo logs after applying the incremental backup, and issues an error message when it does not find them.

When the recovery completes, you may start the managed recovery process again:

SQL> alter database recover managed standby database using current logfile disconnect;

Provided, all FAL settings are correct, your managed recovery will pick-up all logs generated on primary since the incremental backup and you will have fully synchronised standby again.

11.1.0.7

SQL>select  username, profile from dba_users where username like 'XS%';

USERNAME                       PROFILE
------------------------------ ------------------------------
XS$NULL                        DEFAULT

SQL>alter user xs$null profile m_std_profile identified by  Changepass123;

User altered.

connect xs$null/Changepass123
ERROR:
ORA-28000: the account is locked
SQL>alter user xs$null account unlock;

User altered.

Connected.
SYS@MOMPTC2A SQL>connect xs$null/Changepass123
ERROR:
ORA-01017: invalid username/password; logon denied

11.2.0.3

SQL> select  username, profile from dba_users where username like 'XS%';

USERNAME                       PROFILE
------------------------------ ------------------------------
XS$NULL                        DEFAULT

SQL> alter user xs$null profile m_std_profile identified by  Changepass123;
alter user xs$null profile m_std_profile identified by  Changepass123
*
ERROR at line 1:
ORA-01031: insufficient privileges

Onto point 2. Now I am sure that there is a good reason to change a default password as well as locking the account but I fail to see it. My thought process would be that if someone has got into your database with enough privileges to unlock an account then changing a password is not going to be a big obstacle to them. Happy to have some explanation of that requirement. As the title of my  blog states – a lifelong learning experience

As always we are looking for good presentations from UKOUG members so that we can have a really strong database focused agenda.

If anyone wishes to present for the first time this would be a good opportunity but experienced presenters are always welcome. I am responsible for pulling the agenda together this time and hope I am inundated with offers.

The UKOUG link is here if anybody wishes to get more details or offer their support.

As a side note for Yorkshire based Oracle DBAs, Eter Pani has been organising meet-ups at the Adelphi for the last couple of months and the week before last we had 10 people attend for a night of chat about football, beer, and many other topics, as well as small measure of database stuff. It was a really good chance to meet others in the same job in the same area.  You never know it might make the next interview you go for easier if you have had a few beers previously with one of the interviewing panel. There again if you did not get a round in it might not have been a good move.

When I know the date of the next one I will post a note on this blog.

John

However the downside of that is that you can hit non-published bug:12398300 which is a duplicate of bug:12356910 (also non-published). Diskgroup Mount Hangs with RBAL Waiting on ‘GPnP Get Item’ and ‘enq: DD – contention’ [ID 1375505.1].  Note: This issue so far has been reported on RAC 11.2.0.2.3 and 11.2.0.3 environments which is where we saw it (RAC – 11.2.0.3 clusterware – 11.2.0.1 rdbms)

Simply the ALTER DISKGROUP MOUNT just hangs and has to be interrupted (CTRL-C). No errors in the ASM alert log

Killing the ora.gpnpd on the node when ASM is blocked in the gpnp wait, permits not having  to stop the ASM instance. For details, please see Note:1392934.1. Otherwise, restart the ASM instance that is causing the lock condition.

The fix will be included in future 11.2.0.3.x Patch Set Updates (PSUs) but no patches yet exists (at the time this article was written – Nov.9.2011). Also likely there will be patch requests for the fix to be included on top of existing 11.2.0.2 PSU, but none yet exist.

So I think I will stick with what I have always done, best practise or not.

--From Node 1

. oraenv

-- specify ASM instance from node 1

+ASM1

-- sudo -u oracle sqlplus may not work when run the first time so run:

sudo -u oracle ls

sudo -u oracle sqlplus / as sysasm

CREATE DISKGROUP TEST EXTERNAL REDUNDANCY DISK '' [DISK ''];

SELECT STATE, NAME FROM V$ASM_DISKGROUP;

-- from node 2

. oraenv

+ASM2

sudo -u oracle sqlplus / as sysasm

 

ALTER DISKGROUP TEST MOUNT;

SELECT STATE, NAME FROM V$ASM_DISKGROUP;

 

-- if all ok then

-- from node 2

alter diskgroup test dismount;

 

-- from node 1

DROP DISKGROUP TEST;

Now we can add the disk to the desired diskgroup safely.

 

-- From node 1

. oraenv

+ASM1

sudo -u oracle sqlplus / as sysasm

-- check disks visible in v$asm_disk

-- header_status should be CANDIDATE or FORMER

set lines 120 pages 100

column path format a20

SELECT name, path, mode_status, state, header_status, os_mb, free_mb

FROM v$asm_disk ORDER BY name, path;

-- check diskgroups

select GROUP_NUMBER,NAME,STATE,TOTAL_MB,FREE_MB from v$asm_diskgroup;

-- add disks to appropriate diskgroups

alter diskgroup x add disk '/dev/hdiskX';

-- monitor rebalance

set lines 170

select * from v$asm_operation;

select GROUP_NUMBER,NAME,STATE,TOTAL_MB,FREE_MB from v$asm_diskgroup;

11.1.0.7 Patch Set Updates – List Of Fixes In Each PSU [ID 1337836.1]
11.2.0.1 Patch Set Updates – List Of Fixes In Each PSU [ID 1340010.1]
11.2.0.2 Patch Set Updates – List Of Fixes In Each PSU [ID 1340011.1]
11.2.0.3 Patch Set Updates – List Of Fixes In Each PSU [ID 1449750.1] 

However beware if you are looking for a patch that is called Merged or something similar as you have to search the MoS notes for the included patches, not the merged patch itself

 As an example Patch 12640090 is an ASM fix but it is not found in note 1337836.1

But if you look at patch 12640090 in MoS you find the information about the included patches

 (4) Bugs Fixed by This Patch

———————————

The following are the bugs fixed by this patch:

  9007102: KFNMCLIENTSREGISTER12 WHEN HUNG UFG RECOVERS AND REGS FREED GPN

  9029156: ADDITIONAL TRACING FOR ASMB AND ASMB CONNECTIONS

 From note 1337836.1 I can see that 9007102 was included in PSU 8 but 9029156 is not quite as clear but it suggests it is included in PSU 13

Patch Details

 Patch 9029156: ADDITIONAL TRACING FOR ASMB AND ASMB CONNECTIONS 

Last Updated 21-Oct-2012 21:42 (2+ months ago)
Product Oracle Database Family Release Oracle 11.1.0.7.13

Size 104.5 KB
Download Access Extended Support
Classification General
Patch Tag 

Prerequisite Patches
14275623
DATABASE PATCH SET UPDATE 11.1.0.7.13 (INCLUDES CPUOCT2012)

 I understand Oracle have an easier tool for this but it is not generally available. I am sure there is a good reason for that but it escapes me

Merry Xmas to all