Database SIG – Manchester 27th March

The UKOUG have done really well in organising the next Database Server SIG. It is being held at the City of Manchester Stadium – the home of Manchester City. That is the group of players bought on the cheap and managed by Roberto Mancini into the best team in Britain as will be proved when we win the Premier League in April.

They have got two of the worlds’ top Oracle speakers on stage. Actually that is slightly incorrect. They have one of the worlds’ top Oracle speakers talking twice and me also on stage.

The agenda looks very strong

Jonathan Lewis  – Beating the Oracle Optimizer
Phillip Brown – Experiences of a DBA
Jonathan Lewis  – Redo
Peter Homes  – Oracle 10g/11g Automatic Memory management  – cautionary tales
John Hallas – ADRCI (will try and add some new stuff to the talk I gave at the conference)

The date is March 27th and this is the link to it

As we have recently removed AMM from a large Peoplesoft database (10g) and an important EBS (11g)  database in the last couple of months because of problems whereby the constant  resizing of memory chunks eventually caused the databases to hang  I will be interested to hear what Peter has to say. In fact I did post an entry on the problems caused by AMM  – (the EBS database in this case).

I am looking forward to the day

UKOUG 2011 – day 3

The final day of the 2011 UKOUG conference and it was straight in at the deep end with Joel Goodman talking about automatic parallelism in 11GR2. The talk was full of information, as Joel’s talks normally are. He also had time to cover Parallel Bulk Update which groups sets of rows into chunks. Each chunk can have a success or fail independently of other chunks which removes the ‘all or nothing’ approach normally seen with PDML. He has a good blog entry on this which is well worth perusing if you are interested. http://dbatrain.wordpress.com/2011/07/01/add-bulk-to-your-parallel-updates/

 My site is just going down the road with Goldengate so the talk by Pythian’s Marc Fielding on a real life Goldengate migration was very useful. This was a large financial institution where the system was crucial to business continuity and GG was to be used to provide a rapid fallback facility if things went wrong. The main thing I took away from the talk was how small-minded they must be not to provide adequate testing facilities for such a large project. Not being able to use full data sets and similar sized hardware (OK it was a 14TB database) does add a lot of risk and no small matter of frustration to the technicians involved in the migration. Some of the diagnostics that Marc talked about will be very useful to use and I was interested in the alternatives to supplementary logging which may be required if there is no primary key and it is difficult to identify a row specifically.

I did start to listen to another talk but after around 10 people had left I plucked up courage and made a hasty exit myself. It was just not for me.

The best presentation I saw at the conference was Connor Macdonald on a fresh approach to the optimizer statistics. Connor is a real showman and his easy on-stage manner belies the degree of effort he must spending preparing his numerous slides. The set of slides associated with the ITIL process deserved a round of applause by itself and indeed it received one.  This was the second session I went  to that mentioned the value of index key compression and the way it can be calculated by using ‘analyze index validate structure’. A very good presentation that provided food for thought.

My final session was Mike Swing talking about database tuning for packaged apps. He had way too much content and rushed through it much too fast. As several people said to me afterwards, all he really recommended was getting faster disk and more memory.  I liked his presentation style and easy manner but it was a bit light on useful content.

So here endeth day 3. I think this was the conference I have enjoyed the most and got the most from. The presentations were of a top standard and even though I was only interested in the RDBMS stream I had plenty of choice for most time slots. I know that cancellations and changes are hard to avoid but there did seem to be a lot and that made planning harder than it should have been. I think my only constructive critique would be that there were a number of presentations repeated from last year (and some from other SIGs almost 2 years ago). I fully understand that a good presentation is  still a good presentation a year later and not everyone has the chance to have seen it but personally  I am not in favour of too much repeated material.

 

UKOUG 2011 Part Deux

Day 2 of the UKOUG conference at the ICC in Birmingham and back into the fray.

First up was Thomas Presslie talking about Dataguard fast start failover. How he managed to demonstrate transactions and network connectivity using whisky and toilet paper could not be done full justice in a blog – it had to be seen to be believed.

It did make me want to do more with FSFO, especially noting how easy the setup was using OEM. However my belief that the database is only part of the end solution and failing that over to a second datacentre after a network flicker may leave the application stack in a mess does still concern me. Co-incidentally I have a requirement to set up a second standby configuration cascaded from a physical standby but keeping the 3^rd database perhaps one hour behind whilst the standby is in real time apply mode with no lag. That might give us a chance to determine the status of the data before a logical corruption (user error) had occurred. Much more likely to be of value is flashback query but we are going to look at both avenues. It is highly unlikely we would ever be in a position to flashback the database.

Julian Dyke then talked for an hour about RAC trouble-shooting (mostly 11.2.0.2) and the time flew past. I made quite a few notes of things to think about. The pros and cons of putting the scan addresses in /etc/hosts (HPUX) to be used in the event of a DNS failure was one thought. Looking at the exectask function and the scripts used to call various function was another action I took for myself. Another was a big list of asmcmd commands, some of which I did not recognise. I think they must have come in with 11GR2 which I have not really used myself although we are using it on site.

Tanel Poder’s biggest ever problem was next up. I had seen this presentation last year and knew the answer but how he got there was still interesting. The use of the HPUX command kitrace (similar to dtrace on Solaris – see reply below for more details) reminded me that I was going to look at that in some detail but have never got around to it. As my site is likely to be moving away from HPUX sooner rather than later perhaps there is not much point now.

After lunch John Beresniewicz was talking about ASH outliers. Quite mathematically based, which is always a challenge for me but he will be posting a script (possibly via Doug Burn’s blog) which he has developed as another means of dissecting and analysing ASH data.

Michael Salt’s talk on indexes was full of real world examples and there were lots of nice little hints and tips, none of which were earth-shattering but all of which were good practise and I found it a useful reminder of what I should be doing when looking at code. On the same theme two slots later Tony Hasler was presenting a beginners guide to SQL tuning.   I have never seen Tony present before but I really liked both his style and the content. A lot of information thrown in and good explanations of various autotrace outputs. I will definitely be downloading his presentation to run through it and see what I can put to further use. Whilst I do not think I am expert in the field of SQL tuning, indeed far from it, I do like to think I know what to look for. Sometimes listening to others you realise in the same lecture both how much you already know and how little you actually follow best practises. There is no real substitute from looking at code and trying to improve performance. For a lot of us who have a very wide-ranging DBA role then that opportunity to practise odes not appear often enough which is why it is good to review and refresh your approach now and then.

At every conference I like to try and hear something new or touch on an area that is outside my day job. John King’s talk on Edition Based Redefinition was just that. I am not really in a position to take advantage of the ability to let users run differing sets of code and then migrate them across to a new release in a seamless manner, all without any outages or interruption to service. However I could see how useful it could be, especially in the world of the Apps DBA, say for EBS. Apparently no less a person than Tom Kyte referred to EBR as the ‘killer feature’ within 11GR2.  John had an easy, comfortable manner  and the time flew past, so much so that he had to be dragged kicking and screaming from the stage by the next presenter.

All in all another good day, rounded off with a couple of beers with work colleagues and a few presenters, all with plenty of Oracle chat included.

UKOUG 2011 – Day 1

The first day of UKOUG conference at the ICC in Birmingham and there were a lot of familiar faces are around. Due to a lot of traffic I missed the first presentation I wanted to see which was from Kyle Hailey on SQL tuning. I will have to download the presentation later. I did make it to Greg Rahn on the SQL Monitoring report and that was well worth the time spent. Whilst I am familiar with the functionality, he opened my eyes by providing  a number of examples of what he would look at first to try and determine a better resolution. His presentation style was comfortable and he had a small number of examples which covered quite a lot of scenarios. He did point out it was only to be used if you had paid for the Tuning and Diagnostic pack but as it is turned on by default I did wonder how many use it without any further consideration.

The value of UKOUG presentation feedback

One of the benefits of presenting at UKOUG, whether it be a SIG or the Conference is the feedback received from the audience via the questionnaire. I received mine for the ADR presentation I did at the Unix SIG recently. I was pleased to see I received very positive marks and the comments were supportive. Probably the most important factor to me was the marks given for presentation skills which were good and seem to get better with each presentation, which should not be too much of a surprise.

I think self-development is an important part of presenting. Yes I know the altruistic believe it should all be for the audience but I am happy to take away confidence and self-belief from my experiences. I set myself a target originally of doing a SIG presentation and then a conference presentation. My next step was to talk on a technical subject which has now been achieved and my final objective is to try and do an interactive technical demonstration using a laptop. At the moment I have no idea on what that will be about and I have no timescale so I can consider it a work in progress at the moment.

ADR presentation done and dusted

I was pleased with my the presentation on ADR I gave yesterday at the Unix SIG in Reading. The timing was bang on at 60 minutes and the audience had not seen or used many of the features I discussed so that was a bonus. I had the benefit of being able to give the same talk two days earlier to a team of DBAs at work and I learned a lot from the comments I received and that made for a better presentation yesterday.

One question I was asked afterwards was ‘ are Health Monitor and Support Workbench Enterprise Edition features or are they also available in Standard Edition?’. As we do not use Standard Edition I could only hazard a guess that these were tools that were advantageous to Oracle support as well as users and therefore I figured that they would be included in both editions. I have since looked at the list of features for 11GR1 and can see nothing that suggests that the features are licenseable or are different between editions. If anybody knows different then please feel free to correct me.

The presentation is available in PDF format from the UKOUG site,  provided you have a member access that is.

Upcoming ADR presentation at UKOUG

I am presenting a talk around the use of the Automatic Diagnostic Repository at the UKOUG SIG in Reading on 22nd September 2011. I will be covering, amongst other things, the management of files, the Health Monitor, incidents and problems and the Support Workbench utility. I am hoping that, whilst everyone will probably be already aware of ADR, some of the things I mention might be new or have not been fully looked at before. The Health Monitor was new to me and the ease of use of the Support Workbench when raising an SR is certainly the way forward.

I will also be quite critical of how ADR has been delivered, particularly in respect of the management of diagnostic data and the trace and alert logs that are generated. ADR currently lacks features such as the management of the alert log which still needs external management using such commands as logrotate on unix. The standard alert log has not been replaced by an alert log in xml format (log.xml) and the old alert log is only created for backward compatibility and is not guaranteed to be available in the future. Listener logs are not purged and there are ongoing problems in removing core dumps (on HPUX at least). Overall I will be suggesting that ADR is not as automatic as it could be but some of the additional features other than file management are well worth investigating.

I look forward to meeting fellow UKOUG members and hopefully I may learn a few new things about ADR myself if I get good audience interaction.

UKOUG Conference 2011 – Call for papers

The UKOUG Conference to be held in Birmingham (5th – 7th December 2011 ) is looking for presentations on: real life business experiences; in depth product analysis, new product overviews and everything else in between.

 New for 2011: in order to enhance the agenda there are new topic listings, so to get familiar with what to submit your presentation under click here.

Reminder of how to apply
 
1) Complete your Speaker Application form, please note there are new questions this year
2) Submit your abstracts: The judging abstract is to be of 50 – 200 words and a marketing abstract of no more than 400 characters (including spaces and quotation on your intended presentation.
3) Refer to the speaker guidelines for tips, if you have any questions or need a refresher.

As previously, your abstract(s) will then be evaluated by the agenda planning committee, comprised of members of the UK Oracle User Group community and confirmation of whether your abstract has been selected onto the agenda will be sent via email in September.

Submission deadline: Sunday 5th June 2011

As a first-time speaker last year I did not find it  too intimidating, athough I was in one of the smaller  halls, not the main stage areas and  would encourage anyone to try it. I think the best advice would be to ensure that you get chance to practise your speech in advance, either to work colleagues or a UKOUG SIG.  I found that presenting to a group on non-database people at work including  architects, managers and unix admins was very helpful and much better than trying the talk out with fellow DBAs where you felt the need to either miss out the simpler bits or elaborate and get side-tracked from the planned speech.

Good luck to all those who submit entries and to those who are chosen to speak.

How many mistakes can you make when diagnosing a problem

Last week I attended a talk by Daniel Fink at UKOUG around having a plan to sort out problems and perhaps the key point was –‘listen to the problem before setting off down the investigatory path’. I blogged at the time  that I thought it was an interesting  talk and I wanted to try and run through something similar at work.

This blog entry is perhaps a good example of how not to approach a problem but equally holds some technical information that may be of use to others in a similar situation.

I was approached by a developer who asked me about charactersets as he was running a query on two databases both of which had an external table  which came from the same source and yet he was only returning 3 rows out on one database and 4 rows on the other. The column he had narrowed it down to contained a £ sign and so he had assumed it was a characterset issue. In fact his initial mail  stated

Problem field is C21_DPST_RFRNC2….
As a workaround this problem we have used following regular expression when creating a table at source so that it filters out the special characters.
regexp_replace(Regexp_replace(trim(lower(C21_DPST_RFRNC2)),'[^a-z,_, ,A-Z,0-9,@,&,$,£,%,'']',' '),'[[:space:]]{1,}',' ')

 Mistake one – blind acceptance by DBA of what developer had said.

My immediate thought was ‘oh sh……’ we have made a mess of the charactersets at some time and that is going to be a big issue. Given that I have just presented about standards and charactersets was one of the subjects that I touched upon that was not a good place to be.

Archived logs stuck ‘in-memory’ when building a standby

We had to build a new copy of a physical standby database (11.1.0.7) and the RMAN duplicate command had completed  successfully so all we needed to do was to let the archived logs catchup. After an hour or so this was the status as seen from v$archived_log

column "First Time" format A40
column applied format A10
column "Next Time" format A40
set linesize 120
set pagesize 1000

select   thread#, sequence#,       applied,
to_char(first_time,'DD-MON-YY:HH24:MI:SS') "First Time",
to_char(next_time,'DD-MON-YY:HH24:MI:SS') "Next Time"
from      v$archived_log
UNION
select   NULL,NULL,' ',NULL,null FROM DUAL
UNION
select   null,null,
db_unique_name,
database_role,
open_mode
from      v$database
/

and the output was (long list but necessary to see both threads). What baffled us why why there were in-memory logs and yet later logs  had been applied.