Oracle DBA – A lifelong learning experience

In praise of bloggers (along with sql_profiles and RAC interconnect pings)

Posted by John Hallas on March 12, 2014

Today I am going to mention two articles which I came across in the last few days whilst investigating problems and talk about the background to the problems but also praise the articles. I am sure many of us have run the SQL Tuning advisor from within OEM or using the DBMS_SQLTUNE package from the command line and often it recommends a sql_profile that should be applied, (invariably giving 99.98 perceived benefits). Now I see this as both a good thing and a bad thing. Yes it is nice to apply the profile and hopefully fix the problem but I would also like to know what the profile is doing and that is well-hidden.  I agree that there is the option to look at current and new explain plans but sometimes it is difficult to really understand what hints have been used to get the new plan, especially on a statement that has many lines in the explain plan. So why is this important might you ask, just go and select the apply button and forget about it. Well I have two very good reasons why it is important.

Read the rest of this entry »

Posted in Oracle | Tagged: , , , , , , , , , , , , | 1 Comment »

Don’t forget the emctl reload agent command

Posted by John Hallas on March 11, 2014

Just a quick entry to show the use of a command that I had forgotten existed but it seemed to work nicely

Problem  – OEM12C agent had been down for a few days on a non-production server and more than the maximum number of files had been created ready to be uploaded.

emctl status agent

Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent is Not Running
emctl start agent
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Starting agent ........................................................................................................................... started but not ready.

emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 12.1.0.2.0
OMS Version       : 12.1.0.2.0
Protocol Version  : 12.1.0.1.0
Agent Home        : /app/oracle/product/gc12.1/agent12c
Agent Binaries    : /app/oracle/product/gc12.1/core/12.1.0.2.0
Agent Process ID  : 2930
Parent Process ID : 2806
Agent URL         ::1830/emd/main/
Repository URL   ::4900/empbs/upload
Started at        : 2014-03-11 12:45:03
Started by user   : oracle
Last Reload       : (none)
Last successful upload                       : 2014-03-11 12:48:27
Last attempted upload                        : 2014-03-11 12:48:27
Total Megabytes of XML files uploaded so far : 1.07
Number of XML files pending upload           : 3,118
Size of XML files pending upload(MB)         : 8.51
Available disk space on upload filesystem    : 26.60%
Collection Status                            : [COLLECTIONS_HALTED(
  UPLOAD_SYSTEM Threshold (UploadMaxNumberXML: 5000) exceeded with 5001 files)]
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2014-03-11 12:48:11
Last successful heartbeat to OMS             : 2014-03-11 12:48:11
Next scheduled heartbeat to OMS              : 2014-03-11 12:49:11

Edit $ORACLE_HOME/sysman/cinfig.emon.properties and change the entry to 6000 from 5000 for the line

UploadMaxNumberXML=5000

Then instead of the time-consuming process of stopping and starting the agent

Emctl reload agent

That works very quickly and the files were processed few in a few minutes and then I just re-edited the properties file back again and performed a reload

Posted in Grid control and agents, Oracle | Tagged: , | Leave a Comment »

aud$ and ASSM – a good combination

Posted by John Hallas on March 6, 2014

We had a 8 node RAC cluster that was showing evidence of connections timing out. The following AWR report segments indicates a high level of login activity and blocking on the AUD$ table and segments.

ex3

ex1

ex2

None of it conclusive but what was the kicker was the fact that the aud$ table was still in the system tablespace which is manually  managed and therefore automatic segment space management is not coming into play.

Over a 24 hour period there were over 50K connections, all being audited and the aud$ table was on the system tablespace which was manual and not running ASSM.

On all our systems we migrate the aud$ table to an ASSM managed tablespace (normally SYSAUX) after the build but this database had been delivered by a 3rd party and was not under our direct control.

I was pretty certain that moving that table would relieve the symptons of contention on the aud$ segments, which were being exaggerated by the 8 RAC nodes.

The following test case indicates the differences. I have taken the code used from a site which discusses freelist contention and ASSM

Create a test table in  TEST a manually managed tablespace, run some throughput through and monitor through AWR.

Create a new table in an auto managed tablespace, repeat the tests and compare results

drop table test;
 create table test (
   x date,
   y char(255) default 'x'
 )
 storage (freelists 1)
 tablespace test;

exec dbms_workload_repository.create_snapshot();

declare
     l_job number;
 begin
     for i in 1 .. 15
     loop
         dbms_job.submit( l_job, 'do_insert;' );
     end loop;
     commit;
 end;
 /

select substr(job,1,4) "job",
        substr(schema_user,1,10) "user",
        substr(to_char(last_date,'dd.mm.yyyy hh24:mi'),1,16) "last date",
        substr(to_char(next_date,'dd.mm.yyyy hh24:mi'),1,16) "next date",
        substr(broken,1,2) "b",
        substr(failures,1,6) "failed",
        substr(what,1,32) "command"
   from dba_jobs;

The AWR top events from the first run were

Event                                 Waits     Time(s)   (ms)   time Wait Class
------------------------------ ------------ ----------- ------ ------ ----------
buffer busy waits                 1,876,080       2,873      2   63.7 Concurrency
enq: HW - contention                227,951       1,091      5   24.2 Configuration
DB CPU                                              469          10.4
db file sequential read             118,194         108      1    2.4 User I/O
log file switch (checkpoint in           60          10    166     .2 Configuration

followed by the second run using  the tablespace that was ASSM managed

Top 5 Timed Foreground Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Event                                 Waits     Time(s)   (ms)   time Wait Class
------------------------------ ------------ ----------- ------ ------ ----------
DB CPU                                            1,696          24.4
log file switch (checkpoint in        1,775         938    528   13.5 Configuration
db file sequential read             724,504         630      1    9.1 User I/O
log file switch completion            2,344         434    185    6.3 Configuration
buffer busy waits                   987,073         260      0    3.7 Concurrency

The overall result was that we had a similar number of logins, no busy segments on the aud$ table and no timeouts. Job done

Posted in 11g new features, Oracle | Tagged: , , | 5 Comments »

The ‘rfs writes’ event on a DataGuard standby database

Posted by John Hallas on March 5, 2014

It is well known that poor performance on the standby server of a DataGuard pair can affect the performance of the primary database. This post shows an example and how to use the view GV$EVENT_HISTOGRAM to track down an issue.

The databases were 11.2.0.1 on HPUX. I had been seeing alerts from OEM to state that the standby was seeing lag_apply delays when applying redo to standby. Looking at the primary database alert log I could see the entries

ORA-16198: LGWR received timedout error from KSR
LGWR: Attempting destination LOG_ARCHIVE_DEST_2 network reconnect (16198)
LGWR: Destination LOG_ARCHIVE_DEST_2 network reconnect abandoned
Errors in file /app/oracle/diag/rdbms/xxxprd1a/BSMPRD1A/trace/xxxPRD1A_lgwr_24722.trc:
ORA-16198: Timeout incurred on internal channel during remote archival
Error 16198 for archive log file 1 to 'xxxPRD1B'
Destination LOG_ARCHIVE_DEST_2 is UNSYNCHRONIZED

That seemed to correct itself later on but the timeout error was indicative of a network problem  – well at least that was my original hypothesis.

However I have a script which I call rfs_writes.sql which I use on the standby database quite often and once I had run that I was sent in a different direction. Read the rest of this entry »

Posted in Oracle | Tagged: , , , , | Leave a Comment »

Serious BUG alert – RMAN Duplicate overwrites datafiles at original location

Posted by John Hallas on February 11, 2014

Please be aware of the serious bug identified with RMAN duplicate database on the same node with versions 11.2.0.2 and 11.2.0.4

Bug 13498382  ORA-1122 ORA-1208 RMAN Duplicate overwrites datafiles at original location despite db_create_file_dest

This note gives a brief overview of bug 13498382.
The content was last updated on: 07-FEB-2014
Click here for details of each of the sections below.

Affects:

Product (Component) Oracle Server (Rdbms)
Range of versions believed to be affected Versions BELOW 12.2
Versions confirmed as being affected
Platforms affected Generic (all / most platforms affected)

Fixed:

The fix for 13498382 is first included in

Interim patches may be available for earlier versions – click here to check.

Symptoms:

Related To:

Description

New name for datafile created during recover may not be populated during the switch and catalog steps of duplicate command.

You can be affected by this issue if you execute the following steps with RMAN :

1. Take a backup from database using RMAN
    backup database;
    sql 'alter system switch logfile'

2. A datafile is added to the source database, or new tablespace is created
    create tablespace TC;
    sql 'alter system switch logfile'

3. RMAN Duplicated is executed to another DB on the same host.

     connect target sys/pwd@DB1
     connect auxiliary sys/pwd;
     run   {
     set until sequence <seq#>;
     duplicate target database to DB2;
    }  

4. The DB is recovered past the point in time the datafile was added

5. The datafile is created to the original location, instead of created at DB_FILE_CREATE_DEST, overwriting the file at SOURCE

At SOURCE, we see the datafile going OFFLINE after a checkpoint

Read of datafile '+DG/db_unique_name/datafile/tc.1232.769779085' (fno 7) header failed with ORA-01208
Rereading datafile 7 header failed with ORA-01208
Errors in file /diag/rdbms/db_name/db_unique_name/trace/db_unique_name_ckpt_9604.trc:

ORA-63999: data file suffered media failure
ORA-01122: database file 7 failed verification check
ORA-01110: data file 7: '+DG/db_unique_name/datafile/tc.1232.769779085'
ORA-01208: data file is an old version - not accessing current version

 At TARGET / AUX, we see the datafile being created at original location

alter database recover logfile  '+DG/db_unique_name/archivelog/2011_12_13/thread_1_seq_934.1321.769779115'

Media Recovery Log +DG/db_unique_name/archivelog/2011_12_13/thread_1_seq_934.1321.769779115
File #7 added to control file as 'UNNAMED00007'. Originally created as: '+DG/db_unique_name/datafile/tc.1232.769779085'
Errors with log +DG/db_unique_name/archivelog/2011_12_13/thread_1_seq_934.1321.769779115
Recovery interrupted!

Posted in 11g new features | Tagged: , , , | 2 Comments »

Invisible Indexes – a real world example and a bit of LogMiner

Posted by John Hallas on February 11, 2014

We had a requirement to perform a one-off update on a table of around 1.2M rows. There were 11K statements similar to the one below, with varying values and predicates updating a variety of rows from 20 to 500. The potential elapsed time was around 44 hours due to the volume of full table scans involved.

The obvious thing to do was to create an index but this was production and it was urgent so no time to go through the full route to live test cycle.

This seemed a great opportunity to use an invisible index with the theory being that the optimiser would not use it for any other of the currently running work and only our session would see it.

This demo shows how it was tested and what the benefits were along with a couple of thoughts on invisible indexes.

I must admit I wish I could get a consistent method of showing explain plans, sometimes they come out lined up properly, other times they are a mess. This entry is somewhere in between.

Firstly lets run an example update statement to see what access path it uses

SQL> Update JOHN.TEST_TABLE Set WK_WAC_RATE=0.5338 Where SKU_ITEM_KEY=4878 AND TXN_WK_KEY=20140127;

24 rows updated.

Elapsed: 00:00:00.46

-------------------------------------------------------------------------------------------------------------
| Id  | Operation            | Name                 | Rows  | Bytes | Cost (%CPU)| Time     | Pstart| Pstop |
-------------------------------------------------------------------------------------------------------------
|   0 | UPDATE STATEMENT     |                      |    16 |   256 | 13696   (1)| 00:02:45 |       |       |
|   1 |  UPDATE              | TEST_TABLE |       |       |            |          |       |       |
|   2 |   PARTITION RANGE ALL|                      |    16 |   256 | 13696   (1)| 00:02:45 |     1 |   118 |
|*  3 |    TABLE ACCESS FULL | TEST_TABLE |    16 |   256 | 13696   (1)| 00:02:45 |     1 |   118 |
-------------------------------------------------------------------------------------------------------------

Create an index and re-run to see that the index will be used and the results are better, don’t forget  to gather stats.

CREATE INDEX “JOHN”.”INDX_1″ ON “JOHN”.”TEST_TABLE” (“SKU_ITEM_KEY”, “TXN_WK_KEY”);

execute dbms_stats.gather_table_stats(‘JOHN’, TEST_TABLE’,null,10,cascade=>true);

SQL> Update JOHN.TEST_TABLE Set WK_WAC_RATE=0.5338 Where SKU_ITEM_KEY=4878 AND TXN_WK_KEY=20140127;

24 rows updated.

Elapsed: 00:00:00.01

------------------------------------------------------------------------------------------
| Id  | Operation         | Name                 | Rows  | Bytes | Cost (%CPU)| Time     |
------------------------------------------------------------------------------------------
|   0 | UPDATE STATEMENT  |                      |    27 |   432 |    20   (0)| 00:00:01 |
|   1 |  UPDATE           | TEST_TABLE |       |       |            |          |
|*  2 |   INDEX RANGE SCAN| INDX_1               |    27 |   432 |     3   (0)| 00:00:01 |
------------------------------------------------------------------------------------------

Note the index can be made invisible from a command line statement but it does not seem possible if creating using the OEM12 gui

 alter index indx_1 invisible;

SQL> select index_name,visibility from user_indexes;

TEST_TABLE_PK        VISIBLE

INDX_1                         INVISIBLE

Run the query to prove that the invisible index is not used by the optimizer

 SQL> Update JOHN.dwd_rtl_sl_line_item Set WK_WAC_RATE=0.5338 Where SKU_ITEM_KEY=4878 AND TXN_WK_KEY=20140127;

24 rows updated.

-------------------------------------------------------------------------------------------------------------
| Id  | Operation            | Name                 | Rows  | Bytes | Cost (%CPU)| Time     | Pstart| Pstop |
-------------------------------------------------------------------------------------------------------------
|   0 | UPDATE STATEMENT     |                      |    16 |   256 | 13696   (1)| 00:02:45 |       |       |
|   1 |  UPDATE              | TEST_TABLE |       |       |            |          |       |       |
|   2 |   PARTITION RANGE ALL|                      |    16 |   256 | 13696   (1)| 00:02:45 |     1 |   118 |
|*  3 |    TABLE ACCESS FULL | TEST_TABLE |    16 |   256 | 13696   (1)| 00:02:45 |     1 |   118 |
-------------------------------------------------------------------------------------------------------------

Now alter the session so only that session id can take advantage of the index

SQL> ALTER SESSION SET OPTIMIZER_USE_INVISIBLE_INDEXES=TRUE;

Session altered.

SQL>  Update JOHN.TEST_TABLE Set WK_WAC_RATE=0.5338 Where SKU_ITEM_KEY=4878 AND TXN_WK_KEY=20140127;

24 rows updated.

------------------------------------------------------------------------------------------
| Id  | Operation         | Name                 | Rows  | Bytes | Cost (%CPU)| Time     |
------------------------------------------------------------------------------------------
|   0 | UPDATE STATEMENT  |                      |    27 |   432 |    20   (0)| 00:00:01 |
|   1 |  UPDATE           | TEST_TABLE |       |       |            |          |
|*  2 |   INDEX RANGE SCAN| INDX_1               |    27 |   432 |     3   (0)| 00:00:01 |
------------------------------------------------------------------------------------------

The code is running much more efficiently and yet we have not impacted the execution of any other sql that may be running against this table, although the index will need updating with every I,U,D statement, whether or not the index is visible  or not.

Moving to thoughts around how you can tell whether an index is visible or not and when it was made visible. USER_INDEXES will show the state of the index

SQL> select index_name,visibility from user_indexes;

TEST_TABLE_PK        VISIBLE

INDX_1                         INVISIBLE

How do you know when it was made invisible/visible though. Does the LAST_DDL time tell you?

OBJECT_NAM OBJECT_TYPE         CREATED                  LAST_DDL                 VISIBILIT

———- ——————- ———————— ———————— ———

INDX_1     INDEX               10-FEB-14 08:38          10-FEB-14 08:38          INVISIBLE

SQL> alter index indx_1 visible;

 select object_name,object_type, to_char(CREATED,’DD-MON-YY HH24:MI’) Created,

to_char(LAST_DDL_TIME,’DD-MON-YY HH24:MI’) Last_DDL,  Visibility

from user_indexes U, user_objects O where o.object_name = u.index_name

and index_name  = ‘INDX_1′

OBJECT_NAM OBJECT_TYPE         CREATED                  LAST_DDL                 VISIBILIT

———- ——————- ———————— ———————— ———

INDX_1     INDEX               10-FEB-14 08:38          10-FEB-14 10:41          VISIBLE

So yes, if the last DDL on the index was to change it’s visibility then the time can be identified. However what the DDL was cannot be identified and an alternative option is to use logminer functionality.

Ensure supplementary logging is set, perform some actions and then log mine and see if you can identify them.

SELECT SUPPLEMENTAL_LOG_DATA_MIN FROM V$DATABASE;

ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;

EXECUTE DBMS_LOGMNR_D.BUILD(OPTIONS=>DBMS_LOGMNR_D.STORE_IN_REDO_LOGS);

SQL> alter index john.indx_1 invisible;

Index altered.

SQL> alter index john.indx_1 visible;

Index altered.

SQL> alter index john.indx_1 invisible;

Index altered.

alter session set nls_date_format = ‘DD-MON-YYYY HH24:MI:SS’;

execute dbms_start_logmnr( -

starttime => ’10-Feb-2014 12:00:00′, -

endtime => ’10-Feb-2014 12:34:00′, -

options => dbms_logmnr.dict_fom_online_catalog + -

dbms_logmnr.continuous_mine);

select operation, sql_redo,sql_undo from v$logmnr_contents

where seg_owner = ‘JOHN’ and seg_name = ‘INDX_1′ ;

select substr(sql_redo,1,100) sql_redo, to_char(timestamp,’DD-MON-RRRR HH24:MI:SS’) timestamp

from v$logmnr_contents

where seg_owner = upper(‘JOHN’)

and sql_redo like ‘%vis%’;

SQL_REDO          TIMESTAMP

—————————–

alter index john.indx_1 invisible;              10-FEB-2014 12:32:01

alter index john.indx_1 visible;                  10-FEB-2014 12:32:19

alter index john.indx_1 invisible;              10-FEB-2014 12:32:24

This last section on identifying when the index visibility was changed came from an Oracle-L discussion and the suggestion of log mining came from David Fitzjarrel. In fact it was because I had recently read the post that it gave me the idea to use an invisible index.

Posted in 11g new features, Oracle | Tagged: , , , , , , | 5 Comments »

Oracle Restart – going , going , gone?

Posted by John Hallas on January 13, 2014

Oracle restart is an 11GR2 feature which ensures that all services on a standalone installation start up in the correct order. As such it seems to work well. One bugbear I have with it is that it changes the order of entries in the /etc/oratab file. Personally I like my oratab to be ordered in terms of database (most used first), ASM, then agents. In that way when I logon to a box and it automatically sets the SID it picks up the first entry which is commonly the database I want to work with.

Technically it is the oraagent process, which is new in 11GR2 which updates the oratab file, but as that sits under the ohasd directory in the GI home then I consider it to be part of the Restart process. As a bye you can make your oratab file read only as long as you remember to change permissions when adding a new databases or removing one.

To the main purpose of this post. It appears that Oracle Restart is deprecated in 12c although no replacement is on the horizon.

http://docs.oracle.com/cd/E16655_01/server.121/e17642/deprecated.htm#BABBEHJH

“Oracle Restart is deprecated in Oracle Database 12c. Oracle Restart is currently restricted to manage single-instance Oracle databases and Oracle ASM instances only, and is subject to desupport in future releases. Oracle continues to provide Oracle ASM as part of the Oracle Grid Infrastructure installation for Standalone and Cluster deployments.”

That is a bit of a shock, as it is still fully documented in the Oracle 12c docs for installation etc. The MoS note Support Impact of the Deprecation Announcement of Oracle Restart with Oracle Database 12c (Doc ID 1584742.1) gives more detail and tantalisingly seems to suggest that the deprecation may be revoked.

“While typically, a feature is only deprecated once the superseding feature has been announced, on occasion this order cannot be kept. In this case, one may assume that the deprecation announcement will last – without effect – until the superseding feature has been released or the deprecation is revoked. In case of the deprecation announcement of Oracle Restart, the superseding feature has not been released yet. Therefore, Oracle will maintain Oracle Restart in its current form and will continue to fix BUGs for this product until Oracle Restart is officially de-supported, which would constitute the removal of the code.”

It would be reasonable to think that Oracle intended having a replacement ready for the initial release of 12C but it was not available in time and will come out in 12CR2

Posted in 12c new features, ASM, Oracle | Tagged: , , , | Leave a Comment »

2014 technology ambitions – just the two

Posted by John Hallas on January 6, 2014

As this the traditional time to layout resolutions for the year here are my 2 database related ones.

To understand more about some of the newer technologies and to advance my use of APEX as a means of providing reporting information around the systems and people I manage.

Having worked with the base Oracle RDBMS, and latterly SQL Server, for around 20 years I must admit that I am not as familiar with such terms as Hadoop and NoSQL and systems such as MongoDb and Cassandra as I would like to be. My aim is to do some reading and research to get me to a level such that I could in theory respond to this article which I found to be an interesting take on current technologies.

Iggy Fernandez has been writing a set of 12 articles around the subject of NOSQL and I have found them to be both well-written and informative.

In terms of Hadoop, a key phrase I read was “moving computation is more efficient than moving data”. I have not found any great introductory references to Hadoop as yet and most things I Google seem to be quite old (well in computer technology terms at least) – maybe 3-4 years old. However it is an area that seems to be growing especially in industries that generate vast amounts of data – telecoms being one. If anyone has any good sources of Hadoop information please feel free to comment

I am quite familiar with APEX as being a native Web application development framework for the Oracle Database and it can be designed to run large sites such as Liverpool Victoria Insurance  and Oracle Store being just two well-known sites. However I am more interested in the ease and simplicity with which management information reporting can be delivered with little effort and no deep knowledge.

Admittedly, unless you develop themes using CSS or similar, all the applications look very similar and possibly quite basic. However the power that can be delivered in terms of reporting capability is quite significant relative to effort required. We currently use Apex applications to manage many aspects of the database team’s activities – call-out recording, lieu-time, code deployment recording, ongoing activities and many more things and the whole set of applications fit into a nice framework. What I want to look at is to see what other opportunities we have and to spread the use of APEX outside the DBA team into a wider IT Service arena.

Posted in Oracle | Tagged: , , , , | 3 Comments »

2013 blog activity

Posted by John Hallas on December 31, 2013

At the end of the year WordPress produce a brief analysis of activity on my blog

The Louvre Museum has 8.5 million visitors per year. This blog was viewed about 160,000 times in 2013. If it were an exhibit at the Louvre Museum, it would take about 7 days for that many people to see it.

In 2013, there were 25 new posts, growing the total archive of this blog to 176 posts. There were 32 pictures uploaded, taking up a total of 5 MB. That’s about 3 pictures per month.

The busiest day of the year was March 19th with 782 views. The most popular post that day was Purging statistics from the SYSAUX tablespace.

Interestingly numbers  – about 440 hits a day on average although weekend is only ever 125-150 per day and most weekdays are ~550. The pattern always looks the same  – Monday the busiest and each day’s hits reduces towards Friday.

I am pleased I wrote 25 posts, one a fortnight is about right volume for me I think. In my humble opinion my best post was a 2000 word essay I wrote on being a good DBA manager – that was written on the fly with no preparation. On reflection I think 10 commandments was too many as the first couple were a bit weak but were there to make up the 10 in the title. 10 is always top 10, or 10 commandments, 6 is always 6 of the best, I am not sure what you can say about 8.

Finally I will share my photo again of me holding  the football used in the 1966 World Cup Final. I was quite pleased with that.

1966worldcupball

Looking forward to producing more articles and receiving comments in 2014

John

Posted in Oracle | Leave a Comment »

Merry Xmas and a project goes live

Posted by John Hallas on December 20, 2013

I hope all my readers have a Merry Xmas and a Happy 2014.

My company has finally delivered an online delivery solution (we are a major food retailer in the UK  for anyone who does not recognise the branding). It is always nice to see a major project go-live and this is one that has caused some pressure in recent months.

I do think that whilst  the courgettes comment is funny and relevant to the emphasis we put on fresh food it might seem a bit stale (no pun intended) after a while.

 

morrisons_van1

Posted in Uncategorized | Leave a Comment »

 
Follow

Get every new post delivered to your Inbox.

Join 184 other followers