This blog is about a problem I had with an Enterprise Management agent and how I resolved it.
The symptoms are that an agent is not started and yet I can stop and start it successfully. Looking at the trace file $AGENT_HOME/sysman/RACnode/log or $AGENT_HOME/sysman/log if the node is not part of a RAC cluster I see the following error messages and emctl agent status shows lots of files waiting to load
Agent Version : 10.2.0.2.0
OMS Version : 10.2.0.3.0
Protocol Version : 10.2.0.2.0
Agent Home : /u00/app/oracle/OracleHomes/agent10g/mat051.myco.co.uk
Agent binaries : /u00/app/oracle/OracleHomes/agent10g
Agent Process ID : 20241
Parent Process ID : 20220
Agent URL : https://mat051.myco.co.uk:3872/emd/main
Repository URL : https://mat019.myco.co.uk:1159/em/upload
Started at : 2008-03-19 14:19:16
Started by user : oracle
Last Reload : 2008-03-19 14:19:16
Last successful upload : 2008-03-19 14:19:29
Total Megabytes of XML files uploaded so far : 0.07
Number of XML files pending upload : 625
Size of XML files pending upload(MB) : 35.23
Available disk space on upload filesystem : 35.86%
Last successful heartbeat to OMS : 2008-03-20 09:37:26
—————————————————————
Agent is Running and Ready
2008-03-20 09:42:39 Thread-4097833872 ERROR upload: Failed to upload file B0000002.xml, ret = -2
2008-03-20 09:42:39 Thread-4097833872 WARN upload: FxferSend: received http error in header from repository: https://URL/em/upload ERROR-400|ORA-01461: can bind a LONG value only for insert into a LONG column
All sorts of emctl actions failed to help including
emctl clearstate agent
emctl secure agent
I then discovered that several agents all had the same issue so that pointed to the management server
I restarted that with
opmnctl stopall
opmnctl startall
with no success.
I finally shutdown the EM database and rebooted the server and everything sprung back into life. I suspect it was the restart of the database that fixed the problem but I don’t know for certain.
Update 27 March, 2008
Oracle have confirmed that it is an RDBMS issue although mine is 10.1.0.4 and a restart should fix the problem. Full details below
We have Note 469077.1 (still under edit, therefore unpublished) which states:
“New and intermittent errors have started occurring after going to 10.2.0.3 from an older patch
set level or Oracle RDBMS version.
…
These symptoms can eventually be tracked to issues with shared cursors improperly evaluated lengths, which break Oracle i
n various ways. The resultant different errors seen are dependent on the datatyp
e or connectivity used. Some of the errors encountered include:
ORA-932 – inconsistent datatypes expected %s got %s
ORA-1008 – not all variables bound
ORA-1460 – unimplemented or unreasonable conversion requested
ORA-1461 – can bind a LONG value only for insert into a LONG column
ORA-1483 – ORA 1483 invalid length for DATE or NUMBER bind variable
….
While there is more than one potential source for all of the above errors in 10.2.0.3, most will be assoc
iated with a Shared Cursor problem. This note addresses some of the most common
sources found when using 10.2.0.3 If you are not using 10.2.0.3, there are other
known bugs which have similar shared cursor issues which will be covered in oth
er notes to be provided in the future.”