1.

Agent was not starting due to out of memory exception

======> From emagent.nohup

Agent is going down due to an OutOfMemoryError

----- Wed Dec 31 03:42:49 2014::21771::Checking status of EMAgent : 21913 -----
----- Wed Dec 31 03:42:49 2014::21771::EMAgent exited at Wed Dec 31 03:42:49 2014
with return value 57. -----
----- Wed Dec 31 03:42:49 2014::21771::EMAgent will be restarted because of an Out
of Memory Exception. -----
----- Wed Dec 31 03:42:49 2014::21771::Restarting EMAgent. -----

======> from gcagent.log

2014-12-31 03:42:41,315 [1:main] ERROR - agent main threw an error

java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Arrays.java:3209)
at java.lang.String.<init>(String.java:215)
at java.lang.StringBuffer.toString(StringBuffer.java:585)
at oracle.sysman.gcagent.state.StateMgr.loadValue(StateMgr.java:613)
at oracle.sysman.gcagent.state.StateMgr.initStateMgr(StateMgr.java:682)
at oracle.sysman.gcagent.tmmain.TMMain.main(TMMain.java:519)
2014-12-31 03:42:15,728 [1:3305B9] INFO - Handling OOME (FCCounter [count=1,
duration=300000, maxInDuration=0, timestamp=1419997335728], 48178)
2014-12-31 03:42:16,126 [1:3305B9] INFO - agent status is being changed to EXITING
2014-12-31 03:42:16,127 [1:3305B9] INFO - Agent exiting with exit code 57

2. Tried to restart the agent by modifying the java properties in emd.propertie,

but that too didnt help as somehow the agent was autotuning the parameters.

Reference Note : How to Modify or Increase Agent JVM Heap & Permgen Memory -
java.lang.OutOfMemoryError Reported in gcagent.log (Doc ID 1399201.1)

3. As the above workaround didnt work, we disabled autotuning by modifying the

below setting, this took care of teh java heap space error, and the agent started
picking up the correct values for the parameters, but thne it started to crash with
a new error :)

j6-11:agent12c> grep -i autotu emd.properties

enableAutoTuning=flase

4. TOKEN Mismatch issue.

Reference Note: emctl start agent Fails With 'Target Manager failed at Startup:
targets.xml was rejected: loaded with a wrong agent token' Reported in gcagent.log
(Doc ID 1384366.1)

2014-12-31 03:51:17,692 [1:main] ERROR - Agent failed to Startup for Target Manager
in step 2
oracle.sysman.gcagent.tmmain.lifecycle.AgentStartupException: targets.xml was
rejected: loaded with a wrong agent token
to fix this we took a bakcup of targets.xml and then edited the targets.xml, and
removed all the tags except the the below. and tried to restart the agent.
This worked and the agent was started, but as we had removed the target entries
from the xml file, the agent was unable to list/recognize the targets.
To fix that we used the below commands..

cp targets.xml targets.xml_31dec

edit it as per note.

j6-11:agent12c> emctl config agent listtargets

Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.

j6-11:agent12c> emctl config agent addinternaltargets

Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.

j6-11:agent12c> emctl config agent listtargets

Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
[j6-11.idc1.level3.com, host]
[j6-11.idc1.level3.com:3872, oracle_emd]

-- but that too didnt help in retriving all the targets.

5. Now for the fun part :) :)

The issue still didn't get fixed..

In the note EM12c: Troubleshooting Agent Start Failures (emctl start agent) (Doc ID
1554981.1)
found the following

************************
In case the targets.xml is determined to be valid, try the following to correct the
issue:

Remove the AGENT_TOKEN attribute from the Targets tag in the targets.xml file.
Start the agent (the agent will recreate AGENT_TOKEN attribute using the current
encryption key).
********************************

But instead of removing the agent_token completely we used the new value that was
generated in the targets.xml that was re-populated in the above step 4.
So to do this, Re-stored the targets.xml from the backup taken in step 4 and
replaced the AGENT_TOKEN with the new value generated

mv targets.xml targets.xml_targetsmissing
cp targets.xml_31dec targets.xml

then took teh value from targets.xml_targetsmissing and pasated it int eh

targets.xml, after this we were finally able to start the agent and list all the
targets successfully.
f14-24:agent12c> emctl clearstate agent
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
EMD clearstate completed successfully

f14-24:agent12c> emctl upload agent

Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully