TROUBLESHOOTING CONCURRENT MANAGER START-UP
PROBLEMS - Applications 11.0
=======================================================================
IMPORTANT NOTE - Some of these steps are destructive and should not be run on a
Production system without advice from Oracle Support.
INTRODUCTION
============
This note is designed to help diagnose and resolve problems starting the Oracle
Applications Concurrent Manager. There can be many issues which prevent the
CCM from starting and the errors may not always indicate the true problem.
Whilst this note has been written with Oracle Applications 11.0 in mind,
many of these steps are still relevant for 10.7 and 11i.
CHECKS TO MAKE
==============
1) Check the log files for any reported errors. The default name for the
Internal Manager log file is 'CM_<hostname>.log' on NT or >ManagerName>.mgr
on Unix. Also check any other manager log files in the $APPLCSF/$LOG directory
for relevant messages. Do these errors point to any obvious places to check
(e.g. database not running, TNS alias incorrect)? If these files do not exist
at all, there may be permission problems that prevent processes from writing
files to the proper directories.
2) Ensure that you are logged in as the 'applmgr' (or equivalent) user on Unix,
or if on NT that the service is being started by the 'applmgr' (or equivalent)
user.
3) If you are getting errors such as 'establish ICM unable to insert ICM record
APP-1167 APP-1244 Oracle error 1631 in insert_icm_record'
Check your database's tablespace for the FND tables to ensure you enough
free space to create the next extent. Also check the FND tables to ensure no
table has hit the maximum number of extents.
4) Log in to SQLPLUS as database user "apps".
Verify that view fnd_v$process exists: SELECT view_name
FROM all_views
WHERE view_name LIKE '%V$%';
If the view is missing or invalid, rebuild it. From the operating system,
as the applmgr user type:
FNDLIBR FND FNDCPBWV applsys/fnd SYSADMIN 'System Administrator' SYSADMIN
or as the applsys user run the following SQL statement
CREATE VIEW fnd_v$process AS
( SELECT P.Addr, P.Pid, S.Process Spid,
S.Osuser Username, P.Serial#,
S.Terminal, S.Program, P.Background,
P.Latchwait, S.Lockwait
FROM V$process P, V$session S
WHERE P.Addr = S.Paddr );
5) Close down all Concurrent Manager processes and clear database tables/log
files. This process will remove all history of Concurrent Jobs and remove all
scheduled jobs. Please contact Oracle Support for advice before doing this to a
Production installation.
a) Check there are no FNDLIBR, FNDCRM or other Concurrent Manager
processes running. This can be done via the
Concurrent Manager-->Administer screen, or from the operating system -
Note 68993.1 has a list of the process names to look for.
b) In SQLPLUS run the following SQL
UPDATE fnd_concurrent_requests
set phase_code = 'C', status_code = 'X'
where status_code = 'T';
UPDATE fnd_concurrent_queues SET running_processes = 0;
SELECT concurrent_queue_name, control_code, running_processes,
max_processes
FROM fnd_concurrent_queues;
The control_codes should be 'E' for deactivated managers, blank for all others
except FNDICM which should be 'X'. You will need to update the table
manually if you find any exceptions to these values.
For example:-
UPDATE fnd_concurrent_queues SET control_code = 'X'
WHERE concurrent_queue_name = 'FNDICM';
c) Delete all log and output files in FND_TOP/log, APPLCSF/log and APPLCSF/out if these files are no longer needed.
Having to many files in these directories has caused issues previously, so may be worth temporarily removing these files.
6) Check environment variables.
On NT using REGEDT32 check the following keys have these values:
WARNING - ensure you do not change or delete any registry Settings as this
may irrecoverably damage your Oracle or NT Server installation :-
HKEY_LOCAL_MACHINE-->Software-->Oracle-->Applications-->11.0.0-->APPL_CONFIG=<Appl Config Name>
HKEY_LOCAL_MACHINE-->Software-->Oracle-->LOCAL=<SID Name>
HKEY_LOCAL_MACHINE-->Software-->Oracle-->Applications-->11.0.0--><APPL_CONFIG>-->DIAG=Y
HKEY_LOCAL_MACHINE-->Software-->Oracle-->Applications-->11.0.0--><APPL_CONFIG>-->pmon=20
HKEY_LOCAL_MACHINE-->Software-->Oracle-->Applications-->11.0.0--><APPL_CONFIG>-->LOCAL=<SID Name>
HKEY_LOCAL_MACHINE-->Software-->Oracle-->Applications-->11.0.0--><APPL_CONFIG>-->SERVICE_SID=<SID Name>
On Unix check :-
$ORACLE_SID
$LOCAL
If any of these settings are incorrect, reset them to the correct value.
7) Check the program versions
On NT check versions of FND_TOP/bin/FNDLIBR.EXE and AU_TOP/bin/FNDCORE.DLL.
This is achieved by highlighting the program file, then using the right mouse
button to click-->Properties-->Version. Compare this information to that found
in bug 881201 - this patch delivers v11.1.179 of FNDCORE.DLL (pre-req of this
patch is to be on 11.0.3) Alternatively you may need to apply patch 872159
which is FNDCORE.dll v11.0.215
On Unix you can run the following command to determine the version numbers:
'strings -a $FND_TOP\bin\FNDLIBR | grep Header > fndlibr_versions.txt'
This will create a file called 'fndlibr_versions.txt' to send to Oracle Support.
8) Set up SQLNET trace and Database SQL tracing.
a) set TRACE_LEVEL_SERVER=16 in the SQLNET.ORA file on the database server.
b) Set SQL_TRACE=true in the init.ora file
NOTE - the database will need to be restarted for this change to take effect.
It will also generate trace files for every database session, which induces
a great deal of processing overhead as well as a lot of disk writing. Once
the database has been restarted, try starting the Concurrent Manager, then
stop the database and reset 'SQL_TRACE=false' in the init.ora file and
restart the database again.
9) Check the RDBMS version and ensure it is certified; certified combinations
change periodically.
10) Check dual tables: (assuming no-one is in the system and Concurrent Manager processes are not running)
a) select count(*) from sys.dual; There must be one and only one row returned.
b) select count(*) from apps.fnd_dual; There must be at least one row.
11) Check there are no invalid objects in the database. Connect as database
user "system" and run the following query:
SELECT owner, object_name, object_type
FROM dba_objects
WHERE status != 'VALID';
If this returns any rows, we may need to identify why the object is invalid
and get it to compile before proceeding.
12) Check/change Applications profile options.
Check system level profile option 'Concurrent:Use ICM'; by default this should
be set to 'No'. Try changing this value to 'Yes' and see if this changes the
nature of the problem.
13) Check process monitor (PMON) method. Connect as database user "APPS".
SELECT profile_option_value
FROM fnd_profile_option_values
WHERE level_id = 10001
AND level_value = 0
AND application_id = 0
AND profile_option_id =
(SELECT profile_option_id
FROM fnd_profile_options
WHERE profile_option_name = 'CONC_PMON_METHOD');
This should return one row with a value of 'RDBMS' or 'LOCK'
If the value is 'RDBMS' run the script FND_TOP/sql/AFIMPMON.SQL - this will set
the PMON method to LOCK instead of RDBMS.
14) Stop database and restart database server.
If not already shutdown in previous step, stop and restart the database.
If running on NT, restart the NT Server completely.
15) If the above still fails,
if running on NT delete the concurrent manager and recreate from
scratch. This is described in the Applications on NT Installation
manual.
Once the above has been tried, please advise/send in the following information
to Oracle Support:-
a) On NT, versions of FND_TOP/bin/FNDLIBR.EXE and AU_TOP/bin/FNDCORE.DLL.
On Unix run
'strings $FND_TOP/bin/fndlibr| grep Header > fndlibr_version.txt'.
b) SQLNET trace files (svr_*.trc)
c) Database trace files.
d) Confirmation there are no invalid packages for the APPS user.
e) Manager log files.
f) Advise of how many and which Concurrent Manager processes are running.
No comments:
Post a Comment