CONCURRENT MANAGER ISSUES IN APPS R12
Status code and Phase code for Concurrent requests
Here is what the abbreviation for status code and phase code means for Concurrent Requests in Oracle Apps
STATUS_CODE Column:
A Waiting
B Resuming
C Normal
D Cancelled
E Error
F Scheduled
G Warning
H On Hold
I Normal
M No Manager
Q Standby
R Normal
S Suspended
T Terminating
U Disabled
W Paused
X Terminated
Z Waiting
B Resuming
C Normal
D Cancelled
E Error
F Scheduled
G Warning
H On Hold
I Normal
M No Manager
Q Standby
R Normal
S Suspended
T Terminating
U Disabled
W Paused
X Terminated
Z Waiting
PHASE_CODE column
C Completed
I Inactive
P Pending
R Running
Thus a request in "Running" "Normal" state will have phase_code "R" and status_code"R"
Technorati Tags: status_code,phase_code,
CP_STATUS_CODE & CP_PHASE_CODE are the lookup_types in the fnd_lookup_values table.concurrent request,apps
Error 1
Issue:
Concurrent Manager showing status “System Hold, Fix Manager before resetting counters”.
Solution:
To implement the solution, please execute the following steps:
1. Stop all middle tier services including the concurrent managers.
Please make sure that no FNDLIBR, FNDSM, or any dead process is
running.
2. Go to cd $FND_TOP/bin
$ adrelink.sh force=y link_debug=y “fnd FNDLIBR”
$ adrelink.sh force=y link_debug=y “fnd FNDFS”
$ adrelink.sh force=y link_debug=y “fnd FNDCRM”
$ adrelink.sh force=y link_debug=y “fnd FNDSM”
3. Run the CMCLEAN.SQL script from the referenced note below (don’t forget to commit).
Note 134007.1 CMCLEAN.SQL – Non Destructive Script to Clean Concurrent Manager Tables
4. Start the concurrent manager.
5. Retest the issue.
Reference :
SCHEDULE/PRERELEASER MANAGER STATUS : SYSTEM HOLD, FIX MANAGER BEFORE RESETTING [ID 985835.1]
***********************************************************
Error 2
The Concurrent requests are in Pending status appears / No Concurrent Manager defined to execute the request
Problem Description :
Shutting down Internal Concurrent Manager : 10-JUL-2012 06:14:04
The visr12_0709@visr12 internal concurrent manager has terminated successfully - exiting.
When I checked the app node for FNDLIBR process, the process was running but OAM console was showing it as terminated. So, obviously something was not right. So I decided to bounce the concurrent manager
cmclean.sql
The Document Id to get the script from metalink is 134007.1
+—————————————————————————+
Start of log messages from FND_FILE
+—————————————————————————+
In GATHER_SCHEMA_STATS , schema_name= ALL percent= 10 degree = 8 internal_flag= NOBACKUP
stats on table FND_CP_GSM_IPC_AQTBL is locked
stats on table FND_SOA_JMS_IN is locked
stats on table FND_SOA_JMS_OUT is locked
Error #1: ERROR: While GATHER_TABLE_STATS:
object_name=GL.JE_BE_LINE_TYPE_MAP***ORA-20001: invalid column name or duplicate columns/column groups/expressions in method_opt***
Error #2: ERROR: While GATHER_TABLE_STATS:
object_name=GL.JE_BE_LOGS***ORA-20001: invalid column name or duplicate columns/column groups/expressions in method_opt***
Error #3: ERROR: While GATHER_TABLE_STATS:
object_name=GL.JE_BE_VAT_REP_RULES***ORA-20001: invalid column name or duplicate columns/column groups/expressions in method_opt***
+—————————————————————————+
End of log messages from FND_FILE
+—————————————————————————+
Action :
To Check which objects or tables are locked
sql > SELECT OWNER,TABLE_NAME,STATTYPE_LOCKED
FROM DBA_TAB_STATISTICS
WHERE STATTYPE_LOCKED IS NOT NULL;
TO Unlock all the tables in a schema at once :
sql> exec dbms_stats.unlock_schema_stats(‘schema_owner’);
e.g : sql> exec dbms_stats.unlock_schema_stats(‘apps’);
TO Unlock all Individual tables in a schema at once
sql> exec dbms_stats.unlock_schema_stats(‘table_owner’,'table_name’);
e.g : sql > exec dbms_stats.unlock_schema_stats(‘AR’,'AR_REV_REC_QT’);
There are two reasons for that error message:
1 ) There are duplicate rows on FND_HISTOGRAM_COLS table for JE_BE_LINE_TYPE_MAP table.
Because of this problem, FND_STATS tries to gather histogram information using wrong command and it fails with ora-20001 errors.
Following SQL should have returned one row, not two.
SQL> select a.column_name, nvl(a.hsize,254) hsize
from FND_HISTOGRAM_COLS a
where table_name = ‘JE_BE_LINE_TYPE_MAP’
order by column_name;
COLUMN_NAME HSIZE
—————————— ———-
SOURCE 254
SOURCE 254
2) Column does not exist on the table but still listed in FND_HISTOGRAMS_COL table.
Solution:
Find out all duplicates and/or obsolete rows in FND_HISTOGRAM_COLS and delete one of them.
Remember to take backup of the FND_HISTOGRAM_COLS table before deleting any data.
– identify duplicate rows
select table_name, column_name, count(*)
from FND_HISTOGRAM_COLS
group by table_name, column_name
having count(*) > 1;
– Use above results on the following SQL to delete duplicates
delete from FND_HISTOGRAM_COLS
where table_name = ‘&TABLE_NAME’
and column_name = ‘&COLUMN_NAME’
and rownum=1;
– Use following SQL to delete obsoleted rows
delete from FND_HISTOGRAM_COLS
where (table_name, column_name) in
(
select hc.table_name, hc.column_name
from FND_HISTOGRAM_COLS hc , dba_tab_columns tc
where hc.table_name =’&TABLE_NAME’
and hc.table_name= tc.table_name (+)
and hc.column_name = tc.column_name (+)
and tc.column_name is null
);
Solution :
Part 1: Verification
1) Please check if table FND_USER_RESP_GROUPS was backed up to table FND_USER_RESP_GROUPS_OLD in the APPLSYS schema, and verify that there is a SYNONYM FND_USER_RESP_GROUPS_OLD in the APPS schema.
OWNER OBJECT_NAME OBJECT_TYPE
2) Check if object FND_USER_RESP_GROUPS is a view or a table
SQL> SELECT OWNER, OBJECT_NAME, OBJECT_TYPE
OWNER OBJECT_NAME OBJECT_TYPE
FND_USER_RESP_GROUPS should be a view in APPS schema.
If FND_USER_RESP_GROUPS is of type TABLE in the APPLSYS schema
please apply solution from step 3 from metalink note ID 335487.1
Else execute following steps from metalink note ID 316959.1
Part 2: Fix
Run following SQLs
2)SELECT count(*)
3) Back up the wf_role_hierarchies table into a new table you create.
Delete the entries in that table (2 rows exist)
TRUNCATE TABLE applsys.wf_role_hierarchies;
Run affurgol.sql FORCE
SQL> @$FND_TOP/patch/115/sql/affurgol.sql FORCE
Then replace those entries back into wf_role_hierarchies table
SQL> INSERT INTO wf_role_hierarchies
Bounce Apache and try to login as sysadmin. You should be able to see the correct responsibilities.
******************************************************************************
Error 8
• Shutdown the apps services
COMMIT;
EXIT;
• Ran AutoConfig on all tiers, firstly on the DB tier and then the APPS tiers.
Actually FND_CONC_CLONE.SETUP_CLEAN clears up the tables with nodes information and when we run autoconfig, it repopulates these tables with correct node information.
*********************************************************************************
1. First check the CM is up or not by using below any one procedure.
2. If CM in down check the internal manager Logfile in $APPLCSF/APPLLOG location for errors.
3. If any errors related to FNDFS then check the Application listener STATUS , if it is not running start it.
4. If CM is up/running then log file and output file are not able see the user from the front end then check the Application listener is status if it is not running then start it.
**********************************************************************************
Error 10
If any user is complained some request is running long time then
ii. Check any Locks on DB particularly related to that program.
iii. Check any INVALID objects related to that program.
iv. Check any incompatible requests/programs are running for that program.
Login as system administrator responsibility and query the long running request /program. Navigate to concurrent program define query that long running request in the same form select incompatible requests .
********************************************************************************
Error 11
Please check the system profile: Concurrent: GSM Enabled. This should be set to ‘Y’
- Execute: afdcm037.sql Path : $FND_TOP/patch/115/sql
Description of the script:
PL/SQL script to create libraries for FNDSM and FNDSVC and create Managers for Preexisting Nodes. Future nodes will be handled by db trigger.
- Relink the below mentioned executables:
**************************************************************************
Error 13
Note : In the case of a clone instance, it is strongly recommended to clean the non – existing nodes
Error 16
Executing afdcm037.sql – Create The Service Manager “FNDSM”
Error 17
Relink the binaries FNDLIBR, FNDSM, FNDFS and FNDCRM
Error 18
Error 19 :
apply dbtechstack option and clone manually
Solution :
Replaced unzip version with 5.5.2 and it worked
Oracle Apps login page is loaded, but the regular sysadmin and front-end user's password is not working(Even though the passwords aren't changed).
Check if all the background processes are running, particularly "mmon" in case of 10g and 11g: ps -ef | grep mmon
Check Alert log of the database to see if there was an errror during database shutdown.
Concurrent Manager showing status “System Hold, Fix Manager before resetting counters”.
Recently after EBS upgrade to 12.1.3 CM were not functional and was showing status “System Hold, Fix Manager before resetting counters”.Issue:
Concurrent Manager showing status “System Hold, Fix Manager before resetting counters”.
Solution:
To implement the solution, please execute the following steps:
1. Stop all middle tier services including the concurrent managers.
Please make sure that no FNDLIBR, FNDSM, or any dead process is
running.
2. Go to cd $FND_TOP/bin
$ adrelink.sh force=y link_debug=y “fnd FNDLIBR”
$ adrelink.sh force=y link_debug=y “fnd FNDFS”
$ adrelink.sh force=y link_debug=y “fnd FNDCRM”
$ adrelink.sh force=y link_debug=y “fnd FNDSM”
3. Run the CMCLEAN.SQL script from the referenced note below (don’t forget to commit).
Note 134007.1 CMCLEAN.SQL – Non Destructive Script to Clean Concurrent Manager Tables
4. Start the concurrent manager.
5. Retest the issue.
Reference :
SCHEDULE/PRERELEASER MANAGER STATUS : SYSTEM HOLD, FIX MANAGER BEFORE RESETTING [ID 985835.1]
***********************************************************
Error 2
The Concurrent requests are in Pending status appears / No Concurrent Manager defined to execute the request
Problem Description :
The Concurrent request are in Pending status appears / No Concurrent Manager defines to execute the request
The submitted concurrent request are in pending state. The logs states, No Concurrent Manager is running to attend the job. When checked the Admin console for concurrent manager, we found that Internal Concurrent Manager and standard manager were not active.
------------------------******************-----------------------------------------
Error from $APPLCSF -
Shutting down Internal Concurrent Manager : 10-JUL-2012 06:14:04
The visr12_0709@visr12 internal concurrent manager has terminated successfully - exiting.
--------------------------*****************------------------------------------------
Possible cause for Inactive OR No Manager
i) No manager is defined to run the request
or
ii) All managers are locked by run-alone requests
or
iii) Concurrent Manager Service is down
or
iv) No concurrent manager process because of workshift
To check Work Shift for any Concurrent Manager From System Administrator responsibility > Concurrent > Manager > Define > Work Shifts > Processes
Solution :
=======
[oracle@appsnode log]$ adcmctl.sh stop apps/apps
-- starting the Concurrent Manager with diag=Y for detailed info [oracle@appsnode log]$ adcmctl.sh start apps/apps diag=Y
[oracle@appsnode log]$ adcmctl.sh status apps/apps
[oracle@appsnode log]$ ps -ef | grep FNDLIBR
***************************************************************
You are running adcmctl.sh version 120.17.12010000.5
Shutting down concurrent managers for visr12 ...
ORACLE Password:
Submitted request 400227 for CONCURRENT FND SHUTDOWN
adcmctl.sh: exiting with status 0
adcmctl.sh: check the logfile /u01/app/oracle/visr12/inst/apps/visr12_appsnode/logs/appl/admin/log/adcmctl.txt for more information ...
You are running adcmctl.sh version 120.17.12010000.5
Starting concurrent manager for visr12 ...
Starting visr12_0710@visr12 Internal Concurrent Manager
Default printer is noprint
adcmctl.sh: exiting with status 0
adcmctl.sh: check the logfile /u01/app/oracle/visr12/inst/apps/visr12_appsnode/logs/appl/admin/log/adcmctl.txt for more information ...
You are running adcmctl.sh version 120.17.12010000.5
Internal Concurrent Manager is Active.
adcmctl.sh: exiting with status 0
[oracle@appsnode log]$ ps -ef | grep FNDLIBR
oracle 7650 7644 0 06:39 pts/1 00:00:03 FNDLIBR
oracle 7924 7759 0 06:55 ? 00:00:02 FNDLIBR
oracle 7925 7759 0 06:55 ? 00:00:02 FNDLIBR
oracle 7926 7759 0 06:55 ? 00:00:02 FNDLIBR
oracle 7931 7759 0 06:55 ? 00:00:00 FNDLIBR
Once started I opened Concurrent Request form, from OAM > system administrator responsibility > concurrent > request option.
Once the form is opened Navigate to +Concurrent > Manager > Administrator and double click on it to open another form. Here you will see Administer Concurrent Manager screen, from which you can identify running or pending requests.
for more detailed logs, pls check $LOG_HOME/appl/conc OR $APPLCSF
*************************************************************
Error 3
Cannot View Concurrent Manager Log And Out Files in R12
Error Message:
'APP-FND-1634-The log file &FILENAME does not exist or this program is restricted from reading it'
If the concurrent request has completed then please contact your system administrator. APP-FND-1636 NO REPORT AVAILABLE FOR THIS CONCURRENT PROCESS.
Cause:
Profile Option 'RRA:enabled' setting is not correct.
Solution:
To implement the solution, please execute the following steps:
1. Shut down managers and verify no FNDLIBR processes are running.
2. Go into the responsibility: System Administrator
3. Navigate to profile/system
4. Search for profile 'RRA:enabled' and set the value to Yes.
5. Save
6. Restart the managers
7. Retest the issue.
1. Shut down managers and verify no FNDLIBR processes are running.
2. Go into the responsibility: System Administrator
3. Navigate to profile/system
4. Search for profile 'RRA:enabled' and set the value to Yes.
5. Save
6. Restart the managers
7. Retest the issue.
Error 4
Cleaning Concurrent Manager Tables
Non Destructive Script to Clean Concurrent Manager Tablescmclean.sql
Ensure that No FNDLIBR processes are running as detailed within the Troubleshooting
Note 104541.1 and that the Concurrent Manageris down.
You need to issue a commit once the script is run for the changes
to take effect.
*******************************************************************************
Error 5
TROUBLESHOOTING CONCURRENT MANAGER STARTUP PROBLEMS
IMPORTANT NOTE - Some of these steps are destructive and should not be
run on a Production system without advice from Oracle Support.
INTRODUCTION
This note is designed to help diagnose and resolve problems starting the
Oracle Applications Concurrent Manager. There can be many issues which
prevent the
CCM from starting and the errors may not always indicate the true problem.
CHECKS TO MAKE
Check the log files for any reported errors. The default name for the Internal Manager log file is 'CM_.log'
on NT or >ManagerName>.mgr on Unix. Also check any other manager
log files in the $APPLCSF/$LOG directory for relevant messages. Do these
errors point to any obvious places to check (e.g. database not running,
TNS alias incorrect)? If these files do not exist at all, there may be
permission problems that prevent processes from writing
files to the proper directories.
2) Ensure that you are logged in as the 'applmgr' (or equivalent) user
on Unix, or if on NT that the service is being started by the 'applmgr'
(or equivalent) user.
3) If you are getting errors such as 'establish ICM unable to insert ICM record
APP-1167 APP-1244 Oracle error 1631 in insert_icm_record' Check your
database's tablespace for the FND tables to ensure you enough free space
to create the next extent. Also check the FND tables to ensure no
table has hit the maximum number of extents.
4) Log in to SQLPLUS as database user "apps".
Verify that view fnd_v$process exists: SELECT view_name
FROM all_views WHERE view_name LIKE '%V$%';
If the view is missing or invalid, rebuild it. From the operating system, as the applmgr user type:
FNDLIBR FND FNDCPBWV applsys/fnd SYSADMIN 'System Administrator' SYSADMIN
or as the applsys user run the following SQL statement
CREATE VIEW fnd_v$process AS
( SELECT P.Addr, P.Pid, S.Process Spid, S.Osuser Username, P.Serial#,
S.Terminal, S.Program, P.Background, P.Latchwait, S.Lockwait FROM
V$process P, V$session S WHERE P.Addr = S.Paddr );
5) Close down all Concurrent Manager processes and clear database tables/log
files. This process will remove all history of Concurrent Jobs and
remove all scheduled jobs. Please contact Oracle Support for advice when
doing this to a Production installation.
Check there are no FNDLIBR, FNDCRM or other Concurrent Manager processes
running. This can be done via the Concurrent Manager-->Administer
screen, or from the operating system - Note 68993.1 has a list of the
process names to look for.
In SQLPLUS run the following SQL TRUNCATE TABLE fnd_concurrent_processes;
UPDATE fnd_concurrent_requests set phase_code = 'C', status_code = 'X'
where status_code = 'T'; UPDATE fnd_concurrent_queues SET
running_processes = 0; SELECT concurrent_queue_name, control_code,
running_processes, max_processes FROM fnd_concurrent_queues;
The control_codes should be 'E' for deactivated managers, blank for all others
except FNDICM which should be 'X'. You will need to update the table
manually if you find any exceptions to these values. For example:-
UPDATE fnd_concurrent_queues SET control_code = 'X' WHERE concurrent_queue_name = 'FNDICM';
c) Delete all log and output files in FND_TOP/log, APPLCSF/log and APPLCSF/out if these files are no longer needed.
***************************************************************************
Error 6 Gather Schema Statistics fails with Ora-20001 errors after 11G database upgrade :
Gather Schema Statistics” program reported following errors in request log files :+—————————————————————————+
Start of log messages from FND_FILE
+—————————————————————————+
In GATHER_SCHEMA_STATS , schema_name= ALL percent= 10 degree = 8 internal_flag= NOBACKUP
stats on table FND_CP_GSM_IPC_AQTBL is locked
stats on table FND_SOA_JMS_IN is locked
stats on table FND_SOA_JMS_OUT is locked
Error #1: ERROR: While GATHER_TABLE_STATS:
object_name=GL.JE_BE_LINE_TYPE_MAP***ORA-20001: invalid column name or duplicate columns/column groups/expressions in method_opt***
Error #2: ERROR: While GATHER_TABLE_STATS:
object_name=GL.JE_BE_LOGS***ORA-20001: invalid column name or duplicate columns/column groups/expressions in method_opt***
Error #3: ERROR: While GATHER_TABLE_STATS:
object_name=GL.JE_BE_VAT_REP_RULES***ORA-20001: invalid column name or duplicate columns/column groups/expressions in method_opt***
+—————————————————————————+
End of log messages from FND_FILE
+—————————————————————————+
Action :
To Check which objects or tables are locked
sql > SELECT OWNER,TABLE_NAME,STATTYPE_LOCKED
FROM DBA_TAB_STATISTICS
WHERE STATTYPE_LOCKED IS NOT NULL;
TO Unlock all the tables in a schema at once :
sql> exec dbms_stats.unlock_schema_stats(‘schema_owner’);
e.g : sql> exec dbms_stats.unlock_schema_stats(‘apps’);
TO Unlock all Individual tables in a schema at once
sql> exec dbms_stats.unlock_schema_stats(‘table_owner’,'table_name’);
e.g : sql > exec dbms_stats.unlock_schema_stats(‘AR’,'AR_REV_REC_QT’);
There are two reasons for that error message:
1 ) There are duplicate rows on FND_HISTOGRAM_COLS table for JE_BE_LINE_TYPE_MAP table.
Because of this problem, FND_STATS tries to gather histogram information using wrong command and it fails with ora-20001 errors.
Following SQL should have returned one row, not two.
SQL> select a.column_name, nvl(a.hsize,254) hsize
from FND_HISTOGRAM_COLS a
where table_name = ‘JE_BE_LINE_TYPE_MAP’
order by column_name;
COLUMN_NAME HSIZE
—————————— ———-
SOURCE 254
SOURCE 254
2) Column does not exist on the table but still listed in FND_HISTOGRAMS_COL table.
Solution:
Find out all duplicates and/or obsolete rows in FND_HISTOGRAM_COLS and delete one of them.
Remember to take backup of the FND_HISTOGRAM_COLS table before deleting any data.
– identify duplicate rows
select table_name, column_name, count(*)
from FND_HISTOGRAM_COLS
group by table_name, column_name
having count(*) > 1;
– Use above results on the following SQL to delete duplicates
delete from FND_HISTOGRAM_COLS
where table_name = ‘&TABLE_NAME’
and column_name = ‘&COLUMN_NAME’
and rownum=1;
– Use following SQL to delete obsoleted rows
delete from FND_HISTOGRAM_COLS
where (table_name, column_name) in
(
select hc.table_name, hc.column_name
from FND_HISTOGRAM_COLS hc , dba_tab_columns tc
where hc.table_name =’&TABLE_NAME’
and hc.table_name= tc.table_name (+)
and hc.column_name = tc.column_name (+)
and tc.column_name is null
);
*****************************************************************************
Error 7
Problem Statement: Recently I encountered a problem in Oracle Apps 11i (11.5.10.2). When a user login into the application he/she gets error“There are no active responsibilities available for this user “and user is not able to see any responsibility in the home page.
The problem is same even for sysadmin user. Now the challenge is that, if the system administrator is also not able to see any responsibility then its very difficult to administer other users facing same problem. So here is the solution that I found out as per metalink note ID 316959.1
Solution :
Part 1: Verification
Execute following queries for verification as per metalink note ID 335487.1
1) Please check if table FND_USER_RESP_GROUPS was backed up to table FND_USER_RESP_GROUPS_OLD in the APPLSYS schema, and verify that there is a SYNONYM FND_USER_RESP_GROUPS_OLD in the APPS schema.
SELECT OWNER, OBJECT_NAME, OBJECT_TYPE
FROM ALL_OBJECTS
WHERE OBJECT_NAME LIKE ’FND_USER_RESP_GROUPS_OLD’
AND OBJECT_TYPE IN (‘VIEW’,’TABLE’, ‘SYNONYM’);
OWNER OBJECT_NAME OBJECT_TYPE
—————————— —————————— ——————
APPLSYS FND_USER_RESP_GROUPS_OLD TABLE
APPS FND_USER_RESP_GROUPS_OLD SYNONYM
2) Check if object FND_USER_RESP_GROUPS is a view or a table
SQL> SELECT OWNER, OBJECT_NAME, OBJECT_TYPE
FROM ALL_OBJECTS WHERE OBJECT_NAME LIKE ‘FND_USER_RESP_GROUPS’;
OWNER OBJECT_NAME OBJECT_TYPE
—————————— —————————— ——————
APPS FND_USER_RESP_GROUPS VIEW
APPS_MRC FND_USER_RESP_GROUPS SYNONYM
RTREADONLY FND_USER_RESP_GROUPS SYNONYM
FND_USER_RESP_GROUPS should be a view in APPS schema.
If FND_USER_RESP_GROUPS is of type TABLE in the APPLSYS schema
please apply solution from step 3 from metalink note ID 335487.1
Else execute following steps from metalink note ID 316959.1
Part 2: Fix
Run following SQLs
1)SELECT *
FROM Fnd_Responsibility_vl
WHERE Application_Id = 1
AND Responsibility_Name = ‘System Administrator’
AND Start_Date <= Sysdate
AND ( End_Date is NULL OR End_Date > Sysdate );
2)SELECT count(*)
FROM Fnd_User_Resp_Groups;
3) Back up the wf_role_hierarchies table into a new table you create.
CREATE TABLE wf_role_hierarchies_copy AS
SELECT * FROM wf_role_hierarchies;
Delete the entries in that table (2 rows exist)
TRUNCATE TABLE applsys.wf_role_hierarchies;
Run affurgol.sql FORCE
SQL> @$FND_TOP/patch/115/sql/affurgol.sql FORCE
Then replace those entries back into wf_role_hierarchies table
SQL> INSERT INTO wf_role_hierarchies
SELECT * FROM wf_role_hierarchies_copy;
Bounce Apache and try to login as sysadmin. You should be able to see the correct responsibilities.
******************************************************************************
Error 8
CM not starting after Cloning ?
Once after doing Rapid cloning some of concurrent managers were not starting. The issue was that the node was wrongly defined as source node in Administer Manager screen. I then checked the FND_NODES table which wrongly had the source nodes as well apart from the target nodes.
I performed the following action plan which resolved the issue
• Shutdown the apps services
• EXEC FND_CONC_CLONE.SETUP_CLEAN;
COMMIT;
EXIT;
• Ran AutoConfig on all tiers, firstly on the DB tier and then the APPS tiers.
• Start the apps services
Actually FND_CONC_CLONE.SETUP_CLEAN clears up the tables with nodes information and when we run autoconfig, it repopulates these tables with correct node information.
*********************************************************************************
Error 9
Concurrent manager trouble shooting :
1. First check the CM is up or not by using below any one procedure.
i. ps –ef|grep FNDLIBR
ii. adcmctl.sh status apps/appsPWD
iii. Login as system administrator responsibilities go to the below navigation Concurrent -> manager -> Administer -> see the Actual and target for all the managers
2. If CM in down check the internal manager Logfile in $APPLCSF/APPLLOG location for errors.
3. If any errors related to FNDFS then check the Application listener STATUS , if it is not running start it.
4. If CM is up/running then log file and output file are not able see the user from the front end then check the Application listener is status if it is not running then start it.
**********************************************************************************
Error 10
If any user is complained some request is running long time then
i. First check the CM status and ICM status.
ii. Check any Locks on DB particularly related to that program.
iii. Check any INVALID objects related to that program.
iv. Check any incompatible requests/programs are running for that program.
Login as system administrator responsibility and query the long running request /program. Navigate to concurrent program define query that long running request in the same form select incompatible requests .
If any incompatible request is running then find the user name who is running that program inform them regarding the incompatibilities.
********************************************************************************
Error 11
Concurrent Managers issue after clone in 11i
Concurrent managers are not coming up after the clone, having the below mentioned error in the manager log :
Starting STANDARD Concurrent Manager : 28-FEB-2008 08:25:59 Could not initialize the Service Manager FNDSM_MINEDEV_TESTMINE. Verify that mineDEV has been registered for concurrent processing. Routine AFPEIM encountered an error while starting concurrent manager STANDARD with library /d11/app/appltst/mineappl/fnd/11.5.0/bin/FNDLIBR. Check that your system has enough resources to start a concurrent manager process. Contact your system adminis : 28-FEB-2008 08:26:00
Soultion :
Perform the following mentioned steps to resolve the issue:
1. Stop the Internal Concurrent Manager.
2. Connect to the database via SQL*Plus as the APPS user.
3. Execute the following to alter the FNDSM trigger on FND_NODES:
CREATE OR REPLACE TRIGGER fndsm
AFTER INSERT OR UPDATE ON FND_NODES
FOR EACH ROW
BEGIN
if ( :new.NODE_NAME <> 'AUTHENTICATION' ) then
if ( (:new.SUPPORT_CP='Y')
or (:new.SUPPORT_FORMS='Y')
or (:new.SUPPORT_WEB='Y') ) then
fnd_cp_fndsm.register_fndsm_fcq(:new.NODE_NAME);
end if;
if (:new.SUPPORT_CP = 'Y') then
fnd_cp_fndsm.register_fndim_fcq(:new.NODE_NAME);
end if;
end if;
END;
/
4. Ensure to commit the changes.
5. Cleanup the FND_NODES table by executing the following:
SQL> exec FND_CONC_CLONE.SETUP_CLEAN;
6. Run AutoConfig on each node.
Error 12
Concurrent managers are down with status: “System Hold, Fix Manager
All the concurrent managers are down with a status message: system hold, fix manager.
The administer Concurrent manager screen has Actual: 0 and Target: 1 for all managers.
Solution:
Ensure all application services are down.
adrelink.sh force=y “fnd FNDLIBR”
adrelink.sh force=y “fnd FNDSM”
adrelink.sh force=y “fnd FNDFS”
adrelink.sh force=y “fnd FNDCRM”
- Execute cmclean.sql to clean out the concurrent manager tables.
- Start all the application services and restest the issue.
- Start all the application services and restest the issue.
**************************************************************************
Error 13
Issue: Concurrent request running from a very long time
Application version: 11.5.10.2 multi-node (4 nodes)
Database: 11g RAC
Scenario:
Business critical concurrent request is running from more than 2 hours which was supposed to complete within 1 hour.
The standard manager and the manager responsible to execute the critical concurrent request (custom manager) were overloaded with many requests. The server on which these managers run had very high server load and all CPU’s were utilized.
After diagnosis, we had to kill few requests which were running from more than 2 hours and were creating load on the server. These requests were in fact not performing anything and had gone zombie.
The server load came down and CPU’s available were set free to execute new requests/processes.
However the business critical request was still running and wasn’t moving ahead at all. The database session associated was also not executing anything.
Challenge:
Since this being a critical business request, we couldn’t terminate the request coz we had already lost few hours running it. At the same time the database session also wasn’t performing any execution and the whole request was not going anywhere even though it showed running normal status.
Trick:
Not recommended, however can be a life saver at times.
---- Forcibly change the status of the request to pending normal.
---- Kill the database session of the request.
Use the below scripts:
update applsys.fnd_concurrent_requests set phase_code = 'P',
status_code = 'I',
actual_start_date = null,
crm_release_date = null,
controlling_manager=null,
logfile_name = null,
logfile_node_name = null,
outfile_name = null,
outfile_node_name = null,
crm_tstmp = null
where request_id in ('request-id');
alter system kill session 'sid, serial#, @instance-id';
Hence a new database session will get created as soon as the old one is killed and the request starts running. Since the server on which the request is supposed to get processed also had no issues now, the request went through very fast without any issues ;)
Database: 11g RAC
Scenario:
Business critical concurrent request is running from more than 2 hours which was supposed to complete within 1 hour.
The standard manager and the manager responsible to execute the critical concurrent request (custom manager) were overloaded with many requests. The server on which these managers run had very high server load and all CPU’s were utilized.
After diagnosis, we had to kill few requests which were running from more than 2 hours and were creating load on the server. These requests were in fact not performing anything and had gone zombie.
The server load came down and CPU’s available were set free to execute new requests/processes.
However the business critical request was still running and wasn’t moving ahead at all. The database session associated was also not executing anything.
Challenge:
Since this being a critical business request, we couldn’t terminate the request coz we had already lost few hours running it. At the same time the database session also wasn’t performing any execution and the whole request was not going anywhere even though it showed running normal status.
Trick:
Not recommended, however can be a life saver at times.
---- Forcibly change the status of the request to pending normal.
---- Kill the database session of the request.
Use the below scripts:
update applsys.fnd_concurrent_requests set phase_code = 'P',
status_code = 'I',
actual_start_date = null,
crm_release_date = null,
controlling_manager=null,
logfile_name = null,
logfile_node_name = null,
outfile_name = null,
outfile_node_name = null,
crm_tstmp = null
where request_id in ('request-id');
alter system kill session 'sid, serial#, @instance-id';
Hence a new database session will get created as soon as the old one is killed and the request starts running. Since the server on which the request is supposed to get processed also had no issues now, the request went through very fast without any issues ;)
***********************************************************************************
Error 14
1) Running FND_CONC_CLONE.SETUP_CLEAN
Note : In the case of a clone instance, it is strongly recommended to clean the non – existing nodes
FND_NODE table contains the node entries of both SOURCE environment and TARGET environment after clone.
If u want to get rid of the source node entries
a. Stop all middle tier services including the concurrent managers.
b. Stop the database.
c. Start the database.
d. Connect SQLPLUS as APPS user and run the following :
EXEC FND_CONC_CLONE.SETUP_CLEAN;
COMMIT;
EXIT;
e. Run AutoConfig on all tiers, firstly on the DB tier and then the APPS tiers and webtiers to repopulate the required system tables.
Error 16
Executing afdcm037.sql – Create The Service Manager “FNDSM”
NOTE: This script will create libraries for FNDSM and create managers for preexisting nodes.
If Service Manager is not running , To create the Service Manager “FNDSM” Please perform below steps.
a. Log in as applmgr
b. cd to $FND_TOP/patch/115/sql
c. Run the script: afdcm037.sql
Check again that FNDSM entries now exist: select CONCURRENT_QUEUE_NAME from FND_CONCURRENT_QUEUES where CONCURRENT_QUEUE_NAME like ‘FNDSM%’;
Error 17
Relink the binaries FNDLIBR, FNDSM, FNDFS and FNDCRM
It is better to relonk the binaries once, If Concurrent Managers do not start for the cloned Instance.
Go to cd $FND_TOP/bin
$ adrelink.sh force=y “fnd FNDLIBR”
$ adrelink.sh force=y “fnd FNDSM”
$ adrelink.sh force=y “fnd FNDFS”
$ adrelink.sh force=y “fnd FNDCRM”
Error 18
Run CMCLEAN.sql, To Clean Concurrent Manager Tables.
Cleaning out the Concurrent Manager tables is a useful method of making sure that there are no invalid statuses that can prevent the managers from starting.
a. Log in as APPS user
b. Run the script: cmclean.sql
c. COMMIT;
Error 19 :
How do you start/stop/check Concurrent Managers?
Solution:
cd $COMMON_TOP/admin/scripts/context_name/
-> adcmctl.sh start apps/appspwd
-> adcmctl.sh stop apps/appspwd
-> adcmctl.sh status apps/appspwd
-> ps -ef grep FNDLIBR grep applmgr
Error 20
How do you create custom concurrent manager?
Solution:
1. Login to System Administrator Responsibility
2. Navigate to Concurrent > Manager > Define
Manager Field: Custom Manager-
Short Name: CUSTOMCM-
Type: Concurrent Manager-
Program Library: FNDLIBR-
Enter desired Cache-
Work Shifts: Standard-
Enter number of Processes-
Provide Specialization Rules- Save
3. Navigate to Concurrent > Manager > Administer- Activate the Custom Manager
Error 21
How to Start the Concurrent Manager from the Operating system?
solution:
startmgr [parameters]
Example:
startmgr sysmgr="applsys/fnd" mgrname="std"
printer="hqseq1"mailto="jsmith" restart="N" logfile="mgrlog" sleep="90"
pmon="5" quesiz="10"
Parameters:
[sysmgr="fnd_usernamd/fnd_password"] [mgrname="mgrname"]
[printer=printer]
[mailto="userid1 userid2...]
[restart="Nminutes"]
[logfile="log_file_name"]
[sleep="new_check"]
[pmon="manager_check"]
[quesiz="number_check"]
[diag="YN"]
Error 22 :
EachConcurrent Request Phase and Status Meaning?
Solution:
Phase Status Description
PENDING Normal Request is waiting for the next available manager.
PENDING Standby Program to run request is incompatible with other program(s) currently running.
PENDING Scheduled Request is scheduled to start at a future time or date.
PENDING
Waiting A child request is waiting for its Parent request to mark it
ready to run. For example, a request in a request set that runs
sequentially must wait for a prior request to complete.
RUNNING Normal Request is running normally.
RUNNING
Paused Parent request pauses for all its child requests to finish
running. For example, a request set pauses for all requests in the set
to complete.
RUNNING Resuming All requests submitted by the same parent request have completed running. The Parent request resumes running.
RUNNING Terminating Request is terminated by choosing the Cancel Request button in Requests window.
COMPLETED Normal Request completed successfully.
COMPLETED Error Request failed to complete successfully.
COMPLETED Warning Request completed with warnings. For example, a request is generated successfully but fails to print.
COMPLETED Cancelled Pending or Inactive request is cancelled by choosing the Cancel Request button in the Requests window.
COMPLETED Terminated Request is terminated by choosing the Cancel Request button in the Requests window.
INACTIVE Disabled Program to run request is not enabled. Contact your system administrator.
INACTIVE On Hold Pending request is placed on hold by choosing the Hold Request button in the Requests window.
INACTIVE
No Manager No manager is defined to run the request. Check with your
system administrator. A status of No Manager is also given when all
managers are locked by run-alone requests.
Error 23 :
Oracle CM - Standard Manager not Starting up and giving below issue:
Routine AFPESA cannot construct the name of an executable file for REQID
Solution:
Add Custom Top in the ENV file and restart CM
Error 24:
Cloning issue after adcfgclone perl dbTier
RC-50004: Fatal: Error occurred in ApplyDatabase:
Control file creation failed
Solution :
RC-50004: Fatal: Error occurred in ApplyDatabase:
Control file creation failed
Solution :
------------
apply dbtechstack option and clone manually
Error 25:
RC-20200: Fatal -- Unzip issue - -add unzip to the path
Solution :
----------
Replaced unzip version with 5.5.2 and it worked
Error 25:
APP-FND-01564 ORACLE error 942 in afpoload
Cause :
Cause :
=====
apps user accnt locked
apps user accnt locked
Solution :
======
unlock the user of "apps"
alter user apps account unlock;
Error 26:
Oracle Apps login page is loaded, but the regular sysadmin and front-end user's password is not working(Even though the passwords aren't changed).
Check if all the background processes are running, particularly "mmon" in case of 10g and 11g: ps -ef | grep mmon
Check Alert log of the database to see if there was an errror during database shutdown.
*********************************************************************
Error :27
Issue:-Concurrent managers are not starting.
Symptoms :- Below error code in ICM Log.
"Cause: FDUUID failed due to ORA-08004: sequence FND_CONCURRENT_PROCESSES_S.NEXTVAL exceeds MAXVALUE and cannot be instantiated."
Diagnostics:-
SQL> select max(CONCURRENT_PROCESS_ID) from FND_CONCURRENT_PROCESSES;
MAX(CONCURRENT_PROCESS_ID)
--------------------------
9999998
SQL> select FND_CONCURRENT_PROCESSES_S.nextval from dual;
select FND_CONCURRENT_PROCESSES_S.nextval
*
ERROR at line 1:
ORA-08004: sequence FND_CONCURRENT_PROCESSES_S.NEXTVAL exceeds MAXVALUE and
cannot be instantiated
SQL> select name,user from v$database;
NAME USER
--------- ------------------------------
THCUCI APPS
SQL> select * from dba_sequences where SEQUENCE_NAME like'FND_CONCURRENT_PROCESSES_S';
SEQUENCE_OWNER SEQUENCE_NAME MIN_VALUE
----------------------- ------------------- ----------
MAX_VALUE INCREMENT_BY C O CACHE_SIZE LAST_NUMBER
---------- ------------ - - ---------- -----------
APPLSYS FND_CONCURRENT_PROCESSES_S 1
9999999 1 N N 0 10000000
Solution:-
SQL> ALTER SEQUENCE APPLSYS.fnd_concurrent_processes_s MAXVALUE 99999999;
Sequence altered.
No comments:
Post a Comment