ADSTRTAL.sh returns error : TIMED OUT( 100000 ): INTERRUPTED EXCEPTION

Oracle Applications Technology Stack - Version 12.0.6 to 12.1.3 [Release 12.0 to 12.1]
Information in this document applies to any platform..

 Goal

When starting Application Server 10g services with EBusiness Suite Release 12, if the server is heavily loaded you may sometimes receive a timeout message from one or more startup scripts called by adstrtal.sh:-
  adoacorectl.sh
  adformsctl.sh
  adoafmctl.sh
This note describes how you can increase the default timeout values for the startup scripts, which may prevent this issue from occurring.

Fix

The OC4J containers (JVMs) with AS10g are given a certain amount of time to startup.  If the time taken to startup the process is longer than this value, you will see an error returned by the startup script.

NOTE : despite the error being thrown, the process may actually startup correctly.  You can use the following command to check the status of the processes:-
     $ADMIN_SCRIPTS_HOME/adopmnctl.sh status

The error seen for a timeout will normally be similar to that shown below:-

****************************************************
        Executing service control script:
        /oracle/VIS/inst/apps/VIS_myhost/admin/scripts/adoafmctl.sh start
        script returned:
****************************************************
        ERROR : Timed out( 100000 ): Interrupted Exception
        You are running adoafmctl.sh version 120.6.12000000.2
        Starting OPMN managed OAFM OC4J instance  ...
****************************************************

The timeout value is defined in the $ORA_CONFIG_HOME/10.1.3/opmn/conf/opmn.xml file   There are three entries to be concerned about, one for each of the managed OC4J containers (oacore, forms and oafm).  The entry is shown below the appropriate "process-type id=" section and looks like shown below by default:-

<start timeout="600" retry="2"/>

Modifying the timeout value from "600" to "1800" may resolve the timeout error

For example you may see entry similar to shown below for the "oacore" JVM group (lines omitted for brevity)
    <process-type id="oacore" module-id="OC4J" status="enabled" working-dir="$ORACLE_HOME/j2ee/home">
      <module-data>
       ...............
       ...............
      </module-data>
      <start timeout="600" retry="2"/>
      <stop timeout="120"/>
      <restart timeout="720" retry="2"/>
       ...............
       ...............

These default timeout values cannot be changed using Autoconfig variables, so any change needs to be implemented using the "Configuration Customizations" section in Note 387859.1 : Using AutoConfig to Manage System Configurations in Oracle Applications Release 12
If you are running into this issue and want to change the timeout as described in this note, these are the recommended steps:
If your problem is occurring in a TEST environment
1.  Shutdown all AS10g services
2.  Manually edit the opmn.xml file as described above
3.  Restart AS10g services to confirm the timeout message is no longer seen
4.  Implement this new timeout value by modifying the AutoConfig template for opmn.xml

Filename "opmn_xml_1013.tmp" located in $FND_TOP/admin/template

5.  Run AutoConfig to regenerate the configuration files with your changes
6.  Check the changes have taken effect in the opmn.xml file correctly
7.  Restart AS10g services and confirm the timeout message no longer appears
If your problem is occurring in a PRODUCTION environment
1.  Shutdown all AS10g services on a TEST environment
2.  Modify the AutoConfig template for opmn.xml for your TEST environment
3.  Run AutoConfig on TEST environment to regenerate the configuration files with your changes
4.  Check the files in TEST environment to be sure the change has taken effect as expected
5.  Restart AS10g Service on TEST environment and ensure the services all startup correctly
6.  Promote this change to your PROD instance

Note: You may see adoacorectl.sh or other OPMN managed scripts fail with message "exiting with status 150"

This may indicate that the script has not started/stopped within the timeout defined for the service control infrastructure

"s_oacoretimeout" is an AutoConfig variable whose value is used internally by the service control infrastructure to determine how much time it should wait before giving up on a script that starts/stops a service.
The above error likely indicates this OACore timeout has been hit. The default is 100 seconds, but can be increased if necessary.
The variables for the OPMN managed processes are:
s_oacoretimeout
s_formstimeout
s_oafmtimeout

For further information on debugging service startup issues see Note 743518.1 "Starting up AS10g services in an EBusiness Suite Release 12 environment"

For additional reading you can review section "3 Starting and Stopping" in the Oracle Application Server Administrator's Guide 10g Release 3 (10.1.3)