PRCD-1027

having issue with all my listeners files missing. Having this issue below:

-bash-4.1$ srvctl config database -d dev1 -a

PRCD-1027 : Failed to retrieve database dev1

PRCR-1115 : Failed to find entities of type resource that match filters ((NAME == ora.dev1.db) && (TYPE == ora.database.type)) and contain attributes VERSION,ORACLE_HOME,DATABASE_TYPE

Unable to Startup Database and Clusterware After Node Reboot When Using NetXen NIC (Doc ID 1507061.1)

APPLIES TO:

Oracle Server - Enterprise Edition - Version 11.2.0.2 to 11.2.0.3 [Release 11.2]
Linux x86-64

SYMPTOMS

* After node reboot, the database is not up, and cannot be started with srvctl:

srvctl start database -d rcjprod
PRCD-1027 : Failed to retrieve database rcjprod
PRCR-1115 : Failed to find entities of type resource that match filters ((NAME == ora.rcjprod.db) && (TYPE == ora.database.type)) and contain attributes VERSION,ORACLE_HOME,DATABASE_TYPE
Cannot communicate with crsd

* Further investigation shows that the clusterware is actually offline:

$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager

* This issue is occurring in an environment where the cluster interconnect is using at least one NetXen NIC.

CAUSE

* The database is not coming up because the clusterware did not come up on boot.

* The clusterware does not come up because of a connectivity problem between nodes:

The Operating System log file does not show any errors but the traceroute is failing between nodes and CVU reports errors like the following:

CRS-10051:CVU found following errors with Clusterware setup : Node "racnode2" is not reachable
PING RACNODE2.US.ORACLE.COM (10.10.50.77) 56(84) bytes of data.
--- RACNODE2.US.ORACLE.COM ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3001ms

* One or more of the interfaces used for the interconnect is a NetXen NIC, eg:

/etc/sysconfig/network-scripts/ifcfg-eth2 File

-rw-r--r-- 3 root root 222  Sep 24 14:14 2012  Dec 19 10:22 2011  ifcfg-eth2

# NetXen Incorporated NX3031 Multifunction 1/10-Gigabit Server Adapter
DEVICE=eth2 ...

 
* the NetXen NIC is using the "netxen_nic" driver.
 

SOLUTION

The cause is a known issue in RHEL 5 with NetXen NIC cards using the 'netxen_nic' driver

https://access.redhat.com/knowledge/solutions/35299

quote from the RedHat document linked above:

* netxen_nic interfaces fail to receive or send packets after boot until they are restarted manually. Issue is seen with both bonding and non-bonding environment.

* When having at least one netxen network card in a bond, the bond doesn't work after boot. Restarting the interface with "ifdown bond0; ifup bond0" works around the problem.

* The system appears to boot without issue and gives no indication of a network issue, however no network connectivity is available when using the netxen_nic at boot. But performing a "service network restart" or 'ifup', it brings up the interfaces and the network works perfectly from this point forward.

Please contact your Linux support to see if a fix is available.