How to Apply patch on Oracle RAC

 Prerequisite:-

1: Check Oracle service status

[oracle@rac-node1 ~]$crs_stat -t
Name           Type         Target State    Host 
------------------------------------------------------------
ora....SM1.asm application ONLINE ONLINE dbsr...ode1 
ora....E1.lsnr application ONLINE ONLINE dbsr...ode1 
ora....de1.gsd application ONLINE ONLINE dbsr...ode1 
ora....de1.ons application ONLINE ONLINE dbsr...ode1 
ora....de1.vip application ONLINE ONLINE dbsr...ode1 
ora....SM2.asm application ONLINE ONLINE dbsr...ode2 
ora....E2.lsnr application ONLINE ONLINE dbsr...ode2 
ora....de2.gsd application ONLINE ONLINE dbsr...ode2 
ora....de2.ons application ONLINE ONLINE dbsr...ode2 
ora....de2.vip application ONLINE ONLINE dbsr...ode2 
ora.prod.db application    ONLINE ONLINE dbsr...ode1 
ora....b1.inst application ONLINE ONLINE dbsr...ode1 
ora....b2.inst application ONLINE ONLINE dbsr...ode2 
[oracle@rac-node1 ~]$

2: Stop database instance one by one

[oracle@rac-node1 ~]$ srvctl stop instance -i prod1 -d prod
[oracle@rac-node1 ~]$ srvctl stop instance -i prod2 -d prod

3: Stop ASM instance on each node one by one

[oracle@rac-node1 ~]$ srvctl stop asm -n rac-node1
[oracle@rac-node1 ~]$ srvctl stop asm -n rac-node2

4: Stop node applications one by one on each node

[oracle@rac-node1 ~]$ srvctl stop nodeapps -n rac-node1
[oracle@rac-node1 ~]$ srvctl stop nodeapps -n rac-node2

5: Check status again

[oracle@rac-node1 ~]$ crs_stat -t

 Name             Type       Target  State Host 
------------------------------------------------------------
ora....SM1.asm application OFFLINE OFFLINE 
ora....E1.lsnr application OFFLINE OFFLINE 
ora....de1.gsd application OFFLINE OFFLINE 
ora....de1.ons application OFFLINE OFFLINE 
ora....de1.vip application OFFLINE OFFLINE 
ora....SM2.asm application OFFLINE OFFLINE 
ora....E2.lsnr application OFFLINE OFFLINE 
ora....de2.gsd application OFFLINE OFFLINE 
ora....de2.ons application OFFLINE OFFLINE 
ora....de2.vip application OFFLINE OFFLINE 
ora.prod.db application    OFFLINE OFFLINE 
ora....b1.inst application OFFLINE OFFLINE 
ora....b2.inst application OFFLINE OFFLINE 

 [oracle@rac-node1 ~]

Download Patch from Metalink.

Unzip patch folder after that read readme file.

Go to Opatch top.

[oracle@rac-node1 rman]$ cd 9949948
OPatch should be in your path as follows

Set Oracle_Home.

 [oracle@rac-node1 9949948]$ export PATH=$PATH:/u01/app/oracle/product/10.2.0 /db_1/OPatch

Start apply Opatch

[oracle@rac-node1 9949948]$ opatch apply
Invoking OPatch 10.2.0.4.9
Oracle Interim Patch Installer version 10.2.0.4.9
Copyright (c) 2009, Oracle Corporation. All rights reserved.

Oracle Home : /u01/app/oracle/product/10.2.0/db_1
Central Inventory : /u01/app/oracle/OraInventory
 from : /etc/oraInst.loc
OPatch version : 10.2.0.4.9
OUI version : 10.2.0.5.0
OUI location : /u01/app/oracle/product/10.2.0/db_1/oui
Log file location : /u01/app/oracle/product/10.2.0/db_1/cfgtoollogs/opatch/opatch2012-09-20_23-38-29PM.log
Patch history file: /u01/app/oracle/product/10.2.0/db_1/cfgtoollogs/opatch/opatch_history.txt
ApplySession applying interim patch '9949948' to OH '/u01/app/oracle/product/10.2.0/db_1'
Running prerequisite checks...
OPatch detected the node list and the local node from the inventory. OPatch will patch the local system then propagate the patch to the remote nodes.

This node is part of an Oracle Real Application Cluster.
Remote nodes: 'rac-node2' 
Local node: 'rac-node1'
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/oracle/product/10.2.0/db_1')

Is the local system ready for patching? [y|n]

Y  ==> I put yes

User Responded with: Y
Backing up files and inventory (not for auto-rollback) for the Oracle Home
Backing up files affected by the patch '9949948' for restore. This might take a while...
Backing up files affected by the patch '9949948' for rollback. This might take a while...
Patching component oracle.rdbms, 10.2.0.5.0...
Updating archive file "/u01/app/oracle/product/10.2.0/db_1/lib/libserver10.a" with "lib/libserver10.a/ksfd.o"
Running make for target ioracle
ApplySession adding interim patch '9949948' to inventory
Verifying the update...
Inventory check OK: Patch ID 9949948 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 9949948 are present in Oracle Home.
The local system has been patched. You can restart Oracle instances on it.

Patching in rolling mode.

The node 'rac-node2' will be patched next.

Please shutdown Oracle instances running out of this ORACLE_HOME on 'rac-node2'.
(Oracle Home = '/u01/app/oracle/product/10.2.0/db_1')
Is the node ready for patching? [y|n]

y  ==> I put yes for node2

User Responded with: Y
Updating nodes 'rac-node2' 
 Apply-related files are:
 FP = "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/9949948_Aug_13_2010_06_35_20/rac/copy_files.txt"
 DP = "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/9949948_Aug_13_2010_06_35_20/rac/copy_dirs.txt"
 MP = "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/9949948_Aug_13_2010_06_35_20/rac/make_cmds.txt"
 RC = "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/9949948_Aug_13_2010_06_35_20/rac/remote_cmds.txt"
Instantiating the file "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/9949948_Aug_13_2010_06_35_20/rac/copy_files.txt.instantiated" by replacing $ORACLE_HOME in "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/9949948_Aug_13_2010_06_35_20/rac/copy_files.txt" with actual path.
Propagating files to remote nodes...
Instantiating the file "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/9949948_Aug_13_2010_06_35_20/rac/copy_dirs.txt.instantiated" by replacing $ORACLE_HOME in "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/9949948_Aug_13_2010_06_35_20/rac/copy_dirs.txt" with actual path.
Propagating directories to remote nodes...
Instantiating the file "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/9949948_Aug_13_2010_06_35_20/rac/make_cmds.txt.instantiated" by replacing $ORACLE_HOME in "/u01/app/oracle/product/10.2.0/db_1/.patch_storage/9949948_Aug_13_2010_06_35_20/rac/make_cmds.txt" with actual path.
Running command on remote node 'rac-node2': 
cd /u01/app/oracle/product/10.2.0/db_1/rdbms/lib; /usr/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/u01/app/oracle/product/10.2.0/db_1 || echo REMOTE_MAKE_FAILED::>&2

The node 'rac-node2' has been patched. You can restart Oracle instances on it.
There were relinks on remote nodes. Remember to check the binary size and timestamp on the nodes 'rac-node2' .
The following make commands were invoked on remote nodes:
'cd /u01/app/oracle/product/10.2.0/db_1/rdbms/lib; /usr/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/u01/app/oracle/product/10.2.0/db_1
'
OPatch succeeded.

Start Oracle services on both nodes

[oracle@rac-node1 9949948]$ srvctl start nodeapps -n rac-node1
[oracle@rac-node1 9949948]$ srvctl start nodeapps -n rac-node2
[oracle@rac-node1 9949948]$ srvctl start asm -n rac-node1
[oracle@rac-node1 9949948]$ srvctl start asm -n rac-node2
[oracle@rac-node1 9949948]$ crs_stat -t
Name Type Target State Host 
------------------------------------------------------------
ora....SM1.asm application ONLINE ONLINE dbsr...ode1 
ora....E1.lsnr application ONLINE ONLINE dbsr...ode1 
ora....de1.gsd application ONLINE ONLINE dbsr...ode1 
ora....de1.ons application ONLINE ONLINE dbsr...ode1 
ora....de1.vip application ONLINE ONLINE dbsr...ode1 
ora....SM2.asm application ONLINE ONLINE dbsr...ode2 
ora....E2.lsnr application ONLINE ONLINE dbsr...ode2 
ora....de2.gsd application ONLINE ONLINE dbsr...ode2 
ora....de2.ons application ONLINE ONLINE dbsr...ode2 
ora....de2.vip application ONLINE ONLINE dbsr...ode2 
ora.prod.db application OFFLINE OFFLINE 
ora....b1.inst application OFFLINE OFFLINE 
ora....b2.inst application OFFLINE OFFLINE

Start database instances on both nodes:-

[oracle@rac-node1 9949948]$ srvctl start instance -i prod1 -d prod
[oracle@rac-node1 9949948]$ srvctl start instance -i prod1 -d prod
[oracle@rac-node1 9949948]$ crs-stat -t

 Name            Type       Target State     Host 
------------------------------------------------------------
ora....SM1.asm application ONLINE ONLINE dbsr...ode1 
ora....E1.lsnr application ONLINE ONLINE dbsr...ode1 
ora....de1.gsd application ONLINE ONLINE dbsr...ode1 
ora....de1.ons application ONLINE ONLINE dbsr...ode1 
ora....de1.vip application ONLINE ONLINE dbsr...ode1 
ora....SM2.asm application ONLINE ONLINE dbsr...ode2 
ora....E2.lsnr application ONLINE ONLINE dbsr...ode2 
ora....de2.gsd application ONLINE ONLINE dbsr...ode2 
ora....de2.ons application ONLINE ONLINE dbsr...ode2 
ora....de2.vip application ONLINE ONLINE dbsr...ode2 
ora.prod.db application    ONLINE ONLINE dbsr...ode2 
ora....b1.inst application ONLINE ONLINE dbsr...ode1 
ora....b2.inst application ONLINE ONLINE dbsr...ode2

[oracle@rac-node1 script]$ sqlplus /nolog
SQL*Plus: Release 10.2.0.5.0 – Production on Thu Sep 20 23:44:01 2012
Copyright (c) 1982, 2010, Oracle. All Rights Reserved.

SQL>conn /as sysdba
Connected.
SQL>