ORA-16198

ORA-16198 caused by slow flashback log deletion on the physical standby [ID 1163385.1]

Applies to:

Oracle Server - Enterprise Edition - Version: 10.1.0.2 to 11.2.0.1 - Release: 10.1 to 11.2
Information in this document applies to any platform.

Symptoms

ORA-16198 raised on the primary instance of a physical standby configuration.

Cause

The problem was caused by an ARCn process on the standby holding the controlfile transaction (CF) enqueue for extended periods, blocking RFS processes from acquiring the enqueue. This in turn lead to delays writing to the standby redo logs, and eventually an ORA-16198 on the primary because net_timeout was exceeded.

Solution

The ORA-16198 error can be caused by a number of problems. In this case a script was run which sampled V$LOCK and detected any process holding the CF enqueue for more than 2 seconds. Theses processes' stacks were then dumped, and showed the following (excerpt):

...krff_remove_fb_log<-krfgdelfb1<-krareclaim<-krasreclaim...

From this it was deduced that the process was trying to reclaim space in the FRA. We could then cross-reference the time of the ORA-16198 with the standby alert log. The standby alert log showed a large number of flashback logs being deleted from the flash recovery area around this time.

After first testing the theory that the problem was due to flashback log deletion (by disabling flashback database on the standby), the patch for bug 8834425 was applied. This solved the problem.