Blackout configuration on Oracle Enterprise Manager

How to Configure Blackout on OEM???

Blackouts with EMCTL

I have a need to take down a database to perform a cold snapshot of the database’s files. This database is monitored by Enterprise Manager and I do not want to receive an alert when the database is down. So I need to figure out how to start a blackout in Enterprise Manager from the command line which will be included in our backup script.

On the database server, do the following:

cd $AGENT_HOME/bin

./emctl start blackout TestBlackoutEMCTL fgodba.acme.com

Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
Blackout TestBlackoutEMCTL added successfully
EMD reload completed successfully

We can verify the blackout is in effect in Enterprise Manager. Sign on to EM and click on Setup -> Blackouts. You should see something similar to the following:

Select the blackout radio button and click the View button.

We can verify that the blackout name contains the string that I denoted when I started the blackout with the date and time appended to it.
We can also query the status of the blackout on the command line:
./emctl status blackout
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
Blackoutname = TestBlackoutEMCTL
Targets = (fgodba.acme.com:oracle_database,)
Time = ({2012-04-26|13:27:32,|} )
Expired = False

It is now time to end the blackout.

./emctl stop blackout TestBlackoutEMCTL
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
Blackout TestBlackoutEMCTL stopped successfully
EMD reload completed successfully

Setting EM Blackouts from the GUI and Command Line

Oracle Enterprise Manager provides you with the ability to monitor your environments and alert you once specified thresholds have been reached.  Blackouts allow you to suspend monitoring so you do not get notified.   This is useful for scheduled maintenance windows, such as cold backups, where the application and/or database may not be available.

As well, blackouts also suspends data collection for the given targets.  This means that certain metrics such as availability will not be affected.

To create a blackout from the GUI, login to Enterprise Manager, navigate to the target you would like to blackout and at the bottom of the page under Related Links you will see a Blackouts link.  You will be brought to the following page:





On the screen above you can view any blackouts that may currently be in effect as well as create new ones.  To create a new blackout click on the create button.



On this page you can create a name for the blackout, with the default being “Blackout-<timestamp>”.  You can also select the targets you wish to set the blackout for.   As you can see from the screenshot I am going to set a blackout for an infrastructure application server (infra10g).

You can also provide a reason for the blackout by click on the Reason drop down list.  Quite a few are available, from Server Bounce toSecurity Patch.    Jobs can be disabled by deselecting the Run jobs during the blackout checkbox.  If your applying a security patch then you may not want a scheduled backup to run as it will either error or cause problems.

The next screen allows you to select which components within the target will be blacked out.  
I selected a full blackout but you can select certain members if the outage will only affect specific components.



This screen allows you to schedule the blackout.  It can either start immediately or you can choose a date along with a duration.  Blackouts can be repeating as well, so you only have to create one for that monthly maintenance window for example.



The last screen provides a summary and once you have finished reviewing click on the Finish button.

You can also set blackouts from the command line, which is useful if you have some maintenance scripts which are not executed from Enterprise Managers job system.   I’ve only tested this on linux but it should be the same for windows.

To set a blackout for a list of targets:

emctl start blackout <Blackoutname> [<Target_name>[:<Target_Type>]]…. [-d Duration]

To set a blackout for all targets on a host:

emctl start blackout <Blackoutname> [-nodeLevel] [-d <Duration>]

-nodeLevel tells the agent to stop monitoring all targets on the server.

-d Duration allows you to set a duration in the format of [days] hh:mm.   ex. 1 02:05 means the blackout will last for 1 day, 2 hours and 5 minutes.

For example, to use this in a script in which all targets will be unavailable you would start a blackout at the beginning of the script and stop it at the end:

cd $AGENT_HOME/bin 
./emctl start blackout alltargets-myserver –nodeLevel 
<Perform Maintenance Tasks> 
cd $AGENT_HOME/bin 
./emctl stop blackout alltargets-myserver

Troubleshooting

In case you hit issues with blackouts take a look at the following notes:

Subject:     Agent Blackout Initiated By Emctl Command Not Ending      Doc ID:     559577.1

Subject:     EMDiagkit Download and Master Index   Doc ID:     421053.1

Subject:     How to Troubleshoot the EM 10gR1 Blackout Sub-system      Doc ID:     284024.1

Subject:     Troubleshooting Blackouts in EM 10g Grid Control using EMDiag Kit      Doc ID:     300671.1

They provide a lot of information and solutions to different scenarios.   I hit an issue over the weekend in which the blackout didn’t end properly.  When I tried to stop it from the command line:

[oracle@myserver ~]$ /u01/app/oracle/product/agent10g/bin/emctl stop blackout alltargets-myserver 
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0. 
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved. 
Blackout stop Error : Blackout name alltargets-myserver is invalid

When trying to end the blackout via Enterprise Manager:

Error stopping the blackout on "infra10g": ORA-20710: Agent-side blackouts cannot be edited or stopped ORA-06512: at "SYSMAN.MGMT_BLACKOUT_ENGINE", line 501 ORA-06512: at "SYSMAN.MGMT_BLACKOUT_ENGINE", line 3262 ORA-06512: at "SYSMAN.MGMT_BLACKOUT", line 74 ORA-06512: at "SYSMAN.MGMT_BLACKOUT_UI", line 1167 ORA-06512: at line 1 .

To fix this problem I performed the following:
1. Shutdown the agent on the target server myserver 
2. Removed the blackouts.xml file under $AGENT_HOME/sysman/emd 
3. Used note 421053.1 to install the EMDiag kit 
4. Logged in as sysman on the Enterprise Managers repository database and executed the following query:

select blackout_guid, blackout_name 
from mgmt_blackouts;

BLACKOUT_GUID                    BLACKOUT_NAME 
-------------------------------- ---------------------- 
30E2956CA329F0E59FBDF50951F2578E alltargets-myserver

5. Then executed: 

exec mgmt_diag.KillBlackout(HEXTORAW(‘30E2956CA329F0E59FBDF50951F2578E’)); 

6. Restarted the agent on myserver and when I looked in Enterprise manager the blackout had cleared. 

I have seen the command used above for some other scenarios but not this one specifically.   Before executing any commands in your environment please test first.