Zero KB Archive backup

Archive backup of all the databases from a specific linux server was completing successfully with 0 bytes of data backed up.

[Normal] From: BSM@cellsrv01.in.com "dbclient01_DB01_AR"  Time: 03/03/14 07:15:23

        Backup session 2014/03/03-86 started.

[Normal] From: BSM@cellsrv01.in.com "dbclient01_DB01_AR"  Time: 03/03/14 07:23:36

        OB2BAR application on "dbclient01.in.com" successfully started.

[Normal] From: BSM@cellsrv01.in.com "dbclient01_DB01_AR"  Time: 03/03/14 07:23:36

        OB2BAR application on "dbclient01.in.com" disconnected.

[Normal] From: BSM@cellsrv01.in.com "dbclient01_DB01_AR"  Time: 03/03/14 07:23:37

        Backup Statistics:

                Session Queuing Time (hours)         0.00

                -------------------------------------------

                Completed Disk Agents ........          0

                Failed Disk Agents ...........          0

                Aborted Disk Agents ..........          0

                -------------------------------------------

                Disk Agents Total  ...........          0

                ================================

                Completed Media Agents .......          0

                Failed Media Agents ..........          0

                Aborted Media Agents .........          0

                -------------------------------------------

                Media Agents Total  ..........          0

                ================================

                Mbytes Total .................       0 MB

                Used Media Total .............          0

                Disk Agent Errors Total ......          0

What was causing this?? 

Is there any transactions happening in these DBs or not?? 

Any patching on server or DB that prevents the data transfer?? 

Why DP doesn't err anything??

No idea until we checked the seaudit log. The inet service was restarted from an local admin account, from then onwards the DP processes were lacking the root access rights which prevented from reading the data.

05 Mar 2014 04:47:42 D SURROGATE    useradm1     Read       69  2 USER.oradba          /opt/omni/lbin/inet  198.10.10.2       root

Solution:

Once again the inet daemon was restarted from root.

Commands to restart the DP inet service from Linux client:

#/etc/init.d/omni stop

#/etc/init.d/omni start

Tip of the Day!

If you attempt to get the serial number of a device and found "N/A", which means the device is missing/malfunctioning. This also happens when the device is in locked status by BSM/RSM/CSM/UMA/LTT.

C:\>devbra -dev

Tape    HP:Ultrium 1-SCSI  Path: "Tape0:0:0:0"  SN: "N/A"
            Description: CLAIMED:HP LTO drive
            Revision: E38W  Device type: lto [13]  Flags: 0x0011

To confirm the reason, run the below command to identify:

>>omnimm -show_locked_devs | grep <lock name>

If there's a lock, wait for the device to become free. Or you can forcibly free the device by executing the below:

>>omnidbutil -free_locked_devs <lock_name> //this would abort if any session utilizing the device

Once the device is free, run the 'devbra -dev' command and you will get the serial number. If not reboot the media server or the library depends on the error messages. If it still continues to be, replace the drive.

RMAN-03002: failure of shutdown command at 03/02/2014 09:31:22

DP sends the commands from the script (provided in the backup spec) sequentially via. DP Integration to RMAN to start Offline backup of Oracle database as follows.

RMAN> CONNECT TARGET *

2> CONNECT CATALOG *

3> HOST 'exit';

4> run {

5> shutdown immediate;

6> startup mount;

7> allocate channel 'dev_0' type 'sbt_tape'

8> parms 'SBT_LIBRARY=/opt/omni/lib/libob2oracle8_64bit.so,ENV=(OB2BARTYPE=Oracle8,OB2APPNAME=oradb101,OB2BARLIST=Oracle_DB_Spec01)';

9> 

10> send device type 'sbt_tape' 'OB2BARHOSTNAME=Orasrv01';

11> backup incremental level 0

12>  format 'Oracle_DB_Spec01<oradb101_%s:%t:%p>.dbf'

13>  database;

14> backup

15>  format 'Oracle_DB_Spec01<oradb101_%s:%t:%p>.dbf'

16>  current controlfile;

17> alter database open;

18> }

19> EXIT

RMAN-06005: connected to target database: oradb101 (DBID=101)

RMAN-06008: connected to recovery catalog database

RMAN-00571: ==================================================

RMAN-00569: ========= ERROR MESSAGE STACK FOLLOWS =========

RMAN-00571: ==================================================

RMAN-03002: failure of shutdown command at 03/02/2014 09:31:22

ORA-01013: user requested cancel of current operation

Recovery Manager complete.

[Major] From: ob2rman@Orasrv01 "oradb101"  Time: 03/02/14 09:31:22

            External utility reported error.

Execution of the script starts and there was connection established between DP and the Oracle database via. DP Integration. Eventually, once the backup goes smooth the script would bring the database online and complete, if not continue reading the post till end.

The backup has error messages:

RMAN-03002: failure of shutdown command at 03/02/2014 09:31:22

ORA-01013: user requested cancel of current operation

Here, the database is refusing the shutdown command issued by the backup script. Offline backup requires database to be in down and mounted status. Check if the database is engaged by long running tasks if any. 

That's right, when we checked the database was busy with long running oracle jobs which was the root cause to override the shutdown command. Try running the backup when the DB is free from jobs/tasks and during non-business/less or no traffic to DB, after all it's an offline backup.

To check/free locked devices in HP DP

HP DP Command to check/free a device

>> omnimm -show_locked_devs | grep <lock_name>
this command would list all the devices that are being used by the DP cell server. Medium, cartridge, devices

Ex:

Type:               Medium
Name/Id:   c75f:0034:0000:XXXX            //Media ID
Pid:                 23362                                       //Process Id that utilizes the media
Host:               cellsrv01.in.com                    //name of the cell manager (useful in MOM)
Label:             BM1000L3                              //Medium label

Type:         Device        
Name/Id:   ESLE1_D02                          //Drive name
Pid:           25754                                     //Process Id that utilizes the drive
Host:              cellsrv01.in.com                    //name of the cell manager, (useful in MOM)

Type:         Cartridge                     
Name/Id:   ESLE1                                  //Name of the library
Pid:           25836                                   //Process Id that utilises the drive
Host:         cellsrv01.in.com                  //name of the cell manager, (useful in MOM)
Location: 102                           

>> omnidbutil -show_locked_devs
The command would list all the devices, media, cartridge and slots that are in use by Data Protector cell manager.

>> omnidbutil -free_locked_devs <device_name>

Ex:

>> omnidbutil -free_locked_devs ESLE1_D02
Confirm the command by hitting 'Y': Y

After execution of this command, the drive will get released and can be used for any other purpose.

P.S: Omnidbutil / Omnimm commands can be used interchangeably to identify locked devices, to free locked devs, use omnidbutil.

[61:4006] Couldnot connect to inet in order to start BMA@ "Device_name".

Backup Error

[Major] From: BSM@cellsrv01.in.com "winsrvspec01"  Time: 2/28/2014 6:53:57 PM
 [61:4006]   Couldnot connect to inet in order to start BMA@Mediasrv02.in.com "MSL2024_D1".

[Critical] From: BMA@cellsrv01.in.com "MSL2024_D1"  Time: 2/28/2014 6:54:55 PM
[90:1004] Device address not found.

Solution

>> The DP Inet service running in the media server is not reachable from the cell server and the Media Agent couldn't be started. When checked the media server is 'Removed' from production and was shutdown.

>> In this case, the device mapping of the drive should be removed/disabled 'MSL2024_D1' from DP and the backup spec need to be modified accordingly to eliminate failures in future.

>> You can verify the media server reachability from cell manager by pinging, telnet the DP port 5555, can also use omnitcpchk commands.

For ex:

[root@ cellsrv01:/root]

# ping Mediasrv02.in.com 

PING Mediasrv02.in.com: 64 byte packets

----Mediasrv02.in.com PING Statistics----

11 packets transmitted, 0 packets received, 100% packet loss

[root@ cellsrv01:/root]

# /opt/omni/sbin/utilns/omnitcpchk -host Mediasrv02.in.com 

Testing connection Mediasrv02.in.com <---> cellsrv01.in.com....

ERROR!

==============================

TcpCheck failed for host pair:

Mediasrv02.in.com <---> cellsrv01.in.com

[root@ cellsrv01:/root]

# telnet Mediasrv02.in.com 5555

Trying...

telnet: Unable to connect to remote host: Connection timed out

>> The backup would complete using the next device in line, if there are more than one device selected in the backup specification.

  

>> Optionally, the backup would also run with the next zoned drive that has correct SCSI path and available for this backup session.