[81:52] Not a valid mount point => aborting.

[Critical] From: VBDA@winsrv01 "winsrv01 [/G]"  Time: 5/26/2014 9:28:58 AM
[81:52]  /G
                Not a valid mount point => aborting.

An attempt to backup the specific drive (G:/ in this example) connected to the DP client machine was failed and no data will be backed up by DP.

In Windows :

• Login to the windows client machine and check if the drive is accessible.
• Check if the drive is local drive or zoned drive and the zoning still exists.

In case the zoned drive no more exists, you can exclude it in the backup spec. 

If the drive exists on the server and it is accessible, try deleting the \tmp folder on the windows client machine and rerun the backup.

Default path to tmp folder for windows 2003/NT/XP:

C:\Program Files\OmniBack\tmp

Default path to tmp for windows 2008:

"C:\ProgramData\Omniback\tmp"

Hope this helps!!!

[Major] Host for device "ESL01_D01" not found.

[Normal] From: BSM@cellsrv01.in.com "Backupspec_win1"  Time: 3/21/2014 6:04:27 PM

 Backup session 2014/03/21-104 started.

[Major] From: BSM@cellsrv01.in.com "Backupspec_win1"  Time: 3/21/2014 6:04:32 PM

 Host for device ESL01_D01 not found.

[Major] From: BSM@cellsrv01.in.com "Backupspec_win1"  Time: 3/21/2014 6:04:32 PM

 Host for device ESL01_D02 not found.

[Major] From: BSM@cellsrv01.in.com "Backupspec_win1"  Time: 3/21/2014 6:04:32 PM

 Host for device ESL01_D03 not found.

[Normal] From: BSM@cellsrv01.in.com "Backupspec_win1"  Time: 3/21/2014 6:04:32 PM

 Backup Statistics:

         

  Session Queuing Time (hours)         0.00      

  -------------------------------------------    

  Completed Disk Agents ........          0        

  Failed Disk Agents ...........          7        

  Aborted Disk Agents ..........          0        

  -------------------------------------------    

  Disk Agents Total  ...........          7        

  ===========================================    

  Completed Media Agents .......          0        

  Failed Media Agents ..........          0        

  Aborted Media Agents .........          0        

  -------------------------------------------    

  Media Agents Total  ..........          0        

  ===========================================    

  Mbytes Total .................       0 MB      

  Used Media Total .............          0        

  Disk Agent Errors Total ......          0  

This error occurs when the backup device wasn't removed from the devices context following the media server deletion/migration to other cell.

In other words, the media server could have moved to another cell server or might have been decommissioned and by mistake the devices were not removed from the devices list.

Check the media server's existence, if available;

Check the cell server name, it is pointing to;

Mostly, if the orphan device entry is removed, this error will disappear.

[61:2051] All mountpoints on host \"winsrv01.in.com\" are excluded.

Error:

[Major] From: BSM@cellsrv01.in.com \"win2008bck_spec1\"  Time: 06/09/14 19:09:47
[61:2051] All mountpoints on host \"winsrv01.in.com\" are excluded.
Nothing will be backed up.

Reason:

>> This is something related to Inet error (which was discussed Here), but not necessarily be due to network disruptions. This could also be due to platform side issue.

Analysis:

>> Try to ping and telnet the windows server using default port number from cell server or any other server on the same network.

>> In this case, the server was pretty slow and non-responsive. Telnet to the default port was hung, so requested wintel team for server optimization which must fix the error.

>> Double-check by expanding the server's objects from the backup specification (which would hung as well).

Have a Great Day :)

[90:1004] Device address not found.

[Normal] From: BSM@cellsrv01.in.com "Backup_Spec_Win1"  Time: 4/16/2014 12:08:46 PM

 Backup session 2014/04/16-256 started.

[Normal] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:08:55 PM

 STARTING Media Agent "AUT01_D01"

[Normal] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:09:01 PM

 By: UMA@winsrv01.in.com@Changer0:7:0:1

 Loading medium from slot 8 to device Tape1:7:0:0C

[Warning] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:09:57 PM

 The device "AUT01_D01" could not be opened("Device could not be accessed")

[Normal] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:09:57 PM

 Starting the device path discovery process.

[Critical] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:10:00 PM

[90:1004]  Device address not found.

[Normal] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:10:00 PM

 Device path discovery process finished.

[Normal] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:10:00 PM

 By: UMA@winsrv01.in.com@Changer0:7:0:1

 Unloading medium to slot 8 from device Tape1:7:0:0C

[Normal] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:10:39 PM

 ABORTED Media Agent "AUT01_D01"

[Normal] From: BSM@cellsrv01.in.com "Backup_Spec_Win1"  Time: 4/16/2014 12:10:39 PM

 Backup Statistics:

         

  Session Queuing Time (hours)         0.00      

  -------------------------------------------    

  Completed Disk Agents ........          0        

  Failed Disk Agents ...........          4        

  Aborted Disk Agents ..........          0        

  -------------------------------------------    

  Disk Agents Total  ...........          4        

  ===========================================    

  Completed Media Agents .......          0        

  Failed Media Agents ..........          1        

  Aborted Media Agents .........          0        

  -------------------------------------------    

  Media Agents Total  ..........          1        

  ===========================================    

  Mbytes Total .................       0 MB      

  Used Media Total .............          0        

  Disk Agent Errors Total ......          0  

Troubleshooting steps as follows:

>> Logged into the media server and checked for devices claimed in Device Manager. Found the devices.

>> Ran devbra -dev to determine the SCSI address.

>> Found N/A for drive

C:\>devbra -dev

Exch    HP:1x8 G2 AUTOLDR  Path: "Changer0:0:0:1"  SN: "AABBCCDD1E"

        Description: CLAIMED:HP StorageWorks 1x8 Cartridge Autoloader

        Revision: 4.20  Flags: 0x0016  Slots: 8  Drives: 1

        Drive(s) SN:

                "ABCDEFGHIJ"

Tape    HP:Ultrium 3-SCSI  Path: "Tape0:0:0:0"  SN: "N/A"

        Description: CLAIMED:HP LTO3 Drive

        Revision: Q51W  Device type: lto [13]  Flags: 0x0011

>> Checked if the drive is locked by DP

[root@cellsrv01:/root]

# omnimm -show_locked_devs | grep AUT01_D01

[root@ cellsrv01:/root]

>> Stopped DP Inet from Services.msc and Ran LTT.

>> Got error message that "mma.exe" process is accessing the device. Killed the mma.exe process from the Task Manager.

>> rescanned the devices in LTT. Able to detect both autoloader and it's drive this time.

>> Ran devbra -dev, which got the SCSI address.

C:\>devbra -dev

Exch    HP:1x8 G2 AUTOLDR  Path: "Changer0:7:0:1"  SN: "AABBCCDD1E "

        Description: CLAIMED:HP StorageWorks 1x8 Cartridge Autoloader

        Revision: 4.20  Flags: 0x0016  Slots: 8  Drives: 1

        Drive(s) SN:

                "ABCDEFGHIJ "

Tape    HP:Ultrium 3-SCSI  Path: "Tape1:7:0:0C"  SN: "ABCDEFGHIJ "

        Description: CLAIMED:HP LTO3 Drive

        Revision: Q51W  Device type: lto [13]  Flags: 0x0011

>> Ran the backup successfully.

Cause :

The DP media agent was accessing the drive and didn't let any other process to send commands. After the hung process is killed, the drive was accessible for normal operations.

Virtual Device Interface reported error: The object was not open.

Backup of SQL Database fails with the following errors:

[Critical] From: OB2BAR_Main@sqlsrv01.in.com "SIDDBS01"  Time: 3/9/2014 6:31:55 PM

 Error has occurred while executing a SQL statement.

 Error message: 'SQLSTATE:[08004] CODE:(911) MESSAGE:[Microsoft][ODBC SQL Server Driver][SQL Server]Database 'DBN01' does not exist. Make sure that the name is entered correctly.

SQLSTATE:[42000] CODE:(3013) MESSAGE:[Microsoft][ODBC SQL Server Driver][SQL Server]BACKUP DATABASE is terminating abnormally.'

[Critical] From: OB2BAR_Main@sqlsrv01.in.com "SIDDBS01"  Time: 3/9/2014 6:31:55 PM

 Virtual Device Interface reported error:

The object was not open.

 See also Data Protector debug.log and SQL Server error log for details.

[Normal] From: OB2BAR_DBN01@sqlsrv01.in.com "SIDDBS01"  Time: 3/9/2014 6:31:57 PM

 Completed OB2BAR Backup: sqlsrv01.in.com:/SIDDBS01/DBN01/0 "MSSQL"

[Major] From: OB2BAR_DBN01@sqlsrv01.in.com "SIDDBS01"  Time: 3/9/2014 6:31:57 PM

Aborting connection to BSM. Abort code -2.

Solution:

>> Remove the DB 'DBN01' from backup spec, if it doesn't exists in the SQL server.

[OR]

>> Check if the DB was dismounted, If so mount the database 'DBN01' and bring it online.

Trigger the backup and see if it's successful.

[90:801] Active Removable Storage Manager (RSM) service found on local system.

[Normal] From: BMA@mediasrv01.in.com "MSL2024_D02"  Time: 3/23/2014 2:57:16 PM

 STARTING Media Agent "MSL2024_D02"

[Warning] From: BMA@mediasrv01.in.com "MSL2024_D02"  Time: 3/23/2014 2:57:16 PM

[90:801]  Active Removable Storage Manager (RSM) service found on local system.

 [Warning] From: UMA@mediasrv01.in.com "MSL2024"  Time: 3/23/2014 2:57:18 PM

[90:59]   Changer0:0:0:0

 Cannot open exchanger control device ([2] The system cannot find the file specified. )

[Warning] From: UMA@mediasrv01.in.com "MSL2024"  Time: 3/23/2014 2:57:18 PM

 The device "MSL2024" could not be opened ("Device could not be accessed") 

[Normal] From: UMA@mediasrv01.in.com "MSL2024"  Time: 3/23/2014 2:57:18 PM

 Starting the device path discovery process. 

[Warning] From: UMA@mediasrv01.in.com "MSL2024"  Time: 3/23/2014 2:57:19 PM

[90:59]   Changer0:0:0:0

 Cannot open exchanger control device ([2] The system cannot find the file specified. )

[Normal] From: BMA@ mediasrv01.in.com "MSL2024"  Time: 3/23/2014 2:57:21 PM ABORTED Media Agent " MSL2024_D02"

Solution:

Removable Storage Manager (RSM) service on MS Windows 2003 might cause device issues in backup environment.

It is recommended to stop and disable the RSM service on all MS Windows 2003 systems.

RSM Service in Started status

[90:63] Cannot load exchanger medium (Medium error.)

[Normal] From: BMA@winsrv01.in.com "MSL_D01"  Time: 3/24/2014 10:53:43 PM

 By: UMA@winsrv01.in.com@Changer0:0:0:0

 Loading medium from slot 9 to device Tape0:0:0:0

[Major] From: BMA@winsrv01.in.com "MSL_D01"  Time: 3/24/2014 11:03:16 PM

[90:63]   By: UMA@winsrv01.in.com@Changer0:0:0:0

 Cannot load exchanger medium (Medium error.)

[Normal] From: BMA@winsrv01.in.com "MSL_D01"  Time: 3/24/2014 11:03:16 PM

 ABORTED Media Agent "MSL_D01"

[Normal] From: BMA@winsrv01.in.com " MSL_D02"  Time: 3/24/2014 11:03:21 PM

 STARTING Media Agent " MSL_D02"

[Normal] From: BMA@winsrv01.in.com " MSL_D02"  Time: 3/24/2014 11:03:26 PM

 By: UMA@winsrv01.in.com@Changer0:0:0:0

 Loading medium from slot 9 to device Tape1:0:0:0

[Major] From: BMA@winsrv01.in.com " MSL_D02"  Time: 3/24/2014 11:13:10 PM

[90:63]   By: UMA@winsrv01.in.com@Changer0:0:0:0

 Cannot load exchanger medium (Medium error.)

[Normal] From: BMA@winsrv01.in.com " MSL_D02"  Time: 3/24/2014 11:13:10 PM

 ABORTED Media Agent " MSL_D02"

Solution:

Exchanger medium is failing while handling media from particular slot (here it is slot 9). The media has errors that is residing slot 9.

Move to 'Devices & Media' context -> Click Library -> Click Slot -> move to Slot 9.

--- Check the status of media in slot 9.

--- It was poor media. Isolate the media by moving it to separate media pool and unload it from library. Wait for the media to expire and reformat it. If the status is still shows poor even after formatting, it is not advisable to use it henceforth.

--- Clean the affected drives with good cleaning tape and run a test backup. They should be good for backups.

ORA-19588: archived log RECID <> STAMP <> is no longer valid

Error:

RMAN-00571: ===========================================================

RMAN-00569: ======== ERROR MESSAGE STACK FOLLOWS ============

RMAN-00571: ===========================================================

RMAN-03009: failure of backup command on dev_0 channel at 03/08/2014 07:50:43

ORA-19588: archived log RECID 99528 STAMP 841638632 is no longer valid

Recovery Manager complete.

[Major] From: ob2rman@orasrv01.in.com "oradb01"  Time: 03/08/14 07:50:56

 External utility reported error.

RMAN PID=12391

[Major] From: ob2rman@orasrv01.in.com "oradb01"  Time: 03/08/14 07:50:56

 The database reported error while performing requested operation.

RMAN-00571: ===========================================================

 RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

 RMAN-00571: ===========================================================

 RMAN-03009: failure of backup command on dev_0 channel at 03/08/2014 07:50:43

 ORA-19588: archived log RECID 99528 STAMP 841638632 is no longer valid

 Recovery Manager complete.

Problem:

>> The error can be due to invalid/stale archive files which were left uncleared even after the backup completion. Reason could be due to 2 simultaneous backup sessions where one refers to and the other had already backed up and not needed to be backed up.

>> It also happens when RMAN locates an archive file from the archive location that is no longer available in the Controlfile. These store very less info when compared to Recovery log.

Solution:

Execute the command “crosscheck archivelog all”.

The output can be as below:

Crosschecked 826 objects

Now you can re-trigger the backup spec and get a successful backup.

[70:24] A system error occurred when starting the target script or an agent module.

Problem:

Backup of client machine (Win 2003) fails with the following errors.

[Critical] From: INET_thread_vbda.exe@winclient01.in.com \"winclient01.in.com\"  Time: 03/04/14 08:46:39

[70:24] A system error occurred when starting the target script or an agent module. 

The system error code reported is 2 and the message resolves to [2] The system cannot find the file specified.

Solution:

>> Uninstall & re-install DP client. 
>> Import the DP client in the cell server using GUI [or] add the entry in the cell_info file.
>> Start the backup of DP client which should be successful.

RMAN-03002: failure of shutdown command at 03/02/2014 09:31:22

DP sends the commands from the script (provided in the backup spec) sequentially via. DP Integration to RMAN to start Offline backup of Oracle database as follows.

RMAN> CONNECT TARGET *

2> CONNECT CATALOG *

3> HOST 'exit';

4> run {

5> shutdown immediate;

6> startup mount;

7> allocate channel 'dev_0' type 'sbt_tape'

8> parms 'SBT_LIBRARY=/opt/omni/lib/libob2oracle8_64bit.so,ENV=(OB2BARTYPE=Oracle8,OB2APPNAME=oradb101,OB2BARLIST=Oracle_DB_Spec01)';

9> 

10> send device type 'sbt_tape' 'OB2BARHOSTNAME=Orasrv01';

11> backup incremental level 0

12>  format 'Oracle_DB_Spec01<oradb101_%s:%t:%p>.dbf'

13>  database;

14> backup

15>  format 'Oracle_DB_Spec01<oradb101_%s:%t:%p>.dbf'

16>  current controlfile;

17> alter database open;

18> }

19> EXIT

RMAN-06005: connected to target database: oradb101 (DBID=101)

RMAN-06008: connected to recovery catalog database

RMAN-00571: ==================================================

RMAN-00569: ========= ERROR MESSAGE STACK FOLLOWS =========

RMAN-00571: ==================================================

RMAN-03002: failure of shutdown command at 03/02/2014 09:31:22

ORA-01013: user requested cancel of current operation

Recovery Manager complete.

[Major] From: ob2rman@Orasrv01 "oradb101"  Time: 03/02/14 09:31:22

            External utility reported error.

Execution of the script starts and there was connection established between DP and the Oracle database via. DP Integration. Eventually, once the backup goes smooth the script would bring the database online and complete, if not continue reading the post till end.

The backup has error messages:

RMAN-03002: failure of shutdown command at 03/02/2014 09:31:22

ORA-01013: user requested cancel of current operation

Here, the database is refusing the shutdown command issued by the backup script. Offline backup requires database to be in down and mounted status. Check if the database is engaged by long running tasks if any. 

That's right, when we checked the database was busy with long running oracle jobs which was the root cause to override the shutdown command. Try running the backup when the DB is free from jobs/tasks and during non-business/less or no traffic to DB, after all it's an offline backup.

Cannot get information on file '/var/opt/omni/tmp/rcvcat.exp' Error: 2

[Major] From: OB2BAR_DMA@dbsrv01.in.com "DB001"  Time: 2/23/2014 3:05:36 PM
      Cannot get information on file '/var/opt/omni/tmp/rcvcat.exp' Error: 2.

[Major] From: ob2rman@dbsrv01.in.com "DB001"  Time: 02/23/2014 03:05:44 PM
      Backup of recovery catalog failed.

Solution:

>> You would receive this error in linux OS and clustered Oracle DB. The patch DPLNX_00094 would fix this.

Alternatively, you can try the below steps:

>> From the Integration specific option, Enable the option 'Disable recovery catalog auto backup'. This would skip the backup of recovery catalog backup from DP. Rerun the backup now. If you still see the same errors and wonder why, then move on to the next step.

>> The backup script in barlists always takes precedence from the DP GUI options. Open the spec in VI editor and Move to the end of the barlist and edit the script with value "-skip RCVCAT".

Example:
CLIENT "DB001" dbsrv01.in.com
{
        -exec ob2rman.exe
        -args {
                "-skip RCVCAT"                                             //Skips the recovery catalog backup
                "-pre  /home/user/DB001_weekly_full.rman"     //Pre exec scripts if any
                "-backup"                
        }
        -input {
                "run {"
                "allocate channel 'dev_0' type 'sbt_tape'"        //channel allocation
                " parms 'ENV=(OB2BARTYPE=Oracle8,OB2APPNAME=DB001,OB2BARLIST=dbsrv01_DB001_ONLINE)';"
                "backup"
                " format 'dbsrv01_DB001_ONLINE<DB001_%s:%t:%p>.dbf'"
                " current controlfile;"
                "}"
        }
        -profile
}  -protect days 28                                                      //data protection in days

[Critical] [90:63] Cannot load exchanger medium (Target drive is busy.)

It can be interesting in troubleshooting a device issue sometimes. Let's deal with the devices in this post..

Error Message:

[Normal] From: BMA@Mediasrv01.in.com "Drive01"  Time: 2/13/2014 5:41:04 PM

                By: UMA@Mediasrv01.in.com@Changer0:7:0:1

                Loading medium from slot 2 to device Tape0:7:0:0C

[Critical] From: BMA@Mediasrv01.in.com " Drive01"  Time: 2/13/2014 5:41:24 PM

[90:63]                  By: UMA@Mediasrv01.in.com@Changer0:7:0:1

                Cannot load exchanger medium (Target drive is busy.)

From the error message, we could find that the "Target drive is busy". 

>> Login to the media server from where the tape library/autoloader is connected

>> Run devbra -dev, the output would be the drives and exchanger detected from the device manager.

>> Use the uma along with the scsi path of exchanger (here it is Changer0:7:0:1) in cmd line (please see below).

>> Check if there is stuck tape in drive. Move the tape to a free slot either from GUI/CLI and run the backup.

>> there are chances that the library might be fully loaded and the drive too is holding a tape. So manually remove the tape from slot/drive.

Using UMA utility for device management:

Example of fully loaded autoloader which needs manually unload media from drive.

C:\uma -ioctl  Changer0:7:0:1

Changer0:7:0:1> stat d

          1 D1    Full  "" ""

Changer0:7:0:1> stat s

        1001 S1   Full  "" ""

        1002 S2   Full  "" ""

        1003 S3   Full  "" ""

        1004 S4   Full  "" ""

        1005 S5   Full  "" ""

        1006 S6   Full  "" ""

        1007 S7   Full  "" ""

        1008 S8   Full  "" ""

(OR)

Example of unloading media from drive to empty slot.

C:\uma -ioctl  Changer0:7:0:1

Changer0:7:0:1> stat d

          1 D1    Full  "" ""

Changer0:7:0:1> stat s

        1001 S1   Full  "" ""

        1002 S2   Full  "" ""

        1003 S3   Empty  "" ""

        1004 S4   Full  "" ""

        1005 S5   Full  "" ""

        1006 S6   Full  "" ""

        1007 S7   Full  "" ""

        1008 S8   Full  "" ""

Changer0:7:0:1> move D1 S3

Changer0:7:0:1> stat d

          1 D1    Empty  "" ""

>> This is one of many cases for 'Cannot load exchanger medium' error.