[90:114] Cannot unload medium, target slot (255) appears to be occupied

[Warning] From: BMA@linuxsrv155.com "ESL01_D17"  Time: 2/9/2015 1:13:44 PM
[90:54]   /dev/nst1
Cannot open device (Unit not ready.)

[Major] From: BMA@linuxsrv155.com "ESL01_D17"  Time: 2/9/2015 1:13:55 PM
[90:114]   By: UMA@mediasrv01.com@/dev/rchgr/autoch3
Cannot unload medium, target slot (255) appears to be occupied

Solution:

> The above error is due to DP couldn't unload the media from where it was actually loaded to the drive during backup/restore.

> Move the tape manually from the drive to any free slot using UMA by following the steps below. 

> Login to the media server from where the tape library/autoloader is connected.

>> run devbra-dev, the output will be the drives and exchanger detected from the device manager.

>> copy the scsi path of exchanger (here it is /dev/rchgr/autoch3)
>> execute the below command and move the media to any empty slot.
>uma -ioctl /dev/rchgr/autoch3

/dev/rchgr/autoch3> move D17 S13

> If there are no empty slots available, eject full media to I/O slots and move the media stuck in drive to free slot.
> Then retry the backup, which should be successful.

Hope this helps!

Orphan object(s) in the backup specification

Warning message from DP

Warning: While reading mount point information, Data Protector found 2 backup object(s), which reference inexistent mountpoint(s) on client system 'winserver123.abc.com'.
Do you want to leave this(these) orphan object(s) in the backup specification?

Reason:

• Maybe a shared drive that's no longer available on the workstation or were reclaimed.
• If it exists, it's not accessible by the DP software. In such case the data list needs to be updated/changed on what needs to be backed up.

Solution:

• Login to the client workstation and check from the Computer Management -> Storage -> Disk Management
• Check the problematic mountpoint exists and in healthy state. If it doesn't, remove the mountpoint from the backup specification and save it.

[81:52] Not a valid mount point => aborting.

[Critical] From: VBDA@winsrv01 "winsrv01 [/G]"  Time: 5/26/2014 9:28:58 AM
[81:52]  /G
                Not a valid mount point => aborting.

An attempt to backup the specific drive (G:/ in this example) connected to the DP client machine was failed and no data will be backed up by DP.

In Windows :

• Login to the windows client machine and check if the drive is accessible.
• Check if the drive is local drive or zoned drive and the zoning still exists.

In case the zoned drive no more exists, you can exclude it in the backup spec. 

If the drive exists on the server and it is accessible, try deleting the \tmp folder on the windows client machine and rerun the backup.

Default path to tmp folder for windows 2003/NT/XP:

C:\Program Files\OmniBack\tmp

Default path to tmp for windows 2008:

"C:\ProgramData\Omniback\tmp"

Hope this helps!!!

[Major] Host for device "ESL01_D01" not found.

[Normal] From: BSM@cellsrv01.in.com "Backupspec_win1"  Time: 3/21/2014 6:04:27 PM

 Backup session 2014/03/21-104 started.

[Major] From: BSM@cellsrv01.in.com "Backupspec_win1"  Time: 3/21/2014 6:04:32 PM

 Host for device ESL01_D01 not found.

[Major] From: BSM@cellsrv01.in.com "Backupspec_win1"  Time: 3/21/2014 6:04:32 PM

 Host for device ESL01_D02 not found.

[Major] From: BSM@cellsrv01.in.com "Backupspec_win1"  Time: 3/21/2014 6:04:32 PM

 Host for device ESL01_D03 not found.

[Normal] From: BSM@cellsrv01.in.com "Backupspec_win1"  Time: 3/21/2014 6:04:32 PM

 Backup Statistics:

         

  Session Queuing Time (hours)         0.00      

  -------------------------------------------    

  Completed Disk Agents ........          0        

  Failed Disk Agents ...........          7        

  Aborted Disk Agents ..........          0        

  -------------------------------------------    

  Disk Agents Total  ...........          7        

  ===========================================    

  Completed Media Agents .......          0        

  Failed Media Agents ..........          0        

  Aborted Media Agents .........          0        

  -------------------------------------------    

  Media Agents Total  ..........          0        

  ===========================================    

  Mbytes Total .................       0 MB      

  Used Media Total .............          0        

  Disk Agent Errors Total ......          0  

This error occurs when the backup device wasn't removed from the devices context following the media server deletion/migration to other cell.

In other words, the media server could have moved to another cell server or might have been decommissioned and by mistake the devices were not removed from the devices list.

Check the media server's existence, if available;

Check the cell server name, it is pointing to;

Mostly, if the orphan device entry is removed, this error will disappear.

[61:2051] All mountpoints on host \"winsrv01.in.com\" are excluded.

Error:

[Major] From: BSM@cellsrv01.in.com \"win2008bck_spec1\"  Time: 06/09/14 19:09:47
[61:2051] All mountpoints on host \"winsrv01.in.com\" are excluded.
Nothing will be backed up.

Reason:

>> This is something related to Inet error (which was discussed Here), but not necessarily be due to network disruptions. This could also be due to platform side issue.

Analysis:

>> Try to ping and telnet the windows server using default port number from cell server or any other server on the same network.

>> In this case, the server was pretty slow and non-responsive. Telnet to the default port was hung, so requested wintel team for server optimization which must fix the error.

>> Double-check by expanding the server's objects from the backup specification (which would hung as well).

Have a Great Day :)

[90:1004] Device address not found.

[Normal] From: BSM@cellsrv01.in.com "Backup_Spec_Win1"  Time: 4/16/2014 12:08:46 PM

 Backup session 2014/04/16-256 started.

[Normal] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:08:55 PM

 STARTING Media Agent "AUT01_D01"

[Normal] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:09:01 PM

 By: UMA@winsrv01.in.com@Changer0:7:0:1

 Loading medium from slot 8 to device Tape1:7:0:0C

[Warning] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:09:57 PM

 The device "AUT01_D01" could not be opened("Device could not be accessed")

[Normal] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:09:57 PM

 Starting the device path discovery process.

[Critical] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:10:00 PM

[90:1004]  Device address not found.

[Normal] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:10:00 PM

 Device path discovery process finished.

[Normal] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:10:00 PM

 By: UMA@winsrv01.in.com@Changer0:7:0:1

 Unloading medium to slot 8 from device Tape1:7:0:0C

[Normal] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:10:39 PM

 ABORTED Media Agent "AUT01_D01"

[Normal] From: BSM@cellsrv01.in.com "Backup_Spec_Win1"  Time: 4/16/2014 12:10:39 PM

 Backup Statistics:

         

  Session Queuing Time (hours)         0.00      

  -------------------------------------------    

  Completed Disk Agents ........          0        

  Failed Disk Agents ...........          4        

  Aborted Disk Agents ..........          0        

  -------------------------------------------    

  Disk Agents Total  ...........          4        

  ===========================================    

  Completed Media Agents .......          0        

  Failed Media Agents ..........          1        

  Aborted Media Agents .........          0        

  -------------------------------------------    

  Media Agents Total  ..........          1        

  ===========================================    

  Mbytes Total .................       0 MB      

  Used Media Total .............          0        

  Disk Agent Errors Total ......          0  

Troubleshooting steps as follows:

>> Logged into the media server and checked for devices claimed in Device Manager. Found the devices.

>> Ran devbra -dev to determine the SCSI address.

>> Found N/A for drive

C:\>devbra -dev

Exch    HP:1x8 G2 AUTOLDR  Path: "Changer0:0:0:1"  SN: "AABBCCDD1E"

        Description: CLAIMED:HP StorageWorks 1x8 Cartridge Autoloader

        Revision: 4.20  Flags: 0x0016  Slots: 8  Drives: 1

        Drive(s) SN:

                "ABCDEFGHIJ"

Tape    HP:Ultrium 3-SCSI  Path: "Tape0:0:0:0"  SN: "N/A"

        Description: CLAIMED:HP LTO3 Drive

        Revision: Q51W  Device type: lto [13]  Flags: 0x0011

>> Checked if the drive is locked by DP

[root@cellsrv01:/root]

# omnimm -show_locked_devs | grep AUT01_D01

[root@ cellsrv01:/root]

>> Stopped DP Inet from Services.msc and Ran LTT.

>> Got error message that "mma.exe" process is accessing the device. Killed the mma.exe process from the Task Manager.

>> rescanned the devices in LTT. Able to detect both autoloader and it's drive this time.

>> Ran devbra -dev, which got the SCSI address.

C:\>devbra -dev

Exch    HP:1x8 G2 AUTOLDR  Path: "Changer0:7:0:1"  SN: "AABBCCDD1E "

        Description: CLAIMED:HP StorageWorks 1x8 Cartridge Autoloader

        Revision: 4.20  Flags: 0x0016  Slots: 8  Drives: 1

        Drive(s) SN:

                "ABCDEFGHIJ "

Tape    HP:Ultrium 3-SCSI  Path: "Tape1:7:0:0C"  SN: "ABCDEFGHIJ "

        Description: CLAIMED:HP LTO3 Drive

        Revision: Q51W  Device type: lto [13]  Flags: 0x0011

>> Ran the backup successfully.

Cause :

The DP media agent was accessing the drive and didn't let any other process to send commands. After the hung process is killed, the drive was accessible for normal operations.

[61:12500] cannot connect to inet for getting file system list on host

Error:

[Critical]   From: BSM@cellsrv01.in.com "backup_spec_01” Time: 8/16/2014
[61:12500] cannot connect to inet for getting file system list
              on host" winclnt01.in.com".
 

Reason:

Host may be down or Data protector client services (Inet) down.

Troubleshooting steps for Windows clients are as follows:

1. Check if the client (i.e. winclnt01.in.com in above error) is pingable from the cell manager, if not check the issue with platform team.

2. If the server is pingable, telnet the client using port 5555 (eg: #Telnet <Client name> 5555)

3. If the telnet failed, login to the windows client and check whether “Data protector Inet” service is running or not. If not start the “Data Protector Inet” service, Just by right clicking the service and restart it.

4. If the Data protector service is not listed in the “services.msc”, the DP agent might be removed from the client. Push the DP agents on this client from cell server.

5. If the telnet failed using port 5555 even data protector service running, check with platform team to open the port.

Troubleshooting steps for UNIX clients are as follows:

1. Follow the steps 1 and 2 as explained above.

2. If the telnet failed, login to the UNIX client and check for the Omni entries in /etc/services and /etc/inetd.conf using the commands below. If not found, DP client needs to be reinstalled.

[root@unxclnt01:/root]

# egrep omni /etc/services

omni  5555/tcp     # DATA-PROTECTOR

[root@unxclnt01:/root]

# egrep omni /etc/inetd.conf

omni stream tcp nowait root /opt/omni/lbin/inet inet -log /var/opt/omni//log/inet.log

3. Check if the port 5555 is listening using the command below. If not, raise a request to platform team to enable the port and start the inetd daemon.

[root@unxclnt01:/root]

# netstat -an | grep 5555

tcp        0      0  *.5555                 *.*                     LISTEN

4. Check if the ‘inetd’ daemon is running, if not start it using the below commands:

[root@unxclnt01:/root]

# ps -ef | grep inetd

    root  2144     1  0  Aug 10  ?         7:20 /usr/sbin/inetd

# /etc/inetd –s

[90:801] Active Removable Storage Manager (RSM) service found on local system.

[Normal] From: BMA@mediasrv01.in.com "MSL2024_D02"  Time: 3/23/2014 2:57:16 PM

 STARTING Media Agent "MSL2024_D02"

[Warning] From: BMA@mediasrv01.in.com "MSL2024_D02"  Time: 3/23/2014 2:57:16 PM

[90:801]  Active Removable Storage Manager (RSM) service found on local system.

 [Warning] From: UMA@mediasrv01.in.com "MSL2024"  Time: 3/23/2014 2:57:18 PM

[90:59]   Changer0:0:0:0

 Cannot open exchanger control device ([2] The system cannot find the file specified. )

[Warning] From: UMA@mediasrv01.in.com "MSL2024"  Time: 3/23/2014 2:57:18 PM

 The device "MSL2024" could not be opened ("Device could not be accessed") 

[Normal] From: UMA@mediasrv01.in.com "MSL2024"  Time: 3/23/2014 2:57:18 PM

 Starting the device path discovery process. 

[Warning] From: UMA@mediasrv01.in.com "MSL2024"  Time: 3/23/2014 2:57:19 PM

[90:59]   Changer0:0:0:0

 Cannot open exchanger control device ([2] The system cannot find the file specified. )

[Normal] From: BMA@ mediasrv01.in.com "MSL2024"  Time: 3/23/2014 2:57:21 PM ABORTED Media Agent " MSL2024_D02"

Solution:

Removable Storage Manager (RSM) service on MS Windows 2003 might cause device issues in backup environment.

It is recommended to stop and disable the RSM service on all MS Windows 2003 systems.

RSM Service in Started status

[61:2015] Timeout waiting for the devices to get free.

[Critical] From: BSM@cellsrv01.in.com "cellsrv01_IDB"  Time: 02/27/14 02:00:31

[61:2015]  Timeout waiting for the devices to get free.

The session will terminate.

Ø  It is very common in every backup environment to share the same device for different backups. This is an error message due to device contention issue.

Ø  The backup device selected in the backup specification is unavailable for the backup to start. It is in use by another process or by another backup/restore/copy sessions.

Ø   Check the device status using the lock name. (Check Here for the commands used to find a locked device). Wait for the device to be free.

Ø  The backup will be queued for global timeout seconds and will fail if no device is freed / allocated to the backup session.

Ø  The queuing time can be found at the end of backup session from backup statistics. Shown below

  

Backup Statistics:

          

                   Session Queuing Time (hours)         0.00        

                   -------------------------------------------      

                   Completed Disk Agents ........          5          

                   Failed Disk Agents ...........          0          

                   Aborted Disk Agents ..........          0          

                   -------------------------------------------      

                   Disk Agents Total  ...........          5          

                   =====================================     

                   Completed Media Agents .......          1          

                   Failed Media Agents ..........          0          

                   Aborted Media Agents .........          0          

                   -------------------------------------------      

                   Media Agents Total  ..........          1          

                   ===========================================      

                   Mbytes Total .................   17985 MB        

                   Used Media Total .............          1          

                   Disk Agent Errors Total ......          0    

[90:63] Cannot load exchanger medium (Medium error.)

[Normal] From: BMA@winsrv01.in.com "MSL_D01"  Time: 3/24/2014 10:53:43 PM

 By: UMA@winsrv01.in.com@Changer0:0:0:0

 Loading medium from slot 9 to device Tape0:0:0:0

[Major] From: BMA@winsrv01.in.com "MSL_D01"  Time: 3/24/2014 11:03:16 PM

[90:63]   By: UMA@winsrv01.in.com@Changer0:0:0:0

 Cannot load exchanger medium (Medium error.)

[Normal] From: BMA@winsrv01.in.com "MSL_D01"  Time: 3/24/2014 11:03:16 PM

 ABORTED Media Agent "MSL_D01"

[Normal] From: BMA@winsrv01.in.com " MSL_D02"  Time: 3/24/2014 11:03:21 PM

 STARTING Media Agent " MSL_D02"

[Normal] From: BMA@winsrv01.in.com " MSL_D02"  Time: 3/24/2014 11:03:26 PM

 By: UMA@winsrv01.in.com@Changer0:0:0:0

 Loading medium from slot 9 to device Tape1:0:0:0

[Major] From: BMA@winsrv01.in.com " MSL_D02"  Time: 3/24/2014 11:13:10 PM

[90:63]   By: UMA@winsrv01.in.com@Changer0:0:0:0

 Cannot load exchanger medium (Medium error.)

[Normal] From: BMA@winsrv01.in.com " MSL_D02"  Time: 3/24/2014 11:13:10 PM

 ABORTED Media Agent " MSL_D02"

Solution:

Exchanger medium is failing while handling media from particular slot (here it is slot 9). The media has errors that is residing slot 9.

Move to 'Devices & Media' context -> Click Library -> Click Slot -> move to Slot 9.

--- Check the status of media in slot 9.

--- It was poor media. Isolate the media by moving it to separate media pool and unload it from library. Wait for the media to expire and reformat it. If the status is still shows poor even after formatting, it is not advisable to use it henceforth.

--- Clean the affected drives with good cleaning tape and run a test backup. They should be good for backups.

BR0073E Setting of BRBACKUP lock failed

Problem:

BR0051I BRBACKUP 7.20 (25)
BR0055I Start of database backup: benini.qub 2014-03-18 08.00.26
BR0484I BRBACKUP log file: /oracle/SID/sapbackup/benini.qub
BR0071E BRBACKUP currently running or was killed
BR0072I Please delete file /oracle/SID/sapbackup/.lock.brb if BRBACKUP was killed
BR0073E Setting of BRBACKUP lock failed

BR0056I End of database backup: benini.qub 2014-03-18 08.00.26
BR0280I BRBACKUP time stamp: 2014-03-18 08.00.26
BR0054I BRBACKUP terminated with errors
[Major] From: OB2BAR_OMNISAP@orsapsrv1.in.com "OMNISAP"  Time: 03/18/2014 08:00:26 AM
BRBACKUP /usr/sap/SID/SYS/exe/run/brbackup -t online_split -d util_file -c -p initSID.sap.bc -m all -q split -u / returned 3

[Normal] From: BSM@cellsrv01.incom "orsapsrv1_SID01"  Time: 3/18/2014 8:00:27 AM
OB2BAR application on "orsapsrv1.in.com" disconnected.

Solution:

>> Login to the client with SID of the database and check for any brbackup process running.

*****No brbackup process running*****

# ps -ef | grep brbackup

root     17138  9230  0 08:19 pts/0    00:00:00 grep brbackup

>> Check for any progressing DP backups from monitor context. This can also be initiated from DB end too. Wait until the backup completes and then check for the brbackup process again. 

>> Kill the .lock.brb file <path - /oracle/SID/sapbackup/.lock.brb > if there was aborted/hung backup session. Start the backup spec which should complete.

ORA-19588: archived log RECID <> STAMP <> is no longer valid

Error:

RMAN-00571: ===========================================================

RMAN-00569: ======== ERROR MESSAGE STACK FOLLOWS ============

RMAN-00571: ===========================================================

RMAN-03009: failure of backup command on dev_0 channel at 03/08/2014 07:50:43

ORA-19588: archived log RECID 99528 STAMP 841638632 is no longer valid

Recovery Manager complete.

[Major] From: ob2rman@orasrv01.in.com "oradb01"  Time: 03/08/14 07:50:56

 External utility reported error.

RMAN PID=12391

[Major] From: ob2rman@orasrv01.in.com "oradb01"  Time: 03/08/14 07:50:56

 The database reported error while performing requested operation.

RMAN-00571: ===========================================================

 RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

 RMAN-00571: ===========================================================

 RMAN-03009: failure of backup command on dev_0 channel at 03/08/2014 07:50:43

 ORA-19588: archived log RECID 99528 STAMP 841638632 is no longer valid

 Recovery Manager complete.

Problem:

>> The error can be due to invalid/stale archive files which were left uncleared even after the backup completion. Reason could be due to 2 simultaneous backup sessions where one refers to and the other had already backed up and not needed to be backed up.

>> It also happens when RMAN locates an archive file from the archive location that is no longer available in the Controlfile. These store very less info when compared to Recovery log.

Solution:

Execute the command “crosscheck archivelog all”.

The output can be as below:

Crosschecked 826 objects

Now you can re-trigger the backup spec and get a successful backup.

[70:24] A system error occurred when starting the target script or an agent module.

Problem:

Backup of client machine (Win 2003) fails with the following errors.

[Critical] From: INET_thread_vbda.exe@winclient01.in.com \"winclient01.in.com\"  Time: 03/04/14 08:46:39

[70:24] A system error occurred when starting the target script or an agent module. 

The system error code reported is 2 and the message resolves to [2] The system cannot find the file specified.

Solution:

>> Uninstall & re-install DP client. 
>> Import the DP client in the cell server using GUI [or] add the entry in the cell_info file.
>> Start the backup of DP client which should be successful.

Zero KB Archive backup

Archive backup of all the databases from a specific linux server was completing successfully with 0 bytes of data backed up.

[Normal] From: BSM@cellsrv01.in.com "dbclient01_DB01_AR"  Time: 03/03/14 07:15:23

        Backup session 2014/03/03-86 started.

[Normal] From: BSM@cellsrv01.in.com "dbclient01_DB01_AR"  Time: 03/03/14 07:23:36

        OB2BAR application on "dbclient01.in.com" successfully started.

[Normal] From: BSM@cellsrv01.in.com "dbclient01_DB01_AR"  Time: 03/03/14 07:23:36

        OB2BAR application on "dbclient01.in.com" disconnected.

[Normal] From: BSM@cellsrv01.in.com "dbclient01_DB01_AR"  Time: 03/03/14 07:23:37

        Backup Statistics:

                Session Queuing Time (hours)         0.00

                -------------------------------------------

                Completed Disk Agents ........          0

                Failed Disk Agents ...........          0

                Aborted Disk Agents ..........          0

                -------------------------------------------

                Disk Agents Total  ...........          0

                ================================

                Completed Media Agents .......          0

                Failed Media Agents ..........          0

                Aborted Media Agents .........          0

                -------------------------------------------

                Media Agents Total  ..........          0

                ================================

                Mbytes Total .................       0 MB

                Used Media Total .............          0

                Disk Agent Errors Total ......          0

What was causing this?? 

Is there any transactions happening in these DBs or not?? 

Any patching on server or DB that prevents the data transfer?? 

Why DP doesn't err anything??

No idea until we checked the seaudit log. The inet service was restarted from an local admin account, from then onwards the DP processes were lacking the root access rights which prevented from reading the data.

05 Mar 2014 04:47:42 D SURROGATE    useradm1     Read       69  2 USER.oradba          /opt/omni/lbin/inet  198.10.10.2       root

Solution:

Once again the inet daemon was restarted from root.

Commands to restart the DP inet service from Linux client:

#/etc/init.d/omni stop

#/etc/init.d/omni start

Tip of the Day!

If you attempt to get the serial number of a device and found "N/A", which means the device is missing/malfunctioning. This also happens when the device is in locked status by BSM/RSM/CSM/UMA/LTT.

C:\>devbra -dev

Tape    HP:Ultrium 1-SCSI  Path: "Tape0:0:0:0"  SN: "N/A"
            Description: CLAIMED:HP LTO drive
            Revision: E38W  Device type: lto [13]  Flags: 0x0011

To confirm the reason, run the below command to identify:

>>omnimm -show_locked_devs | grep <lock name>

If there's a lock, wait for the device to become free. Or you can forcibly free the device by executing the below:

>>omnidbutil -free_locked_devs <lock_name> //this would abort if any session utilizing the device

Once the device is free, run the 'devbra -dev' command and you will get the serial number. If not reboot the media server or the library depends on the error messages. If it still continues to be, replace the drive.