[90:114] Cannot unload medium, target slot (255) appears to be occupied

[Warning] From: BMA@linuxsrv155.com "ESL01_D17"  Time: 2/9/2015 1:13:44 PM
[90:54]   /dev/nst1
Cannot open device (Unit not ready.)

[Major] From: BMA@linuxsrv155.com "ESL01_D17"  Time: 2/9/2015 1:13:55 PM
[90:114]   By: UMA@mediasrv01.com@/dev/rchgr/autoch3
Cannot unload medium, target slot (255) appears to be occupied

Solution:

> The above error is due to DP couldn't unload the media from where it was actually loaded to the drive during backup/restore.

> Move the tape manually from the drive to any free slot using UMA by following the steps below. 

> Login to the media server from where the tape library/autoloader is connected.

>> run devbra-dev, the output will be the drives and exchanger detected from the device manager.

>> copy the scsi path of exchanger (here it is /dev/rchgr/autoch3)
>> execute the below command and move the media to any empty slot.
>uma -ioctl /dev/rchgr/autoch3

/dev/rchgr/autoch3> move D17 S13

> If there are no empty slots available, eject full media to I/O slots and move the media stuck in drive to free slot.
> Then retry the backup, which should be successful.

Hope this helps!

[Major] Host for device "ESL01_D01" not found.

[Normal] From: BSM@cellsrv01.in.com "Backupspec_win1"  Time: 3/21/2014 6:04:27 PM

 Backup session 2014/03/21-104 started.

[Major] From: BSM@cellsrv01.in.com "Backupspec_win1"  Time: 3/21/2014 6:04:32 PM

 Host for device ESL01_D01 not found.

[Major] From: BSM@cellsrv01.in.com "Backupspec_win1"  Time: 3/21/2014 6:04:32 PM

 Host for device ESL01_D02 not found.

[Major] From: BSM@cellsrv01.in.com "Backupspec_win1"  Time: 3/21/2014 6:04:32 PM

 Host for device ESL01_D03 not found.

[Normal] From: BSM@cellsrv01.in.com "Backupspec_win1"  Time: 3/21/2014 6:04:32 PM

 Backup Statistics:

         

  Session Queuing Time (hours)         0.00      

  -------------------------------------------    

  Completed Disk Agents ........          0        

  Failed Disk Agents ...........          7        

  Aborted Disk Agents ..........          0        

  -------------------------------------------    

  Disk Agents Total  ...........          7        

  ===========================================    

  Completed Media Agents .......          0        

  Failed Media Agents ..........          0        

  Aborted Media Agents .........          0        

  -------------------------------------------    

  Media Agents Total  ..........          0        

  ===========================================    

  Mbytes Total .................       0 MB      

  Used Media Total .............          0        

  Disk Agent Errors Total ......          0  

This error occurs when the backup device wasn't removed from the devices context following the media server deletion/migration to other cell.

In other words, the media server could have moved to another cell server or might have been decommissioned and by mistake the devices were not removed from the devices list.

Check the media server's existence, if available;

Check the cell server name, it is pointing to;

Mostly, if the orphan device entry is removed, this error will disappear.

[90:1004] Device address not found.

[Normal] From: BSM@cellsrv01.in.com "Backup_Spec_Win1"  Time: 4/16/2014 12:08:46 PM

 Backup session 2014/04/16-256 started.

[Normal] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:08:55 PM

 STARTING Media Agent "AUT01_D01"

[Normal] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:09:01 PM

 By: UMA@winsrv01.in.com@Changer0:7:0:1

 Loading medium from slot 8 to device Tape1:7:0:0C

[Warning] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:09:57 PM

 The device "AUT01_D01" could not be opened("Device could not be accessed")

[Normal] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:09:57 PM

 Starting the device path discovery process.

[Critical] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:10:00 PM

[90:1004]  Device address not found.

[Normal] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:10:00 PM

 Device path discovery process finished.

[Normal] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:10:00 PM

 By: UMA@winsrv01.in.com@Changer0:7:0:1

 Unloading medium to slot 8 from device Tape1:7:0:0C

[Normal] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:10:39 PM

 ABORTED Media Agent "AUT01_D01"

[Normal] From: BSM@cellsrv01.in.com "Backup_Spec_Win1"  Time: 4/16/2014 12:10:39 PM

 Backup Statistics:

         

  Session Queuing Time (hours)         0.00      

  -------------------------------------------    

  Completed Disk Agents ........          0        

  Failed Disk Agents ...........          4        

  Aborted Disk Agents ..........          0        

  -------------------------------------------    

  Disk Agents Total  ...........          4        

  ===========================================    

  Completed Media Agents .......          0        

  Failed Media Agents ..........          1        

  Aborted Media Agents .........          0        

  -------------------------------------------    

  Media Agents Total  ..........          1        

  ===========================================    

  Mbytes Total .................       0 MB      

  Used Media Total .............          0        

  Disk Agent Errors Total ......          0  

Troubleshooting steps as follows:

>> Logged into the media server and checked for devices claimed in Device Manager. Found the devices.

>> Ran devbra -dev to determine the SCSI address.

>> Found N/A for drive

C:\>devbra -dev

Exch    HP:1x8 G2 AUTOLDR  Path: "Changer0:0:0:1"  SN: "AABBCCDD1E"

        Description: CLAIMED:HP StorageWorks 1x8 Cartridge Autoloader

        Revision: 4.20  Flags: 0x0016  Slots: 8  Drives: 1

        Drive(s) SN:

                "ABCDEFGHIJ"

Tape    HP:Ultrium 3-SCSI  Path: "Tape0:0:0:0"  SN: "N/A"

        Description: CLAIMED:HP LTO3 Drive

        Revision: Q51W  Device type: lto [13]  Flags: 0x0011

>> Checked if the drive is locked by DP

[root@cellsrv01:/root]

# omnimm -show_locked_devs | grep AUT01_D01

[root@ cellsrv01:/root]

>> Stopped DP Inet from Services.msc and Ran LTT.

>> Got error message that "mma.exe" process is accessing the device. Killed the mma.exe process from the Task Manager.

>> rescanned the devices in LTT. Able to detect both autoloader and it's drive this time.

>> Ran devbra -dev, which got the SCSI address.

C:\>devbra -dev

Exch    HP:1x8 G2 AUTOLDR  Path: "Changer0:7:0:1"  SN: "AABBCCDD1E "

        Description: CLAIMED:HP StorageWorks 1x8 Cartridge Autoloader

        Revision: 4.20  Flags: 0x0016  Slots: 8  Drives: 1

        Drive(s) SN:

                "ABCDEFGHIJ "

Tape    HP:Ultrium 3-SCSI  Path: "Tape1:7:0:0C"  SN: "ABCDEFGHIJ "

        Description: CLAIMED:HP LTO3 Drive

        Revision: Q51W  Device type: lto [13]  Flags: 0x0011

>> Ran the backup successfully.

Cause :

The DP media agent was accessing the drive and didn't let any other process to send commands. After the hung process is killed, the drive was accessible for normal operations.

[90:801] Active Removable Storage Manager (RSM) service found on local system.

[Normal] From: BMA@mediasrv01.in.com "MSL2024_D02"  Time: 3/23/2014 2:57:16 PM

 STARTING Media Agent "MSL2024_D02"

[Warning] From: BMA@mediasrv01.in.com "MSL2024_D02"  Time: 3/23/2014 2:57:16 PM

[90:801]  Active Removable Storage Manager (RSM) service found on local system.

 [Warning] From: UMA@mediasrv01.in.com "MSL2024"  Time: 3/23/2014 2:57:18 PM

[90:59]   Changer0:0:0:0

 Cannot open exchanger control device ([2] The system cannot find the file specified. )

[Warning] From: UMA@mediasrv01.in.com "MSL2024"  Time: 3/23/2014 2:57:18 PM

 The device "MSL2024" could not be opened ("Device could not be accessed") 

[Normal] From: UMA@mediasrv01.in.com "MSL2024"  Time: 3/23/2014 2:57:18 PM

 Starting the device path discovery process. 

[Warning] From: UMA@mediasrv01.in.com "MSL2024"  Time: 3/23/2014 2:57:19 PM

[90:59]   Changer0:0:0:0

 Cannot open exchanger control device ([2] The system cannot find the file specified. )

[Normal] From: BMA@ mediasrv01.in.com "MSL2024"  Time: 3/23/2014 2:57:21 PM ABORTED Media Agent " MSL2024_D02"

Solution:

Removable Storage Manager (RSM) service on MS Windows 2003 might cause device issues in backup environment.

It is recommended to stop and disable the RSM service on all MS Windows 2003 systems.

RSM Service in Started status

[61:2015] Timeout waiting for the devices to get free.

[Critical] From: BSM@cellsrv01.in.com "cellsrv01_IDB"  Time: 02/27/14 02:00:31

[61:2015]  Timeout waiting for the devices to get free.

The session will terminate.

Ø  It is very common in every backup environment to share the same device for different backups. This is an error message due to device contention issue.

Ø  The backup device selected in the backup specification is unavailable for the backup to start. It is in use by another process or by another backup/restore/copy sessions.

Ø   Check the device status using the lock name. (Check Here for the commands used to find a locked device). Wait for the device to be free.

Ø  The backup will be queued for global timeout seconds and will fail if no device is freed / allocated to the backup session.

Ø  The queuing time can be found at the end of backup session from backup statistics. Shown below

  

Backup Statistics:

          

                   Session Queuing Time (hours)         0.00        

                   -------------------------------------------      

                   Completed Disk Agents ........          5          

                   Failed Disk Agents ...........          0          

                   Aborted Disk Agents ..........          0          

                   -------------------------------------------      

                   Disk Agents Total  ...........          5          

                   =====================================     

                   Completed Media Agents .......          1          

                   Failed Media Agents ..........          0          

                   Aborted Media Agents .........          0          

                   -------------------------------------------      

                   Media Agents Total  ..........          1          

                   ===========================================      

                   Mbytes Total .................   17985 MB        

                   Used Media Total .............          1          

                   Disk Agent Errors Total ......          0    

[90:63] Cannot load exchanger medium (Medium error.)

[Normal] From: BMA@winsrv01.in.com "MSL_D01"  Time: 3/24/2014 10:53:43 PM

 By: UMA@winsrv01.in.com@Changer0:0:0:0

 Loading medium from slot 9 to device Tape0:0:0:0

[Major] From: BMA@winsrv01.in.com "MSL_D01"  Time: 3/24/2014 11:03:16 PM

[90:63]   By: UMA@winsrv01.in.com@Changer0:0:0:0

 Cannot load exchanger medium (Medium error.)

[Normal] From: BMA@winsrv01.in.com "MSL_D01"  Time: 3/24/2014 11:03:16 PM

 ABORTED Media Agent "MSL_D01"

[Normal] From: BMA@winsrv01.in.com " MSL_D02"  Time: 3/24/2014 11:03:21 PM

 STARTING Media Agent " MSL_D02"

[Normal] From: BMA@winsrv01.in.com " MSL_D02"  Time: 3/24/2014 11:03:26 PM

 By: UMA@winsrv01.in.com@Changer0:0:0:0

 Loading medium from slot 9 to device Tape1:0:0:0

[Major] From: BMA@winsrv01.in.com " MSL_D02"  Time: 3/24/2014 11:13:10 PM

[90:63]   By: UMA@winsrv01.in.com@Changer0:0:0:0

 Cannot load exchanger medium (Medium error.)

[Normal] From: BMA@winsrv01.in.com " MSL_D02"  Time: 3/24/2014 11:13:10 PM

 ABORTED Media Agent " MSL_D02"

Solution:

Exchanger medium is failing while handling media from particular slot (here it is slot 9). The media has errors that is residing slot 9.

Move to 'Devices & Media' context -> Click Library -> Click Slot -> move to Slot 9.

--- Check the status of media in slot 9.

--- It was poor media. Isolate the media by moving it to separate media pool and unload it from library. Wait for the media to expire and reformat it. If the status is still shows poor even after formatting, it is not advisable to use it henceforth.

--- Clean the affected drives with good cleaning tape and run a test backup. They should be good for backups.

Tip of the Day!

If you attempt to get the serial number of a device and found "N/A", which means the device is missing/malfunctioning. This also happens when the device is in locked status by BSM/RSM/CSM/UMA/LTT.

C:\>devbra -dev

Tape    HP:Ultrium 1-SCSI  Path: "Tape0:0:0:0"  SN: "N/A"
            Description: CLAIMED:HP LTO drive
            Revision: E38W  Device type: lto [13]  Flags: 0x0011

To confirm the reason, run the below command to identify:

>>omnimm -show_locked_devs | grep <lock name>

If there's a lock, wait for the device to become free. Or you can forcibly free the device by executing the below:

>>omnidbutil -free_locked_devs <lock_name> //this would abort if any session utilizing the device

Once the device is free, run the 'devbra -dev' command and you will get the serial number. If not reboot the media server or the library depends on the error messages. If it still continues to be, replace the drive.

[61:4006] Couldnot connect to inet in order to start BMA@ "Device_name".

Backup Error

[Major] From: BSM@cellsrv01.in.com "winsrvspec01"  Time: 2/28/2014 6:53:57 PM
 [61:4006]   Couldnot connect to inet in order to start BMA@Mediasrv02.in.com "MSL2024_D1".

[Critical] From: BMA@cellsrv01.in.com "MSL2024_D1"  Time: 2/28/2014 6:54:55 PM
[90:1004] Device address not found.

Solution

>> The DP Inet service running in the media server is not reachable from the cell server and the Media Agent couldn't be started. When checked the media server is 'Removed' from production and was shutdown.

>> In this case, the device mapping of the drive should be removed/disabled 'MSL2024_D1' from DP and the backup spec need to be modified accordingly to eliminate failures in future.

>> You can verify the media server reachability from cell manager by pinging, telnet the DP port 5555, can also use omnitcpchk commands.

For ex:

[root@ cellsrv01:/root]

# ping Mediasrv02.in.com 

PING Mediasrv02.in.com: 64 byte packets

----Mediasrv02.in.com PING Statistics----

11 packets transmitted, 0 packets received, 100% packet loss

[root@ cellsrv01:/root]

# /opt/omni/sbin/utilns/omnitcpchk -host Mediasrv02.in.com 

Testing connection Mediasrv02.in.com <---> cellsrv01.in.com....

ERROR!

==============================

TcpCheck failed for host pair:

Mediasrv02.in.com <---> cellsrv01.in.com

[root@ cellsrv01:/root]

# telnet Mediasrv02.in.com 5555

Trying...

telnet: Unable to connect to remote host: Connection timed out

>> The backup would complete using the next device in line, if there are more than one device selected in the backup specification.

  

>> Optionally, the backup would also run with the next zoned drive that has correct SCSI path and available for this backup session.

[Critical] Duplicate BARCODE information from Media Agent

[Normal] From: MSM@cellservero1.in.com "MSL2024_L01"  Time: 22.2.2014 10:33:56

      Media session 2014/02/22-179 started.

[Normal] From: UMA@mediaserver01.in.com "MSL2024_L01"  Time: 22.2.2014 10:34:07

      STARTING Media Agent "MSL2024_L01"

[Critical] From: MSM@cellservero1.in.com "MSL2024_L01"  Time: 22.2.2014 10:34:10

Duplicate BARCODE information from Media Agent, Not updating the Repository

[Normal] From: UMA@mediaserver01.in.com "MSL2024_L01"  Time: 22.2.2014 10:34:09

      COMPLETED Media Agent "MSL2024_L01"

======================================================

                      23 cartridges out of 23 successfully scanned.

======================================================

>> The Library model is MSL2024, but there were only 23 slots scanned and the barcode scan failed.

>> Check the Repository, It shows only 23 slots, add the 24th slot.

>> Now barcode info has valid argument to scan.



Guest Post done by: Anbu

[Critical] [90:63] Cannot load exchanger medium (Target drive is busy.)

It can be interesting in troubleshooting a device issue sometimes. Let's deal with the devices in this post..

Error Message:

[Normal] From: BMA@Mediasrv01.in.com "Drive01"  Time: 2/13/2014 5:41:04 PM

                By: UMA@Mediasrv01.in.com@Changer0:7:0:1

                Loading medium from slot 2 to device Tape0:7:0:0C

[Critical] From: BMA@Mediasrv01.in.com " Drive01"  Time: 2/13/2014 5:41:24 PM

[90:63]                  By: UMA@Mediasrv01.in.com@Changer0:7:0:1

                Cannot load exchanger medium (Target drive is busy.)

From the error message, we could find that the "Target drive is busy". 

>> Login to the media server from where the tape library/autoloader is connected

>> Run devbra -dev, the output would be the drives and exchanger detected from the device manager.

>> Use the uma along with the scsi path of exchanger (here it is Changer0:7:0:1) in cmd line (please see below).

>> Check if there is stuck tape in drive. Move the tape to a free slot either from GUI/CLI and run the backup.

>> there are chances that the library might be fully loaded and the drive too is holding a tape. So manually remove the tape from slot/drive.

Using UMA utility for device management:

Example of fully loaded autoloader which needs manually unload media from drive.

C:\uma -ioctl  Changer0:7:0:1

Changer0:7:0:1> stat d

          1 D1    Full  "" ""

Changer0:7:0:1> stat s

        1001 S1   Full  "" ""

        1002 S2   Full  "" ""

        1003 S3   Full  "" ""

        1004 S4   Full  "" ""

        1005 S5   Full  "" ""

        1006 S6   Full  "" ""

        1007 S7   Full  "" ""

        1008 S8   Full  "" ""

(OR)

Example of unloading media from drive to empty slot.

C:\uma -ioctl  Changer0:7:0:1

Changer0:7:0:1> stat d

          1 D1    Full  "" ""

Changer0:7:0:1> stat s

        1001 S1   Full  "" ""

        1002 S2   Full  "" ""

        1003 S3   Empty  "" ""

        1004 S4   Full  "" ""

        1005 S5   Full  "" ""

        1006 S6   Full  "" ""

        1007 S7   Full  "" ""

        1008 S8   Full  "" ""

Changer0:7:0:1> move D1 S3

Changer0:7:0:1> stat d

          1 D1    Empty  "" ""

>> This is one of many cases for 'Cannot load exchanger medium' error.

[90:54] Cannot unload medium, target slot (255) appears to be occupied

[Warning] From: BMA@server01.na.com "ESLE01_D1"  Time: 2/9/2014 1:13:44 PM
[90:54]   /dev/nst1
Cannot open device (Unit not ready.)

[Major] From: BMA@server01.na.com "ESLE01_D1"  Time: 2/9/2014 1:13:55 PM
[90:114]   By: UMA@cellserver01.na.com@/dev/autoch3
Cannot unload medium, target slot (255) appears to be occupied

> The above error is due to DP couldn't unload the media from where it was actually loaded to the drive.

> Move the tape manually from the drive using UMA
/dev/autoch3> move D3 S13

> Then retry the backup, which should be successful/the error will not repeat.