[90:114] Cannot unload medium, target slot (255) appears to be occupied

[Warning] From: BMA@linuxsrv155.com "ESL01_D17"  Time: 2/9/2015 1:13:44 PM
[90:54]   /dev/nst1
Cannot open device (Unit not ready.)

[Major] From: BMA@linuxsrv155.com "ESL01_D17"  Time: 2/9/2015 1:13:55 PM
[90:114]   By: UMA@mediasrv01.com@/dev/rchgr/autoch3
Cannot unload medium, target slot (255) appears to be occupied

Solution:

> The above error is due to DP couldn't unload the media from where it was actually loaded to the drive during backup/restore.

> Move the tape manually from the drive to any free slot using UMA by following the steps below. 

> Login to the media server from where the tape library/autoloader is connected.

>> run devbra-dev, the output will be the drives and exchanger detected from the device manager.

>> copy the scsi path of exchanger (here it is /dev/rchgr/autoch3)
>> execute the below command and move the media to any empty slot.
>uma -ioctl /dev/rchgr/autoch3

/dev/rchgr/autoch3> move D17 S13

> If there are no empty slots available, eject full media to I/O slots and move the media stuck in drive to free slot.
> Then retry the backup, which should be successful.

Hope this helps!

Restarting DP Cell Manager

How to restart DP cell manager

When the cell manager hangs or any kind of malfunction requires DP cell manager to be reset the following steps could help! 

1. To check the status of cell manager services
/opt/omni/sbin/omnisv –status

2. Run the following cmd to Stop all DP related services (CRS, RDS and MMD)
/opt/omni/sbin/omnisv –stop

3. Double check whether all the services went down
/opt/omni/sbin/omnisv –status

4. Check if any omni related process is running. If not move to next step
Ps –ef |grep omni    ( Command to check any omni processes are in hung status)
kill -9 <PID>            ( kill the process if any using its Process ID)
Ps –ef |grep omni    ( double check and move to next step)

5. Start the cell manager services now
/opt/omni/sbin/omnisv –start

6. Check all the services are up. If not follow steps 2, 3 & 4
/opt/omni/sbin/omnisv –status

7. Once the services are up, run the below command before any backup is triggered after the restart. This will clear all the in-progress sessions that went to hung status during the services restart.
/opt/omni/sbin/omnidbutil –clear

Please use these steps with caution!

[90:1004] Device address not found.

[Normal] From: BSM@cellsrv01.in.com "Backup_Spec_Win1"  Time: 4/16/2014 12:08:46 PM

 Backup session 2014/04/16-256 started.

[Normal] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:08:55 PM

 STARTING Media Agent "AUT01_D01"

[Normal] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:09:01 PM

 By: UMA@winsrv01.in.com@Changer0:7:0:1

 Loading medium from slot 8 to device Tape1:7:0:0C

[Warning] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:09:57 PM

 The device "AUT01_D01" could not be opened("Device could not be accessed")

[Normal] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:09:57 PM

 Starting the device path discovery process.

[Critical] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:10:00 PM

[90:1004]  Device address not found.

[Normal] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:10:00 PM

 Device path discovery process finished.

[Normal] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:10:00 PM

 By: UMA@winsrv01.in.com@Changer0:7:0:1

 Unloading medium to slot 8 from device Tape1:7:0:0C

[Normal] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:10:39 PM

 ABORTED Media Agent "AUT01_D01"

[Normal] From: BSM@cellsrv01.in.com "Backup_Spec_Win1"  Time: 4/16/2014 12:10:39 PM

 Backup Statistics:

         

  Session Queuing Time (hours)         0.00      

  -------------------------------------------    

  Completed Disk Agents ........          0        

  Failed Disk Agents ...........          4        

  Aborted Disk Agents ..........          0        

  -------------------------------------------    

  Disk Agents Total  ...........          4        

  ===========================================    

  Completed Media Agents .......          0        

  Failed Media Agents ..........          1        

  Aborted Media Agents .........          0        

  -------------------------------------------    

  Media Agents Total  ..........          1        

  ===========================================    

  Mbytes Total .................       0 MB      

  Used Media Total .............          0        

  Disk Agent Errors Total ......          0  

Troubleshooting steps as follows:

>> Logged into the media server and checked for devices claimed in Device Manager. Found the devices.

>> Ran devbra -dev to determine the SCSI address.

>> Found N/A for drive

C:\>devbra -dev

Exch    HP:1x8 G2 AUTOLDR  Path: "Changer0:0:0:1"  SN: "AABBCCDD1E"

        Description: CLAIMED:HP StorageWorks 1x8 Cartridge Autoloader

        Revision: 4.20  Flags: 0x0016  Slots: 8  Drives: 1

        Drive(s) SN:

                "ABCDEFGHIJ"

Tape    HP:Ultrium 3-SCSI  Path: "Tape0:0:0:0"  SN: "N/A"

        Description: CLAIMED:HP LTO3 Drive

        Revision: Q51W  Device type: lto [13]  Flags: 0x0011

>> Checked if the drive is locked by DP

[root@cellsrv01:/root]

# omnimm -show_locked_devs | grep AUT01_D01

[root@ cellsrv01:/root]

>> Stopped DP Inet from Services.msc and Ran LTT.

>> Got error message that "mma.exe" process is accessing the device. Killed the mma.exe process from the Task Manager.

>> rescanned the devices in LTT. Able to detect both autoloader and it's drive this time.

>> Ran devbra -dev, which got the SCSI address.

C:\>devbra -dev

Exch    HP:1x8 G2 AUTOLDR  Path: "Changer0:7:0:1"  SN: "AABBCCDD1E "

        Description: CLAIMED:HP StorageWorks 1x8 Cartridge Autoloader

        Revision: 4.20  Flags: 0x0016  Slots: 8  Drives: 1

        Drive(s) SN:

                "ABCDEFGHIJ "

Tape    HP:Ultrium 3-SCSI  Path: "Tape1:7:0:0C"  SN: "ABCDEFGHIJ "

        Description: CLAIMED:HP LTO3 Drive

        Revision: Q51W  Device type: lto [13]  Flags: 0x0011

>> Ran the backup successfully.

Cause :

The DP media agent was accessing the drive and didn't let any other process to send commands. After the hung process is killed, the drive was accessible for normal operations.

[61:12500] cannot connect to inet for getting file system list on host

Error:

[Critical]   From: BSM@cellsrv01.in.com "backup_spec_01” Time: 8/16/2014
[61:12500] cannot connect to inet for getting file system list
              on host" winclnt01.in.com".
 

Reason:

Host may be down or Data protector client services (Inet) down.

Troubleshooting steps for Windows clients are as follows:

1. Check if the client (i.e. winclnt01.in.com in above error) is pingable from the cell manager, if not check the issue with platform team.

2. If the server is pingable, telnet the client using port 5555 (eg: #Telnet <Client name> 5555)

3. If the telnet failed, login to the windows client and check whether “Data protector Inet” service is running or not. If not start the “Data Protector Inet” service, Just by right clicking the service and restart it.

4. If the Data protector service is not listed in the “services.msc”, the DP agent might be removed from the client. Push the DP agents on this client from cell server.

5. If the telnet failed using port 5555 even data protector service running, check with platform team to open the port.

Troubleshooting steps for UNIX clients are as follows:

1. Follow the steps 1 and 2 as explained above.

2. If the telnet failed, login to the UNIX client and check for the Omni entries in /etc/services and /etc/inetd.conf using the commands below. If not found, DP client needs to be reinstalled.

[root@unxclnt01:/root]

# egrep omni /etc/services

omni  5555/tcp     # DATA-PROTECTOR

[root@unxclnt01:/root]

# egrep omni /etc/inetd.conf

omni stream tcp nowait root /opt/omni/lbin/inet inet -log /var/opt/omni//log/inet.log

3. Check if the port 5555 is listening using the command below. If not, raise a request to platform team to enable the port and start the inetd daemon.

[root@unxclnt01:/root]

# netstat -an | grep 5555

tcp        0      0  *.5555                 *.*                     LISTEN

4. Check if the ‘inetd’ daemon is running, if not start it using the below commands:

[root@unxclnt01:/root]

# ps -ef | grep inetd

    root  2144     1  0  Aug 10  ?         7:20 /usr/sbin/inetd

# /etc/inetd –s

[61:2015] Timeout waiting for the devices to get free.

[Critical] From: BSM@cellsrv01.in.com "cellsrv01_IDB"  Time: 02/27/14 02:00:31

[61:2015]  Timeout waiting for the devices to get free.

The session will terminate.

Ø  It is very common in every backup environment to share the same device for different backups. This is an error message due to device contention issue.

Ø  The backup device selected in the backup specification is unavailable for the backup to start. It is in use by another process or by another backup/restore/copy sessions.

Ø   Check the device status using the lock name. (Check Here for the commands used to find a locked device). Wait for the device to be free.

Ø  The backup will be queued for global timeout seconds and will fail if no device is freed / allocated to the backup session.

Ø  The queuing time can be found at the end of backup session from backup statistics. Shown below

  

Backup Statistics:

          

                   Session Queuing Time (hours)         0.00        

                   -------------------------------------------      

                   Completed Disk Agents ........          5          

                   Failed Disk Agents ...........          0          

                   Aborted Disk Agents ..........          0          

                   -------------------------------------------      

                   Disk Agents Total  ...........          5          

                   =====================================     

                   Completed Media Agents .......          1          

                   Failed Media Agents ..........          0          

                   Aborted Media Agents .........          0          

                   -------------------------------------------      

                   Media Agents Total  ..........          1          

                   ===========================================      

                   Mbytes Total .................   17985 MB        

                   Used Media Total .............          1          

                   Disk Agent Errors Total ......          0    

BR0073E Setting of BRBACKUP lock failed

Problem:

BR0051I BRBACKUP 7.20 (25)
BR0055I Start of database backup: benini.qub 2014-03-18 08.00.26
BR0484I BRBACKUP log file: /oracle/SID/sapbackup/benini.qub
BR0071E BRBACKUP currently running or was killed
BR0072I Please delete file /oracle/SID/sapbackup/.lock.brb if BRBACKUP was killed
BR0073E Setting of BRBACKUP lock failed

BR0056I End of database backup: benini.qub 2014-03-18 08.00.26
BR0280I BRBACKUP time stamp: 2014-03-18 08.00.26
BR0054I BRBACKUP terminated with errors
[Major] From: OB2BAR_OMNISAP@orsapsrv1.in.com "OMNISAP"  Time: 03/18/2014 08:00:26 AM
BRBACKUP /usr/sap/SID/SYS/exe/run/brbackup -t online_split -d util_file -c -p initSID.sap.bc -m all -q split -u / returned 3

[Normal] From: BSM@cellsrv01.incom "orsapsrv1_SID01"  Time: 3/18/2014 8:00:27 AM
OB2BAR application on "orsapsrv1.in.com" disconnected.

Solution:

>> Login to the client with SID of the database and check for any brbackup process running.

*****No brbackup process running*****

# ps -ef | grep brbackup

root     17138  9230  0 08:19 pts/0    00:00:00 grep brbackup

>> Check for any progressing DP backups from monitor context. This can also be initiated from DB end too. Wait until the backup completes and then check for the brbackup process again. 

>> Kill the .lock.brb file <path - /oracle/SID/sapbackup/.lock.brb > if there was aborted/hung backup session. Start the backup spec which should complete.

Batch script for " IDB Maintenance & Resolving the Velocis Error " for Windows Servers…

Solution : copy the below script and paste in notepad and save it as " IDB_velosis.bat" file and just click on the file ... 

Your Velocis error and IDB maintenance will be completed in Just-a-click!

Note:- Modify the script according to where [Which Drive/Path] we installed the DataProtector 

Copy the text which in blue color 

echo # IDB Maintenance & Resolving the Velosis Error #

echo # Resolving the Velosis Error #
cd \

D:

cd Program files\omniback\bin

omnisv -status

omnisv -stop

taskkill /IM vbda.exe /F /T
taskkill /IM bsm.exe /F /T
taskkill /IM dbsm.exe /F /T
taskkill /IM vrda.exe /F /T
taskkill /IM uma.exe /F /T
taskkill /IM crs.exe /F /T
taskkill /IM rds.exe /F /T
taskkill /IM mmd.exe /F /T

cd \

cd Program Files\OmniBack\tmp

del CRS.pid
del dbcheck.cdb
del dbcheck.mmdb
del lic.ctx
del mmd.ctx

cd \

cd Program Files\OmniBack\db40\logfiles\syslog

del *.chg
del *.chk

cd \

cd Program Files\OmniBack\db40\datafiles\catalog

rename rdm.bil rdm.bil.old
rename rdm.chi rdm.chi.old

cd \

cd Program Files\OmniBack\bin

echo # Now Bring up the databae

omnisv -start

omnidbutil -clear

omnidbutil -free_locked_devs

echo # IDB Maintanence

omnidbutil -purge -messages 30 -force
omnidbutil -purge -sessions 30 -force
omnidbutil -purge -dcbf -force
omnidbutil -purge -filenames -force

exit

End of the script

Try the same in Testing environment prior to Production, All the best :)

Tip of the Day!

If you attempt to get the serial number of a device and found "N/A", which means the device is missing/malfunctioning. This also happens when the device is in locked status by BSM/RSM/CSM/UMA/LTT.

C:\>devbra -dev

Tape    HP:Ultrium 1-SCSI  Path: "Tape0:0:0:0"  SN: "N/A"
            Description: CLAIMED:HP LTO drive
            Revision: E38W  Device type: lto [13]  Flags: 0x0011

To confirm the reason, run the below command to identify:

>>omnimm -show_locked_devs | grep <lock name>

If there's a lock, wait for the device to become free. Or you can forcibly free the device by executing the below:

>>omnidbutil -free_locked_devs <lock_name> //this would abort if any session utilizing the device

Once the device is free, run the 'devbra -dev' command and you will get the serial number. If not reboot the media server or the library depends on the error messages. If it still continues to be, replace the drive.

To check/free locked devices in HP DP

HP DP Command to check/free a device

>> omnimm -show_locked_devs | grep <lock_name>
this command would list all the devices that are being used by the DP cell server. Medium, cartridge, devices

Ex:

Type:               Medium
Name/Id:   c75f:0034:0000:XXXX            //Media ID
Pid:                 23362                                       //Process Id that utilizes the media
Host:               cellsrv01.in.com                    //name of the cell manager (useful in MOM)
Label:             BM1000L3                              //Medium label

Type:         Device        
Name/Id:   ESLE1_D02                          //Drive name
Pid:           25754                                     //Process Id that utilizes the drive
Host:              cellsrv01.in.com                    //name of the cell manager, (useful in MOM)

Type:         Cartridge                     
Name/Id:   ESLE1                                  //Name of the library
Pid:           25836                                   //Process Id that utilises the drive
Host:         cellsrv01.in.com                  //name of the cell manager, (useful in MOM)
Location: 102                           

>> omnidbutil -show_locked_devs
The command would list all the devices, media, cartridge and slots that are in use by Data Protector cell manager.

>> omnidbutil -free_locked_devs <device_name>

Ex:

>> omnidbutil -free_locked_devs ESLE1_D02
Confirm the command by hitting 'Y': Y

After execution of this command, the drive will get released and can be used for any other purpose.

P.S: Omnidbutil / Omnimm commands can be used interchangeably to identify locked devices, to free locked devs, use omnidbutil.

[61:4006] Couldnot connect to inet in order to start BMA@ "Device_name".

Backup Error

[Major] From: BSM@cellsrv01.in.com "winsrvspec01"  Time: 2/28/2014 6:53:57 PM
 [61:4006]   Couldnot connect to inet in order to start BMA@Mediasrv02.in.com "MSL2024_D1".

[Critical] From: BMA@cellsrv01.in.com "MSL2024_D1"  Time: 2/28/2014 6:54:55 PM
[90:1004] Device address not found.

Solution

>> The DP Inet service running in the media server is not reachable from the cell server and the Media Agent couldn't be started. When checked the media server is 'Removed' from production and was shutdown.

>> In this case, the device mapping of the drive should be removed/disabled 'MSL2024_D1' from DP and the backup spec need to be modified accordingly to eliminate failures in future.

>> You can verify the media server reachability from cell manager by pinging, telnet the DP port 5555, can also use omnitcpchk commands.

For ex:

[root@ cellsrv01:/root]

# ping Mediasrv02.in.com 

PING Mediasrv02.in.com: 64 byte packets

----Mediasrv02.in.com PING Statistics----

11 packets transmitted, 0 packets received, 100% packet loss

[root@ cellsrv01:/root]

# /opt/omni/sbin/utilns/omnitcpchk -host Mediasrv02.in.com 

Testing connection Mediasrv02.in.com <---> cellsrv01.in.com....

ERROR!

==============================

TcpCheck failed for host pair:

Mediasrv02.in.com <---> cellsrv01.in.com

[root@ cellsrv01:/root]

# telnet Mediasrv02.in.com 5555

Trying...

telnet: Unable to connect to remote host: Connection timed out

>> The backup would complete using the next device in line, if there are more than one device selected in the backup specification.

  

>> Optionally, the backup would also run with the next zoned drive that has correct SCSI path and available for this backup session.

Cannot get information on file '/var/opt/omni/tmp/rcvcat.exp' Error: 2

[Major] From: OB2BAR_DMA@dbsrv01.in.com "DB001"  Time: 2/23/2014 3:05:36 PM
      Cannot get information on file '/var/opt/omni/tmp/rcvcat.exp' Error: 2.

[Major] From: ob2rman@dbsrv01.in.com "DB001"  Time: 02/23/2014 03:05:44 PM
      Backup of recovery catalog failed.

Solution:

>> You would receive this error in linux OS and clustered Oracle DB. The patch DPLNX_00094 would fix this.

Alternatively, you can try the below steps:

>> From the Integration specific option, Enable the option 'Disable recovery catalog auto backup'. This would skip the backup of recovery catalog backup from DP. Rerun the backup now. If you still see the same errors and wonder why, then move on to the next step.

>> The backup script in barlists always takes precedence from the DP GUI options. Open the spec in VI editor and Move to the end of the barlist and edit the script with value "-skip RCVCAT".

Example:
CLIENT "DB001" dbsrv01.in.com
{
        -exec ob2rman.exe
        -args {
                "-skip RCVCAT"                                             //Skips the recovery catalog backup
                "-pre  /home/user/DB001_weekly_full.rman"     //Pre exec scripts if any
                "-backup"                
        }
        -input {
                "run {"
                "allocate channel 'dev_0' type 'sbt_tape'"        //channel allocation
                " parms 'ENV=(OB2BARTYPE=Oracle8,OB2APPNAME=DB001,OB2BARLIST=dbsrv01_DB001_ONLINE)';"
                "backup"
                " format 'dbsrv01_DB001_ONLINE<DB001_%s:%t:%p>.dbf'"
                " current controlfile;"
                "}"
        }
        -profile
}  -protect days 28                                                      //data protection in days