Wednesday, July 30, 2014

[81:52] Not a valid mount point => aborting.

[Critical] From: VBDA@winsrv01 "winsrv01 [/G]"  Time: 5/26/2014 9:28:58 AM
[81:52]  /G
                Not a valid mount point => aborting.

An attempt to backup the specific drive (G:/ in this example) connected to the DP client machine was failed and no data will be backed up by DP.

In Windows :

  • Login to the windows client machine and check if the drive is accessible.
  • Check if the drive is local drive or zoned drive and the zoning still exists.

In case the zoned drive no more exists, you can exclude it in the backup spec. 

If the drive exists on the server and it is accessible, try deleting the \tmp folder on the windows client machine and rerun the backup.

Default path to tmp folder for windows 2003/NT/XP:
C:\Program Files\OmniBack\tmp

Default path to tmp for windows 2008:
"C:\ProgramData\Omniback\tmp"

Hope this helps!!!

Sunday, June 22, 2014

[Major] Host for device "ESL01_D01" not found.

[Normal] From: BSM@cellsrv01.in.com "Backupspec_win1"  Time: 3/21/2014 6:04:27 PM
 Backup session 2014/03/21-104 started.

[Major] From: BSM@cellsrv01.in.com "Backupspec_win1"  Time: 3/21/2014 6:04:32 PM
 Host for device ESL01_D01 not found.

[Major] From: BSM@cellsrv01.in.com "Backupspec_win1"  Time: 3/21/2014 6:04:32 PM
 Host for device ESL01_D02 not found.

[Major] From: BSM@cellsrv01.in.com "Backupspec_win1"  Time: 3/21/2014 6:04:32 PM
 Host for device ESL01_D03 not found.

[Normal] From: BSM@cellsrv01.in.com "Backupspec_win1"  Time: 3/21/2014 6:04:32 PM

 Backup Statistics:
         
  Session Queuing Time (hours)         0.00      
  -------------------------------------------    
  Completed Disk Agents ........          0        
  Failed Disk Agents ...........          7        
  Aborted Disk Agents ..........          0        
  -------------------------------------------    
  Disk Agents Total  ...........          7        
  ===========================================    
  Completed Media Agents .......          0        
  Failed Media Agents ..........          0        
  Aborted Media Agents .........          0        
  -------------------------------------------    
  Media Agents Total  ..........          0        
  ===========================================    
  Mbytes Total .................       0 MB      
  Used Media Total .............          0        
  Disk Agent Errors Total ......          0  


This error occurs when the backup device wasn't removed from the devices context following the media server deletion/migration to other cell.

In other words, the media server could have moved to another cell server or might have been decommissioned and by mistake the devices were not removed from the devices list.

Check the media server's existence, if available;
Check the cell server name, it is pointing to;

Mostly, if the orphan device entry is removed, this error will disappear.


Thursday, June 12, 2014

[61:2051] All mountpoints on host \"winsrv01.in.com\" are excluded.

Error:

[Major] From: BSM@cellsrv01.in.com \"win2008bck_spec1\"  Time: 06/09/14 19:09:47
[61:2051] All mountpoints on host \"winsrv01.in.com\" are excluded.
Nothing will be backed up.

Reason:

>> This is something related to Inet error (which was discussed Here), but not necessarily be due to network disruptions. This could also be due to platform side issue.

Analysis:

>> Try to ping and telnet the windows server using default port number from cell server or any other server on the same network.

>> In this case, the server was pretty slow and non-responsive. Telnet to the default port was hung, so requested wintel team for server optimization which must fix the error.

>> Double-check by expanding the server's objects from the backup specification (which would hung as well).

Have a Great Day :)

Tuesday, June 10, 2014

TSM Insufficient system resources exist to complete the requested service


Error:

ANS9999E ntrc.cpp(928): Received Win32 RC 1450 (0x000005aa) from FileRead(): ReadFile '\\xxxx\. Error description: Insufficient system resources exist to complete the requested service.

Solution:

>> Edit the dsm.opt file by adding the param "MEMORYEFFICIENTBACKUP YES".
>> Save it.
>> Try a manual backup and check if the backup completes.

If not, this error needs attention from platform end.

>> Windows OS has the paging file space memory that needs attention. 
>> Issue can be resolved by adding/modifying "PoolUsageMaximum & PagedPoolSize" in registry

For more info, please refer the article from microsoft.

http://support.microsoft.com/kb/304101

Your backup should be successful, once this is fixed.

Have a nice Day!!

Tuesday, April 22, 2014

[90:1004] Device address not found.

[Normal] From: BSM@cellsrv01.in.com "Backup_Spec_Win1"  Time: 4/16/2014 12:08:46 PM
 Backup session 2014/04/16-256 started.

[Normal] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:08:55 PM
 STARTING Media Agent "AUT01_D01"

[Normal] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:09:01 PM
 By: UMA@winsrv01.in.com@Changer0:7:0:1
 Loading medium from slot 8 to device Tape1:7:0:0C

[Warning] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:09:57 PM
 The device "AUT01_D01" could not be opened("Device could not be accessed")

[Normal] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:09:57 PM
 Starting the device path discovery process.

[Critical] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:10:00 PM
[90:1004]  Device address not found.

[Normal] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:10:00 PM
 Device path discovery process finished.

[Normal] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:10:00 PM
 By: UMA@winsrv01.in.com@Changer0:7:0:1
 Unloading medium to slot 8 from device Tape1:7:0:0C

[Normal] From: BMA@winsrv01.in.com "AUT01_D01"  Time: 4/16/2014 12:10:39 PM
 ABORTED Media Agent "AUT01_D01"

[Normal] From: BSM@cellsrv01.in.com "Backup_Spec_Win1"  Time: 4/16/2014 12:10:39 PM

 Backup Statistics:
         
  Session Queuing Time (hours)         0.00      
  -------------------------------------------    
  Completed Disk Agents ........          0        
  Failed Disk Agents ...........          4        
  Aborted Disk Agents ..........          0        
  -------------------------------------------    
  Disk Agents Total  ...........          4        
  ===========================================    
  Completed Media Agents .......          0        
  Failed Media Agents ..........          1        
  Aborted Media Agents .........          0        
  -------------------------------------------    
  Media Agents Total  ..........          1        
  ===========================================    
  Mbytes Total .................       0 MB      
  Used Media Total .............          0        
  Disk Agent Errors Total ......          0  


Troubleshooting steps as follows:

>> Logged into the media server and checked for devices claimed in Device Manager. Found the devices.
>> Ran devbra -dev to determine the SCSI address.
>> Found N/A for drive

C:\>devbra -dev

Exch    HP:1x8 G2 AUTOLDR  Path: "Changer0:0:0:1"  SN: "AABBCCDD1E"
        Description: CLAIMED:HP StorageWorks 1x8 Cartridge Autoloader
        Revision: 4.20  Flags: 0x0016  Slots: 8  Drives: 1
        Drive(s) SN:
                "ABCDEFGHIJ"

Tape    HP:Ultrium 3-SCSI  Path: "Tape0:0:0:0"  SN: "N/A"
        Description: CLAIMED:HP LTO3 Drive
        Revision: Q51W  Device type: lto [13]  Flags: 0x0011

>> Checked if the drive is locked by DP

[root@cellsrv01:/root]
# omnimm -show_locked_devs | grep AUT01_D01

[root@ cellsrv01:/root]

>> Stopped DP Inet from Services.msc and Ran LTT.
>> Got error message that "mma.exe" process is accessing the device. Killed the mma.exe process from the Task Manager.

>> rescanned the devices in LTT. Able to detect both autoloader and it's drive this time.

>> Ran devbra -dev, which got the SCSI address.

C:\>devbra -dev

Exch    HP:1x8 G2 AUTOLDR  Path: "Changer0:7:0:1"  SN: "AABBCCDD1E "
        Description: CLAIMED:HP StorageWorks 1x8 Cartridge Autoloader
        Revision: 4.20  Flags: 0x0016  Slots: 8  Drives: 1
        Drive(s) SN:
                "ABCDEFGHIJ "

Tape    HP:Ultrium 3-SCSI  Path: "Tape1:7:0:0C"  SN: "ABCDEFGHIJ "
        Description: CLAIMED:HP LTO3 Drive
        Revision: Q51W  Device type: lto [13]  Flags: 0x0011

>> Ran the backup successfully.

Cause :

The DP media agent was accessing the drive and didn't let any other process to send commands. After the hung process is killed, the drive was accessible for normal operations.





Sunday, April 13, 2014

[61:12500] cannot connect to inet for getting file system list on host

Error:

[Critical]   From: BSM@cellsrv01.in.com "backup_spec_01” Time: 8/16/2014
[
61:12500] cannot connect to inet for getting file system list
              on host" winclnt01.in.com".
 
Reason:

Host may be down or Data protector client services (Inet) down.


Troubleshooting steps for Windows clients are as follows:

1. Check if the client (i.e. winclnt01.in.com in above error) is pingable from the cell manager, if not check the issue with platform team.

2. If the server is pingable, telnet the client using port 5555 (eg: #Telnet <Client name> 5555)

3. If the telnet failed, login to the windows client and check whether “Data protector Inet” service is running or not. If not start the “Data Protector Inet” service, Just by right clicking the service and restart it.

4. If the Data protector service is not listed in the “services.msc”, the DP agent might be removed from the client. Push the DP agents on this client from cell server.

5. If the telnet failed using port 5555 even data protector service running, check with platform team to open the port.

Troubleshooting steps for UNIX clients are as follows:

1. Follow the steps 1 and 2 as explained above.

2. If the telnet failed, login to the UNIX client and check for the Omni entries in /etc/services and /etc/inetd.conf using the commands below. If not found, DP client needs to be reinstalled.

[root@unxclnt01:/root]
# egrep omni /etc/services
omni  5555/tcp     # DATA-PROTECTOR

[root@unxclnt01:/root]
# egrep omni /etc/inetd.conf
omni stream tcp nowait root /opt/omni/lbin/inet inet -log /var/opt/omni//log/inet.log

3. Check if the port 5555 is listening using the command below. If not, raise a request to platform team to enable the port and start the inetd daemon.

[root@unxclnt01:/root]
# netstat -an | grep 5555
tcp        0      0  *.5555                 *.*                     LISTEN

4. Check if the ‘inetd’ daemon is running, if not start it using the below commands:

[root@unxclnt01:/root]
# ps -ef | grep inetd
    root  2144     1  0  Aug 10  ?         7:20 /usr/sbin/inetd

# /etc/inetd –s




Tuesday, April 1, 2014

Virtual Device Interface reported error: The object was not open.

Backup of SQL Database fails with the following errors:

[Critical] From: OB2BAR_Main@sqlsrv01.in.com "SIDDBS01"  Time: 3/9/2014 6:31:55 PM
 Error has occurred while executing a SQL statement.
 Error message: 'SQLSTATE:[08004] CODE:(911) MESSAGE:[Microsoft][ODBC SQL Server Driver][SQL Server]Database 'DBN01' does not exist. Make sure that the name is entered correctly.
SQLSTATE:[42000] CODE:(3013) MESSAGE:[Microsoft][ODBC SQL Server Driver][SQL Server]BACKUP DATABASE is terminating abnormally.'

[Critical] From: OB2BAR_Main@sqlsrv01.in.com "SIDDBS01"  Time: 3/9/2014 6:31:55 PM
 Virtual Device Interface reported error:
The object was not open.

 See also Data Protector debug.log and SQL Server error log for details.

[Normal] From: OB2BAR_DBN01@sqlsrv01.in.com "SIDDBS01"  Time: 3/9/2014 6:31:57 PM
 Completed OB2BAR Backup: sqlsrv01.in.com:/SIDDBS01/DBN01/0 "MSSQL"

[Major] From: OB2BAR_DBN01@sqlsrv01.in.com "SIDDBS01"  Time: 3/9/2014 6:31:57 PM

Aborting connection to BSM. Abort code -2.

Solution:

>> Remove the DB 'DBN01' from backup spec, if it doesn't exists in the SQL server.

[OR]

>> Check if the DB was dismounted, If so mount the database 'DBN01' and bring it online.

Trigger the backup and see if it's successful.

Friday, March 28, 2014

[90:801] Active Removable Storage Manager (RSM) service found on local system.

[Normal] From: BMA@mediasrv01.in.com "MSL2024_D02"  Time: 3/23/2014 2:57:16 PM
 STARTING Media Agent "MSL2024_D02"

[Warning] From: BMA@mediasrv01.in.com "MSL2024_D02"  Time: 3/23/2014 2:57:16 PM
[90:801]  Active Removable Storage Manager (RSM) service found on local system.

 [Warning] From: UMA@mediasrv01.in.com "MSL2024"  Time: 3/23/2014 2:57:18 PM
[90:59]   Changer0:0:0:0
 Cannot open exchanger control device ([2] The system cannot find the file specified. )

[Warning] From: UMA@mediasrv01.in.com "MSL2024"  Time: 3/23/2014 2:57:18 PM
 The device "MSL2024" could not be opened ("Device could not be accessed") 

[Normal] From: UMA@mediasrv01.in.com "MSL2024"  Time: 3/23/2014 2:57:18 PM
 Starting the device path discovery process. 

[Warning] From: UMA@mediasrv01.in.com "MSL2024"  Time: 3/23/2014 2:57:19 PM
[90:59]   Changer0:0:0:0
 Cannot open exchanger control device ([2] The system cannot find the file specified. )
[Normal] From: BMA@ mediasrv01.in.com "MSL2024"  Time: 3/23/2014 2:57:21 PM ABORTED Media Agent " MSL2024_D02"

Solution:

Removable Storage Manager (RSM) service on MS Windows 2003 might cause device issues in backup environment.

It is recommended to stop and disable the RSM service on all MS Windows 2003 systems.

RSM Service in Started status

Thursday, March 27, 2014

[61:2015] Timeout waiting for the devices to get free.

[Critical] From: BSM@cellsrv01.in.com "cellsrv01_IDB"  Time: 02/27/14 02:00:31
[61:2015]  Timeout waiting for the devices to get free.
The session will terminate.

Ø  It is very common in every backup environment to share the same device for different backups. This is an error message due to device contention issue.

Ø  The backup device selected in the backup specification is unavailable for the backup to start. It is in use by another process or by another backup/restore/copy sessions.

Ø   Check the device status using the lock name. (Check Here for the commands used to find a locked device). Wait for the device to be free.

Ø  The backup will be queued for global timeout seconds and will fail if no device is freed / allocated to the backup session.

Ø  The queuing time can be found at the end of backup session from backup statistics. Shown below
  

Backup Statistics:
          
                   Session Queuing Time (hours)         0.00        
                   -------------------------------------------      
                   Completed Disk Agents ........          5          
                   Failed Disk Agents ...........          0          
                   Aborted Disk Agents ..........          0          
                   -------------------------------------------      
                   Disk Agents Total  ...........          5          
                   =====================================     
                   Completed Media Agents .......          1          
                   Failed Media Agents ..........          0          
                   Aborted Media Agents .........          0          
                   -------------------------------------------      
                   Media Agents Total  ..........          1          
                   ===========================================      
                   Mbytes Total .................   17985 MB        
                   Used Media Total .............          1          
                   Disk Agent Errors Total ......          0    





Tuesday, March 25, 2014

[90:63] Cannot load exchanger medium (Medium error.)

[Normal] From: BMA@winsrv01.in.com "MSL_D01"  Time: 3/24/2014 10:53:43 PM
 By: UMA@winsrv01.in.com@Changer0:0:0:0
 Loading medium from slot 9 to device Tape0:0:0:0

[Major] From: BMA@winsrv01.in.com "MSL_D01"  Time: 3/24/2014 11:03:16 PM
[90:63]   By: UMA@winsrv01.in.com@Changer0:0:0:0
 Cannot load exchanger medium (Medium error.)

[Normal] From: BMA@winsrv01.in.com "MSL_D01"  Time: 3/24/2014 11:03:16 PM
 ABORTED Media Agent "MSL_D01"

[Normal] From: BMA@winsrv01.in.com " MSL_D02"  Time: 3/24/2014 11:03:21 PM
 STARTING Media Agent " MSL_D02"

[Normal] From: BMA@winsrv01.in.com " MSL_D02"  Time: 3/24/2014 11:03:26 PM
 By: UMA@winsrv01.in.com@Changer0:0:0:0
 Loading medium from slot 9 to device Tape1:0:0:0

[Major] From: BMA@winsrv01.in.com " MSL_D02"  Time: 3/24/2014 11:13:10 PM
[90:63]   By: UMA@winsrv01.in.com@Changer0:0:0:0
 Cannot load exchanger medium (Medium error.)

[Normal] From: BMA@winsrv01.in.com " MSL_D02"  Time: 3/24/2014 11:13:10 PM
 ABORTED Media Agent " MSL_D02"


Solution:

Exchanger medium is failing while handling media from particular slot (here it is slot 9). The media has errors that is residing slot 9.

Move to 'Devices & Media' context -> Click Library -> Click Slot -> move to Slot 9.

--- Check the status of media in slot 9.

--- It was poor media. Isolate the media by moving it to separate media pool and unload it from library. Wait for the media to expire and reformat it. If the status is still shows poor even after formatting, it is not advisable to use it henceforth.

--- Clean the affected drives with good cleaning tape and run a test backup. They should be good for backups.


Tuesday, March 18, 2014

IDB on Exclusive mode

Cannot open Internal Database in exclusive mode


Problem: Cannot open Internal Database in exclusive mode
Cannot backup internal database because another database check in progress
 
Solution 1: Bring down the omni services by typing the below commands

/etc/init.d/omni stop in unix
<omni dirc >/bin> omnisv -stop
Check if there is any hung process
do " ps -ef | grep omni "
if there is any hung process kill the hung sessions

use Kill -9 <process ID >

if there is no hung session , Bring up the omni services up 

/etc/init.d/omni start
check the services are up & running or not 
if all the services are up & running 

go to /opt/omni/sbin 

omnidbutil -clear ( this command will kill the ghost sessions )
you will get the message " Done ! "

to check the IDB database check is still running or not , go to " /opt/omni/sbin "
use " omnidbcheck " command

Now you will not get the message " Database check is in Process "

Now re initiate the IDB backup .... This will run the IDB backup successfully

Solution 2: If still the issue is not resolve by solution 1

log on to the cell manager and go the below path
/var/opt/omni/tmp

you can see a file name " tmp_dbcheck.lk" , Remove that file by using the command

rm tmp_dbcheck.lk

Restart the DP Services , your issue will resolve after performing any of the solutions


BR0073E Setting of BRBACKUP lock failed

Problem:

BR0051I BRBACKUP 7.20 (25)
BR0055I Start of database backup: benini.qub 2014-03-18 08.00.26
BR0484I BRBACKUP log file: /oracle/SID/sapbackup/benini.qub
BR0071E BRBACKUP currently running or was killed
BR0072I Please delete file /oracle/SID/sapbackup/.lock.brb if BRBACKUP was killed
BR0073E Setting of BRBACKUP lock failed

BR0056I End of database backup: benini.qub 2014-03-18 08.00.26
BR0280I BRBACKUP time stamp: 2014-03-18 08.00.26
BR0054I BRBACKUP terminated with errors
[Major] From: OB2BAR_OMNISAP@orsapsrv1.in.com "OMNISAP"  Time: 03/18/2014 08:00:26 AM
BRBACKUP /usr/sap/SID/SYS/exe/run/brbackup -t online_split -d util_file -c -p initSID.sap.bc -m all -q split -u / returned 3

[Normal] From: BSM@cellsrv01.incom "orsapsrv1_SID01"  Time: 3/18/2014 8:00:27 AM
OB2BAR application on "orsapsrv1.in.com" disconnected.

Solution:

>> Login to the client with SID of the database and check for any brbackup process running.

*****No brbackup process running*****

# ps -ef | grep brbackup
root     17138  9230  0 08:19 pts/0    00:00:00 grep brbackup

>> Check for any progressing DP backups from monitor context. This can also be initiated from DB end too. Wait until the backup completes and then check for the brbackup process again. 

>> Kill the .lock.brb file <path - /oracle/SID/sapbackup/.lock.brb > if there was aborted/hung backup session. Start the backup spec which should complete.


Sunday, March 16, 2014

ORA-19588: archived log RECID <> STAMP <> is no longer valid

Error:

RMAN-00571: ===========================================================
RMAN-00569: ======== ERROR MESSAGE STACK FOLLOWS ============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on dev_0 channel at 03/08/2014 07:50:43
ORA-19588: archived log RECID 99528 STAMP 841638632 is no longer valid

Recovery Manager complete.
[Major] From: ob2rman@orasrv01.in.com "oradb01"  Time: 03/08/14 07:50:56
 External utility reported error.

RMAN PID=12391

[Major] From: ob2rman@orasrv01.in.com "oradb01"  Time: 03/08/14 07:50:56
 The database reported error while performing requested operation.

RMAN-00571: ===========================================================
 RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
 RMAN-00571: ===========================================================
 RMAN-03009: failure of backup command on dev_0 channel at 03/08/2014 07:50:43
 ORA-19588: archived log RECID 99528 STAMP 841638632 is no longer valid

 Recovery Manager complete.

Problem:

>> The error can be due to invalid/stale archive files which were left uncleared even after the backup completion. Reason could be due to 2 simultaneous backup sessions where one refers to and the other had already backed up and not needed to be backed up.

>> It also happens when RMAN locates an archive file from the archive location that is no longer available in the Controlfile. These store very less info when compared to Recovery log.

Solution:

Execute the command “crosscheck archivelog all”.

The output can be as below:

Crosschecked 826 objects


Now you can re-trigger the backup spec and get a successful backup.

Batch script for " IDB Maintenance & Resolving the Velocis Error " for Windows Servers…

Solution : copy the below script and paste in notepad and save it as " IDB_velosis.bat" file and just click on the file ... 



Your Velocis error and IDB maintenance will be completed in Just-a-click!


Note:- Modify the script according to where [Which Drive/Path] we installed the DataProtector 




Copy the text which in blue color 

echo # IDB Maintenance & Resolving the Velosis Error #

echo # Resolving the Velosis Error #
cd \

D:

cd Program files\omniback\bin

omnisv -status

omnisv -stop

taskkill /IM vbda.exe /F /T
taskkill /IM bsm.exe /F /T
taskkill /IM dbsm.exe /F /T
taskkill /IM vrda.exe /F /T
taskkill /IM uma.exe /F /T
taskkill /IM crs.exe /F /T
taskkill /IM rds.exe /F /T
taskkill /IM mmd.exe /F /T

cd \

cd Program Files\OmniBack\tmp

del CRS.pid
del dbcheck.cdb
del dbcheck.mmdb
del lic.ctx
del mmd.ctx

cd \

cd Program Files\OmniBack\db40\logfiles\syslog

del *.chg
del *.chk

cd \

cd Program Files\OmniBack\db40\datafiles\catalog

rename rdm.bil rdm.bil.old
rename rdm.chi rdm.chi.old

cd \

cd Program Files\OmniBack\bin

echo # Now Bring up the databae

omnisv -start

omnidbutil -clear

omnidbutil -free_locked_devs

echo # IDB Maintanence

omnidbutil -purge -messages 30 -force
omnidbutil -purge -sessions 30 -force
omnidbutil -purge -dcbf -force
omnidbutil -purge -filenames -force

exit


End of the script

Try the same in Testing environment prior to Production, All the best :)