ORA-19588: archived log RECID ***** STAMP ********* is no longer valid

Error log:

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on dev_0 channel at 03/08/2014 07:50:43
ORA-19588: archived log RECID 99528 STAMP 841638632 is no longer valid

Recovery Manager complete.
[Major] From: ob2rman@dbserver01.com "stin"  Time: 03/08/14 07:50:56
External utility reported error.

RMAN PID=12391

[Major] From: ob2rman@dbserver01.com "stin"  Time: 03/08/14 07:50:56
The database reported error while performing requested operation.

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
 RMAN-03009: failure of backup command on dev_0 channel at 03/08/2014 07:50:43
 ORA-19588: archived log RECID 99528 STAMP 841638632 is no longer valid

Recovery Manager complete.

Cause:

-> Archive log backup fails due to invalid/stale archive files which are left uncleared after the backup completion.

-> It can be when there is two consecutive backups happening. For instance, an archive backup gets triggered while there's active full backup. The overlaps would cause this error.

Solution:

Database admin should run "crosscheck archivelog all" to remove the files that were backed up already.

Hope that helps!

TSM Insufficient system resources exist to complete the requested service

Error:

ANS9999E ntrc.cpp(928): Received Win32 RC 1450 (0x000005aa) from FileRead(): ReadFile '\\xxxx\. Error description: Insufficient system resources exist to complete the requested service.

Solution:

>> Edit the dsm.opt file by adding the param "MEMORYEFFICIENTBACKUP YES".

>> Save it.

>> Try a manual backup and check if the backup completes.

If not, this error needs attention from platform end.

>> Windows OS has the paging file space memory that needs attention. 
>> Issue can be resolved by adding/modifying "PoolUsageMaximum & PagedPoolSize" in registry

For more info, please refer the article from microsoft.

http://support.microsoft.com/kb/304101

Your backup should be successful, once this is fixed.

Have a nice Day!!

TSM Admin Tasks

This post would give an idea of the commands and tasks that a backup administrator would carry-on everyday from TSM server end.

Ø  q event * t=a begind=-1 endd=t

Ø  BACKUP DB type=full dev=<device_class_name> wait=yes

Ø  BACKUP DEVCONFIG

Ø  BACKUP VOLHISTORY

Ø  PREPARE source=dbbackup devclass=<device_class_name> wait=yes

Ø  DELETE VOLHISTORY todate=today-7 type=all

Ø  EXPIRE INVENTORY wait=yes duration=240

Ø  RECLAIM STGPOOL <stgpool_name> th=75 du=120

(Repeat this command for all the storage pools)

Ø  QUERY STGPOOL

Ø  QUERY MOUNT

Ø  QUERY SESSION

Ø  select count(*) as Scratch, library_name from libvolumes where status='Scratch' group by library_name

(To check for scratch volumes count in each library)

Ø  select source_name as source, destination_name as target,library_name as library, device from paths where online = 'NO'

(To check any offline paths)

Ø  QUERY DRIVE

Ø  QUERY ACTLOG s=error

Ø  QUERY ACTLOG s=failure

Ø  QUERY ACTLOG s="Server out of data storage space

Backup status in TSM

Full syntax:
query event <domain_name> <schedule_name> type=client begindate=mm/dd/yyyy begintime=00:00 enddate=mm/dd/yyyy endtime=hh:mm exceptionsonly=yes <format=detailed>

These commands needs to be run from the TSM server to check the pervious day's missed/failed backups of BA-client/admin schedules. Below is the short form of the same:

Short form:
q event * * begind=-1 endd=t ex=yes

Next we going to check the status of administrative schedules such as DB backup, reclamation.. etc

Full syntax:
query event <schedule_name> type=administrative begindate=mm/dd/yyyy begintime=00:00 enddate=mm/dd/yyyy endtime=hh:mm exceptionsonly=yes <format=detailed>

Short form:
q event * t=a begind=-1 endd=t ex=yes