ORA-19588: archived log RECID ***** STAMP ********* is no longer valid

Error log:

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on dev_0 channel at 03/08/2014 07:50:43
ORA-19588: archived log RECID 99528 STAMP 841638632 is no longer valid

Recovery Manager complete.
[Major] From: ob2rman@dbserver01.com "stin"  Time: 03/08/14 07:50:56
External utility reported error.

RMAN PID=12391

[Major] From: ob2rman@dbserver01.com "stin"  Time: 03/08/14 07:50:56
The database reported error while performing requested operation.

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
 RMAN-03009: failure of backup command on dev_0 channel at 03/08/2014 07:50:43
 ORA-19588: archived log RECID 99528 STAMP 841638632 is no longer valid

Recovery Manager complete.

Cause:

-> Archive log backup fails due to invalid/stale archive files which are left uncleared after the backup completion.

-> It can be when there is two consecutive backups happening. For instance, an archive backup gets triggered while there's active full backup. The overlaps would cause this error.

Solution:

Database admin should run "crosscheck archivelog all" to remove the files that were backed up already.

Hope that helps!

ORA-19588: archived log RECID <> STAMP <> is no longer valid

Error:

RMAN-00571: ===========================================================

RMAN-00569: ======== ERROR MESSAGE STACK FOLLOWS ============

RMAN-00571: ===========================================================

RMAN-03009: failure of backup command on dev_0 channel at 03/08/2014 07:50:43

ORA-19588: archived log RECID 99528 STAMP 841638632 is no longer valid

Recovery Manager complete.

[Major] From: ob2rman@orasrv01.in.com "oradb01"  Time: 03/08/14 07:50:56

 External utility reported error.

RMAN PID=12391

[Major] From: ob2rman@orasrv01.in.com "oradb01"  Time: 03/08/14 07:50:56

 The database reported error while performing requested operation.

RMAN-00571: ===========================================================

 RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

 RMAN-00571: ===========================================================

 RMAN-03009: failure of backup command on dev_0 channel at 03/08/2014 07:50:43

 ORA-19588: archived log RECID 99528 STAMP 841638632 is no longer valid

 Recovery Manager complete.

Problem:

>> The error can be due to invalid/stale archive files which were left uncleared even after the backup completion. Reason could be due to 2 simultaneous backup sessions where one refers to and the other had already backed up and not needed to be backed up.

>> It also happens when RMAN locates an archive file from the archive location that is no longer available in the Controlfile. These store very less info when compared to Recovery log.

Solution:

Execute the command “crosscheck archivelog all”.

The output can be as below:

Crosschecked 826 objects

Now you can re-trigger the backup spec and get a successful backup.

RMAN-03002: failure of shutdown command at 03/02/2014 09:31:22

DP sends the commands from the script (provided in the backup spec) sequentially via. DP Integration to RMAN to start Offline backup of Oracle database as follows.

RMAN> CONNECT TARGET *

2> CONNECT CATALOG *

3> HOST 'exit';

4> run {

5> shutdown immediate;

6> startup mount;

7> allocate channel 'dev_0' type 'sbt_tape'

8> parms 'SBT_LIBRARY=/opt/omni/lib/libob2oracle8_64bit.so,ENV=(OB2BARTYPE=Oracle8,OB2APPNAME=oradb101,OB2BARLIST=Oracle_DB_Spec01)';

9> 

10> send device type 'sbt_tape' 'OB2BARHOSTNAME=Orasrv01';

11> backup incremental level 0

12>  format 'Oracle_DB_Spec01<oradb101_%s:%t:%p>.dbf'

13>  database;

14> backup

15>  format 'Oracle_DB_Spec01<oradb101_%s:%t:%p>.dbf'

16>  current controlfile;

17> alter database open;

18> }

19> EXIT

RMAN-06005: connected to target database: oradb101 (DBID=101)

RMAN-06008: connected to recovery catalog database

RMAN-00571: ==================================================

RMAN-00569: ========= ERROR MESSAGE STACK FOLLOWS =========

RMAN-00571: ==================================================

RMAN-03002: failure of shutdown command at 03/02/2014 09:31:22

ORA-01013: user requested cancel of current operation

Recovery Manager complete.

[Major] From: ob2rman@Orasrv01 "oradb101"  Time: 03/02/14 09:31:22

            External utility reported error.

Execution of the script starts and there was connection established between DP and the Oracle database via. DP Integration. Eventually, once the backup goes smooth the script would bring the database online and complete, if not continue reading the post till end.

The backup has error messages:

RMAN-03002: failure of shutdown command at 03/02/2014 09:31:22

ORA-01013: user requested cancel of current operation

Here, the database is refusing the shutdown command issued by the backup script. Offline backup requires database to be in down and mounted status. Check if the database is engaged by long running tasks if any. 

That's right, when we checked the database was busy with long running oracle jobs which was the root cause to override the shutdown command. Try running the backup when the DB is free from jobs/tasks and during non-business/less or no traffic to DB, after all it's an offline backup.

Cannot get information on file '/var/opt/omni/tmp/rcvcat.exp' Error: 2

[Major] From: OB2BAR_DMA@dbsrv01.in.com "DB001"  Time: 2/23/2014 3:05:36 PM
      Cannot get information on file '/var/opt/omni/tmp/rcvcat.exp' Error: 2.

[Major] From: ob2rman@dbsrv01.in.com "DB001"  Time: 02/23/2014 03:05:44 PM
      Backup of recovery catalog failed.

Solution:

>> You would receive this error in linux OS and clustered Oracle DB. The patch DPLNX_00094 would fix this.

Alternatively, you can try the below steps:

>> From the Integration specific option, Enable the option 'Disable recovery catalog auto backup'. This would skip the backup of recovery catalog backup from DP. Rerun the backup now. If you still see the same errors and wonder why, then move on to the next step.

>> The backup script in barlists always takes precedence from the DP GUI options. Open the spec in VI editor and Move to the end of the barlist and edit the script with value "-skip RCVCAT".

Example:
CLIENT "DB001" dbsrv01.in.com
{
        -exec ob2rman.exe
        -args {
                "-skip RCVCAT"                                             //Skips the recovery catalog backup
                "-pre  /home/user/DB001_weekly_full.rman"     //Pre exec scripts if any
                "-backup"                
        }
        -input {
                "run {"
                "allocate channel 'dev_0' type 'sbt_tape'"        //channel allocation
                " parms 'ENV=(OB2BARTYPE=Oracle8,OB2APPNAME=DB001,OB2BARLIST=dbsrv01_DB001_ONLINE)';"
                "backup"
                " format 'dbsrv01_DB001_ONLINE<DB001_%s:%t:%p>.dbf'"
                " current controlfile;"
                "}"
        }
        -profile
}  -protect days 28                                                      //data protection in days

[12:8301] Oracle Server home directory not found.

[Normal] From: BSM@cellserver01.in.com "Oracle_oradb01_Spec01"  Time: 2/13/2014 7:00:06 AM

          Backup session 2014/02/13-5 started.

[Normal] From: BSM@cellserver01.in.com "Oracle_oradb01_Spec01"  Time: 2/13/2014 7:00:12 AM

          OB2BAR application on "oradbsrv01.in.com" successfully started.

[Major] From: ob2rman@oradbsrv01.in.com "oradb01"  Time: 02/13/14 07:00:20

          [12:8301] Oracle Server home directory not found.

[Normal] From: BSM@cellserver01.in.com "Oracle_oradb01_Spec01"  Time: 2/13/2014 7:00:21 AM

          OB2BAR application on "oradbsrv01.in.com " disconnected.

[Normal] From: BSM@cellserver01.in.com "Oracle_oradb01_Spec01"  Time: 2/13/2014 7:00:21 AM

Error message Interpretation:

From the error message "Oracle Server home directory not found." we can identify that the home dir was migrated to some other location. Had to double check with the DBAs.

>> Yes, the database was actually moved to another node and the integration was reconfigured to backup the new migrated database. 

>> The old backup spec was disabled and can be removed after the last version of DB backup expires.

ORA-19511: Error received from media manager layer, error text: Vendor specific error: OB2_StartObjectBackup() failed ERR(-17)

RMAN-03090: Starting backup at 22-FEB-14

RMAN- 0 : channel dev_0: starting incremental level 0 datafile backupset

RMAN- 010: channel dev_0: specifying datafile(s) in backupset

RMAN- 522: input datafile fno=00007 name=/var/opt/vgdb/u01/oracle/DB007/ DB007_DB7_datas01.dbf

RMAN- 522: input datafile fno=000  name=/var/opt/vgdb/u01/oracle/DB007/ DB007_DB7_indexs01.dbf

RMAN- 522: input datafile fno=00001 name=/var/opt/vgdb07/u01/oracle/DB007/ DB007_system01.dbf

RMAN- 522: input datafile fno=00002 name=/var/opt/vgdb07/u01/oracle/DB007/ DB007_undotbs101.dbf

RMAN- 522: input datafile fno=00003 name=/var/opt/vgdb06/u01/oracle/DB007/ DB007_data01s01.dbf

RMAN- 522: input datafile fno=00004 name=/var/opt/vgdb07/u01/oracle/DB007/ DB007_indx01s01.dbf

RMAN- 522: input datafile fno=00005 name=/var/opt/vgdb07/u01/oracle/DB007/ DB007_tools01.dbf

RMAN- 522: input datafile fno=00006 name=/var/opt/vgdb06/u01/oracle/DB007/ DB007_users01.dbf

RMAN- 038: channel dev_0: starting piece 1 at 22-FEB-14

RMAN- 031: released channel: dev_0

RMAN-00571: ===========================================================

RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571: ===========================================================

RMAN-03009: failure of backup command on dev_0 channel at 02/22/2014 21:36:50

ORA-19506: failed to create sequential file, name="Oracle8_Spec01_DB007_ONLINE< DB00701_677:840231409:1>.dbf", parms=""

ORA-27028: skgfqcre: sbtbackup returned error

ORA-19511: Error received from media manager layer, error text:

   Vendor specific error: OB2_StartObjectBackup() failed ERR(-17)

Recovery Manager complete.

[Major] From: ob2rman@oraserver01 "DB00701"  Time: 02/22/14 21:36:50

 External utility reported error.

RMAN PID=7092

[Major] From: ob2rman@oraserver01 "DB00701"  Time: 02/22/14 21:36:50

 The database reported error while performing requested operation.

Solution:

>> The backup data is destined to file library which ran out of free space.

>> Created free space and reran the backup, which was successful.

Oracle Errors 'ORA-01031' during backup

[Normal] From: BSM@cellserver01.co.in "Oraclespec01"  Time: 01/17/11 08:38:07

        OB2BAR application on "Oraserver01.co.in" successfully started.

[Normal] From: ob2rman@Oraserver01.co.in "Oradb01"  Time: 01/17/11 08:38:09

        Starting backup of target database.

[Major] From: ob2rman@Oraserver01.co.in "Oradb01"  Time: 01/17/11 08:38:10

        The database reported error while performing requested operation.

ERROR:

 ORA-01031: insufficient privileges

 CONNECT: dporadb01/*****@Oradb01

[Major] From: ob2rman@Oraserver01.co.in "Oradb01"  Time: 01/17/11 08:38:10

        Backup of target database failed.

[Normal] From: BSM@cellserver01.co.in "Oraclespec01"  Time: 01/17/11 08:38:11

        OB2BAR application on "Oraserver01.co.in" disconnected.

Reason :

This error message "ORA-01031: insufficient privileges" means the authentication failure to access the database for backup.

The password file must be the problem, if not the DB username is not removed. The password file might have expired or changed or do not have permissions.

Solution :

Please have your DB team to remove/recreate the password file.

Command would look like the below:

ORAPWD FILE=filename [ENTRIES=numusers] [FORCE={Y|N}] [IGNORECASE={Y|N}] [NOSYSDBA={Y|N}]

Reference:

http://docs.oracle.com/cd/B28359_01/server.111/b28310/dba007.htm#ADMIN10241

Hope this helps!