Home / Engineering Calculator / Calculator: Water Flow Rate through Piping

Asm Health Checker Found 1 New Failures Updated __top__ | Simple

Troubleshooting "ASM Health Checker Found 1 New Failures Updated"

If you are an Oracle Database Administrator, seeing the alert "ASM Health Checker found 1 new failures updated" in your logs or monitoring dashboard (like Enterprise Manager) can be a bit jarring. This message is the Oracle Automatic Storage Management (ASM) framework’s way of telling you that its internal diagnostic engine has detected an issue that could compromise the health of your storage layer.

Here is a deep dive into what this error means, why it happens, and how to resolve it. What is the ASM Health Checker?

The ASM Health Checker is a proactive diagnostic utility that runs within the Oracle Grid Infrastructure. It constantly monitors the state of ASM disk groups, metadata consistency, and background processes.

When it detects a discrepancy—such as a corrupted metadata block, a disk timeout, or an offline disk—it logs a "failure." The "Updated" status usually means the health check engine has refreshed its findings and confirmed that the issue is persistent and requires administrator intervention. Common Causes for This Alert

While the message itself is a general notification, the "1 new failure" usually stems from one of the following:

Disk Connectivity Issues: A physical disk or LUN has become unreachable or is experiencing intermittent latency.

Metadata Corruption: Inconsistency in the ASM Allocation Units (AU) or disk headers.

Disk Group Imbalance: A rebalance operation failed or was interrupted, leaving the disk group in a "degraded" state.

Offline Disks: A disk was dropped or taken offline due to I/O errors, but the redundancy (if using Normal or High redundancy) kept the database running. Step-by-Step Resolution Guide 1. Identify the Specific Failure

The alert message is just the "headline." You need to find the specific error code (like ORA-15032 or ORA-15078).

Check the Alert Log: Navigate to your ASM diagnostic trace folder and check the alert_+ASM.log.

Use ADRCI: Run the command adrci and use show alert to see the most recent incidents and their specific impact. 2. Query the ASM Views

Log into your ASM instance via SQL*Plus (sqlplus / as sysasm) and run the following to see the status of your disks:

SELECT group_number, name, state, type FROM v$asm_diskgroup; SELECT path, header_status, mode_status, state FROM v$asm_disk; Use code with caution.

Look for any disks where the header_status is CANDIDATE (instead of MEMBER) or mode_status is OFFLINE. 3. Check for Ongoing Rebalances

Sometimes the health checker flags a failure if a rebalance is stuck. SELECT * FROM v$asm_operation; Use code with caution.

If an operation is hanging, you may need to investigate the underlying I/O subsystem. 4. Run a Manual Check (The "Check" Command)

You can force ASM to verify the consistency of a disk group to see if it clears the error or provides more detail: ALTER DISKGROUP CHECK ALL; Use code with caution. Proactive Tips to Prevent Future Failures

Monitor I/O Latency: Often, the health checker finds a "failure" simply because a storage array is too slow. Monitor your OS-level tools like iostat or sar.

Update Grid Infrastructure: Ensure you are on the latest RU (Release Update), as Oracle frequently releases patches for ASM Health Checker "false positives."

Verify Redundancy: Always ensure your critical disk groups are at least on "Normal" redundancy to allow the health checker to find and fix issues without taking the database offline.

The "ASM Health Checker found 1 new failures updated" alert is a call to action. It usually indicates a physical storage hiccup or a metadata inconsistency. By checking the ASM alert logs and querying v$asm_disk, you can usually pinpoint the culprit disk and bring it back online or replace it before a total outage occurs.

ASM Health Checker Found 1 New Failures Updated: What It Means and How to Resolve It

Automatic Storage Management (ASM) is a vital component of Oracle databases, responsible for managing storage resources and providing a layer of abstraction between the database and the underlying storage devices. The ASM health checker is a built-in tool that monitors the health and performance of ASM instances, alerting administrators to potential issues before they become critical problems.

If you've received a notification that the "ASM health checker found 1 new failures updated," it's essential to understand what this message means and take prompt action to resolve the issue. In this article, we'll delve into the details of ASM health checking, explore the possible causes of this error, and provide step-by-step guidance on how to troubleshoot and fix the problem. asm health checker found 1 new failures updated

Understanding ASM Health Checking

The ASM health checker is a continuous monitoring process that checks the health and performance of ASM instances. It collects data on various aspects of ASM operations, including:

  1. Disk usage and availability
  2. Disk performance and latency
  3. ASM instance stability and connectivity
  4. Database and storage interactions

The health checker uses this data to identify potential issues, such as disk failures, performance bottlenecks, or configuration problems. When an issue is detected, the health checker updates the ASM alert log with a failure message, indicating the type and severity of the problem.

What Does "ASM Health Checker Found 1 New Failures Updated" Mean?

When you receive a notification that the "ASM health checker found 1 new failures updated," it means that the ASM health checker has detected a new issue with the ASM instance or one of its associated disks. The failure message is updated in the ASM alert log, indicating that a new problem has been identified.

The failure message may indicate a variety of issues, including:

  1. Disk failure: A disk has failed or is no longer accessible, impacting ASM operations.
  2. Performance issue: A disk or ASM instance is experiencing performance problems, such as high latency or low throughput.
  3. Configuration problem: A configuration error or inconsistency has been detected, affecting ASM operations.
  4. Connection issue: A problem has been identified with the connection between the ASM instance and the database or storage devices.

Causes of ASM Health Checker Failures

There are several possible causes for ASM health checker failures, including:

  1. Disk errors or failures: Physical disk errors or failures can cause ASM health checker failures.
  2. ASM configuration issues: Incorrect or inconsistent ASM configuration can lead to health checker failures.
  3. Performance bottlenecks: Performance issues with disks, ASM instances, or database operations can trigger health checker failures.
  4. Connectivity problems: Issues with connections between ASM instances, databases, and storage devices can cause health checker failures.

How to Troubleshoot and Resolve ASM Health Checker Failures

To troubleshoot and resolve ASM health checker failures, follow these steps:

  1. Check the ASM alert log: Review the ASM alert log to understand the specific failure message and the component that triggered the failure.
  2. Verify ASM disk status: Check the status of ASM disks using the ASMCMD command-line tool or the Oracle Enterprise Manager.
  3. Investigate disk performance: Analyze disk performance metrics to identify potential bottlenecks or issues.
  4. Review ASM configuration: Verify ASM configuration settings to ensure consistency and correctness.
  5. Check database and storage connections: Verify connections between the ASM instance, database, and storage devices.

Step-by-Step Troubleshooting Guide

Here's a more detailed, step-by-step guide to troubleshooting ASM health checker failures:

Step 1: Check the ASM Alert Log

  1. Connect to the ASM instance using the ASMCMD command-line tool or Oracle Enterprise Manager.
  2. Review the ASM alert log to understand the specific failure message and the component that triggered the failure.

Step 2: Verify ASM Disk Status

  1. Use the ASMCMD command-line tool to list ASM disks: asmcmd lsdisks
  2. Verify that all disks are listed and have a status of "ONLINE" or "NORMAL".

Step 3: Investigate Disk Performance

  1. Use Oracle Enterprise Manager or other monitoring tools to analyze disk performance metrics, such as:
    • Disk usage and capacity
    • Disk I/O throughput and latency
    • Disk error rates
  2. Identify potential bottlenecks or issues with disk performance.

Step 4: Review ASM Configuration

  1. Verify ASM configuration settings using the ASMCMD command-line tool or Oracle Enterprise Manager.
  2. Check for consistency and correctness in ASM configuration, including:
    • ASM instance parameters
    • Disk group configurations
    • ASM file and directory structures

Step 5: Check Database and Storage Connections

  1. Verify connections between the ASM instance, database, and storage devices.
  2. Check for any issues with network connectivity, authentication, or authorization.

Resolving ASM Health Checker Failures

Once you've identified the root cause of the ASM health checker failure, take corrective action to resolve the issue. This may involve:

  1. Replacing failed disks: Replace failed disks or take corrective action to repair disk errors.
  2. Adjusting ASM configuration: Adjust ASM configuration settings to optimize performance or resolve inconsistencies.
  3. Resolving performance bottlenecks: Address performance bottlenecks or issues with disk, ASM instance, or database operations.
  4. Restoring connections: Restore connections between ASM instances, databases, and storage devices.

By following these steps, you can troubleshoot and resolve ASM health checker failures, ensuring the stability and performance of your Oracle database and ASM environment.

Conclusion

The "ASM health checker found 1 new failures updated" message indicates a potential issue with the ASM instance or one of its associated disks. By understanding the causes of ASM health checker failures and following a step-by-step troubleshooting guide, you can identify and resolve issues before they become critical problems. Regular monitoring and maintenance of ASM instances and disks can help prevent health checker failures and ensure optimal performance and stability of your Oracle database and storage environment.

The message "ASM health checker found 1 new failures updated" typically appears in the Oracle Automatic Storage Management (ASM) alert logs when a background check detects a serious issue with disk group availability or redundancy.

A formal review of this failure should include an investigation of the root cause and an immediate assessment of data risk. Initial Assessment & Risk Level

High Priority: This message often precedes a disk group going into an INTERMEDIATE or OFFLINE state. Troubleshooting "ASM Health Checker Found 1 New Failures

Data Integrity Check: If you are using External Redundancy, a single disk failure can make the entire disk group unrecoverable ("toast").

Redundancy Impact: In Normal Redundancy setups, the system may still be running but is now vulnerable to a second failure until full redundancy is restored. Failure Review Checklist To conduct a thorough review, perform the following steps: Identify the Specific Failure

Check the ASM alert log for accompanying error codes (e.g., ORA-15000 to ORA-15999).

Look for "Write Failed" or "I/O error" warnings to see if a physical disk has dropped. Verify Disk Status

Run crsctl stat res -t to check if disk group resources are in a STABLE or INTERMEDIATE state.

Query V$ASM_DISK to find disks with a status of OFFLINE or HUNG. Analyze Metadata Health

If the disk group won't mount, use the kfed utility (e.g., kfed read ) to check for corrupted metadata or invalid disk headers. Evaluate Capacity

Check REQUIRED_MIRROR_FREE_MB. If your usable space is negative, the system may not have enough room to rebalance data and restore redundancy after this failure. Recommended Actions

If a disk is missing: Verify physical hardware or multipathing configurations to ensure the device path (e.g., /dev/sdg1) is still visible to the OS.

If space is low: Avoid adding more data until the failure is resolved, as further writes may lead to ORA-15041 (disk group out of space).

For permanent failures: You may need to drop the failed disk and add a replacement, then monitor the ARB0 background process to ensure a successful rebalance. KB88485 - My Oracle Support

ASM Health Checker alert "found 1 new failures updated" typically indicates that the BIG-IP system's internal monitoring has detected a specific resource or service failure within the Application Security Manager (ASM)

. This is often triggered when a monitored resource crosses a predefined threshold or a critical daemon stops responding. Immediate Review Checklist To review and resolve this failure, follow these steps: Identify the Failure Source : Navigate to Security > Reporting > Settings > ASM Alerts

in the Configuration utility. This screen displays which specific health alert was triggered (e.g., CPU usage, memory limits, or database connectivity). Check Daemon Health : Verify if critical ASM processes like asm_config_server are running. You can check this via the command line using tmsh show /sys service Investigate Recent Changes

: Review the audit logs for recent maintenance activities, such as software upgrades, re-licensing, or configuration loads, which are common triggers for ASM health failures. Examine MySQL Database Status

: ASM relies heavily on an internal MySQL database. Check for database corruption or space issues by running tmsh load sys config verify or reviewing /var/log/asm for SQL-related errors. Utilize iHealth Diagnostics : Generate a file and upload it to the F5 iHealth portal

. This will automatically compare your system state against known bugs and best practices to pinpoint the exact failure. Common Root Causes

Configuring BIG-IP ASM system resource alerts using ... - My F5

It sounds like you're referencing a log or output from an ASM health check (likely Oracle Automatic Storage Management). A useful review would typically include:

If you share the actual failure text or log snippet, I can help interpret it and recommend next steps.

Step 4: Prevent Future Issues

Conclusion

The "asm health checker found 1 new failures updated" alert requires immediate attention to prevent data loss, performance degradation, or system downtime. By understanding the cause, taking corrective action, and implementing preventive measures, database administrators can ensure the reliability and performance of their Oracle databases. Always refer to Oracle documentation or consult with Oracle Support for specific guidance tailored to your environment.

The message "ASM Health Checker found 1 new failures" is a critical alert typically generated by Oracle Automatic Storage Management (ASM). It indicates that the background health monitor has detected a significant issue within the storage layer that could impact database availability. Immediate Diagnostic Steps

To identify the specific cause, you should immediately examine the ASM alert log and current disk status:

Check the Alert Log: Look for ORA- errors (like ORA-15130 or ORA-15063) in the trace file directory: Disk usage and availability Disk performance and latency

Path: /u01/app/oracle/diag/asm/+asm//trace/alert_.log.

Verify Diskgroup Status: Run the following command in the ASM instance to see which group is affected:

SQL> SELECT name, state, offline_disks FROM v$asm_diskgroup;.

Check Individual Disk Health: Identify if a specific disk has dropped or is hung:

SQL> SELECT path, header_status, mode_status FROM v$asm_disk;. Common Causes & Solutions KB88485 - My Oracle Support

The message "ASM Health Checker found 1 new failures" typically appears in the Oracle Automatic Storage Management (ASM) alert log when a critical issue—such as a disk failure or a forced diskgroup dismount—is detected. This is part of Oracle's fault diagnosability infrastructure designed to capture diagnostic data at the first sign of trouble. Immediate Actions to Take

If you see this message, follow these steps to identify and resolve the failure:

Check the ASM Alert Log: Review the alert log (often located in /u01/app/grid/diag/asm/+asm/+ASM/trace/alert_+ASM.log) for errors preceding the health checker message, such as ORA-15130 (diskgroup being dismounted) or ORA-15032.

Run ADRCI: Use the ADR Command Interpreter (ADRCI) to view the specific "incident" or "problem" that was logged. Command: adrci> show problem or adrci> show incident

Verify Diskgroup Status: Log into the ASM instance and check if any diskgroups are offline or if disks have been dropped. SQL> select name, state from v$asm_diskgroup;

SQL> select name, header_status, mode_status from v$asm_disk;

Investigate I/O Failures: Look for hardware-level issues, such as storage path failures, SAN/NFS connectivity problems, or OS-level permission changes that might have caused the disk to go offline. Common Causes

Disk Path Failure: The OS can no longer see the physical storage device.

Forced Dismount: ASM may force a dismount if too many disks in a failure group are lost, exceeding the redundancy limit.

Communication Issues: In a RAC environment, network or heartbeat failures between nodes can trigger ASM health alerts.

For automated assistance, you can use tools like Oracle ORAchk to run a comprehensive health check on your entire Oracle stack.

The coffee hadn’t even finished brewing when Sarah saw the notification on her primary dashboard: “ASM Health Checker found 1 new failure updated.”

In the world of database administration, "1 new failure" is rarely just a number; it’s a riddle. She logged into the terminal, the cursor blinking like a nervous heartbeat. As she ran the diagnostic tool, the system confirmed the dread: Disk Group 'DATA_01' was reporting a predictive failure on a single member.

She knew the routine. Oracle ASM is designed to handle this—it’s built for redundancy. But "1 failure" is the first domino. The Investigation

Sarah pulled up the alert logs. The health checker hadn't just found a flaw; it had flagged a PST (Parallel Server Tree) write failure.

The Symptom: One disk was lagging, its I/O response times ballooning into the hundreds of milliseconds.

The Automation: The health checker had already updated the status, signaling the ASM instance to prepare for a "drop and rebalance". The Turning Point

She watched as the background process, ARB0, kicked into gear. The data began its silent migration, flowing away from the dying hardware and onto the healthy disks in the group. The "1 failure" was no longer a threat; it was a task being solved by the very software that discovered it.

2. Disk Path Failures (Multipath Issues)

In multipath environments (e.g., DM-Multipath on Linux, PowerPath on AIX), a loss of one path to a disk does not immediately offline the disk. However, the ASM Health Checker detects increased I/O latency or path errors and reports a new failure, even if the disk remains online.

Summary

The Automatic Storage Management (ASM) health check utility has identified 1 new failure since the last successful check. This report details the failure and recommended actions.

Immediate Steps to Diagnose the Failure

When you see "ASM Health Checker found 1 new failures updated" in the ASM alert log, follow this systematic diagnostic procedure.