WAIT STATS - high Wait Time on ASYNC_IO_COMPLETION

Question

I currently look at my wait stats. I followed best practices explained by Paul Randal and go through the documents he references.

Looking at the statistic slices I take once / day for a while now from all production servers, I see significant high values for Avg_Wait_S, meaning the relation of total Wait in Seconds divided by WaitCounts. The value for wait count is very low compared to all the other wait types (the lowest), but on each of my servers the AvgWait_S for this type is between 120 - 2493 seconds! This looks like a massive wait time.

I read this can anyhow be caused by backups. But I wonder if this is kind of a 'normal' value? How to get more details about it?

Here is a sample resultset of my wait stats:

WAITTYPE            Wait_S      Resource_S  Signal_S    WaitCount   Percentage  AvgWait_S   AvgRes_S    AvgSig_S 
BACKUPIO            46151.80    45093.07    1058.74     22819151    26.55       0.0020      0.0020      0.0000 
CXPACKET            45057.81    27926.59    17131.22    20387211    25.92       0.0022      0.0014      0.0008 
BACKUPBUFFER        16658.83    15867.71    791.12      8993341     9.58        0.0019      0.0018      0.0001 
PAGEIOLATCH_SH      15326.95    15284.65    42.30       2131848     8.82        0.0072      0.0072      0.0000 
ASYNC_IO_COMPLETION 14203.17    14203.17    0.00        9           8.17        1578.1303   1578.1303   0.0000 
WRITELOG            8570.20     8377.81     192.39      2136964     4.93        0.0040      0.0039      0.0001 
PAGEIOLATCH_EX      7691.32     7673.31     18.01       1777069     4.43        0.0043      0.0043      0.0000 
SOS_SCHEDULER_YIELD 4548.90     43.44       4505.47     13294384    2.62        0.0003      0.0000      0.0003 
LCK_M_S             3018.76     3018.06     0.71        1704        1.74        1.7716      1.7712      0.0004 
ASYNC_NETWORK_IO    2678.83     2147.78     531.05      4372825     1.54        0.0006      0.0005      0.0001 
LCK_M_SCH_S         2612.09     2612.08     0.01        37          1.50        70.5971     70.5968     0.0003

Answer 1

As mentioned

The value for wait count is very low compared to all the other wait types (the lowest)

the count being only 9, i.e. the wait has only showed up for 9 times, during the run of gathering wait stats for a complete day, wont be a concern here for me.

However, its worth checking the other wait types BACKUPIO and BACKUPBUFFER .

This wait type generally shows up when you are taking the backup over the network with not a good NW connectivity or on the tape etc (i.e. any other extremely slow backup system)

We had this similar wait on one of our SQL server 2005 for backing up the database approx 200 GB in size. As the size being large and we did not had compression feature available at that time, witness lots of waits. we had to go with third party compression tool and saw those wait types count going down.So i believe it could be a good start if you start checking on the backups. Make sure you use the SQL server compression feature for backing up the databases.

However just to check if this wait type is causing any problems, you need to check for any issues in backup throughput.

May be you can go around and test the backups by doing native backups on same network as compared to other network and check for other counter values.

Also, you can refer A cause of high-duration ASYNC_IO_COMPLETION waits for more explanation by Paul on this wait.