How to best manage Index Builds with Stats Updates?

Question

This is a follow-up to what I read on Is there any reason to stop transaction log backups during a maintenance window? in the answer proposed by sp_BlitzErik.

I use Ola Hallengren's indexing script and have the settings specified as per the below. I run this once a week through an Agent job.

@Databases nvarchar(max),
@FragmentationLow nvarchar(max) = NULL,
@FragmentationMedium nvarchar(max) = 'INDEX_REORGANIZE,INDEX_REBUILD_ONLINE,INDEX_REBUILD_OFFLINE',
@FragmentationHigh nvarchar(max) = 'INDEX_REBUILD_ONLINE,INDEX_REBUILD_OFFLINE',
@FragmentationLevel1 int = 5,
@FragmentationLevel2 int = 30,
@PageCountLevel int = 500,

I know as apart of an Index Re-build an update stats is done automatically, but I have the following database properties set against all my databases which I believe should take care of updating stats:

Auto Create Statistics = True
Auto Update Statistics = True
Auto Update Statistics Asynchronously = True

What is generally best practice here for these settings and regularly updating stats though? Should you update stats on a nightly basis? I'm not sure how to measure whether stats should be updated or not which is why I have those database properties set.

I see in the answer by sp_BlitzErik he mentions "You're still going to want to update stats regularly though, and you can do that with this command:"

with the following command, but update stats regularly is very general.

EXECUTE dbo.IndexOptimize @Databases = 'USER_DATABASES',
                          @FragmentationLow = NULL,
                          @FragmentationMedium = NULL,
                          @FragmentationHigh = NULL,
                          @UpdateStatistics = 'ALL',
                          @OnlyModifiedStatistics = 'Y',
                          @StatisticsSample = NULL,
                          @LogToTable = 'Y';

Answer 1

I use Ola Hallengren's indexing script and have the settings specified as per the below. I run this once a week through an Agent job.

Let's break down the command you're running:

@FragmentationLevel1 int = 5,
@FragmentationLevel2 int = 30,
@PageCountLevel int = 500,

Even once a week, this is absurdly aggressive. I know the 5/30% comes from ancient Microsoft advice, but it's just that --ancient. It's time to move on.

You're taking the time to defragment indexes that are 500 8KB pages. That's a 4MB table. If your hardware has issues reading 4MB into memory, or keeping a 4MB table in memory, the answer isn't index maintenance. That's why I crank this up to at least 5000.

Modern hardware like SANs, SSDs, and non-32-bit servers that can house amounts of RAM > 4GB simply don't have the same data access issues that spinning platter drives from the early 2000s had.

We're talking record players vs. CDs.

Index rebuilds should happen extremely infrequently, and only to either correct a problem with an index, or change a setting. Here's the thing: 30% fragmentation isn't a real problem. It's just something DBAs do because they've been told to, and it's something they can measure.

So here's my question to you: how much time and resources does your server expend on index maintenance, and how much does that reduce time and resources consumed by queries?

If you can measure that, you can rebuild and reorg as much as you want.

... but I have the following database properties set against all my databases which I believe should take care of updating stats...

Kinda sorta. Auto-update stats (even async) happens when 20% of the table data + 500 rows (assuming the table is > 500 rows) changes. If you have a table with a million rows in it, that's 200k rows. You can dynamically decrease that threshold by using Trace Flag 2371, but auto update stats uses the default sampling algorithm, which may not be enough for tables with heavily skewed data.

but update stats regularly is very general.

Well, yeah, I'm answering questions about a server I've never looked at. I prefer nightly stats updates, but I've seen servers that needed them more regularly than that.

So what should you do?

1. Start by dialing back index maintenance to the commands I posted in the question you're referencing

   If you run into problems, decrease fragmentation thresholds until they stop. If no one says anything, run index maintenance less frequently, until maybe, just maybe, you never run it at all.

2. Update stats nightly

   Start with the default threshold. If you find that the full stats update that occurs with a rebuild is necessary, use the CommandLog table to find which tables and indexes were regularly being rebuilt, and start focusing on those. They'll typically be indexes for "big" tables, with tens of millions of rows.

This is about as specific as I can get about a server I've never seen. You're going to have to take the experimentation from here.

See also my post Why most of you should leave Auto-Update Statistics on