“pre_get_posts” firing on every query

Question

How can I change arguments for the main query only, and not affect other queries?

add_filter('pre_get_posts', 'custom_post_count');
function custom_post_count($query){
  $query->set('posts_per_page', 5);
  return $query;
};

Because this action is called inside the get_posts method of WP_Query, this code will alter the posts_per_page argument for all loops, not just the main, so passing this argument to WP_Query is useless...