Question: When should I define a service?

We've been getting stuck into Drupal 8 development and have been grappling with the concepts of services, dependency injection and containers.

One question that's come up has been:

When should I define a service?

Sometimes it's really obvious, suppose you are defining a new way to connect to some 'resource' a bit like a database, yeah that should be a service so that I can swap it out etc. Maybe you are providing a way to optimize images, yeah, that should be a service that accepts an image and returns an optimized version of that image.

But, should all classes you write, also be a service?

Looking at core, we can see that no, they shouldn't. Most (all?) route controllers, plugins etc. are not defined as a services. But they have factories that are defined as a service. Where is the line?

Drupal core provides some classes, like ConfigImporter, that aren't in the service container? Why?
Should they be a service, or should code wanting to use them instantiate them directly?

We have our opinions, but we'd love to hear yours. Add a comment to this article to let us know what guides your decision as to whether something should be a defined service or not.