Python nose unit tests generating too many clients already

Question

I'm using python 3.3, pyramid, sqlalchemy, psygopg2. I'm using a test postgres db for the unit tests. I have 101 unit tests set up for nose to run. On test 101 I get:

nose.proxy.OperationalError: (OperationalError) FATAL: sorry, too many clients already

It seems from the traceback that the exception is being thrown in

......./venv/lib/python3.3/site-packages/SQLAlchemy-0.8.2-py3.3.egg/sqlalchemy/pool.py", line 368, in __connect

   connection = self.__pool._creator()

Perhaps tearDown() is not running after each test? Isn't the connection pool limit for Postgresql 100 at one time?

Here's my BaseTest class:

class BaseTest(object):
    def setup(self):
        self.request = testing.DummyRequest()
        self.config = testing.setUp(request=self.request)
        self.config.scan('../models')
        sqlalchemy_url = 'postgresql://<user>:<pass>@localhost:5432/<db>'
        engine = create_engine(sqlalchemy_url)
        DBSession = scoped_session(sessionmaker(extension=ZopeTransactionExtension()))
        DBSession.configure(bind=engine)
        Base.metadata.bind = engine
        Base.metadata.create_all(engine)
        self.dbsession = DBSession

    def tearDown(self):
        testing.teardown()

My test classes inherit from BaseTest:

class TestUser(BaseTest):
    def __init__(self, dbsession = None):
        if dbsession:
            self.dbsession = dbsession
    
    def test_create_user(self):
        ......
        ......

One of the test classes tests a many-to-many relationship, so in that test class I first create the records needed to satisfy the foreign key relationships:

from tests.test_user import TestUser
from tests.test_app import TestApp
class TestAppUser(BaseTest):
    def __init__(self, dbsession = None):
        if dbsession:
            self.dbsession = dbsession
    
    def create_app_user(self):
        test_app = TestApp(self.dbsession)
        test_user = TestUser(self.dbsession)
        test_app.request = testing.DummyRequest()
        test_user.request = testing.DummyRequest()
        app = test_app.create_app()
        user = test_user.create_user()
        ......

I'm passing the dbsession into the TestApp and TestUser classes...I'm thinking that is the source of the problem, but I'm not sure.

Any help is greatly appreciated. Thanks.

Answer 1

Pyramid has nothing to do with SQLAlchemy. There is nowhere in Pyramid's API where you would link any of your SQLAlchemy configuration in a way that Pyramid would actually care. Therefore, Pyramid's testing.tearDown() does not do anything with connections. How could it? It doesn't know they exist.

You're using scoped sessions with a unit test, which really doesn't make a lot of sense because your unit tests are probably not threaded. So now you're creating threadlocal sessions and not cleaning them up. They aren't garbage collected because they're threadlocal. You also aren't manually closing those connections so the connection pool thinks they're still being used.

Is there a reason you need the ZopeTransactionExtension in your tests? Are you using the transaction package in your tests, or pyramid_tm? In a test if you don't know what something does then it shouldn't be there. You're calling create_all() from your setUp() method? That's going to be slow as hell introspecting the database and creating tables on every request. Ouch.

class BaseTest(object):
    def setUp(self):
        self.request = testing.DummyRequest()
        self.config = testing.setUp(request=self.request)
        self.config.scan('../models')
        sqlalchemy_url = 'postgresql://<user>:<pass>@localhost:5432/<db>'
        self.engine = create_engine(sqlalchemy_url)
        Base.metadata.create_all(bind=self.engine)
        self.sessionmaker = sessionmaker(bind=self.engine)
        self.sessions = []

    def makeSession(self, autoclose=True):
        session = self.sessionmaker()
        if autoclose:
            self.sessions.append(session)

    def tearDown(self):
        for session in self.sessions:
            session.close()
        self.engine.dispose()
        testing.teardown()