Gracefully handle stabilizer branching limits

[billti]

See comment at #3594 . This isn't easy to plumb through currently, but we should look at making the error handling more graceful if limits are exceeded.