Let's first take a look at your current position. You have said that your reviews have generally been positive. Does that mean that a few of these reviews were not positive? Or perhaps they were not glowing enough to make you feel that you are really on the partnership track? Even if you felt you had a strong shot at making partner, you never really can consider yourself a "lock." I know of far too many associates that were told that they would definitely make partner, only to find out that this prediction was not exactly on target.
Is your current firm an "up or out" environment or will you be guaranteed a job at the firm even if you are not voted into the partnership? If it is the latter, would you want to remain at your firm or would you still be interested in moving on? What does "substantial client" mean? If substantial to you means $1,000,000 in annual billings or more, your chances for becoming a partner are much brighter. However, if substantial means something like $25,000, your top-ranked firm is not going to be impressed. In fact, they may not even want this business, since it's so small.
As a mid/upper level associate, I think you were right to put yourself on the job market. If you are at all concerned about your future with your firm, it is time to see what else is out there before you are considered too senior to make a lateral move.
I would not reject an offer simply on the basis that it comes from a smaller firm. What's important is the quality of the firm's lawyers and the work they do, not the size of the firm. Potential employers in your city or another you'd like to move to will evaluate your employer on this basis.
While its size is not necessarily a problem, I do have a concern that this firm can't seem to keep its partners. I am not so certain that, as an associate, you are going to be able to get the answers you need about the firm's stability. It is difficult though not impossible for an associate to obtain financial information about a potential employer.
Keep in mind that it could be the former partners have become judges, or moved in-house as a general counsel to a client. In these situations you can expect the law firm to tell you about this right away, as these are matters to brag about. Since you have not heard about any departures due to these types of reasons, I doubt that all the departures were amicable.
Another point for you to consider is your significant client. Do you think this client will want to follow you to a 25-attorney firm? Are there enough attorneys at this firm to staff the client's needs? This needs to be a big consideration on your part.
Unless this firm has many compelling reasons for you to join them, I would strongly urge you to continue interviewing. The prospect of becoming a partner is never a "shoo-in," particularly when a firm is not stable. Some shaky firms have been known to ask attorneys to join the partnership simply to shore up their finances with the money new partners pay to buy into the partnership.
With your experience and portable business in tow, you should have many excellent prospects. Keep looking! Best wishes!
Summary: Whether you stay at your current firm or join the smaller firm, you cannot be guaranteed a win-win situation.