As Gustavo showed us:
A timeout of client inactivity while the instance is in the passive state. The timeout is specified by the Deployer in an EJB Container implementation specific way.
So, no doubt - from the passive state there is no way to invoke the ejbRemove method. It is also being shown in the 'Stateful Session Bean State Diagram' on page #84.
Further down on page #89 on 2.1 spec:
If the session bean instance allocates resources in the ejbCreate<METHOD> method and/or in the business methods, and normally releases the resources in the ejbRemove method, these resources will not be automatically released in the above scenarios. The application using the session bean should provide some clean up mechanism to periodically clean up the unreleased resources.
Not exactly exciting to know that we can't rely on the fact the ejbRemove is being called every time. At least the specification is very clear about it.
Regards,
Dan