You could have pretty extensive UI generated for custom authentication module based on the callbacks defined in the module's xml properties file. Module xml file defines the set of callbacks (based on Auth_Module_Properties.dtd) and since Auth UI is completely driven by callbacks, it renders these callbacks via JSP pages.
OpenSSO support all standard JAAS callbacks + additional ones that we have implemented like "HttpCallback", "RedirectCallback" and "PagePrpertiescalback". As long as custom authentication module uses these callbacks in its xml file, Auth UI layer can understand and render those. With this set of callbacks, you can achieve extensive UI where you can exercise HTTP protocol level negotiation parameters, redirect to third party urls, change page branding like images, headers, templates (jsp pages), page time outs, etc.. in addition to what stand JAAS callbacks support. You can also do multi-step authentication if custom authentication module defines multiple set of "Callbacks" elements for multiple page orders or page states.
But if custom auth module wants to define its own custom callback, then Auth UI would not be able to render that callback since it would not know about it. If this case is required then one can use auth remote API to authenticate to custom auth module and process and render that custom callback in the client application itself. Auth remote API does support custom callbacks communication from auth module to the API caller, as per remote-auth.dtd.