Tony's Blog

Recently there was a case where an application used C_WrapKey() with an RSA key pair, on a Niagara-based system running Solaris 10, and the return code was CKR_MECHANISM_INVALID. Since RSA is supported by both ncp and pkcs11_softtoken that is an unexpected error code to see.

When looking through a truss of the output, metaslot calls the ncp crypto accelerator to do the wrapping operation; however, as displayed by 'cryptoadm list -mv', ncp does not support wrapping. This situation was fixed in OpenSolaris under bug 6543159. OpenSolaris now checks if the provider supports the operation before calling it. If it does not support the operation, the provider is skipped. But this is Solaris 10, which does not have the 6543159 fix. So when the C_WrapKey operations fails on ncp, it return CKR_MECHANISM_INVALID. So we know why the error exists, but why is metaslot returning that error code?

Metaslot was built so that one provider fails, it will try another provider if available. So continuing down the truss output, metaslot does failover to pkcs11_softtoken. But the C_WrapKey call fails with an error of CKR_WRAPPING_KEY_INCONSISTENT. Now we are on to something.. Both providers failed

So why did we not see pkcs11_softtoken's correct error code, but saw ncp's incorrect error code? Well it was a design decision for metaslot. We can only report one error code.. Metaslot makes the assumption by using the first error code, CKR_MECHANISM_INVALID. In 99% of the cases, where both providers support an operation it's safe to bet that they will both have the same error code. This is just that 1%.

So after fixing the CKR_WRAPPING_KEY_INCONSISTENT issue in the application, the operation worked. Under the hood, it still fails on the ncp call, but succeeds on the pkcs11_softtoken call and that success is returned.

If such an error happens to you, the best way to find the true error code is to run the below:

$ truss -f -u pkcs11_softtoken*::   2>&1 | grep C_WrapKey