Lawrence Spracklen's Blog

I've been gradually expanding the crypto wiki (which can be found here); adding additional info and some code examples. Please let me know what additional information would be useful to add, how the wiki could be improved, and even add your own thoughts....

In my recent CommunityOne Microparallelism presentation, one of the cases studies discusses how to convert high ILP code on superscalar processors into the TLP implementations on CMT processors. The case study is discussed with reference to the SPARC implementation of SHA-1, which I wrote several years ago. The code, tuned for sun4u processors, can actually be found in OpenSolaris here. The message expansion portion of the SHA-1 computation is performed in parallel with the compression function portion using the VIS instructions. The SIMD nature of the VIS instructions is not leveraged, merely the fact that they allow integer operations to be performed on the FP pipelines. As a result, the IPC on a UltraSPARC IV+ processor is increased from around 2 to almost 4 -- improving performance by over 1.7X...

On CMT processors, such as T2, this doesn't deliver optimal performance. However, given the low inter-thread synchronization costs, one can consider performing these two portions of the SHA-1 computation using two threads:

I've been spending some time over recent years looking into the potential for leveraging compression on various off-chip links. In July's issue of IEEE Transactions on Computing, there is an article covering some of this work - focussing specifically on links to memory. The paper can be found here.

Basically, the article illustrates that on a wide variety of workloads (SPECcpu, media and commercial), a number of simple compression techniques will yield significant reductions in the amount of data that needs to be transmitted over the link (for both reads and write-backs). Accordingly, leveraging these techniques can deliver significant benefits - cost reduction, power reduction and potentially even performance improvements.

I've started a wiki to capture the more pertinent info on UltraSPARC crypto performance in a more organized form.

A brief synopsis of how to leverage the UltraSPARC hardware cryptographic accelerators from your application.

Introduction

Sun's UltraSPARC T1, T2 and T2Plus processors support high-performance hardware cryptographic accelerators on chip. These accelerators can significantly reduce the normally significant overheads associated with cryptography and secure operation.

On the UltraSPARC T1, T2 and T2plus processors, there is a cryptographic accelerator per each core, such that an 8-core processor provides 8 accelerators. The algorithms supported by these accelerators vary with processor and are illustrated in the following table:

The public-key operations are performed by the accelerator's modular arithmetic unit, while symmetric cipher and cryptographic hash operations are performed by the accelerator's cipher and hash unit (CHU). The UltraSPARC T1 accelerators are composed of just a MAU, while the UltraSPARC T2/T2plus accelerators have both MAU and CHU, both of which can operate in parallel. The accelerators operate at the core frequency (in parallel with the core) and are capable of delivering cryptographic performance that is typically an order of magnitude better than can be achieved on traditional processors in software, as is illustrated in the following table:

This article describes how to code your application such that it can leverage these hardware accelerators. Many important applications will already leverage the UltraSPARC hardware accelerators, either directly out-of-the-box or with minimal configuration. These include; the Sun Studio webserver, the Apache webserver, KSSL and IPsec to name but a few. More details of how to configure these applications are provided in a Sun cryptographic blueprint [1].

Using the UltraSPARC hardware cryptographic accelerators

Access to the cryptographic accelerators is controlled by the Solaris Cryptographic Framework. For non-privileged applications, access is via the userland cryptographic framework (UCF), while for kernel modules (such as KSSL or IPsec) access is via the kernel cryptographic framework (KCF). This article focuses on the userland cryptographic framework.

The Userland Cryptographic Framework exposes a PKCS#11 [2] compliant API to non-priv userland applications. Applications can interact directly with the UCF via the PKCS#11 interface, or indirectly via:

• Java Cryptographic Framework (JCE)

• OpenSSL

• Network Security Services (NSS)

The remainder of this article focuses on how to interact with the UCF directly and indirectly via JCE, OpenSSL and NSS.

Direct interaction with UCF

For PKCS#11 compliant applications, libpkcs11.so is the gateway to the UCF, and its just a simple matter of linking against this library [located in /usr/lib]. Given the fairly widespread use of the PKCS11 interface, especially with respect to traditional off-chip cryptographic accelerators (such as Sun's SCA6000 card), many applications already leverage PKCS#11. If an application doesn't already use the PKCS#11 interface, it is pretty straightforward to modify the application, with documents showing example implementations readily available [3].

Offload via OpenSSL

If the application uses OpenSSL for its cryptographic requirements (and many do), access to the accelerators can be achieved by using a version of OpenSSL that has been modified to support the PKCS#11 engine. A patched version of OpenSSL is supplied with Solaris 10 and is located in /usr/sfw/lib, allowing application compilation as follows:

cc -fast -I /usr/sfw/include -L /usr/sfw/lib -lcrypto aes_test.c -o aes_test.out

For operations that are to be offloaded, it is necessary to restrict use to the EVP_ functions and explicitly indicate the use of the PKCS11 engine; something like the following works for bulk ciphers (the process for RSA is similar):

ENGINE *e;

ENGINE_load_builtin_engines();

e = ENGINE_by_id("pkcs11");

ENGINE_set_default_ciphers(e);

EVP_CIPHER_CTX_init (&ctx);

EVP_EncryptInit (&ctx, EVP_des_cbc (), key, iv);

EVP_EncryptUpdate (.....);

PKCS#11 engine patches are available from OpenSSL.org for a number of different versions of OpenSSL, if the version of OpenSSL that ships with Solaris isn't suitable [4].

Offload via JCE

For applications that utilize the Java Cryptographic Extensions (JCE), the application should simply be configured to utilize the SunPKCS11-Solaris provider. Accordingly, in order for applications to use the hardware accelerators automatically, it is just necessary to ensure that sun.security.pkcs11.SunPKCS11 is configured as the first provider in $JAVA_HOME/jre/lib/security/java.security file.

The SunPKCS11-Solaris provider can also be explicitly selected as follows:

String provider = "SunPKCS11-Solaris";

Cipher aescipher = Cipher.getInstance("AES/ECB/NoPadding", provider);

It should be noted that the SunPKCS11-Solaris provider currently only offloads a subset of the chaining modes supported by the hardware, so make sure that the chaining mode and padding mode are supported [5]. The modes supported by the hardware accelerators are illustrated in the following table:

Offloading via NSS

In order for NSS to use the hardware cryptographic accelerators, the Solaris cryptographic framework should be added as a provider for NSS. This is achieved by modifying the appropriate NSS security databases. As an example, the following illustrates how firefox can offload RSA operations to the hardware:

/usr/sfw/bin/modutil -dbdir /home/sprack/.mozilla/firefox/r5s548iw.default/ -add "Solaris Crypto Framework" -libfile /usr/lib/libpkcs11.so -mechanisms RSA

/usr/sfw/bin/modutil -dbdir /home/sprack/.mozilla/firefox/r5s548iw.default/ -enable "Solaris Crypto Framework"

The use of the mechanism option indicates that the Solaris Cryptographic Framework should be the default provider for RSA operations [6].

Observability

When operations are submitted to the cryptographic framework, the cryptographic framework will, as appropriate, route processing for these operations to the Niagara cryptographic provider (ncp) device driver for public-key operations, and the Niagara-2 cryptographic provider (n2cp) device driver for symmetric cipher and cryptographic hash operations. These device drivers then perform the actual offload to the hardware accelerators and return the results to the framework. The interaction between these drivers and the cryptographic frame is controlled via cryptoadm.

kstat can be used to provide insight into the cryptographic operations that ncp and n2cp are handling, as follows:

kstat -m ncp | less

kstat -m n2cp | less

Additionally, cputrack can be utilized to determine the activity of the hardware accelerators directly (use cputrack -h to determine which counters to track).

Concluding thoughts

Cryptographic processing overheads are finding their way into an ever wider array of applications as security becomes ever more important. By providing on-chip hardware cryptographic accelerators, the UltraSPARC processors can vastly reduce these overheads, and in many situations enable respectable performance even when operating securely.

Via the Cryptographic Framework Solaris provides a simple way via which applications can leverage the benefits of the UltraSPARC hardware accelerators, while continuing to ensure application portability

References

[1] Using the cryptographic accelerators in the UltraSPARC T1 and T2 processors

[2] PKCS #11: Cryptographic Token Interface Standard

[3] The Solaris cryptographic framework

[4] Miscellaneous OpenSSL Contributions

[5] Sun PKCS#11 Provider's Supported Algorithms

[6] Configuring Solaris Cryptographic Framework and Sun Java System Web Server 7 on Systems With UltraSPARC T1 Processors

I've started looking at how to leverage the UltraSPARC T2 hardware cryptographic accelerators to improve MySQL performance and there are a couple of interesting opportunities;

1. SSL is used to secure communication between a potentially remote MySQL client and the MySQL server. One option is to modify the appropriate SSL libraries to use the T2 hardware accelerators where appropriate -- pretty straight forward. Another option that I'm currently investigating is trying to use the Solaris Kernel SSL proxy (KSSL). KSSL already uses the UltraSPARC T2 HW crypto accelerators, and so could be a very elegant solution to offloading MySQL SSL processing.

2. A variety of operations are supported by MySQL to secure database information, such as aes_encrypt() and des_decrypt(). Support for DES and SHA1 are also provided. Again, it is fairly straight forward to modify this code to use the T2 hardware accelerators were appropriate.

More details/results to follow as I continue investigating.

Just been playing with crypto on a 2-way UltraSPARC T2plus (Victoria Falls) system. The system, with 16-cores running at 1.2GHz was running Nevada 89, and my crypto microbenchmarks scaled very nicely. Able to hit HW peak performance (~6.8GB/s/system with AES-128-CBC), with suitable object sizes and a bunch of threads. More details as time allows.

I've just been experimenting with Java Cryptographic Framework (JCE) on the UltraSPARC T2 processor and it is important to remember which algorithms/modes/padding are supported for offload to the cryptographic hardware. While the UltraSPARC T2 processor supports most common chaining modes, offloads from JCE occur via the SunPKCS11-Solaris provider. The supported algorithms/modes/padding are somewhat more restrictive and are listed here. If a none supported mode is specified, the operation will not be offloaded to the HW, but will be performed in software.

If the SunPKCS11-Solaris provider is explicitly selected:

String provider = "SunPKCS11-Solaris";

Cipher aescipher = Cipher.getInstance("AES/ECB/NoPadding", provider);

then an exception is taken when a non supported mode is requested.

Thanks to everyone that attended my CommunityOne presentation earlier this week. The slides can be downloaded from here (search for Microparallelism) [Username: contentbuilder Password: doc789]-- there are significantly more Microparallelism examples than contained in the recent MultiCore Expo presentation.

As stated in an earlier entry, when running on an UltraSPARC T2 processor, applications using the Java cryptographic extensions (JCE) should (when applicable) automatically leverage the on-chip cryptographic accelerators.

Following a recent conversation with a Java Guru, you should check the following, if you experience problems:

Java on Solaris automatically sets SunPKCS11-Solaris (which calls into
the Solaris Crypto Framework) as the default security provider, so you
need to do nothing.

This begins from some version of J2SE 5.0. You can go look at the
${java.home}/lib/security/java.security file. There should be one line
look like:

security.provider.1=sun.security.pkcs11.SunPKCS11
${java.home}/lib/security/sunpkcs11-solaris.cfg

Interesting article on using AES from Java can be found here

I typically witter on about crypto performance at the microbenchmark level, but I was recently browsing the SPECweb05 results and I was impressed to see how the T2 performs, especially on the Banking workload, which is 100% HTTPS:

Intel 2-chip http://www.spec.org/web2005/results/res2008q1/web2005-20080225-00104.txt
Intel 4-chip http://www.spec.org/web2005/results/res2007q4/web2005-20071203-00101.html

Opteron http://www.spec.org/web2005/results/res2008q2/web2005-20080409-00107.txt
T2 http://www.spec.org/web2005/results/res2008q2/web2005-20080408-00105.txt

Pretty Impressive! So a single-socket UltraSPARC T2 processor provides equivalent performance to 4-socket x64 systems containing Quad-core processors! On a per socket basis, T2 outperforms the competition by over 2.7X!

Now, this performance leadership is not all down to the HW crypto support – I'm sure the onchip NICs, and abundance of threads help somewhat too. However, the cryptographic overheads associated with HTTPS are pretty significant – RSA ops for session establishment and then RC4 and MD5 (these are the algorithms used for SPECweb2005 anyway) operations to secure and authenticate the subsequent traffic. In fact, looking at the following figures:

Figure 1: Relative costs in an HTTPS transaction for different file sizes. Referenced from here

Figure 2: Typical breakdown of overheads for SPECweb2005 banking

it is apparent that a significant proportion of the total application-level overheads are associated with cryptographic processing. Its therefore not surprising that providing HW support to accelerate cryptographic processing provides a significant performance advantage to the UltraSPARC T2 processor on SPECweb05 banking...

Its nice to see that the good microbenchmark numbers actually translate into significant gains at an application level....

My slides from Multicore are now available on the OpenSPARC website here.

The other day I was looking for a C code example of illustrating how to leverage the softtoken key store when directly interacting with the Solaris crypto framework. There's substantial documentation available but I couldn't find a basic example. So here's what I concocted:

1. Configure my softtoken keystore via the command line:

	pktool setpin keystore=pkcs11 
	pktool genkey label=test_key keytype=aes keylen=128 
	pktool list objtype=key 

where the first operation updates the passphrase required to access the keystore. If the keystore doesn't exist, the keystore is first created. The default passphrase is “changeme”. The second operation creates a 128-bit AES key and installs it in the keystore – the label associated with the key is “test_key”. The third operation displays the contents of the keystore, so it is possible to confirm that the key has been created correctly.

2. Use the AES key in the keystore from a application that is directly using the Solaris crytpto framework:

#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <security/cryptoki.h>
#include <security/pkcs11.h>

int
main()
{
  CK_RV rv;
  CK_ULONG found_keys;
  CK_MECHANISM mechanism;
  CK_OBJECT_HANDLE hKey, key_list[1];
  CK_SESSION_HANDLE hSession;
  CK_UTF8CHAR label[] = {"test_key"};

  unsigned char ivec[16];
  unsigned char userPIN[] = {"mykeystore"};

  mechanism.mechanism = CKM_AES_CBC;
  mechanism.pParameter = ivec;
  mechanism.ulParameterLen = 16;

  rv = C_Initialize(NULL);
  if (rv != CKR_OK) {
      fprintf(stdout, "C_Init: rv = 0x%.8X\n", rv);
      exit(1);
  }

  /*Use metaslot i.e. slot 0*/
  rv = C_OpenSession(0, CKF_SERIAL_SESSION | CKF_RW_SESSION,
                     NULL, NULL, &hSession);
  if (rv != CKR_OK) {
      fprintf(stdout, "C_openSess: rv = 0x%.8X\n", rv);
      exit(1);
  }

  /*Log in using the correct passphrase*/
  rv = C_Login(hSession, CKU_USER, userPIN, sizeof(userPIN));
  if (rv != CKR_OK) {
      fprintf(stdout, "C_Login: rv = 0x%.8X\n", rv);
      exit(1);
  }

  /* Get the key object, where lable is the label of the
   * key we which to leverage
   */
  CK_ATTRIBUTE template[] = {
      {CKA_LABEL, label, sizeof(label)-1}
  };

  rv = C_FindObjectsInit(hSession, template,
                         sizeof (template) / sizeof (CK_ATTRIBUTE));
  if (rv != CKR_OK) {
      fprintf(stdout, "C_FindObjectsInit: rv = 0x%.8X\n", rv);
      exit(1);
  }

  rv = C_FindObjects(hSession, key_list, 1, &found_keys);

  hKey = key_list[0];

  if (rv != CKR_OK) {
      fprintf(stdout, "C_FindObjects: rv = 0x%.8X\n", rv);
      exit(1);
  }

  if (found_keys != 1)
  {
      fprintf(stdout, "C_FindObjects found %d objects\n", found_keys);
      exit(1);
  }

  /* Initialize the encryption operation in the session */
  rv = C_EncryptInit(hSession, &mechanism, hKey);
  if (rv != CKR_OK) {
      fprintf(stdout, "C_EncryptInit: rv = 0x%.8X\n", rv);
      exit(1);
  }

  .
  .
  .

in the above example it is assumed that the phasephrase is set to “mykeystore”.

The keys stored in the Solaris softtoken keystore are encrypted and they are also checked for integrity. The PBKDF2 function defined in PKCS#5 is used for generating the keys from the passphrase.

I have a more detailed presentation on Microparallelism at the upcoming CommunityOne conference (May 5th). Details here.

As I've mentioned in previous entries, Sun's latest UltraSPARC T2 Plus processors, which are launched today, continue to provide hardware acceleration for a wide variety of important cryptographic operations.

Acceleration is provided in an identical manner to the original UltraSPARC T2 processor - each core has its own hardware cryptographic accelerator that provides support for public-key operations (RSA, DSA, DH, ECC), bulk ciphers (RC4, DES, 3DES, AES-{128/192/256}) and secure hashes (MD5, SHA-1, SHA-256). For the bulk ciphers the currently supported chaining modes are ECB, CBC and CFB64 for DES/3DES and ECB, CBC, and CTR for AES.

The Sun SPARC Enterprise T5240 and T5140 Servers both support 2 UltraSPARC T2 Plus processors for a total of up to 16 cryptographic accelerators per system. Access to the accelerators is via the Solaris Cryptographic Framework (either directly, or indirectly via Java, OpenSSL or NSS) and the framework will automatically load balance requests across the 16 accelerators.

For both the T5240 and the T5140 the accelerators provide an aggregate throughput of up to 80Gb/s of AES-128 (enabling wire-speed encryption), and over 70,000 RSA-1024 sign operations/sec. And this performance can be delivered while the processor is largely idle and available for other processing, essentially eliminating the normally significant overheads associated with crypto processing (zero-cost security!).