The 2009.Q3 software update provides a new iSCSI software stack via the COMSTAR framework. COMSTAR is a Solaris framework which centralizes and simplifies the deployment of SCSI targets. COMSTAR supports many different SCSI protocols, including iSCSI, fibre channel, iSER, with support for more protocols under development. In 2009.Q3, the COMSTAR iSCSI port provider has replaced the older iSCSI target daemon. This new COMSTAR iSCSI provider replaces the entire iSCSI stack, from the management of iSCSI targets and initiators down to the interactions with the underlying ZFS volume. COMSTAR provides improved performance, as well as a more flexible management model. In this blog entry, I'll explain the new management model as well as talk about the advantages of using this new framework.

With the iSCSI target daemon in the 2009.Q2 update, every LUN was advertised within its own iSCSI target. As a result, initiators could hit built-in limits which prevent discovery of all necessary targets. Moreover, managing these targets did not scale well as the number of LUNs on the system increased. Access to a LUN was controlled by that LUN's ACL, a list of initiators which were able to access that LUN.

The administrative model used with the COMSTAR framework has some key differences and advantages. This model breaks this one-to-one mapping of LUN to target, and instead allows an administrator to advertise multiple LUNs behind a single target. Each LUN participates in a target group, a set of iSCSI targets which all advertise that LUN. In addition, each LUN has its own initiator group, a list of initiators not unlike the previous software's initiator ACL. This group defines which initiators may access LUNs bound to that initiator group, as well as the CHAP parameters to expect if CHAP authentication is enabled.

Target and initiator groups define sets of targets and initiators which are associated with LUNs. Besides any user-created target and initiator groups, LUNs may be associated with the default target and initiator group. These default groups contain all targets and all initiators, respectively. While using the default target and initiator group can be useful for evaluation purposes, their use is discouraged since their use may expose the LUN to unwanted hosts.

In addition to the flexible mapping of LUNs to targets, the COMSTAR iSCSI stack also allows targets to be bound to network interfaces:

This binding allows fine separation of iSCSI traffic¹ from other data protocols. In addition, iSCSI traffic can be routed over the fastest interfaces, leaving other slow interfaces available management traffic.

Best Practices

When upgrading to 2009.Q3, you'll need to rebuild your iSCSI configuration in this new model. With that reconfiguration in mind, here are some best practices for deploying COMSTAR iSCSI. Keep in mind that IP and SCSI have similar abstractions, and you may find yourself able to solve a problem at the IP level (e.g., creating an LACP aggregation and binding a target to that aggregation's interface) or the SCSI level (e.g., creating a target for each of several interfaces and adding those targets to the same target group).

• Use a single iSCSI target per target group. As the example above indicates, you may also combine targets into a single target group, but I've found it simplest to create a LACP aggregation for a single target.
• Don't use the default initiator group -- that group grants every initiator access to a LUN. A renegade or misconfigured initiator may discover an incorrect LUN; explicitly using an initiator group prevents any damage.
• Avoid using the default target group beyond evaluation.
• If you know a LUN must have LU number 0, create that LUN first. LUNs which are given automatically-assigned LU numbers can be given number zero.
• Don't enable the write cache setting on a LUN unless you completely understand the ramifications of doing so. Consult your application's documentation for information about using this setting.

Upgrading to 2009.Q3

Once you've applied the 2009.Q3 software update, any iSCSI configuration from 2009.Q2 and earlier is set aside and the SAN environment must be rebuilt. All LUNs are still intact and available; however, you'll need to define new iSCSI targets and (optionally) initiators to access those LUNs. Here's a quick and dirty set of steps to get up and running with 2009.Q3:

1. Create one or more iSCSI targets bound to the proper interfaces. This configuration is available in the new Configuration > SAN section.
2. Assign the iSCSI targets you created in the first step to a target group.
3. For each LUN, change its target group to the group created above.
4. (Optional) Create initiator definitions, assign those initiators to initiators groups, then bind LUNs to those initiator groups.
5. (Optional) Configure any CHAP or RADIUS authentication.
6. Once you've validated with the configuration, apply the COMSTAR deferred update to allow complete PGR support.

Additional Information

• Roch explains the performance advantages of the COMSTAR stack.
• COMSTAR project on OpenSolaris.
• Consult the documenation bundled with the 2009.Q3 update for details on deploying iSCSI LUNs with COMSTAR.

---

[1] In some circumstances, traffic to or from and iSCSI target on one interface may travel on a different interface. With certain network configurations (like multiple default routes on the system), the routing configuration will dictate that traffic must use certain interfaces which conflict with the iSCSI configuration.

As discussed in other blog entries, many of the problems we solved in building the Sun Storage 7000 series were not specific to the NAS market. Moreover, the appliance kit framework we built solves many problems associated with an appliance in any domain -- that is, in addition to providing new innovative features, the appliance kit exposes and clarifies existing Solaris abstractions. In that vein, we needed a system to manage which users function as administrators of the appliance. We wanted the capability to pull users from an existing directory service, such as NIS or LDAP. Moreover, administrators need their own properties which were specific to our framework, such as whether they required a session annotation, or were a kiosk user. Finally, administrators have several user preferences which customize their environment to their liking, such as their initial login screen or character locale.

The 7000 series functions as a NIS and LDAP client¹ and queries those directory services for information about users. These directory services consolidate the management of users in that single place, and allow a user's password and other properties (UID, home directory, etc.) to remain consistent throughout an entire organization. Adding a administrator from a directory service allows that user to login to the 7000 series and administer the appliance. Attributes and passwords for these directory users are stored in a directory service, and the appliance authenticates those administrators through a directory service. Administrators can also be added locally, in which case passwords for those users are stored locally on the appliance. These local users must be added to each appliance separately.

Best Practices

When our team was discussing how users should administer the appliance, we decided that the best practice was to add all administrators from a directory service and forego adding any local users. We established this best practice for the following reasons:

• Simplified management of users. If an employee changes his password, that change will take effect on all machines.

• No UID conflicts. Because local users are given UIDs starting from 2000000000 in the order in which they're added, two local users on different appliances can have different UIDs, or even worse, the same UID can represent two different users. Even though these local users should not be used to access data, local users are no different than any other user on a Solaris system -- they can be used in establishing permissions on the root directory of a filesystem. If a particular user requires the same UID across several machines, shouldn't that user be stored in a directory service?

• No orphaned accounts. When an employee leaves a company, only the user account in the directory service must be deleted. The administrative accounts added to the appliance will not function if that user cannot be resolved in the directory service. These users are called degraded users, and while they should be cleaned up as part of normal housekeeping, they pose no security risk since they cannot login.

We've had some beta customers request the ability to prevent any local users on an appliance and restrict all administrators (other than root, of course) to be stored in a directory service. We'll work on this feature for the next software update. The appliance kit also has a rich set of fine-grained access controls for administrators which closely resembles RBAC in its design -- I'll cover this feature in a later entry.

---

[1] The 7000 series can also join an Active Directory domain and communicate with a domain controller for the purposes of authenticating CIFS clients. However, support for pulling administrators from Active Directory is not currently supported and will be coming a future software update.

As I described in my earlier blog entry on kiosk users, our team has been in contact with evaluation and beta customers to understand the problems they face as storage administrators. These administrators are responsible for the security of the storage infrastructure, and must understand how and why changes are made to the storage configuration. Many IT departments use a centralized ticket management system to manage open issues and requests. When a storage consumer reports a problem and requests some configuration change, that user is given a unique identifier to track the issue in this database. The ticket database contains all the details of the request, like the person requesting the change, the date and time of the request, which administrator will make the change, and the current status of the change.

One of our larger beta customers requested the ability to tag the audit log with a user-defined string which annotates that particular session. Certain users, upon logging in, must provide a session annotation:

Then, that annotation is saved with every audit record generated in that session. Users who are required to indicate this annotation may change their annotations in the middle of the session, but can never provide an empty session annotation:

Customers who have a centralized ticket management system can have their administrators provide the ticket identifier upon logging in. With those annotations, the auditors who examine the audit log gain a richer understanding of those actions, as each audit record can be correlated with the information in the database.

While the Fishworks team was developing our new storage server, we enlisted the help of several dozen experts in the enterprise storage field to help us define useful and necessary features. In addition to meeting the basic needs of storage administrators, we also wanted to provide additional features which administrators would find convenient. Across many enterprise domains, we continually heard one common complaint: storage servers were being blamed for performance problems, when the real performance problem lay somewhere else in the IT infrastructure.

At large companies, the storage infrastructure is consolidated into a single team of administrators who manage storage for the entire company. This team of administrators has service agreements which stipulate certain capacity, performance, and uptime requirements for other groups, and they must address problems if those service levels are not maintained. Storage administrators complained that they often spent time debugging performance problems with a particular group's application, only to find the storage was performing exactly as prescribed. Storage administrators could not gain any insight into a particular server's operation; instead, they wasted hours proving to application developers that the storage was not the source of the problem.

Our revolutionary analytics interface helps storage administrators understand how the box is performing. The dashboard page provides a summary view of key metrics, including a weather metaphor for each statistic. While this information is useful to storage administrators, it can also be useful to application developers who are actually using the storage. With this idea in mind, we created a new kind of administrator: a kiosk user. Kiosk users are created using the normal user dialog:

Storage administrators can add the application developers as kiosk users, or create a kiosk account for an entire team. By setting the screen to which kiosk users are restricted, administrators control the level of access application developers have to analytics data. For example, a kiosk user may be able to only view the dashboard, but not more specific worksheets. Likewise, a different kiosk user may be restricted to a worksheet for a particular project instead of having access to analytics data across all projects.

We anticipate that our customers will find this kiosk feature most useful for granting access to the dashboard and analytics data, though there are many other useful scenarios:

• Clients who have connectivity problems can use the services pages to check the state of system and data services.
• By viewing the audit logs, auditors can understand who accessed the appliance what changes those users have made.
• Network administrators can inspect the networking and routing configuration to understand and troubleshoot any problems.

One important note about kiosk users: even though they are restricted to viewing a certain screen, a malicious Javascript client can still make XML-RPC calls. A kiosk user cannot navigate to any other screen in the UI; however, that user will be able to see the results from raw XML-RPC calls which are not associated with that screen. Because of the appliance's fine-grained access control, a kiosk user with no roles or authorizations will not be able to change any configuration. Do not make the mistake of assuming a kiosk user will not be able to view the current shares, users, or other configuration parameters even if their kiosk screen would not normally allow them to access that data.

I hope our customers are able to find many and varied uses for kiosk users; please share your experiences with them in the comments.

Mike Shapiro and Bryan Cantrill have given a great overview of our new storage appliance, the Sun Storage Server 7xxx Series. I'm going to dive right in and give a summary of how our appliance fits into existing Windows environments, and explain what features we've created to help Windows administrators. Because our Sun Storage Server 7xxx series is built on top of Solaris, we're able to leverage many innovative Solaris technologies. The recent addition of a native Solaris CIFS server provides a new CIFS service tightly integrated with the Solaris kernel. This tight integration allows a richer set of functionality for CIFS clients, including seamless integration with NFSv4 clients.

Active Directory integration

Our storage appliance can join an Active Directory domain, which involves creating an account for that computer in the requisite location of the Active Directory database. Once that account has been established and the appliance has performed the necessary authentication, the appliance can query the Active Directory database for information about Windows users, groups, and other important objects. Because the appliance can join the Active Directory domain in this manner, no changes are needed to a customer's environment to support CIFS clients.

In addition to delegating client authentication to an Active Directory domain controller, the CIFS service can also operate in workgroup mode. A Windows workgroup is a collection of computers which authenticate users locally instead of storing user information in a centralized database like Active Directory or LDAP. We expect that workgroup mode will be used mainly for evaluation purposes, as any environment with more than about a dozen users would find workgroup mode extremely unwieldy.

Credential management

One of the goals of the Solaris CIFS project was to simplify how NFSv4 and CIFS clients access files concurrently. Because NFSv4 and CIFS clients use different tokens to represent the credentials of a user or group (UID/GIDs and SIDs, respectively), the underlying filesystem must have knowledge of both types of credentials to accurately allow and deny access. Existing solutions involve storing one type of credential, and then performing a translation of that credential to the other form. While this approach solves the problem of the disjoint sets of identities, it is brittle and has a large administrative burden.

When we set out to solve the problem of credential management and mapping more cleanly, Mike developed a plan to unify UID/GID and SID credentials on the ZFS filesystem. The resulting plan changed how access tokens were stored in the kernel and in ZFS, and allowed the identity mapping service to translate Windows credentials to Unix credentials and vice versa.

The identity mapping service maps Windows identities to Unix identities to allow CIFS and NFSv4 clients to share the same identity. If only CIFS or NFS clients are accessing a particular share, or if CIFS and NFS clients are accessing disjoint directories within a share, then no identity mapping configuration is necessary. If NFS and CIFS clients must share the same identity (i.e. if one employee wants to access her home directory over both protocols), then the identity mapping service needs to know how to correlate those two identities. When the identity mapping service sees a request for a user or group which is not already mapped, there are three methods of resolving that mapping: directory-based mapping, name-based mapping, and ephemeral mapping.

Directory-based identity mapping allows annotation of a user's object in both an LDAP and Active Directory database. The LDAP object is annotated with attributes about that user's Windows identity, and the AD object is annotated with information about that user's Unix identity. When the identity mapping service is configured to use directory-based mapping, the service will lookup these additional attributes in the corresponding directory, and create a mapping based on those attributes. The identity mapping service can be configured to use directory-based mapping in either direction: mapping from Windows to Unix ("AD-only"), Unix to Windows ("LDAP-only"), or both directions ("mixed"). This approach to identity mapping is the most scalable, as these attributes will only need to be created once per identity, and are stored in only one place for each mapping direction.

Name-based identity mapping uses various identity mapping rules which map users to other users. These identity mapping rules can either map a particular Windows user or groups to a particular Unix user or group, or can create more general mappings which map across all users or groups in an Active Directory domain. Again, this name-based mapping approach is only needed when NFS and CIFS clients share the same identity.

Finally, the identity mapping service falls back on ephemeral mapping if a mapping cannot be handled through directory-based or name-based mapping. An ephemeral mapping is a temporary mapping for a Windows user which is not persistent across reboots, but is stored persistently on disk. That is, when an appliance reboots, the same Windows user may map to a different ephemeral UID (UIDs above 2^31 are reserved for ephemeral UIDs.), but that Windows user will have the same access to files as before the reboot. Because these ephemeral UIDs are transient and not static, NFS clients cannot use these same UIDs. However, ephemeral UIDs and GIDs are perfectly valid when defining permissions on shares, and you may see them occasionally in a share's ACL if the appliance cannot communicate with an Active Directory server.

I hope you've received a good overview of some of Windows features and how identities are managed to create a truly native interoperable environment. Once the identity mapping scheme has been established, NFSv4 and CIFS clients can access common files with no restrictions or limitations. In the future, I'll blog about some other interesting Windows features, including:

• Our DTrace analytics system which can observe CIFS traffic like never before.
• Our full support for NT ACLs and ACL inheritance.
• Autohome shares, which are home directory shares created on-demand for all users.