Tuesday December 05, 2006

OpenDS Access Control Requirements

Monday November 20, 2006

Installing OpenDS Made Easier

Monday November 06, 2006

Exploring OpenDS Made Easier

Friday October 13, 2006

Contributing to OpenDS

• If you're not already, become a member of the OpenDS project. You can do that by going to https://opends.dev.java.net/servlets/ProjectMembershipRequest and request the User role. There are other roles that appear on this page, but unfortunately we don't have much control over what shows up in the list. We don't use the Observer, Developer, or Content Developer roles in our project, so please don't request them, and you'll find it's pretty unlikely that we will give out the Project Owner role so please don't request it either.
  
  
• You can subscribe to one or more of our e-mail mailing lists at https://opends.dev.java.net/servlets/ProjectMailingListList. If you're interested in helping develop OpenDS, then the lists you might find most useful include dev (general discussion among OpenDS developers), commits (notifications of commits made into our repository), and precommit-notification (requests to review and comment on code before it gets committed).
  
  
• Check out and build OpenDS from our repository. You can find basic instructions for checking out the code and building it in HTML form in our documentation depot. This page also contains links to the OpenDS Source Guide in OpenDocument and PDF forms, which contains a lot more detailed information.
  
  
• Read our Contribute to OpenDS page and our Governance Model to understand the processes that we have in place.
  
  
• Sign and submit the Sun Contributor Agreement document, which is necessary before we will accept any contributed changes. Once you have completed this, then you can send a message to contribution-request@opends.dev.java.net to request the Contributor role in the project.

Congratulations to the Apache Directory Project

Friday September 22, 2006

Improving Test Coverage in OpenDS

Friday September 08, 2006

Why Run the Directory Server on Solaris?

• Solaris runs on lots of different systems, on the SPARC, x86, and x64 processors. Sun makes excellent hardware and our systems are frequently setting new world records in performance, but Solaris is certified on systems from lots of other hardware vendors and our hardware compatibility list contains more systems than most other UNIX or Linux vendors.
  
  
• Compatibility and stability is taken very seriously. If you develop your applications based on public interfaces (and we have tools to help you do that) in current versions of Solaris, then they will continue to work in future versions. This is not necessarily the case on other platforms.
  
  
• Solaris is available for use at no cost whatsoever if you don't need support. If you do need support, then the pricing plans are frequently less expensive than the offerings from other vendors.
  
  
• Solaris is continually and rapidly improving. Solaris 10 has many significant improvements over previous releases, and it is continuing to get better.

• Solaris runs on systems with anywhere from 1 to 144 CPU cores, and anywhere from hundreds of megabytes to over a terabyte of memory. These constraints are primarily based on hardware that is currently available and not necessarily limitations in the operating system.
  
  
• Solaris understands subtle but important differences between different types of hardware concurrency models (SMP, CMT, NUMA, dual/multi-core, HyperThreading, etc.) and can optimize its behavior accordingly.
  
  
• Solaris has excellent resource management capabilities that make it possible to control how computing resources are allocated between processes. This can be done through fixed-size processor sets, variable-size resource pools, and through various scheduler models like the fair-share scheduler, the fixed-priority scheduler, and the real-time scheduler. An upcoming CPU cap implementation may also provide another mechanism for achieving fine-grained control.
  
  
• Networking performance was significantly improved in the Solaris 10 release, and work is still ongoing to make it even faster and more scalable.
  
  
• In many areas, the libumem memory manager provides dramatically improved performance (especially when compared with previous alternatives like mtmalloc), in addition to providing many features that can help identify memory leaks and other related problems. Upcoming enhancements to the virtual memory subsystem should even further improve performance, especially when working with very large caches.
  
  
• UFS filesystem performance was dramatically improved in Solaris 9 updates and has been carried through to Solaris 10. In many cases (and particularly for the kinds of workloads that Directory Server has), ZFS is notably faster than UFS and offers many more attractive features like compression and snapshots that work well in a directory environment.
  
  
• We do much more Directory Server testing on Solaris (both SPARC and x86/x64 systems) than on any other platform. If there is a performance problem with the server on Solaris, we are more likely to find it and fix it before it is released than on other platforms.

• The introduction of process rights management (also called least privilege) in Solaris 10 makes it very easy to provide the Directory Server with exactly the privileges that it needs to operate, and even to take away capabilities that it doesn't need. Even if an exploitable security hole were found in the Directory Server, least privilege can be used to severely constrain what a potential attacker could do. For older Solaris systems, role-based access control (RBAC) can be used to limit the need to have root access when starting the server, although it is not as flexible as process rights management.
  
  
• Solaris containers (also called zones) can provide tightly-constrained environments for applications like Directory Server to operate in a manner that isolates it from anything else that might be running on the system. When used in conjunction with features like resource pools, containers can help limit the impact of denial-of-service attacks or other forms of resource exhaustion.
  
  
• The Solaris cryptographic framework provides a centralized mechanism for encryption and message digest operations, particularly those using the PKCS#11 framework. Although the Directory Server is currently unlikely to benefit from hardware SSL acceleration in most cases, it can still use this mechanism for secure key storage using FIPS 140-2 compliant devices. In addition, future developments in hardware (e.g., the "free encryption" capabilities of the Niagara 2 processor) may provide performance benefits. Note that you will not be able to take full advantage of the Solaris cryptographic framework with the Directory Server until our upcoming 6.0 release.
  
  
• Solaris BSM provides fine-grained auditing capabilities for keeping track of what happens on the system.
  
  
• Additional security-related enhancements that will be available in the near future include Trusted Extensions (which adds labeling capabilities to Solaris 10), ZFS encryption, and a key management framework.
  
  
• Solaris 9 has been evaluated at EAL4 for the CAPP and RBACPP protection profiles. Solaris 10 is currently under EAL4+ evaluation for the same profiles, and Solaris 10 with Trusted Extensions is also being evaluated with the LSPP protection profile.

• DTrace provides complete observability into virtually all aspects of the system in a way that is unmatched by any competing offerings. Even for external users without access to the Directory Server source code, it is very useful to be able to collect information about the underlying system and the way that the Directory Server interacts with it. For cases in which there is a problem with the Directory Server, engineering may be able to provide custom DTrace scripts for more detailed analysis of a problem without a significant impact on the running server.
  
  
• All of the standard performance analysis tools are available (e.g., vmstat, iostat, mpstat, etc.), but a number of additional tools are also provided, including lockstat/plockstat, prstat, cpustat, and trapstat. These can help identify potential problems or bottlenecks that may be responsible for unnecessarily low performance.
  
  
• The coreadm utility provides a mechanism for managing core file creation, and offers a way for allowing setuid/setgid processes like the Directory Server to dump core (which is not allowed by most UNIX-based operating systems).
  
  
• The service management framework (SMF) provides a mechanism for ensuring that services are started appropriately when the system boots, and can also monitor processes and restart them if a failure is detected. SMF can also provide further integration with process rights management to offer greater control over what rights are granted to and removed from individual processes.
  
  
• Much of the Solaris source code is available through OpenSolaris (and more code is being released all the time), which can make it easier to understand how the underlying system works. The new Solaris Internals books also go into great detail on the design and implementation of the operating system. In addition, OpenSolaris provides direct access to the engineers responsible for major components of Solaris.

Friday September 01, 2006

OpenDS Weekly Builds

1. Download the latest weekly build and unzip the contents of the file to wherever you want to put it.
   
   
2. Go into the OpenDS-0.1-build006 directory (or whatever directory was created by the version you are using) and run ./setup.sh on UNIX-based systems, or setup.bat on Windows. Follow the prompts to create the initial configuration for the server (just pressing ENTER to accept all the defaults, other than setting the root user password, will usually result in a working configuration).
   
   
3. Run the bin/start-ds.sh script (or for testing purposes you may prefer "bin/start-ds.sh -nodetach", which will cause the server to run in the foreground without detaching from the console) on UNIX-based systems, or "bin\start-ds.bat" on Windows (which will automatically create a separate console output window).

I have just uploaded OpenDS 0.1 Build 006, built from revision 262 of 
our source tree, to our weekly builds folder.  The direct link to 
download this file is:

https://opends.dev.java.net/files/documents/4926/39978/OpenDS-0.1-build006.zip

Notable changes that have been integrated since build 005 include:

Revision 211 (Issue #603) -- Add a mechanism that may be used to 
determine the operating system on which the server is running.  It 
includes code to try and identify the following operating systems:  AIX, 
FreeBSD, HP-UX, Linux, Mac OS X, Solaris, Windows, and z/OS.  We may be 
able to use this information to tailor the operation of the server to 
the underlying platform in certain cases.


Revision 227 (Issue #601) -- Provide a mechanism for setting file 
permissions.  On UNIX-based systems, we will attempt to use the chmod 
command if possible, which will work properly on both Java 5 and Java 6 
and will also provide the greatest flexibility.  On other systems, if 
the server is running with Java 6 then it will attempt to use the new 
methods added to the java.io.File class.


Revision 230 (Issue #588) -- Add initial MakeLDIF support, which can be 
used to generate entries based on a template.  This is similar to the 
MakeLDIF tool provided with SLAMD (although it is not yet completely 
implemented, and there may be cases in which the two will not be 100% 
compatible), although this version includes better validation of the 
template before generating entries.  Further, this capability has been 
directly integrated into the import utility so that it is possible to 
import data that is being generated on the fly rather than from an 
existing LDIF file.


Revision 242 (Issue #614) -- Update the client-side tools to help 
eliminate a potential race condition that could result from the reuse of 
LDAP message IDs.  On some systems, using the StartTLS extended 
operation could cause the client to reuse a message ID of an operation 
that was still being processed by the server, which would cause the 
server to reject the new request.


Revision 655 (Issue #604) -- Update the synchronization code to provide 
initial detection and resolution for conflicts that can occur when 
making updates on multiple servers at the same time.  Some of these 
conflicts include concurrent replacement of the same attribute in the 
same entry, modifying an entry on one server while deleting or renaming 
it on another, and concurrent creation of two entries with the same DN.


Revision 261 (Issue #618) -- Fix a bug that could impact indexed subtree 
searches using subInitial filters.

Friday August 25, 2006

LDAP/Directory Server Standards and Specifications

Thursday August 17, 2006

The truth about nsslapd-search-tune

1. The server will use the indexes to put together a candidate list containing the IDs of all the entries that could match the filter. This phase was discussed in an earlier post.
2. The server will retrieve each entry in the candidate list, check to see if it actually matches the search criteria, and if so return it to the client.

(member=uid=john.doe,ou=People,dc=example,dc=com)

(&(cn=My Group)(member=uid=john.doe,ou=People,dc=example,dc=com))

• 1 -- This controls whether the feature will actually be enabled. If it is not set, then there will always be a secondary check against the filter before returning each entry. If it is set, then this secondary check can be skipped under the right conditions.
• 8 -- This controls whether to allow skipping the secondary filter test even if an attribute used in the filter is also included in the values returned to the client. More on this later.
• 16 -- This controls whether to allow skipping the secondary filter test for compound filters (i.e., those containing AND or OR components). If it is not set, then only simple equality filters will be candidates for skipping the check.
• 32 -- This controls whether to ensure that index processing is always applied to all components in a compound AND filter. If it is set, then it can disable certain optimizations in the index processing code that could allow short-circuiting out of the index evaluation, which may degrade performance in some areas but will increase the conditions under which it is possible to bypass the secondary filter check.

Friday August 11, 2006

OpenDS, Java, and Performance

• Java is not an interpreted language. The HotSpot JIT compiler translates the bytecode into native machine code and then executes that. It is true that when the JVM is just started and the program initially starts running that it may be interpreted for a period of time, but once it has been compiled then it is running natively.
  
  
• It is true that garbage collection does run in the background. However, if it's done right then this has a negligible impact on performance. In fact, in many some ways the memory management approach that Java uses can be faster than most native applications because it's optimized for this kind of usage and allocating memory for small temporary objects is very efficient. Further, there are several different garbage collector models that you can choose from based on your requirements.
  
  
• Java is an extremely observable language. It's easy to see what's going on inside the JVM, and there are a plethora of tools that can be used to help analyze the performance of Java applications. There are interfaces that profilers can plug into, and it is also possible to use DTrace on Solaris 10 that can be used to examine code running in production. Similarly, we've even built a simple profiler into OpenDS itself that you can use to collect data about what's going on while the server is running (and without the need to restart). All of this makes it very easy to identify the portions of the code that can benefit most from performance optimization to help us know where to spend our time.
  
  
• There are a lot of configurable options available for use with the JVM, and some of them can have a significant impact on performance. This can include settings that impact memory usage, garbage collection, native compilation, lock behavior, etc. This can provide a number of benefits for different kinds of systems without the need for us to write any special code to take advantage of them.
  
  
• Java can also help us write code more quickly, since there are a number of IDEs that can help with rapid development, and we also don't need to spend as much time worrying about things like memory management and cross-platform compatibility.

Monday August 07, 2006

Interesting new features in OpenDS

• Configuration Cloning (Issue #164) -- When creating a new instance of the server, it will be possible to clone the configuration of an existing instance.
• Configuration Archiving (Issues #165, 166) -- The server will keep track of all of the configuration changes that are made over time and make it easy to determine who made what changes and when.

• Incremental Backup and Restore (Issue #15) -- It will be possible to perform an incremental backup including only the database changes since the last incremental or full backup.
• Compressed Backup and LDIF Export (Issues #16, 26) -- It will be possible to have the server compress database backups and LDIF exports, and to restore from the compressed archives.
• Encrypted Backup and LDIF Export (Issues #17, 27) -- It will be possible to have the server encrypt database backups and LDIF exports, and to restore from the encrypted archives.
• Hashed and/or Signed Backup and LDIF Export (Issues #18, 28) -- It will be possible to include a cryptographic digest and/or digital signature with a backup or LDIF export, and to verify that hash or signature upon performing a restore.

• Index Verification Tool (Issue #42) -- We will include a tool with the server that will allow you to ensure that all of the indexes are in sync with the data.
• Multiple Base DNs (Issue #48) -- It will be possible to include multiple base DNs in a single backend, which can help avoid the need to create a new backend for each suffix.
• Better Database Portability (Issue #67) -- While the Sun Java System Directory Server allows for binary copy initialization in some limited cases, there will be fewer restrictions for OpenDS. In particular, there will no longer be requirements to keep the database stored in the same location on the filesystem, nor to use the same operating system or CPU architecture.
• Database Preloading (Issue #60) -- There will be an option to allow the server to automatically preload portions of the database on startup so that the information will be read into caches for faster availability.
• Better Debugging Options (Issues #69, 70) -- The server will make information available to clients about the processing performed as part of a search operation (e.g., how many potential candidate entries there are, and how candidate lists are merged).
• NFS Support (Issue #566) -- The server will support the ability to store the database on an NFS volume (although it will not be supported to access the database from multiple systems concurrently).

• Run Without an Entry Cache (Issue #562) -- The server will provide the ability to run without any kind of entry cache (in fact, this is the default configuration).
• Cache Filtering (Issue #563) -- The server will provide the ability to define include and exclude filters that can be used to control the kinds of entries that may be stored in the entry cache.
• Better Memory Management Options (Issues #564,565) -- Entry cache sizing should be relatively simple and should not require much interaction from the administrator.

• Multiple Request Handlers (Issue #149) -- The server will provide the ability to have multiple threads used to read requests from clients, which can provide for better performance and scalability.
• Ability to Disconnect Clients (Issue #429) -- The server will provide the ability for administrators to terminate client connections if it is deemed necessary (e.g., a particular client is sending a large number of inefficient requests to the server).
• On-the-Fly Management (Issue #568) -- The server will provide the ability to enable, disable, create, and destroy connection handlers on the fly without the need to restart.
• Stop Accepting without Disconnecting (Issue #150) -- The server will provide the ability to configure a connection handler so that it can stop accepting new connections without terminating existing connections, which can be used as a kind of graceful shutdown mechanism.

• Pluggable Keystores (Issue #412-415, 569) -- The server will provide the ability to obtain its SSL certificates from a variety of locations, including JSSE and NSS keystores, PKCS#12 files, and PKCS#11 devices. An API will also be provided to allow further extending this capability.
• Multiple Root DNs (Issue #419) -- The server will provide the ability to have multiple root DNs in the server. This will allow for the use of different passwords (and potentially different password policies) for each root user, and will provide a better audit trail for operations performed by root users.
• Strong Authentication for Root DNs (Issue #420) -- Root user accounts will be represented as entries, which will allow them to contain the information necessary to be used to perform SASL authentication.
• Access Controls for Bind Operations (Issue #421, 440) -- The server will provide the ability to enforce access controls for bind operations, including binds by root users.
• Hashed/Encrypted Compare-Only Attributes (Issue #467) -- The server will provide the ability to create attributes whose values are hashed or encrypted, and will not allow general searches against values for those attributes.
• Scoped Privileges (Issues #468, 469) -- The server will provide a privileges capability, which can be used to assign capabilities to normal users that would normally only be available to root users. Further, those privileges may be scoped to constrain their effects in some way (e.g., a user may be exempted from access control evaluation, but only for a particular branch of the directory tree).

• Pass-Through Authentication (Issues #294-300) -- The server will provide the ability to allow user passwords to be stored in other repositories (e.g., other directory servers, IMAP/POP3, Access Manager, Kerberos servers, etc.) and have authentication interact with that external repository to verify the user's password.
• New Password Storage Schemes (Issues #310,312,314-318) -- The server will support a number of new password storage schemes, including the SHA-2 algorithms, as well as reversible schemes like 3DES and AES.
• Storage Scheme Deprecation/Migration (Issue #323) -- The server will support the ability to automatically deprecate certain storage schemes so that if a user authenticates to the server and has a password encoded in one or more deprecated schemes it will be automatically re-encoded using the default schemes.
• Idle Account Lockout (Issue #307) -- The server will provide the ability to automatically lock accounts that have not been used for longer than a specified length of time.
• Account Expiration (Issue #543) -- The server will provide the ability to define an expiration time for accounts, so that they cannot be used after that time. This could be useful for temporary accounts like those used by contractors.
• Pluggable Password Validators (Issues #334-341) -- The server will provide an API for defining password validators and will include a number of default implementations, including those based on the length of the password, the types of characters in the password, whether the password is in the user's password history, whether the password is contained in a given dictionary, and whether the password matches the value of any attribute in the user's entry.
• Account Status Change Notification (Issue #483) -- The server will provide a mechanism for notifying users of significant changes to their account, including warning about an upcoming expiration, a password change or administrative reset, or a lockout due to failed attempts.

• Multiple Concurrent Loggers (Issue #178) -- The server will allow administrators to configure multiple loggers of each type. For example, it will be possible to have multiple access loggers, with one capturing all information, another only logging failed operations, a third only logging write operations, etc.
• Log Filtering (Issue #183) -- The server will make it possible to define log filters, which can be used to control which operations get written to a log.
• Centralized Logging (Issues #180-182) -- The server will provide the ability for muultiple server instances to log to a centralized repository (e.g., to a central database, or to a shared filesystem).
• Compressed Logging (Issue #185) -- The server will provide the ability to write log information in compressed form.
• Encrypted Logging (Issue #186) -- The server will provide the ability to write log information in encrypted form.
• Signed/Hashed Logging (Issue #187) -- The server will provide the ability to hash and/or sign log records as a means of tamper detection.
• Rotation Actions (Issues #193-196) -- The server will provide the ability to perform various operations when a log file is rotated (e.g., compress on rotate, encrypt on rotate, sign on rotate, etc.).
• Unique Message IDs (Issue #203) -- Every message that may be written to the server's error log or that may be included in a response sent to the client will be assigned a unique integer ID.

• Better Monitor Coverage (Issues #206-216) -- The server will provide the ability to publish more monitor information, so that it is easier to understand its performance and health state.
• Integrating with External Systems (Issues #226-232) -- We will provide instructions that can be used to allow the server to integrate with external monitoring frameworks like Big Brother, HP OpenView, Nagios, BMC PATROL, and Tivoli Monitoring.
• Pluggable Notification Mechanisms (Issues #219-224) -- The server will provide a notification mechanism so that administrators can be made aware of any significant problems or notable events that are encountered during processing.

Tuesday August 01, 2006

Introducing the OpenDS Directory Service

Monday July 31, 2006

Posts now mirrored at www.directorymanager.org

Friday July 28, 2006

Understanding the Backend Monitor Entries

• currentEntryCacheCount -- This specifies the number of entries that are currently held in the entry cache for the backend.
  
  
• currentEntryCacheSize -- This specifies the amount of memory (in bytes) that the server believes the backend's entry cache is consuming. This number may be somewhat inaccurate (see my earlier post on cache sizing for why this is the case), although the accuracy should be improved in the upcoming Directory Server 6 release.
  
  
• database -- This specifies the type of database that is in use for the backend. This should always be "ldbm database".
  
  
• dbEntryCount -- This specifies the number of entries in the backend database (i.e., the number of unique keys in the id2entry database file).
  
  
• dbFileCacheHit-# -- This specifies the number of times the server requested an entry from the associated database file and the data was already contained in the database cache.
  
  
• dbFileCacheMiss-# -- This specifies the number of times the server requested an entry from the associated database file and the data was not contained in the database cache.
  
  
• dbFileName-# -- This specifies the path to a database file for the backend. Note that the "-#" part of this and other dbFile* attribute names actually indicates an integer value. Each database file accessed by the server will be given a different number, and the same number will be used for the DB file name, hit and miss count, and page in and out count so that those statistics can be associated with the appropriate database file.
  
  
• dbFilePageIn-# -- This specifies the number of times that content from the associated database file had to be paged into the cache.
  
  
• dbFilePageOut-# -- This specifies the number of times that content from the associated database file was paged out of the cache (presumably to make room in the cache for another database page that needed to be paged in).
  
  
• entryCacheHitRatio -- This specifies the percentage of the time that a request to retrieve an entry from the entry cache actually finds that entry in the cache. It will be equal to 100 * entryCacheHits / entryCacheTries.
  
  
• entryCacheHits -- This specifies the total number of times that a request to retrieve an entry from the entry cache actually finds that entry in the cache.
  
  
• entryCacheTries -- This specifies the total number of times that a request was made to retrieve an entry from the entry cache, regardless of whether or not the target entry was in the cache.
  
  
• ldapEntryCount -- This specifies the number of LDAP entries in the backend database. The difference between this and the dbEntryCount attribute is that the ldapEntryCount does not include special entries like tombstones that are not generally accessible over LDAP.
  
  
• maxEntryCacheCount -- This specifies the maximum number of entries that may be held in the entry cache, or -1 if there is no upper limit to the number of entries to store.
  
  
• maxEntryCacheSize -- This specifies the maximum amount of memory in bytes that the server will try to use for the entry cache. Again, this may be a little inaccurate, but Directory Server 6 should be better than 5.2 in this area.
  
  
• nextID -- This specifies the entry ID that will be used for the next entry that is added to the backend database. This is generally not that useful for monitoring, but may be helpful for some corner cases when debugging problems.
  
  
• readOnly -- This indicates whether the backend is configured to be read-only. Note that even if the backend itself is not configured in read-only mode, the server-wide read-only configuration may be enabled which has the same effect but across all backends.