Directory Masters Event 2008 (Day 1)

The 2008 United States Directory Masters event was held in Somerset, NJ on May 12th & 13th, 2008. This event is a gathering of LDAP and Directory Server engineers, architects, and marketing personnel. We talk about the market leading Directory Server Enterprise Edition product from Sun, we talked about LDAP in the industry, and identity management topics. We also played some music and drank some beer!

Sun dominates the Directory Services communities in Financials, Telecommunications, and other industry verticals. Customers are storing more and more types of data in Directory Server, and accessing many different types of data using the LDAP protocol, such as Web 2.0 and SOA. Sun Java System Directory Server also dominates the identity management storage and access markets in the extranet.

Directory Server Enterprise Edition

We start with a presentation about the roadmap and new features available in Directory Server Enterprise Edition. Directory Server Enterprise Edition, encompasses the Directory Server 6, the Directory Proxy Server 6, Directory Editor, the Directory Service Control Center (DSCC), and Identity Synchronization for Windows.

Directory Server Enterprise Edition (DSEE) is a service-focused product. The days of focusing on an individual server are now in the past. New command line interfaces to enable easy scripting, and a new browser-based administration GUI makes the LDAP admin's job much easier and enable the service-focused paradigm. Service manageability has been enhanced by adding the browser-based administration GUI and adding scriptable command line utilities.

Directory Proxy Server

The Directory Proxy Server (DPS) enables virtual directory capabilities in the form of a consolidated virtual view of disparate data sources, configurable load balancing and automatic fail-over, firewall-like security, prevents trawling of critical identity data, protect private directory information and still publish public information, referral handling, data abstraction, DN re-writing and DN mapping, operational load-balancing, operations-based routing, and schema mapping. Directory Proxy Server provides multiple views to service disparate application needs and security requirements. Virtual views are constructed dynamically, so there is no need for costly and error-prone synchronization. Virtual data views also smash many political barriers to data access. Virtual data views also have data transformation capabilities. Directory Proxy Server supports client affinity also, meaning that replication latency is no longer a factor in LDAP client operations and that LDAP operations from the client perspective can pass the ACID test. Directory Proxy Server makes statistics available via the cn=monitor suffix: operation counters, status, subsystem status, and debugging info are all available.

DPS supports virtual views of data sources such as LDIF, JDBC, and LDAP.

Directory Proxy Server 6's failover mechanism hides commanded or uncommanded component outages from LDAP clients. The DPS has two health-check mechanisms, reactive (no periodic polling) and proactive (periodic polling), customizable health criteria, and silent retry (when possible) on transient errors.

The DPS Load-balancing capabilities comes in 4 forms: proportional, saturation, failover and hash. Proportional is the percentage-based load balancing. Saturation switches traffic to Server 'A' when a Server 'B' is "saturated". Failover always sends traffic to the best primary server, and hashed means identical operations to the same server.

Directory Proxy Server also provides a customizable data distribution facility which gives support for massively scalable environments and tremendous performance capabilities due to unlimited horizontal scalability. Data distribution also allows customers to use smaller, discreet datasets allowing for quicker backup and recovery. There are three distribution algorithms: lexicographic, numeric, and regular expression.

Operational routing routes LDAP operations to a selected dataview. A Dataview is a portion of the DIT identified by a viewbase (base DN). Operations are routed independently of each other, go to the best, most-specific data view, and searches might span multiple dataviews.

Directory Proxy Server supports client affinity (sticky routing), which obviates replication latency (loose consistency) issues, resource limits, request/response filtering policies, and other quality of service mechanisms.

Directory Proxy Server supports three different authentication methods:

• replays client request
• proxied authentication
• DPS uses fixed DN for authentication

Directory Proxy Server supports ACIs at the proxy level, the syntax is similar to Directory Server 6 ACI syntax, except that access to entry content is not supported.

Directory Server

The Directory Server in DSEE 6 provides performance enhancements in a number of key areas. We see customers doing many more updates via LDAP, and Directory Server update performance has been dramatically increased in response.

In Directory Server 6, there is no limit on the number of suppliers (masters) that can be deployed for a database, the ALLIDSTHRESHOLD is now configurable per database, There is now a last-login time plugin, the Directory Service Control Center can be, as of 6.1, deployed as a WAR file, password policy now supports grace logins, safe password changes and LDAP controls for account status and error reporting, as of DSEE 6.3, SUSE and HP-UX 11.23 (PA-RISC) are supported, Directory Server's frozen mode allows for snapshots of a database, using ZFS, this means quick backups of even huge datasets, Directory Server as of 6.1 supports database compaction allowing databases to be shrunk (offline).

Prioritized replication is a fantastic new feature: passwords, or any attributes, can be replicated more quickly than other attributes, it's based on a prioritized replication rule. Directory Server has tremendous performance capabilities in searches also.

Directory Server 6 has performance enhancements in the area of large static groups, ACI processing, enhancements of connection management, replication optimization, and database updates. As of 6.3, replication meta-data has been reduced.

Identity Synchronization for Windows supports integration with Active Directory, reducing deployment times (saving money and decreasing time-to-market), bidirectional password synchronization with AD, and bidirectional entry creation, deletion, and activation.

The Directory Editor (DE) is a web application that gives a browser-based interface to LDAP data, which is fantastic for delegated administration. DE runs in any servlet 2.2-compliant container - I run it in Glassfish.

The Directory Server Resource Kit (DSRK) provides a number of interesting tools for performance testing, debugging, deployment utilities, and some sample LDAP applications. SLAMD is also available for benchmarking and resource testing.

Useful Directory Server Links

* Official Product Page
* Directory Server Enterprise Edition at docs.sun.com
* Sun Wiki
* Bigadmin
* SLAMD: A distributed load generation engine
* ZFS

Glossary

* LDAP: Lightweight Directory Access Protocol
* LDIF: Directory Interchange Format
* DSEE: Directory Server Enterprise Edition
* DSCC: Directory Service Control Center
* DS: Directory Server
* DPS: Directory Proxy Server
* DE: Directory Editor
* DSRK: Directory Server Resource Kit
* ZFS: Zettabyte File System
* DIT: Directory Information Tree
* ACI: Access Control Instruction
* DN: distinguished name
* ACID: Atomicity, Consistency, Isolation, Durability

[posted with ecto]

Technorati Tags: availability, collaboration, directoryserver, DSEE, identity, java, ldap, MMR, networking, opends, openSource, protocols, replication, security, software, sun, tools, zfs

Posted at 07:05AM May 13, 2008 by tgardner in LDAP  |  Comments[0]

cache.pl

 # ./cache.pl --root /var/opt/sun/ds6 --pwdFile ~/ds6pwd  --bindDN 'cn=dmanager'
localhost 22:23:29 hitratio:99 pagein:560 pageout:561 roevict:0 rwevict:0
localhost 22:23:30 hitratio:99 pagein:560 pageout:561 roevict:0 rwevict:0
localhost 22:23:31 hitratio:99 pagein:560 pageout:561 roevict:0 rwevict:0
localhost 22:23:32 hitratio:99 pagein:560 pageout:561 roevict:0 rwevict:0

=== snip ===
#! /usr/bin/perl -w

#
# terry.gardner@sun.com
#
# 08-FEB-2008
#

# Parameter and variables

my ( $lines, $linesPerPage, $loadPerConnection );

#
# set up some default values
#
$port      = 389;
$root      = "/ldap/";
$instance  = "slapd-native";
$delay     = 1;
$hostname  = "localhost";
$pidFile   = $root . "/logs/pid";
$bindDN    = "cn=directory manager";
$pwdFile   = "ds6pwd";
$ldapsearch = "/opt/sun/dsrk6/bin/ldapsearch";
#
# parameters that cannot be modified by
# command line options
#
$dbCacheDN="cn=monitor,cn=ldbm database,cn=plugins,cn=config";

sub displayUsageHints() {
  print "cache.pl prints a tabular display of some directory server\n";
  print "        operational information.\n";
  print "\n";
  print "Usage:\n";
  print "cache.pl [OPTIONS]\n";
  print "\n";
  print "[--hostname hostname]           LDAP server hostname, default: $hostname\n";
  print "[--port port]                   LDAP server port, default: $port\n";
  print "[--instance instance]           LDAP server instance, default: $instance\n";
  print "[--delay delay]                 delay in seconds between samples, default: $delay\n";
  print "[--pwdFile filename]            file containing the root DN password, default: $pwdFile";
  print "[--bindDN bindDN]               root DN, default $bindDN";
  print "[--ldapsearch ldapsearchBinary] ldapsearch binary, default $ldapsearch";
  print "[--help]                        display ds_monitor.pl command invocation hints\n";
}

#
# process command line options
#
$sn = 0;
for (@ARGV) {
  if (/--port/) {
    $port = $ARGV[++$sn];
    ++$sn;
  } elsif (/--hostname/) {
    $hostname = $ARGV[++$sn];
    ++$sn;
  } elsif (/--root/) {
    $root = $ARGV[++$sn];
    ++$sn;
  } elsif (/--instance/) {
    $instance = $ARGV[++$sn];
    ++$sn;
  } elsif (/--delay/) {
    $delay = $ARGV[++$sn];
    ++$sn;
  } elsif (/--pidFile/) {
    $pidFile = $ARGV[++$sn];
    ++$sn;
  } elsif (/--pwdFile/) {
    $pwdFile = $ARGV[++$sn];
    ++$sn;
  } elsif (/--bindDN/) {
    $bindDN = $ARGV[++$sn];
    ++$sn;
  } elsif (/--ldapsearch/) {
    $ldapsearch = $ARGV[++$sn];
    ++$sn;
  } elsif (/--help/) {
    &displayUsageHints;
    exit;
  }
}

$scratchFile = "/tmp/" . "cache-" . `date +%H%m%S`;

$lines = 0;
$linesPerPage = 30;

for (;;) {
  # database cache
  $searchBase = $dbCacheDN;
  $cmd = "$ldapsearch -D '$bindDN' -j '$pwdFile' -s base -h $hostname -p $port -b '$searchBase' '(objectClass=*)'  > $scratchFile";
  `$cmd`;

  open(scratchFile,$scratchFile) || die "Failed to open $scratchFile: $!";

  while ()
  {
    chomp();

    s/://;

    ($attribute,$value) = split(" ",$_,2);

    if (/dbcachehitratio/) {
      $dbcachehitratio = $value;
    } elsif (/dbcachepagein/) {
      $dbcachepagein = $value;
    } elsif (/dbcachepageout/) {
      $dbcachepageout = $value;
    } elsif (/dbcacheroevict/) {
      $dbcacheroevict = $value;
    } elsif (/dbcacherwevict/) {
      $dbcacherwevict = $value;
    }

    $attribute = "";
  }


  close(scratchFile);
  
  # Get the current time
  ($sec,$min,$hour,$junk,$junk,$junk,$junk,$junk,$junk) = localtime(time);

  printf "%9.9s %2.2d:%2.2d:%2.2d hitratio:%d pagein:%d pageout:%d roevict:%d rwevict:%d\n",
       $hostname,
       $hour,
       $min,
       $sec,
       $dbcachehitratio,
       $dbcachepagein,
       $dbcachepageout,
       $dbcacheroevict,
       $dbcacherwevict;

  sleep $delay;

}
=== snip ===

Posted at 10:22AM May 02, 2008 by tgardner in LDAP  |  Comments[0]

ds_monitor.pl

Here's ds_monitor.pl. This little Perl script prints out some interesting Directory Server operational statistics. I'll try to keep an updated version at my tools.

#! /usr/bin/perl -w

#
# terry.gardner@sun.com
#
# 20-JUL-2005
# 20-SEP-2005
# 21-SEP-2005
# 08-FEB-2008
#

# Parameter and variables

my ( $lines, $linesPerPage, $loadPerConnection );

#
# set up some default values
#
$port      = 10389;
$root      = "/ldap";
$instance  = "instance-name";
$delay     = 1;
$hostname  = "localhost";
$pidFile   = $root . "/logs/pid";

#
# parameters that cannot be modified by
# command line options
#
$searchBase= "cn=monitor";

sub displayUsageHints() {
  print "ds_monitor.pl prints a tabular display of some directory server\n";
  print "        operational information.\n";
  print "\n";
  print "Usage:\n";
  print "ds_monitor.pl [OPTIONS]\n";
  print "\n";
  print "[--hostname hostname]            LDAP server hostname, default: $hostname\n";
  print "[--port port]                    LDAP server port, default: $port\n";
  print "[--instance instanceName]        LDAP server instance, default: $instance\n";
  print "[--delay delayBetweenIterations] delay between samples, default: $delay\n";
  print "[--pidFile file ]                file containing the PID of the running directory server, default: $pidFile\n";
  print "[--help]                         display ds_monitor.pl command invocation hints\n";
}

#
# process command line options
#
$sn = 0;
for (@ARGV) {
  if (/--port/) {
    $port = $ARGV[++$sn];
    ++$sn;
  } elsif (/--hostname/) {
    $hostname = $ARGV[++$sn];
    ++$sn;
  } elsif (/--root/) {
    $root = $ARGV[++$sn];    
    ++$sn;
  } elsif (/--instance/) {
    $instance = $ARGV[++$sn];
    ++$sn;
  } elsif (/--delay/) {
    $delay = $ARGV[++$sn];
    ++$sn;
  } elsif (/--pidFile/) {
    $pidFile = $ARGV[++$sn];
    ++$sn;
  } elsif (/--help/) {
    &displayUsageHints;
    exit;
  }
}

#
# check to see if the $pidFile can be opened
#
open(pidFile,$pidFile) or die "cannot open $pidFile: $!";
close(pidFile);

#
# read directory server pid, used
# later for process size
#
$pid = `cat $pidFile`; chomp $pid;

$scratchFile = "/tmp/" . "ds_monitor" . `date +%H%m%S`;

$lines = 0;
$linesPerPage = 30;

$lastCurrentConnections = 0;
$currentConnections = 0;

for (;;) {

  `ldapsearch -s base -h $hostname -p $port -b $searchBase '(objectClass=*)' > $scratchFile`;
  $size = `ps -p $pid -o vsz | tail -1`; chomp $size;

  $uptime = `uptime`;
  ($junk,
   $junk,
   $junk,
   $junk,
   $junk,
   $junk,
   $junk,
   $junk,
   $junk,
   $cpu) = split(" ",$uptime,9999);

  $cpu =~ s/,//g;

  open(scratchFile,$scratchFile) || die "Failed to open $scratchFile: $!";

  while ()
  {
    chomp();

    s/://;

    ($attribute,$value) = split(" ",$_,2);

    if (/currentconnections/) {
      $lastCurrentConnections = $currentConnections;
      $currentConnections = $value;
    } elsif (/totalconnections/) {
      $totalConnections = $value;
    } elsif (/readwaiters/) {
      $readWaiters = $value;
    } elsif (/request-que-backlog/) {
      $requestQueueBacklog = $value;
    } elsif (/connectionpeak/) {
      $connectionPeak = $value;
    }

    $attribute = "";
  }


  close(scratchFile);
  
  # Get the current time
  ($sec,$min,$hour,$junk,$junk,$junk,$junk,$junk,$junk) = localtime(time);

  $diffConnections = $currentConnections - $lastCurrentConnections;

  printf "%9.9s %2.2d:%2.2d:%2.2d size:%d la:%.2f tc:%d peak:%d cc:%d (%d) rq:%d rw:%d\n",
       $hostname,
       $hour,
       $min,
       $sec,
       $size,
       $cpu,
       $totalConnections,
       $connectionPeak,
       $currentConnections,
       $diffConnections,
       $requestQueueBacklog,
       $readWaiters;

  sleep $delay;

}

Technorati Tags: dsee, ldap, perl, script, software, tools

Posted at 01:17PM May 01, 2008 by tgardner in LDAP  |  Comments[1]

New content - exclusive master/supplier MMR deployments at the Sun wiki

There is a new article on the Wiki regarding the benefits of exclusive master/supplier deployments with DSEE6 at the Sun wiki.

Directory Server Enterprise Edition has eliminated the limit of 4 suppliers in multi-master replication topologies. There are a number of advantages to using all suppliers in an environment. Read the article to find out more.

[posted with ecto]

Technorati Tags: availability, directoryserver, MMR, software

Posted at 12:32PM Apr 30, 2008 by tgardner in LDAP  |  Comments[3]

Sun continues pursuit of Java for iPhone

Sun continues pursuit of Java for iPhone:
Sun continues to pursue talks with Apple to have Java applications run on Apple's iPhone while acknowledging a third party's efforts toward the same goal.

[posted with ecto]

Technorati Tags: ldap, iPhone

Posted at 09:52AM Apr 25, 2008 by tgardner in LDAP  |  Comments[0]

Definitions of LDAP

Some definitions of LDAP on the internet ....

RFC2251
Wikipedia
Gracion
LDAPMan
SearchMobileComputing
webopedia
about.com
Hitachi
The Dev Shed
docs.sun.com
BitPipe
TechWeb
OpenDS Wiki

Technorati Tags: ldap, protocols, software

Posted at 08:50AM Apr 25, 2008 by tgardner in LDAP  |  Comments[0]

IRC Channels #dsee and #slamd

There are two IRC channels on irc.freenode.net, #dsee and #slamd, that have been started to answer questions informally about Directory Server Enterprise Edition and SLAMD.

Technorati Tags: directoryserver, dsee, irc, ldap, slamd

Posted at 04:32AM Mar 20, 2008 by tgardner in LDAP  |  Comments[0]

LDAP Error Codes

LDAP Error Codes are listed in tabular form for convenience here on the Sun Wiki.

Technorati Tags: ldap

Posted at 03:34AM Feb 21, 2008 by tgardner in LDAP  |  Comments[0]

Correction to Sun Java Systems wiki article

I've made a correction to the OpenDS Replication marker at the Sun Java Systems wiki.

[posted with ecto]

Technorati Tags: blogs, ldap, opensource, wiki

Posted at 01:21PM Feb 18, 2008 by tgardner in LDAP  |  Comments[0]

Indexes in the Sun Java System Directory Server.

New content regarding behavior of indexes has been posted in the Sun wiki Directory Server indexes article.

[posted with ecto]

Technorati Tags: directoryserver, dsee, software, wiki

Posted at 12:56AM Feb 16, 2008 by tgardner in LDAP  |  Comments[0]