To demonstrate what VLV control provides what is missing in
Simple Paged Results specification, let us examine the data that
is loaded in OpenDS.
|
|
|
I have loaded OpenDS with 10001 entries of sample data.
|
Loading sample data into OpenDS is very simple.
|
Assume that OpenDS is installed under "/opt/OpenDS" directory.
|
cd /opt/OpenDS/bin
./stop-ds
./import-ldif -n userRoot -A /opt/OpenDS/config/MakeLDIF/example.template
./start-ds
|
The tool that I am using here is the "ldapsearch" that is
shipped with OpenDS which is capable of running LDAP
searches with both Simple Paged Results control and VLV
Control.
For demonstration purposes, let us add an administrator account.
$ ./ldapmodify -p 1389 -D "cn=directory manager" -w password -a
dn: uid=admin,ou=people,dc=example,dc=com
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
sn: Administrator
cn: Directory Administrator
userPassword: password
|
Add an ACI to grant full access to "admin" user
$ ./ldapmodify -p 1389 -D "cn=directory manager" -w password
dn: dc=example,dc=com
changetype: modify
add: aci
aci: (targetattr = "*")(version 3.0; acl "Admin Access"; allow(all)
userdn = "ldap:///uid=admin,ou=people,dc=example,dc=com";)
|
Grant read permission on Simple Paged Results Control for
user "admin".
The OID for Simple Paged Results control is
1.2.840.113556.1.4.319.
$ ./dsconfig -h localhost -p 1389 -D "cn=Directory Manager" -w password set-access-control-
handler-prop \
--add global-aci:"(targetcontrol=\"1.2.840.113556.1.4.319\")(version 3.0; acl \"Allow Simple Paged
Results Access\"; allow(read) userdn = \"ldap:///uid=admin,ou=people,dc=example,dc=com\";)" -n
|
Let us run the ldapsearch with Simple Paged Results control
./ldapsearch -p 1389 -b "ou=people,dc=example,dc=com" -s one -D
"uid=admin,ou=people,dc=example,dc=com" -w password --simplePageSize 1000
"objectclass=inetOrgPerson"
SEARCH operation failed
Result Code: 50 (Insufficient Access Rights)
Additional Information: You do not have sufficient privileges to perform an unindexed search
|
By default, the OpenDS does not allow search with more than
4000 candidates unless you are using cn=directory manger
One way to fix this is to change the index entry limit to a desired
value and another way of accomplishing the same is to create an
administrative account and grant unindexed-search privilege.
I chose the former approach. Not because of any particular
merit though.
So, let us change the default index limit of the index
objectCalss to 20,000.
$ ./dsconfig -h localhost -p 1389 -D "cn=directory manager" -w password
-n set-local-db-index-prop \ --backend-name userRoot --index-name objectclass
--set index-entry-limit:20000
$ ./stop-ds
$ ./rebuild-index -i objectclass -b dc=example,dc=com
$ ./start-ds
|
Also, grant unlimited lookthrough limit to "admin" user. Here is the
more information on lookthrough limit.
$ ./ldapmodify -p 1389 -D "cn=directory manager" -w password
dn: uid=admin,ou=people,dc=example,dc=com
changetype: modify
add: ds-rlim-lookthrough-limit
ds-rlim-lookthrough-limit: -1
|
Let us try the ldapsearch again.
./ldapsearch -p 1389 -b "ou=people,dc=example,dc=com" -s one -D
"uid=admin,ou=people,dc=example,dc=com" -w password --simplePageSize 2000
"objectclass=inetOrgPerson" dn
dn: uid=user.0,ou=People,dc=example,dc=com
dn: uid=user.1,ou=People,dc=example,dc=com
dn: uid=user.2,ou=People,dc=example,dc=com
dn: uid=user.3,ou=People,dc=example,dc=com
.
.
.
SEARCH operation failed
Result Code: 4 (Size Limit Exceeded)
Additional Information: This search operation has sent the maximum of 1000 entries to the client
|
The above command requests the OpenDS to return all entries in
pages with 2000 entries per page. However, the client receives LDAP
error code 4 (Size Limit Exceeded). Which means that the
server can't send send 2000 entries back to the client as requested.
The size limit is the constraint that the OpenDS server imposes with
default value set to 1000. More information on size limit is here.
So, let us tune our ldapsearch and decrease the page size to 1000.
$ ./ldapsearch -p 1389 -b "ou=people,dc=example,dc=com" -s one -D
"uid=admin,ou=people,dc=example,dc=com" -w password --simplePageSize 1000
"objectclass=inetOrgPerson" dn
dn: uid=user.0,ou=People,dc=example,dc=com
dn: uid=user.1,ou=People,dc=example,dc=com
dn: uid=user.2,ou=People,dc=example,dc=com
dn: uid=user.3,ou=People,dc=example,dc=com
dn: uid=user.4,ou=People,dc=example,dc=com
dn: uid=user.5,ou=People,dc=example,dc=com
dn: uid=user.6,ou=People,dc=example,dc=com
dn: uid=user.7,ou=People,dc=example,dc=com
.
.
.
|
We got our paged results. All of 10002 entries are returned with
1000 entries at a time. I have noticed that some LDAP server
administrators use Simple Paged Control to return all
the entries to generate reports, perform other operations on the data.
Now, let us examine VLV control and see how it can provide us
with a better paged results where a client can request the server to
"sort" on an attribute for the results and where to "start"
in the results set.
The
VLV Control allows a client to request the server in a
manageable
list of entries.
The
Control provides following options to the client.
1. before:after:index:count
before: Specify the number of entries before the target entry
after: Specify the number of entries after the target entry
index: Offset to the target entry. Number 1 means first entry
count: Number of the size of result set for a search filter.
2. before:after:index
before: Specify the number of entries before the target entry
after: Specify the number of entries after the target entry
index: string within the result set (index=worrell)
Again, the ldapsearch is the tool to use VLV control . In the following
command the ldapsearch sends a VLV request to the server with
following options (-G 0:1:1:0)
before: 0 - Zero entries before the target
after: 1 - One entry after the target
index: 1 - First Entry is the target.
Count: 0 - Unknown. Let the server determine the size.
The server returns two entries two entries one before the target and
the target itself. Most importantly, the result set is sorted on
"sn" attribute.
$ ./ldapsearch -p 1389 -b "ou=people,dc=example,dc=com" -s one -D
"uid=admin,ou=people,dc=example,dc=com" -w password -G 0:1:1:0 --sortOrder sn
"objectclass=inetOrgPerson"
dn: uid=user.4,ou=People,dc=example,dc=com
objectClass: person
objectClass: organizationalperson
objectClass: inetorgperson
objectClass: top
postalAddress: Aartjan Aalders$60403 South Street$Salem, MS 22403
postalCode: 22403
uid: user.4
description: This is the description for Aartjan Aalders.
userPassword: {SSHA}HBLw+/8K4Tf1fjzCyDyyCuhEmAO4+lL2Cmz5rw==
employeeNumber: 4
initials: AVA
givenName: Aartjan
pager: +1 165 230 1073
mobile: +1 096 889 4907
cn: Aartjan Aalders
telephoneNumber: +1 804 202 6921
sn: Aalders
street: 60403 South Street
homePhone: +1 326 592 0306
mail: user.4@example.com
l: Salem
st: MS
dn: uid=user.5,ou=People,dc=example,dc=com
objectClass: person
objectClass: organizationalperson
objectClass: inetorgperson
objectClass: top
postalAddress: Abagael Aasen$33591 Main Street$Lansing, NV 51577
postalCode: 51577
uid: user.5
description: This is the description for Abagael Aasen.
userPassword: {SSHA}3OoD2VVJOou9klYCX2i6ayO2LbY2tUF8x5Z3nw==
employeeNumber: 5
initials: AQA
givenName: Abagael
pager: +1 331 917 5915
mobile: +1 258 717 0685
cn: Abagael Aasen
telephoneNumber: +1 039 625 4980
sn: Aasen
street: 33591 Main Street
homePhone: +1 052 066 4526
mail: user.5@example.com
l: Lansing
st: NV
# VLV Target Offset: 1
# VLV Content Count: 10002
|
Let us look at another example. Request the server to return the target
entry that whose "sn" attribute begins with "jensen" and one
followed by the target.
$ ./ldapsearch -p 1389 -b "ou=people,dc=example,dc=com" -s one -D
"uid=admin,ou=people,dc=example,dc=com" -w password -G 0:1:jensen --sortOrder sn
"objectclass=inetOrgPerson"
dn: uid=user.5814,ou=People,dc=example,dc=com
objectClass: person
objectClass: organizationalperson
objectClass: inetorgperson
objectClass: top
postalAddress: Mollie Jensen$10019 First Street$Hattiesburg, SD 58443
postalCode: 58443
uid: user.5814
description: This is the description for Mollie Jensen.
userPassword: {SSHA}yAKGDBMBn0qTPIksRITI+CxaGem+AH0BzkLYrQ==
employeeNumber: 5814
initials: MHJ
givenName: Mollie
pager: +1 891 013 7026
mobile: +1 139 787 4408
cn: Mollie Jensen
telephoneNumber: +1 880 751 8601
sn: Jensen
street: 10019 First Street
homePhone: +1 016 249 2054
mail: user.5814@example.com
l: Hattiesburg
st: SD
dn: uid=user.5815,ou=People,dc=example,dc=com
objectClass: person
objectClass: organizationalperson
objectClass: inetorgperson
objectClass: top
postalAddress: Molly Jensenworth$31942 Fifteenth Street$Portland, NM 00430
postalCode: 00430
uid: user.5815
description: This is the description for Molly Jensenworth.
userPassword: {SSHA}2Bi8fv6syMvpWYqtO+o7sbe67zdereieAaT/gQ==
employeeNumber: 5815
initials: MTJ
givenName: Molly
pager: +1 327 826 0387
mobile: +1 896 260 6042
cn: Molly Jensenworth
telephoneNumber: +1 004 038 6469
sn: Jensenworth
street: 31942 Fifteenth Street
homePhone: +1 044 105 6790
mail: user.5815@example.com
l: Portland
st: NM
# VLV Target Offset: 5816
# VLV Content Count: 10002
|
It is now clear that with VLV control the client can determine the
result set, server side sort attribute and the page size. If you are writing
a LDAP application with a large amount of data the VLV control is the
best option.
For example, for an Address Book application you can efficiently
provide paged results based on what user inputs in the "Last Name"
field instead of going through multiple pages with potentially unwanted
data in it.
In order to get the above example to work, OpenDS needs some
configuration.
We need to tell the server how to handle VLV requests before start
using VLV control in your application(s). First provide VLV control
access to "admin" user VLV Control OID: 2.16.840.1.113730.3.4.9
./dsconfig -h localhost -p 1389 -D "cn=Directory Manager" -w password
set-access-control-handler-prop --add global-aci:"(targetcontrol=\"2.16.840.1.113730.3.4.9\")(version 3.0;
acl \"Allow Simple Paged Results Access\"; allow(read)
userdn = \"ldap:///uid=admin,ou=people,dc=example,dc=com\";)" -n
|
Create VLV control configuration for the search operation. Please note
that the "Base DN", "Search Scope", "Search Filter" and the
"Sort Attribute" should match the configuration in order to make
VLV to work.
Following dsconfig command adds the following VLV configuration.
Search Base: ou=people,dc=example,dc=com
Search Scope: One Level
Search Filter: "objectclass=inetOrgPerson"
Sort Attribute: sn
$ ./dsconfig -h localhost -p 1389 -D "cn=Directory Manager" -w password
create-local-db-vlv-index --backend-name userRoot --index-name People_Search
--set sort-order:sn --set scope:single-level --set base-dn:ou=people,dc=example,dc=com
--set filter:objectclass=inetOrgPerson -n
|
After creating the VLV configuration, the index should be generated
before using the VLV control.
$ ./stop-ds
$ ./rebuild-index -i vlv.People_Search -b dc=example,dc=com
$ ./start-ds
|
If you are reviewing the DSEE (Directory Server Enterprise
Edition 6.3) and would like to get the above VLV configuration
workingon the Sun Directory Server 6.3 perform the following
configuration.
Create the VLV configuration LDIF file "vlv.ldif" with following entry.
dn: cn=getsn,cn=example,cn=ldbm database,cn=plugins,cn=config
objectclass: top
objectclass: vlvSearch
cn: getsn
vlvBase: ou=People,dc=example,dc=com
vlvScope: 1
vlvfilter: (objectclass=inetorgperson))
aci: (target="ldap:///cn=getsn,cn=example,cn=ldbm database,cn=plugins,cn=config")(targetattr="*")
(version 3.0;acl "Config";allow(read,search,compare)(userdn="ldap:///anyone");)
dn: cn=sortsn,cn=getsn,cn=example,cn=ldbm database,cn=plugins,cn=config
objectclass: top
objectclass: vlvIndex
vlvsort: sn
cn: sortsn
|
Use ldapmodify to add the configuration to the Server.
$ ./ldapmodify -h localhost -p 1389 -D "cn=Directory Manager" -w password -a -f vlv.ldif
|
Now, generate the VLV index. Assuming that the Directory Server
is installed under /opt/DSEE63 and the actual Directory Server instance
is running under /opt/ds directory.
$ cd /opt/DSEE63/ds6/bin
$ ./dsadm stop /opt/ds
$ ./dsadm reindex -l -t sortsn /opt/ds dc=example,dc=com
$ ./dsadm start /opt/ds
|
All set. Now, you can run the ldapsearch (shipped with OpenDS
and also that is shipped with DSEE). I am going to provide an
example using the "ldapsearch" that is shipped with DSEE. You
can find the "ldapsearch" under /opt/DSEE63/dsrk6/bin directory
if you have installed using zip distribution.
$ ./ldapsearch -p 1389 -b "ou=people,dc=example,dc=com" -s one -D
"uid=admin,ou=people,dc=example,dc=com" -w password -G 0:5:jensen -S sn -x
"objectclass=inetOrgPerson" dn
version: 1
dn: uid=kjensen, ou=People, dc=example,dc=com
dn: uid=bjensen, ou=People, dc=example,dc=com
dn: uid=gjensen, ou=People, dc=example,dc=com
dn: uid=jjensen, ou=People, dc=example,dc=com
dn: uid=ajensen, ou=People, dc=example,dc=com
dn: uid=bjense2, ou=People, dc=example,dc=com
|
The Directory Editor which is part of DSEE an
example of application that uses VLV control for searching
the Directory Server.
One difference that I have noticed when using Simple Paged
Results control with ldapsearch, all the entries (10002 in this
case) are returned for the search filter "objectclass=inetorgperson"
with 1000 entries per page. However, when the VLV control
is used in the ldpasearch, I received only first page. I have noticed
that for some administrators it could be desirable to fetch all the
entries page wise just like ldpasearch has pulled all entries when
used with Simple Paged Results. If you are a developer/administrator
who likes to build tools around LDAP for administration
purposes to accomplish certain tasks then you can use
standard LDAP SDK for Java to write any LDAP applications
in Java.
I have written a Java class using LDAP SDK for Java
which is part of the DSEE (Directory Server Enterprise Edition),
can be downloaded from here.
The LDAP SDK for Java can be downloaded separately here.
I have used NetBeans IDE 6.1 to build the jar file.
VLVFetchAll.java - Performs a LDAP search using VLV Control
and fetches all the entries page wise.
You will need LDAP SDK for Java to compile. Get it from here.
If you have installed the DSEE then it will be part of DSRK.
VLVFetchAll.jar - Binary. Built using NetBeans IDE 6.1.
Usage: java VLVFetchAll [-h] [-p] [-b] [-D ] [-w ] [-S] [-s] [-P] [-f]
-h - LDAP Server host. Default: localhost
-p - LDAP Server port. Default: 389
-b - Base DN for search. Default: rootDSE
-D - Bind DN. Default: anonymous
-w - Bind Password. Defualt: blank
-s - Scope fo the search. Default: one
Allowed values: base,one,sub
-S - Attribute(s) to sort. Default: sn
-f - Search filter. Mandatory option
-A - Space delimited list of attributes. Default: all attributes
-P - VLV pages size. Default: 2000
Example:
java -jar VLVFetchAll -h sundir.example.com -b "ou=people,dc=example,dc=com" -D
"uid=admin,ou=people,dc=example,dc=com" -w password -f
"(objectclass=posixAccount)"-S "cn uid" -P 1000
In the above example all the entries are fetched that match the filter (objectClass=posixAccount)
Please note that the VLV configuration should already exist. If the Directory Server is configured
for LDAP naming services for Unix then the above example would work without any issues
|
If you are a System Administrator with lots of Perl background,
then you can use Net::LDAP to write a Perl script that exactly
does the same as the Java class above.