Version 37 (modified by jgarcia@…, 5 years ago) (diff)

added attribute type schema 'version' for caching stuff in DataTable.pm

Content

  1. Ldap Backend
    1. Planning
    2. Work in progress
      1. Ldap #1
      2. Ldap #2
      3. Ldap #3
      4. Ldap #4
      5. Ldap #5
      6. Ldap #6
      7. Ldap #7
    3. Ldap Object Structure
      1. Organizational elements needed
      2. Example Ldap tree
    4. Ldap directory servers tested
      1. Benchmark Results
    5. Ldap information
      1. Useful RFCs
      2. Useful Links
      3. Useful commands

Ldap Backend

Work in progress... (probably you won't be able to understand most of the contents of this page, at least nowadays... in the end the contents could make sense, i hope so...)

So, as i said before, Viewer Discretion is Advised!

The purpose of this work is to replace gconf for Ldap (exactly OpenLdap) as the storage configuration engine.

Planning

  • (1) Generation of Ldap Schemas 4,5d
    • Type of attribute types (Ldap Syntaxes) for Ebox::Types::*
      • Relationship between Ldap Type <=> Ebox::Type::xxx.yyy Ldap #1 - 1,5d
        • Ip+netmask in 2 strings or in another type?
        • Is there a type for MacAddress representation?
      • New Types? Automatic or manual generation? /client/XXXX/src/Ebox/YYYY/Types/*.pm Ldap #2 - 0,5d
    • objectclass for each model defined in a TableDescription /client/XXXX/src/Ebox/YYYY/Model/*.pm #3 - 2,5d
      • Auto generation of Ldap schema given a TableDescription (new()->table())
        • We are going to mock the behaviour, in the future we should use the ModelManager
        • objectclass with attributes namespaced
        • OIDs?!
        • First generate attributetypes, secondly (and finally) the objectclasses
  • (2) Initial Ldap Backend Implementation 9,5d
    • Replace GconfModule for LdapStuff without EBox::Module integration #4 - 3d
    • Modify Types/* #5 - 1,5d
    • Modify DataTable #6 - 2,5d
      • Find operation in another file or class
      • Do not know how to store, make Table operations, not object ones
    • Modify Row #7 - 2,5d
      • Store (create/modify) the data
      • Get info from each attribute and create a query with LdapStuff for doing the dirty work
  • (3) Creation on the fly of Ldap schemas and organizational units and schemas
    • Check what it's already in the Ldap server
    • Create what it's neccesary
    • Use cn=config (intrepid's slapd works that way)
      • Problem found: how the hell can i insert entries there? (The client does not have sufficient access to perform the requested operation)
  • TODO
    • Refactor Types/* according to #5

Work in progress

Ldap #1

Relationship betwen ldap Type <=> Ebox::Type

  • Templates to create attributetype of some concrete SYNTAX.
    • Definition 
      AttributeTypeDescription = "(" whsp
   numericoid whsp     ; AttributeType identifier
 [ "NAME" qdescrs ]             ; name used in AttributeType
 [ "DESC" qdstring ]            ; description
 [ "OBSOLETE" whsp ]
 [ "SUP" woid ]                 ; derived from this other
                                ; AttributeType
 [ "EQUALITY" woid              ; Matching Rule name
 [ "ORDERING" woid              ; Matching Rule name
 [ "SUBSTR" woid ]              ; Matching Rule name
 [ "SYNTAX" whsp noidlen whsp ] ; Syntax OID
 [ "SINGLE-VALUE" whsp ]        ; default multi-valued
 [ "COLLECTIVE" whsp ]          ; default not collective
 [ "NO-USER-MODIFICATION" whsp ]; default user modifiable
 [ "USAGE" whsp AttributeUsage ]; default userApplications
 whsp ")"
    • Types supported
      • Boolean
      • String
      • Integer
      • Float
      • Datetime
    • Templates 
      attributetype ( __OID__ NAME '__NAME__'
 ___OPTIONS___
)

OPTIONS depends on which type we want to represent:

Boolean
  EQUALITY booleanMatch
  SYNTAX 1.3.6.1.4.1.1466.115.121.1.7

String (DirectoryString) UTF-8 encoding of ISO-10646 (Unicode)
  EQUALITY caseIgnoreMatch
  SUBSTR caseIgnoreSubstringsMatch
  ORDERING caseIgnoreOrderingMatch
  SYNTAX 1.3.6.1.4.1.1466.115.121.1.15

Integer
  EQUALITY integerMatch
  ORDERING integerOrderingMatch
  SYNTAX 1.3.6.1.4.1.1466.115.121.1.27


Float
  There is no standard for saving them, so save them as String

Datetime
    EQUALITY generalizedTimeMatch
    ORDERING generalizedTimeOrderingMatch
    SYNTAX 1.3.6.1.4.1.1466.115.121.1.24
  • Every Ebox's Types has a method (see Ldap #2 ) description() with a hash with tuples name => type
  • A function "somewhere" gets a EBox::Type (and a Module and Model name) and generates the attributetype schema definitions
    • For types that have more than one field (e.g. PortRange) the function generates more than one attributetype
    • Special cases will be:
      • Composite: do nothing
      • HasMany: No need for declare an attributetype, a hasmany field in a model, means that the foreign EBoxModel will be as a child of the current Ebox Model (the one which the HasMany attribute, but this's not concern for the schema generation)

Ldap #2

New Types? Automatic or manual generation

  • Every EBox::Type has to implement a description() method which returns a reference to a Hash with tuples name => type.
    • With this information (how the type need to be stored), we'll be able to generate schemas for every possible type created
    • e.g.: 
      package EBox::Type::Boolean;

sub description() {
    my $self = shift;
    return { 
        $self->fieldName() => EBox::Ldap::Types::Boolean 
    };
}

      package EBox::Type::Port;

sub description() {
    my $self = shift;
    return {
        'initial' => EBox::Ldap::Types::Integer,
        'end'     => EBox::Ldap::Types::Integer,
        'type'    => EBox::Ldap::Types::String
    };
}

Ldap #3

Auto generation of Ldap schema

  • For each model, firstly generate attribute types, finally generate object classes.
  • Attribute type
    • See Ldap #1 and Ldap #2
    • An attributetype depends on:
      • OID, which depends on:
        • EBox Module which the model belongs to
      • Type: this information is given by description method
      • Name, formed like "ebox$moduleName$modelName$attrName", which depends on:
        • EBox Module which the model belongs to
        • EBox Model which the attribute belongs to
  • Object class
    • Definition 
      ObjectClassDescription = "(" whsp
 numericoid whsp      ; ObjectClass identifier
 [ "NAME" qdescrs ]
 [ "DESC" qdstring ]
 [ "OBSOLETE" whsp ]
 [ "SUP" oids ]       ; Superior ObjectClasses
 [ ( "ABSTRACT" / "STRUCTURAL" / "AUXILIARY" ) whsp ]
                      ; default structural
 [ "MUST" oids ]      ; AttributeTypes
 [ "MAY" oids ]       ; AttributeTypes
  whsp ")"
    • All object class generated will be just like: 
      objectclass ( ___OID___
   NAME '___NAME___'
   DESC 'Model foo for module bar'
   SUP top
   STRUCTURAL
   MUST ( [uid|___ATTRIBUTES___] )
   MAY  ( ___ATTRIBUTES___ ) )
    • An objectclass depends on:
      • OID, which depends on:
        • EBox Module which the model belongs to
      • Name, formed like "ebox$moduleName$modelName"
      • Attributes
        • If there is some must attribute, this will be part of the RDN (the first one if there are > 1)
          • An attribute is declared as must when its unique method returns true and only has 1 field (See Ldap #2).
        • Otherwise, an uid attribute will be defined (the entries will look like: uid=1,cn=foo..., uid=2,cn=foo,.., uid=20,cn=foo...) (See Ldap Object Structure).

Ldap #4

Ldap stuff for dealing with Ldap Server

  • Reuse EBox::Ldap singleton class
    • It could be useful add methods like This methods will be added in schemaGeneration (EBox::Ldap will be used):
      • existsSchemaFor($moduleName, $modelName)
        • Should we create schema for model $modelName and module $moduleName?
      • oidStateFor($moduleName)
        • Returns the next free oid for both attributetypes and objectclasses inside the oid for the module $moduleName (e.g. if we have defined the module Samba with 1 model and 10 attributes, this method will return {attribute => 11, object => 2 }
      • nextFreeOid()
        • Returns the next oid to define a new ebox module (each module has a concrete subtree of the ebox's oids)

Ldap #5

EBox's Types behaviour

  • Do not know anything about how could they store, just indicate which fields will have persistence (with description method)
  • They should know how restore themselves (A default behavior will be provided, which it'll be ok for most of the types).
    • You can override restoreFromHash() or individual setters for each value of keys %{ $self->description }, setSomeAttribute(). 
      sub description() {
    my $self = shift;
    return {
        'initial port' => EBox::Ldap::Types::Integer,
        'end port'     => EBox::Ldap::Types::Integer,
        'type'         => EBox::Ldap::Types::String
    };
}
...
# Method: setInitialPort
# 
#    Override default behavior 
#
sub setInitialPort {
   my ($self, $initialPort) = @_;

   $self->{initial} = $initialPort; # silly example (this is the default behavior)
}
  • They should know what differences are between their data and another values (A default behavior will be provided), for this you can override compareToHash() or individual cmp functions like cmpSomeAttribute().
  • You can also override the default getters (with methods like someAttribute())
  • From the getters and cmps methods the type will have a method to obtain a hash with the diferences between its data and the data of another object (of the same type). This hash will be used for storing it to the ldap server.

Ldap #6

Modify !DataTable

  • Stuff done
    • Fixed issue when you have an enabled field and also set to true the enableProperty
    • Refactored notify related methods to another file (DataTable/Notificator?.pm), DataTable?.pm inherit from it.
  • Tips
    • $self->{'directory'} is going to be $self->{'dn'} (the full dn, not the concrete rdn of the instance)
    • ldapmodule replaces gconfmodule and it'll work with EBox::Ldap perl module at low level.
  • Issues
    • name() VS nameFromClass()
      • a) nameFromClass (let's automate as much as possible)
      • b) name (let's the programmer choose the name)
    • Remove gconfModule
      • How can we obtain the Module 's name?
        • a) it's the ->modelDomain(), isn't it? (EBox::DNS::Model::DomainTable has not the 'modelDomain' key, WTF?)
          • It seems $self->{'gconfmodule'}->name() is used for getting the module name as well
            • We are going to get the module's name from $self->{'ldapmodule'}->name(). This way is not longer necessary to specified modelDomain key in the tableDescription of a model.
        • b) As parameter in the new()

Ldap #7

Ldap Object Structure

Besides the schemas for both object classes and attribute types, we are going to need some schemas and entries to organize the contents

Organizational elements needed

  • Schemas
    • attribute types
      • eboxPkIndex 
        attributetype ( EBox::Ldap::OIDs::EBoxInternalOID.'.1.1'
    NAME 'eboxPkIndex'
    EQUALITY caseIgnoreMatch
    SUBSTR caseIgnoreSubstringsMatch
    ORDERING caseIgnoreOrderingMatch
    SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 )

attributetype ( EBox::Ldap::OIDs::EBoxInternalOID.'.1.2'
    NAME 'eboxDataVersion'
    EQUALITY integerMatch
    ORDERING integerOrderingMatch
    SYNTAX 1.3.6.1.4.1.1466.115.121.1.27 )
    • object classes
      • eboxModule 
        objectClass ( EBox::Ldap::OIDs::EBoxInternalOID.'.2.1'
    NAME 'eboxModule' 
    DESC 'container for EBox modules'
    SUP  top STRUCTURAL
    MUST ( cn )
    MAY  ( description ) )
      • eboxModel 
        objectClass ( EBox::Ldap::OIDs::EBoxInternalOID.'.2.2'
    NAME 'eboxModel' 
    DESC 'container for EBox models'
    SUP  top STRUCTURAL
    MUST ( cn )
    MAY  ( description $ eboxPkIndex $ eboxDataVersion ) )
  • Entries
    • dc=ebox (it's been already created somewhere...) (dcObject, organizationalUnit) 
      dn: dc=ebox
objectClass: dcObject
objectClass: organizationalUnit
dc: ebox
ou: ebox
    • ou=modules,dc=ebox (organizationalUnit) 
      dn: ou=modules,dc=ebox
objectClass: organizationalUnit
ou: modules
    • Foreach MODULE
      • cn=samba,ou=modules,dc=ebox (eboxModule) 
        dn: cn=samba,ou=modules,dc=ebox
objectClass: eboxModule
cn: samba
description: foobar
      • Foreach MODEL (when some (the first) object is being created)
        • cn=sambaShare,cn=samba,ou=modules,dc=ebox (eboxModel) 
          dn: cn=sambaShare,cn=samba,ou=modules,dc=ebox
objectClass: eboxModel
cn: sambaShare
description: foo foo foo
eboxPkIndex: []
eboxDataVersion: 0

Example Ldap tree

  • dc=ebox
    • ou=modules,dc=ebox
      • ou=moduleName,ou=modules,dc=ebox
        • cn=modelName,ou=moduleName,ou=modules,dc=ebox [keys, ...]
          • uid=1,cn=modelName,ou=moduleName,ou=modules,dc=ebox
            • cn=someSubmodelName,ou=moduleName,ou=modules,dc=ebox [keys,]
              • eboxModuleNameSubmodelNameAttrName=foobarizate,cn=modelName,ou=moduleName,ou=modules,dc=ebox
              • ...
              • eboxModuleNameSubmodelNameAttrName=jate,cn=modelName,ou=moduleName,ou=modules,dc=ebox
          • uid=2,cn=modelName,ou=moduleName,ou=modules,dc=ebox
            • cn=someSubmodelName,ou=moduleName,ou=modules,dc=ebox
              • ...
          • ...
          • uid=100,cn=modelName,ou=moduleName,ou=modules,dc=ebox
        • cn=anotherModelName,ou=moduleName,ou=modules,dc=ebox [keys, ..]
          • eboxModuleNameAnotherModelNameAttrName=Foobar,cn=anotherModelName,ou=moduleName,ou=modules,dc=ebox
          • ...
          • eboxModuleNameAnotherModelNameAttrName=lalala,cn=anotherModelName,ou=moduleName,ou=modules,dc=ebox
      • ...

Ldap directory servers tested

There are some schemas and code for each one, but in brief:

  • OpenLdap (Ubuntu 8.04)
    • In the last version (the one with Ubuntu 8.10) the schema are saved in the Ldap itself.
  • Apache Directory Server
    • Same performance (in the very simple load test used...)
  • Fedora Directory Server
    • More performance (+10-20%) and some additional features but less integration.

So, for now, OpenLdap wins (but it will be almost trivial change it in the future)

Benchmark Results

# OpenLdap VS MySQL, Quad core 6600. 10k entries, 4 values
       s/iter  LDAP     DB
LDAP     3.62   --     -98%
DB   7.00e-02  5074%    --

# OpenLdap VS MySQL , Dual core E2140, 10k entries, 4 values
     s/iter  LDAP    DB
LDAP   7.29    --  -98%
DB    0.172 4151%    --

#################################################

# OpenLdap Dual core E2140, 2k entries, 104 values
       s/iter
LDAP     3.51

# OpenLdap, Dual core E2140, 2k entries, 4 values
       s/iter
LDAP     1.28

#################################################

# OpenLdap vs FedoraDS, Dual core E2140, 10k entries 4 values
          s/iter  OpenLDAP Fedora DS
OpenLDAP    6.85        --      -14%
Fedora DS   5.86       17%        --

# OpenLdap vs FedoraDS, Dual core E2140, 2k entries 104 values
          s/iter  OpenLDAP Fedora DS
OpenLDAP    3.44        --      -13%
Fedora DS   3.01       14%        --

################################################
####################  Ruby  ####################
################################################

# OpenLdap vs MySQL, Quad core 6600, 10k entries 4 values
           user     system      total        real
LDAP  10.440000   0.790000  11.230000 ( 11.566221)
DB     0.350000   0.020000   0.370000 (  0.368707)

Ldap information

We are all f*king noobs unless the opposite will be proved

Useful RFCs

Useful commands

(if you don't have knowledge of ldap... or you're as dumb as i am)

  • Remove db (be careful with the permissions!) 
    $ sudo rm /var/lib/ldap/__db.* /var/lib/ldap/*.bdb /var/lib/ldap/log.*
  • Check configuration 
    $ sudo slaptest -v
  • See all supported stuff 
    $ ldapsearch -H ldap://localhost -xLLL -s base -b "" +
  • Add a ldif file 
    $ ldapadd -H ldap://127.0.0.1:9999  -x -w passForAdmin -D 'cn=admin,dc=local' -f file_with_data.ldif
-v # verbose mode
-c # continue if fails
  • Delete stuff 
    $ ldapdelete -r -x -w passForAdmin -H ldap://localhost:9999 -D 'cn=admin,dc=local' "ou=sampleModel,cn=sample,ou=modules,ou=ebox,dc=local"
-r # recursive
  • Search stuff 
    $ ldapsearch -H ldap://127.0.0.1:9999 -xLLL -w passForAdmin -D 'cn=admin,dc=local' "(objectclass=*)" 
-E pr=3 # pagination