Last modified 3 years ago

Objective: create downloadable PDF reports

This feature would be accessible through the Control Center.

The user should choose between a full report or selecting the modules in which he is interested. He must also choose the time period of the report (the granularity is of one day). He must choose to have the report for specific Zentyal servers or/and for the network aggregate.

In the report there is historical data and actual state date (mostly actual configuration), some users are only interested in historical data to show to their manager so actual data msut be switched off at request.

Architecture

Each Zentyal module we have a method report() that will return the required data. The data origin will be the logs, the RRDs and other sources.

Daily the Zentyal will pick data for all modules and sent it to the CC in a CSV format.

In the CC the data will be received and stored in a database. Each different dataset will have its own table.

The report will be generated upon demand with the data stored.

The data stored we will used too in the virtual CIO project.

The CC's software portion don't need to be written in Perl

Report contents

  • the graphs and tabular data for the modules and the time period selected.
  • alerts for the modules selected
  • summary page of the server. This has the problem that in the time period the configuration can change several times. One option is to use the actual and forget from previous changes. This is easiest bz old changes are NOT recorded.
  • tops for web proxy

Contents by module

antivirus

  • we can put the last av database update

alerts

  • log alert counts; not only this is not implemented but it can overlap with other areas in this case we can skip it. Unless we think th number of alerts is interesting for personal management of the sysadmins.

asterisk

  • calls graph. Currently there is not log of this

ca

  • we can put the expiration datqa of the CA certificate
  • certificates issued and expired count

captiveportal

  • accesses. Currently there is not log

dhcp

  • log which contain leases and releases. Currently is not summarizes so we had to summarize it first
  • number of hosts active, medium and maximum (can be extracted from the previous item)

dns

  • requests. Currently there is not log
  • request per register. Not implemented
  • number of hosts in the domain. Maybe this should be shared with DHCP so a complete inventory of the domain is possible

ebackup

  • it would be cool to have the dates of the last backups. However at the moment there is not any log.

ebox

  • graph reporting uptime (there is a 'State' alert for this)
  • graph reporting service's uptime (there is already a alert for this)
  • RAID events (there is an alert for this, maybe we should log some RAID events)
  • last date of configuration backup

egroupware

  • accesses. Not log implemented

firewall

  • summarized log data (dropped packets)
  • top sources of dropped packages. Not implemented, we summarize only the number of dropped packets
  • open ports in each interface (using nmap?)

ids

  • summarized log (number of alerts)
  • top attackers and attack types (this ata is not currently logged)

improxy

  • traffic summary. Currently there is not log

jabber

  • traffic summary. Currently there is not log

l7-protocols

  • rules with description and traffic

mail

  • log data., currently not summarized. We could summarize it to show the aggregate number of messages sent and received.
  • traffic size. Currently not implemented

mailfilter

  • summarized report of smtp filter (number of messages in each category)
  • summarized report of transparent pop filter (number of messages in each category)

monitor

  • monitoring graphics. They are system load, cpu usage, temperature, file system usage and physical memory usage They re stored as rrds.

network

  • connection speed (feature neither implemented nor logged)
  • gateway uptimes. (there is already a alert for gateways not accessible)

ntp

  • whether is enabled and which ntp sever we are using

objects

*nothing to report here

openvpn

  • "Client Connection" and "Connection to server" events. Currently the logs are not summarized
  • speed connection in each VPN (feature neither implemented nor logged)
  • total traffic size in each VPN (feature neither implemented nor logged)
  • number of clients in each VPN (feature neither implemented nor logged)
  • nets reachable through the other point of a Zentyal to Zentyal conenction (not implemented)

printers

  • "Queued" log events. Currently arent summarized
  • Queued events per printer and pages printed per printer. Not implemented

radius

  • Number of successful and failed authorizations?. Currently there isn't any log

remoteservices

  • Status of subscription. Time left in subscription?

samba

  • Samba access logs. Currently not summarized
  • Samba antivirus logs. Currently not summarized
  • Samba quarantine logs. Currently not summarized

services

  • Nothing here...

software

  • show the version numbers of zentyal components and whether updates are avaialble
  • we can give the number of system packages needing updating or list them

squid

  • summarized request data
  • tops:

Tops for web proxy

  • 10, 20 or 25 more visited domains
  • for each domain 10 user whom visited the most?. This schema has consolidation problems bz the inherent data loss but could do the trick
  • top of N user which most visit banned and filtered domains
  • it may be good idea to be able to specify domain exceptions to the top list (i.e: intranet site, punch-in application, etc)

trafficshaping

  • statistics. Not implemented. Use the "tc -s" command

usersandgroups

  • Slave or master mode
  • log slave oeprations?
  • number of users and groups

webserver

  • number of accesses. Currently there is not log

Send automatic report

Maybe it would be a good idea to automatically email the report to configured addresses, allowing in this fashion some monitorization without having real administration access

PDF creation

Seeing the difficult to use PDF::Creator and the like, we will try to create a html and then use html2pdf or some similar tool to convert to PDF. See also:

Graphs

We use a JS library to generate graphs that could not export to any graphical file so we must look for other method to plot our data. The data itself is stored in a postgresql database so we can use a external tool.

Perl is lacking of "pretty" plot libraries (we have used in the past the Chart perl library) so we will not use a perl package.

One option would be to convert the data to rrd and use the library already developed to export in PNG format, but I will rather to avoid the data conversion to rrd.

Other options:

  • gnuplot
  • sphinx

Example

(commentaries between parenthesis) (All the graph could include tables showing the data. How many entries to show then?)

Zentyal Report

week 18th-27th July 2009 (time period) . Generated on [date generated[

Full report (in case of a partial report we will put Partial report and the modules used)

hostname

server statistics

[uptime graph]

[graphs from monitor module]

(what to do here if monitor module is disabled?)

[RAID events ] (if RAID is installed)

disaster recovery

[last date of configuration backup]

[last date of full backup] (from ebackup module)

(in both case after the date we will have a statement of the type "one day ago", "two weeks ago" ,etc..)

(add remote backups if available)

subscription services

[ time left in the subscription ]

software updates

[ version number of Zentyal components and available updates]

[number of components that could be updated ]

network

interfaces list

ethX

status (up o down)

xxx.xxx.xxx.xxx/z

[speed graph]

[traffic shaping statistics ]

gateway list

[gateway data]

[gateway availability graph]

(maybe it should be a good idea to put networks reacheable with Zentyal-to-Zentyal connections here instead that in the VPN section)

alerts

(no sure about this section if there is a lot overlap we can discard it)

[graph of alerts history]

firewall

[list of open ports]

[packet dropped graph]

[top adropped sources chart]

[l7 portocols rules]

ids

[service uptime graph]

[alert graph]

[top attackers and attack types]

domain

[ list of configured domains]

[ using data from dns and dhcp modules try to list hosts under each domain]

[DNS server uptime graph ]

[ DNS request graphs ]

dhcp

[DHCP server uptime graph ]

[DHCP requests and releases graphs ]

[DHCP hosts actives, medium and maximum ]

users

[Mode used]

[ If master last operations upon the slave ]

[ If slave , address of the master and date of the last operation ]

[Users and group count ]

openvpn

[service uptime graph]

servers list

server name

VPN address

[ Speed connection graph ]

[ Traffic graph ]

[ Graph with number of clients connected ]

Zentyal to Zentyal nodes

(we will list here both servers with Zentyal to Zentyal option enabled and clients)

server name

type (server or client)

VPN address

[Other ends address/es ] (a server could have more than one client)

[Network reacheable using the other end[

[ Speed connection graph ]

[ Traffic graph ]

ntp

[NTP server used for synchronization ]

mail

[service uptime graphic ]

[ graphic showing traffic in number of sent and received messages ]

[ graphic showing traffic in size ]

antivirus

[last update message ]

mail filter

[service uptime graphic]

[smtp filter graph]

[pop3 transparent proxy graph]

HTTP proxy

[service uptime graphic]

[ graph of request ]

[ tops of domains requested and users ]

Web server

[service uptime graphic]

[requests graphs per domain hosted]

samba

[samba service uptime traffic ]

[ samba access graphs ] [ samba antivirus and quaratine graphs ] (in the logs are different but maybe we could combine them

printers

( i dont know whether we can use uptime graphic here, bz i think that this use the same service than samba. I am wrong?)

[total queue graph]

[printers list]

[printer queue graph]

[printed pages]

jabber

[jabber service uptime graph ]

[ access and traffic graph ]

improxy

[ improxy service uptime ]

[access graph ]

Radius

[radius service uptime ]

[accesses graph] (show too failed accesses?)

Captive portal

[captive portal uptime]

[access graph ]

Egroupware

[egroupware service uptime ]

[access graph]

Tasks

Report creation framework

A way to define the layout of the report so it could call the appropriate methods in each module. There is some difficulty because in the sample they are data for different modules scattered in various sections. Anyway this precludes that modules themselves define all the rendering of its data. It seems that makes sense to use a template based solution. This template should have methods to easily request the needed bits of each module.

Improvements in Log modules

We will request log data form the report so we need methods to make this easier. We have the EBox::logs::Search method to retrieve entries but we need another to retrieve the graphics. We need also something to make the "top" lists, maybe something similar to the consolidation mechanism

Improvements in modules

A lot of the bit of informations in the draft are not implemented!. So we should decide which ones are a priority and try to implement them