Bitlogg

The client is one of India’s leading financial services groups with operations that span more than 40 different lines of business and subsidiaries. All financial services provided by them heavily depends on sound functioning of IT Systems. All such services offered by them also have to comply with various regulatory rules imposed by the respective regulators related to audit controls. In view of regulatory requirements, application should have audit trails of business record of all transactions and activities, including login and logout events and exceptions.
Requirements
The client has about 40 applications, running on a large server farm. Servers could be placed at more than one data centers. Number of applications and number of servers could grow in future. Most applications are on DotNET platform on Windows Server OS. Few others are on different platforms / OS. Number of such applications and number of servers within a server farm could grow in future. All such applications generate or are expected to generate copious amount of logs and audit trails. Logs are application specific and could be at different severity levels from an alert (highest) to information to debug (lowest) level. Audit logs are related to entry, modification, or deletion of information to a specific customer account or transaction and must be recorded and archived in tamper proof manner for compliance. In order to handle such requirement, a central log server is desired to archive logs. In addition to centralized archiving of logs, Central Audit Log System must also provide real-time monitoring and routine reports.
Solution
Bitlogg is used by the business applications to send event logs. These include Dotnet based and non-Dotnet based business applications. Server side components of Bitlogg include:
  • Web Service: Web service accept batched logs (aggregated by the Adaptor forDotnet applications) and event logs generated by non-Dotnet applications. Web service feeds the data to Log Digest Service.
  • Log Digest Service: This service processes log events, validates it, and updates the database backend. Log Digest Service also provides statistical information to web based control panel for status.
  • Web based Control Panel: This provides a web based interface to the Super Administrator (to manage the Bitlogg server, updates masters, update access rights) and Application Owners and Compliance Officers to view relevant logs. This component gets the data from the back-end database. Web based control panel performs all authentications against identity server (AD).
  • Database back-end: This is a mix of RDBMS (to maintain masters, access rights and other related information) and high performance search platform Elasticsearch.
Business Application side components includes
  • Log Adaptor: Log Adaptor accepts log events from DotNET application, and hand it over to the Central Audit Log Server either immediately, or by buffering those logs, and sending it in a batch.
  • Command Line Interface: to switch between Production and Diagnosis mode: This is a command line utility to switch between production and diagnosis mode. This utility is used only by an authorized administrator.

 

Technologies

The ELK Stack is a collection of three open-source products — Elasticsearch, Logstash, and Kibana — all developed, managed and maintained by Elastic. Elasticsearch is a NoSQL database that is based on the Lucene search engine. Logstash is a log pipeline tool that accepts inputs from various sources, executes different transformations, and exports the data to various targets. Kibana is a visualization layer that works on top of Elasticsearch.

We were an early adopters of the stack when Elasticsearch, Logstash and Kibana were different open source projects and not part of Elastic, which was not formed by then.

We wrote various custom filters for Logstash for processing events. We had to customize logstash-forwarder, a Logstash client written in Go. It would run on Windows servers to read the logs written on the local disk and then buffer and forward to Logstash for further processing and storage.

 

Salient features
  • Central Application Master maintains a list of all applications running across the servers and Log server specific attributes like archive and purge policy.
  • Attribute list master maintains global and application specific attributes, its types and validation criteria
  • Dotnet based business applications use the Log Adaptor forDotnet based applications installed on same physical server as the applications to log events with lowest possible latency time.
  • Non-Dotnet based Business Applications and a Log Adaptor use a web service to log events to the central log server.
  • Log digest utility parses event logs, validates it against set of acceptable attributes and stores structured data in the database.
  • A web-based control panel allows an administrator to define filters and monitor events in real-time for events generated from selected application/servers/action/severity/customer account etc.
  • A web-based control panel to monitor application heartbeat and raise an alarm in case logs are not received in time, or logs failing to parse, or application servers/log server is running out of disk space to store logs, etc.
  • A command line utility which can only be used by authorized user to switch between diagnosis mode and production mode to ensure that logs generated during diagnosis purpose are not reported in compliance reports.
  • A web-based control panel to view and download logs based on filters. Logs could be filtered based on duration/application/servers/action/severity/customer account and any other attribute maintained by the system.
  • Old logs archival.
  • Purge old logs based on age of respective events.
  • All authentications are performed against AD configured within the client’s infrastructure.
  • No direct access to manipulate log database even to an authorized business user.
Benefits
A central log server brings various benefits in terms of improved compliance to preserve audit trails, reduces manual overheads in handling archive trails, mitigates risk of possible tampering, improved monitoring, etc.
Centralized repository of logs: It’s easy to (partially) loose logs generated by heterogeneous applications, developed and maintained by different teams having varied skills and sensitivity to audit trails, inadequate resources on servers, etc. Such issues can cause problem during system audits. With a central log server, once logs are within central log server, they are safe from local problems on the application servers like low disk space, or disk corruptions, or malwares, accidental or planned formatting of the system without backup, etc.
Automate Log rotation: A Dotnet adaptor sends event logs to the Web Service as they are generated. In case of exceptions when event logs cannot be sent out to the system, they are stored on a local server and later sent out to the Central Audit Log System. This frees up Infrastructure team from routine jobs of moving logs from each application server to the central server. Web based UI Key statistical information and log reports are be available from the web-based control panel of the Central Audit Log System. This reduces business application owners dependence on infrastructure team to view logs collected for their own applications.
AD based authentication: Bitlogg only uses AD (already deployed within the client’s infrastructure) for user authentication. This reduces the overhead of accounts and password policy issues.
Open source OS and database back-end: A Central Log Server uses open-source software components for OS and database back-end, and any other reusable libraries. Thus there is no license cost for software OS and database.
Structured logs within a searchable database: A Central Audit Log Server Application maintains structured data in back-end. This helps in querying database from any other server to extend functionality in future.
Source code and technical details: Merce has provided final schema, relationship among various entities, and a source code and details about software stack to integrate with other applications and extend application.