- Web Service: Web service accept batched logs (aggregated by the Adaptor forDotnet applications) and event logs generated by non-Dotnet applications. Web service feeds the data to Log Digest Service.
- Log Digest Service: This service processes log events, validates it, and updates the database backend. Log Digest Service also provides statistical information to web based control panel for status.
- Web based Control Panel: This provides a web based interface to the Super Administrator (to manage the Bitlogg server, updates masters, update access rights) and Application Owners and Compliance Officers to view relevant logs. This component gets the data from the back-end database. Web based control panel performs all authentications against identity server (AD).
- Database back-end: This is a mix of RDBMS (to maintain masters, access rights and other related information) and high performance search platform Elasticsearch.
- Log Adaptor: Log Adaptor accepts log events from DotNET application, and hand it over to the Central Audit Log Server either immediately, or by buffering those logs, and sending it in a batch.
- Command Line Interface: to switch between Production and Diagnosis mode: This is a command line utility to switch between production and diagnosis mode. This utility is used only by an authorized administrator.
The ELK Stack is a collection of three open-source products — Elasticsearch, Logstash, and Kibana — all developed, managed and maintained by Elastic. Elasticsearch is a NoSQL database that is based on the Lucene search engine. Logstash is a log pipeline tool that accepts inputs from various sources, executes different transformations, and exports the data to various targets. Kibana is a visualization layer that works on top of Elasticsearch.
We were an early adopters of the stack when Elasticsearch, Logstash and Kibana were different open source projects and not part of Elastic, which was not formed by then.
We wrote various custom filters for Logstash for processing events. We had to customize logstash-forwarder, a Logstash client written in Go. It would run on Windows servers to read the logs written on the local disk and then buffer and forward to Logstash for further processing and storage.
- Central Application Master maintains a list of all applications running across the servers and Log server specific attributes like archive and purge policy.
- Attribute list master maintains global and application specific attributes, its types and validation criteria
- Dotnet based business applications use the Log Adaptor forDotnet based applications installed on same physical server as the applications to log events with lowest possible latency time.
- Non-Dotnet based Business Applications and a Log Adaptor use a web service to log events to the central log server.
- Log digest utility parses event logs, validates it against set of acceptable attributes and stores structured data in the database.
- A web-based control panel allows an administrator to define filters and monitor events in real-time for events generated from selected application/servers/action/severity/customer account etc.
- A web-based control panel to monitor application heartbeat and raise an alarm in case logs are not received in time, or logs failing to parse, or application servers/log server is running out of disk space to store logs, etc.
- A command line utility which can only be used by authorized user to switch between diagnosis mode and production mode to ensure that logs generated during diagnosis purpose are not reported in compliance reports.
- A web-based control panel to view and download logs based on filters. Logs could be filtered based on duration/application/servers/action/severity/customer account and any other attribute maintained by the system.
- Old logs archival.
- Purge old logs based on age of respective events.
- All authentications are performed against AD configured within the client’s infrastructure.
- No direct access to manipulate log database even to an authorized business user.