Month: January 2023

Some log management tips

Post author By Kanti Kalyan Arumilli
Post date 30 January 2023

Recently, I have been writing on log management tools and techniques. Very recently, I am even evaluating Grafana Loki on-premise. I would write a review in few days regarding Grafana Loki. As of now from server hardware requirements, log volume ingestion standpoint Grafana seems excellent compared with ELK stack and GrayLog.

This blog post is a general blog post. For proper log management, we need different components.

Log ingestion client
Log ingestion server
Log Viewer
Some kind of long-term archiver that can restore certain logs on required basis (Optional)

Log Ingestion Client:

FluentD is the best log ingestion client for several reasons. Every log ingestion stack have their own log ingestion clients. ELK Stack has LogBeats, MetricBeats etc… GrayLog does not have a client of its own but supports log ingestion via Gelf / RSysLog etc… Grafana Loki has PromTail.

FluentD can collect logs from various sources and ingest into various destinations. Here is the best part – multiple destinations based on rules. For example certain logs can be ingested into Log servers and uploaded to S3. Very easy to configure and customize and there are plenty of plugins for sources, destinations and even customizing logs such as adding tags, extracting values etc… Here is a list of plugins.

FluentD can ingest into Grafana Loki, ELK stack, GrayLog and much more. If you use FluentD, if the target needs to be changed, its just a matter of configuration.

Log Ingestion Server:

ELK vs GrayLog vs Grafana Loki vs Seq and several others. As of now, I have evaluated ELK, GrayLog and Grafana Loki.

Log Viewer:

Grafana front end with Loki backend, GrayLog, Kibana frontend with ElasticSearch backend in ELK stack.

Long-Term Archiving:

ELK stack has lifecycle rules for backing up and restoring. GrayLog can be configured to close indexes and re-open on a necessary basis. Grafana Loki has retention and compactor settings. However, I have not figured out how to re-open compacted gz files on a necessity basis.

Apart from these, I am using Graphite for metrics. I do have plans for ingesting additional metrics. As of now, I am using the excellent hosted solution provided by Grafana. As of now, in the near-term I don’t have plans for self-hosting metrics. But Grafana front-end supports several data sources.

I am thinking of collecting certain extra metrics without overloading the application (might be an after-thought or might not be). I am collecting NGinx logs in json format. The URL, upstream connect, upstream response time are being logged. Now, by parsing these logs, the name of the ASP.Net MVC controller, name of the Action Method, the HTTP verb can be captured. Now, I can use these as metrics. I can very easily add metrics at the database layer in the application. With these metrics, I can easily identify bottlenecks, slow performing methods and even monitor average response times etc… and set alerts.

The next few days or weeks would be about the custom metric collection based on logs. You can expect few blog posts on some FluentD configuration, C# code etc… FluentD does have some plugins for collecting certain metrics but we will look into some C# code for parsing, sending metrics into Graphite.

Here is a screenshot from the self-hosted Grafana front-end for Loki logs:

Grafana showing Loki logs for PodDB

Here is a screenshot from Grafana.com hosted showing Graphite metrics

Graphite Solr Backend Server CPU usage

I am hoping this blog posts helps someone. Some C# code for working with Logs, Metrics and Graphite over the next few days / weeks.

–

Mr. Kanti Kalyan Arumilli

Arumilli Kanti Kalyan, Founder & CEO

B.Tech, M.B.A

Founder & CEO, Lead Full-Stack .Net developer

ALight Technology And Services Limited

ALight Technologies USA Inc

Youtube

Facebook

Phone / SMS / WhatsApp on the following 3 numbers:

+91-789-362-6688, +1-480-347-6849, +44-07718-273-964

+44-33-3303-1284 (Preferred number if calling from U.K, No WhatsApp)

kantikalyan@gmail.com, kantikalyan@outlook.com, admin@alightservices.com, kantikalyan.arumilli@alightservices.com, KArumilli2020@student.hult.edu, KantiKArumilli@outlook.com and 3 more rarely used email addresses – hardly once or twice a year.

Tags Logging

.Net C# Logging NLog

NLog FallbackGroup Target

Post author By Kanti Kalyan Arumilli
Post date 26 January 2023

In the past I have written few blog posts on NLog and discussed several techniques:

Programatically configuring NLog in C#

NLog in .Net Applications

Some NLog configuration examples for writing into different log management systems

How to log debug level logs only when an exception occurs in ASP.Net using NLog

And I have discussed about a possibility of capturing more information in logs only when needed such as in the case of errors or exceptions in the following blog post:

An abnormal way of logging – open for discussion

My use-case explanation:

I am planning to use Gelf logging for easier compatibility reasons. Gelf logs can be ingested into pretty much every major centralized logging platforms such as: Kibana, GrayLog, Seq, Grafana. Some would require some intermediary software to accept Gelf formatted logs and some can directly ingest Gelf formatted logs. However, for various reasons, sometimes the logging server might not be available, specifically when the log ingestors are not in a cluster. Log files can be easily ingested into the above mentioned centralized logging agents using different sofware.

Based on the above use-case I wanted to use Gelf for directly logging into the centralized logging server and as a failover, I want to write the logs to a file that would get ingested at a later point by some other software.

Now, by combing the previous post example, we can achieve AspNetBuffering and ingest different levels of logs only when errors occur. The code samples should be very easy to understand.

Read the How to log debug level logs only when an exception occurs in ASP.Net using NLog prior to continuing.

<extensions>
    <add assembly="NLog.Web.AspNetCore"/>
    <add assembly="NLog.Targets.Gelf"/>
</extensions>

<targets>
    <target xsi:type="AspNetBufferingWrapper" name="aspnetbuffer" bufferGrowLimit="100000" growBufferAsNeeded="true">
        <target xsi:type="PostFilteringWrapper" defaultFilter="level &gt;= LogLevel.Info">
            <target xsi:type="FallbackGroup" returnToFirstOnSuccess="true">
                <target xsi:type="gelf" endpoint="tcp://logs.local:12201" facility="console-runner" sendLastFormatParameter="true">
                    <parameter name="param1" layout="${longdate}"/>
                    <parameter name="param2" layout="${callsite}"/>
                 </target>
		<target xsi:type="File" fileName="c:\temp\nlog-AspNetCore-all-${shortdate}.log" layout="${longdate}|${event-properties:item=EventId:whenEmpty=0}|${level:uppercase=true}|${logger}|${message} ${exception:format=tostring}" />
              </target>
              <when exists="level &gt;= LogLevel.Warn" filter="level &gt;= LogLevel.Debug"/>
       </target>
    </target>
</targets>

<rules>
    <logger name="*" minlevel="Debug" writeTo="aspnetbuffer" />
</rules>

In the above code we have wrapped Gelf logger, File logger inside a FallBackGroup logger. The FallBackGroup logger is wrapped inside a PostFilteringWrapper. The PostFilteringWrapper is wrapped inside a AspNetBufferingWrapper.

In the above code in the <rules> section we are sending all Debug and above logs to the AspNetBufferingWrapper.

Now AspNetBufferingWrapper buffers the log messages for an entire request, response cycle and sends the log messages to the PostFilteringWrapper.

The PostFilteringWrapper sees if there are any Warnings or above loglevel, if yes sends all the messages that have Debug and above loglevels. Else sends Info and above messages. The target of PostFilteringWrapper is the FallbackGroup logger which receives these messages.

The FallBackGroup logger attempts to use the Gelf logger, if the Gelf logger is unable to process the messages, the logs are sent to the File logger.

–

Mr. Kanti Kalyan Arumilli

Arumilli Kanti Kalyan, Founder & CEO

B.Tech, M.B.A

Founder & CEO, Lead Full-Stack .Net developer

ALight Technology And Services Limited

ALight Technologies USA Inc

Youtube

Facebook

Phone / SMS / WhatsApp on the following 3 numbers:

+91-789-362-6688, +1-480-347-6849, +44-07718-273-964

+44-33-3303-1284 (Preferred number if calling from U.K, No WhatsApp)

Tags .Net, C#, Logging, NLog

.Net C# Logging NLog

How to log debug level logs only when an exception occurs in ASP.Net using NLog

Post author By Kanti Kalyan Arumilli
Post date 26 January 2023

In the past I have written few blog posts on NLog and discussed several techniques:

Programatically configuring NLog in C#

NLog in .Net Applications

Some NLog configuration examples for writing into different log management systems

And I have discussed about a possibility of capturing more information in logs only when needed such as in the case of errors or exceptions in the following blog post:

An abnormal way of logging – open for discussion

This blog post is for Asp.Net Core only.

This should be possible by combining AspNetBufferingWrapper and PostFilteringWrapper.

Sample configuration provided below:

AspNetBufferingWrapper:

AspNetBufferingWrapper buffers all the messages in a ASP.Net request and sends all the messages to the wrapped target.

Remember to set this logger properly. This involves:

Adding the NLog.Web.AspNetCore nuget package
Properly configuring nlog.config file
Registering the middleware

dotnet add package NLog.Web.AspNetCore --version 5.2.1

<extensions>
    <add assembly="NLog.Web.AspNetCore"/>
</extensions>

using NLog.Web;

app.UseMiddleware<NLogBufferingTargetWrapperMiddleware>();

PostFilteringWrapper:

This wrapper evaluates a specified condition and filters logs, then sends the logs to the wrapped target:

<target xsi:type="PostFilteringWrapper" defaultFilter="level &gt;= LogLevel.Info">
    <target .... />
    <when exists="level &gt;= LogLevel.Warn" filter="level &gt;= LogLevel.Debug"/>
</target>

The above configuration by default logs Info and above logs, but if there is a Warn or higher, logs debug or higher. For this to work properly obviously this logger has to receive Debug messages otherwise there is no point in using this logger.

Now combing these two loggers, here is an example:

<extensions>
    <add assembly="NLog.Web.AspNetCore"/>
</extensions>

<targets>
    <target xsi:type="AspNetBufferingWrapper" name="aspnetbuffer" bufferGrowLimit="1000" growBufferAsNeeded="true">
        <target xsi:type="PostFilteringWrapper" defaultFilter="level &gt;= LogLevel.Info">
            <target xsi:type="File" fileName="c:\temp\nlog-AspNetCore-all-${shortdate}.log"
 layout="${longdate}|${event-properties:item=EventId:whenEmpty=0}|${level:uppercase=true}|${logger}|${message} ${exception:format=tostring}" />
        <when exists="level &gt;= LogLevel.Warn" filter="level &gt;= LogLevel.Debug"/>
        </target>
    </target>
</targets>

<rules>
    <logger name="*" minlevel="Debug" writeTo="aspnetbuffer" />
</rules>

Hoping this post helps someone!

Happy development.

–

Mr. Kanti Kalyan Arumilli

Arumilli Kanti Kalyan, Founder & CEO

B.Tech, M.B.A

Founder & CEO, Lead Full-Stack .Net developer

ALight Technology And Services Limited

ALight Technologies USA Inc

Youtube

Facebook

Phone / SMS / WhatsApp on the following 3 numbers:

+91-789-362-6688, +1-480-347-6849, +44-07718-273-964

+44-33-3303-1284 (Preferred number if calling from U.K, No WhatsApp)

Tags .Net, C#, Logging, NLog

Logging

Some centralized logging tools

Post author By Kanti Kalyan Arumilli
Post date 26 January 2023

I have evaluated few different centralized logging tools, specifically the following:

Grafana Loki
Kibana
Graylog
Seq

In the short-term, I am using Graylog, but in the next few years, I might choose a different option.

The key features I have been looking for are:

Lower hardware requirements – for a small startup without any revenue yet, I didn’t want to spend too much.
Customizable retention period of logs
Being able to backup logs to some cheaper storage such as S3 rather than having 100’s of GB on EBS volumes.
Easily able to restore a smaller portion of logs for a certain period and be able to search.
Being able to ingest various types of logs

Let me explain my personal requirements clearly.

I want to ingest all the logs from all possible sources i.e system logs, software logs such as web server, mysql audit logs, custom application logs. Currently my applications and servers are generating approximately 800Mb of logs per day. That would be about 25Gb per month and 300Gb per year. I want to retain logs for a longer period in archives for various reasons. I currently don’t have any products that need to meet compliance requirements. I arbitrarily choose 400 days worth of log retention and the logs need to be immutable. Once the logs are ingested, the logs need to be stored for 400 days and should not be modified. The reason being in the future if I need to meet compliance requirements, it would be easy to change the retention period and the integrity of the logs can be verified.

I have personally evaluated the following:

Grafana Loki hosted at https://www.grafana.com.
Self-Hosted ELK stack
Self-Hosted Open Source version of Graylog
AWS Cloudwatch

I have read about but have not evaluated the following yet:

Self-Hosted Seq
Self-Hosted Grafana Loki

Given the above I will tell you the advantages and disadvantages of each solution.

Grafana Loki hosted:

Grafana has a very generous free tier with 50Gb log ingestion and 14 days retention. The paid customized plans pricing was not clear. Considering the logs are hosted by a 3rd party, I would hope they would introduce some additional security measures such as allowing log ingestion from only certain IP’s etc… Even if the API keys are stolen or spied upon, the hackers cannot pollute the log data.

Self-Hosted ELK stack:

This is one very great solution but the setup and versions compatibility is very problematic. Self-Hosted ELK stack is a little heavy on resources. But definitely worth for SME’s who have the budget for the required hardware and few Server Admin professionals on team. As of now, because of the R&AW harassment, impersonation, I don’t know when I would launch commercial products. And these are recurring expenses, not one time expenses, so I am trying to set myself for success with smaller monthly server expenses. I wish these psycho human rights violators get arrested. There are ways to export backups into S3, almost a perfect solution

GrayLog OpenSource:

GrayLog is a bit heavy on system resources but requires lesser resources compared with ELK stack. Indexes can be closed but backing up and restoring are not directly part of the application. Probably part of the GrayLog paid version.

AWS CloudWatch:

AWS Cloudwatch is perfect if there is a need for compliance with retention policies and immutability. CloudWatch logs can be exported into S3 buckets. S3 buckets can be configured to be immutable for compliance reasons and S3 lifecycle policies can be defined for removal of data etc… But querying data is a little problematic compared with the ease of other solutions.

Seq:

Seq has a free version, seemed to be light-weight. Very easy to write extensions using C# (My primary development language). There is no direct plugin for for exporting data into S3 but a customizable plugin might be possible. There are plugins for writing into an Archive file. The Archive file can be exported to S3 periodically. Trying on localhost is very easy – pull a docker image and run the docker image. No complicated setup.

Self-hosted Grafana Loki:

I think pretty much all the capabilities of hosted Grafana Loki might be possible. However, I haven’t tried yet.

In all the above solutions, logs could be tampered by hackers except with AWS Cloudwatch. Once ingested, the logs stay there un-tampered. If Admin’s account gets hacked, the retention period can be changed or log streams might be deleted, but cannot be tampered.

As of now, I have not yet found the perfect solution for my requirements, but I am hoping this blog post helps some people in deciding between various different centralized logging solutions based upon your own requirements.

–

Mr. Kanti Kalyan Arumilli

Arumilli Kanti Kalyan, Founder & CEO

B.Tech, M.B.A

Founder & CEO, Lead Full-Stack .Net developer

ALight Technology And Services Limited

ALight Technologies USA Inc

Youtube

Facebook

Phone / SMS / WhatsApp on the following 3 numbers:

+91-789-362-6688, +1-480-347-6849, +44-07718-273-964

+44-33-3303-1284 (Preferred number if calling from U.K, No WhatsApp)

Tags Logging

AWS

Private Hosted Zone in Route 53 on AWS

Post author By Kanti Kalyan Arumilli
Post date 24 January 2023

Cross Post – https://kantikalyan.medium.com/private-hosted-zone-in-route-53-on-aws-3b31975b6fb1

Although ALight Technology And Services Limited is a very small startup as of now but internally there are few different servers hosted on AWS. I had the need for a private DNS server. This blog post is about setting up a Private Hosted Zone in Route 53 on AWS.

*Hosted Zones cost $0.5 per month and additional for the DNS queries but worth instead of going through the trouble of setting up own DNS servers on few different EC2 instances (alternate and slightly cheaper way, but not very reliable and lots of sysadmin work).

Here are the steps:

Go to Route 53 and create a new Hosted Zone with the Type – Private Hosted Zone option.
Associate the necessary VPC’s, be careful not to have overlapping addresses. The VPV’s need to have DNS Hostnames and DNS Resolution enabled.
Now log in into your EC2 instance and do a nslookup.

> nslookup my.local //Assuming you have setup a record as my.local in Route 53.

If you get the IP resolved great! If not, use this webpage for troubleshooting – https://aws.amazon.com/premiumsupport/knowledge-center/route-53-fix-dns-resolution-private-zone/

For me the 4th part solved my error – Review custom settings in resolv.conf.

Hoping the above blog post helps someone.

Although, I have AWS Certified Architect Certificate and do have knowledge of AWS Route 53 Private Hosted Zone, this effort has allowed me to gain hands-on experience!

Happy development :)!

–

Mr. Kanti Kalyan Arumilli

Arumilli Kanti Kalyan, Founder & CEO

B.Tech, M.B.A

Founder & CEO, Lead Full-Stack .Net developer

ALight Technology And Services Limited

ALight Technologies USA Inc

Youtube

Facebook

Phone / SMS / WhatsApp on the following 3 numbers:

+91-789-362-6688, +1-480-347-6849, +44-07718-273-964

+44-33-3303-1284 (Preferred number if calling from U.K, No WhatsApp)

Tags AWS

.Net AWS C#

How to export logs from cloudwatch to S3.

Post author By Kanti Kalyan Arumilli
Post date 14 January 2023

There might be a business case for log retention whether it’s for compliance or any other reason.

This is the approach being used by ALight Technology And Services Limited for longer term log retention.

I have created a bucket in S3 with the following retention policies:

AWS S3 Object Lock Policy

I personally don’t have to follow compliance yet, but nothing wrong in implementing compliance policies.

I have also defined a life-cycle policy to transition objects into Standard-IA (Infrequent Access) after 30 days.

Now I am developing a Lambda that would create Export tasks in CloudWatch once a week.

Here are some relevant C# code snippets:

var _client = new AmazonCloudWatchLogsClient(RegionEndpoint.EUWest2);
// Initialized AmazonCloudWatchLogsClient

var response = await _client.DescribeLogGroupsAsync();
// Get a list of LogGroups

foreach(var logGroup in response.LogGroups)
{
    var prefix = $"{from}-{to}-{logGroup.LogGroupName}";
    // You can define your own prefix
   
    var exportResult = await _client.CreateExportTaskAsync(new
        CreateExportTaskRequest
{
            Destination = "<NAME_OF_S3_BUCKET>",
            DestinationPrefix = prefix,
            From = GetUnixMilliSeconds(from),
            LogGroupName = logGroup.LogGroupName,
            TaskName = prefix,
            To = GetUnixMilliSeconds(to),
        })
    };

The above code is pretty much self-explantory. Here is a code snippet for getting Unix MilliSeconds from epoch.

long GetUnixMilliSeconds(DateTime dateTime)
{
    var _epoch = new DateTime(1970, 1, 1, 0, 0, 0, 0);
    return (dateTime.Ticks - _epoch.Ticks) / 10000;
}

Happy development!

Stay away from psycoSpies!

–

Mr. Kanti Kalyan Arumilli

Arumilli Kanti Kalyan, Founder & CEO

B.Tech, M.B.A

Founder & CEO, Lead Full-Stack .Net developer

ALight Technology And Services Limited

ALight Technologies USA Inc

Youtube

Facebook

Phone / SMS / WhatsApp on the following 3 numbers:

+91-789-362-6688, +1-480-347-6849, +44-07718-273-964

+44-33-3303-1284 (Preferred number if calling from U.K, No WhatsApp)

Tags .Net, AWS, C#

.Net C# Security

C# code for reading sensitive information from Console

Post author By Kanti Kalyan Arumilli
Post date 13 January 2023

I had a need to generate random passwords and / keys and update various config files. For example, keys and passwords used by log ingesting utilities such as FileBeat, PromTail, MetricBeat etc…

In earlier blog posts, I have mentioned, that at this point log ingestion, retention and major alerts implementation is complete. So, obviously the next part is securing the keys.

I know the hacker spies – India’s psychopath R&AW spies can and are seeing any plain-text items on screen and if I am not wrong, they might have even hacked into my accounts several times. Yes, they say they are investigation teams etc… bull-shit but in reality they are corrupted and are the criminals i.e greedy investigators / spies who did crime and are trying to get away from crime.

Anyway, because I know how the “prying eyes” equipment works, I need to defend myself from the hacker spies as much as possible. For more info about this scam: https://www.simplepro.site.

Here is a small C# code snippet for reading from console without echoing back:

string GetSensitiveText()
{
    StringBuilder password = new StringBuilder();
    ConsoleKeyInfo keyInfo = Console.ReadKey(true);

    while (keyInfo.Key != ConsoleKey.Enter)
    {
        password.Append(keyInfo.KeyChar);

        keyInfo = Console.ReadKey(true);
    }

    return password.ToString();
}

Now everyone knows how to do open a file, read content and replace content. A simple program can be developed that would take the path of config file, old value, new value and replace.

i.e for example during test, alpha modes if a key is “KEY” and then later if you use a random password generator that would generate password and copy into memory, this type of small tool can help with replacing “KEY” with the “RAND0M P@$$W0rd”.

Some code sample:

Console.WriteLine("Enter filepath:");
var fileName = Console.ReadLine();
var sr = new StreamReader(fileName);
var content = sr.ReadToEnd();
sr.Close();
Console.WriteLine("Enter Search Phrase:");
var searchPhrase = Console.ReadLine();
var matchedIndex = content.IndexOf(searchPhrase);
if(matchedIndex >= 0)
{
    Console.WriteLine("Match found.");
    Console.WriteLine("Enter replacement text:");
    var replacementText = GetSensitiveText();

    var sw = new StreamWriter(fileName);
    sw.Write(content.Replace(searchPhrase, replacementText));
    sw.Flush();
    sw.Close();
}

We prompt for the path to the config file, prompt for the search text. If the search text is found, we prompt for the secret i.e the replace text. But, we don’t echo the new sensitive info to the Console. Then the search text is replaced with new sensitive info and then we write the contents back to the file.

Happy secure coding! 🙂

–

Mr. Kanti Kalyan Arumilli

Arumilli Kanti Kalyan, Founder & CEO

B.Tech, M.B.A

Founder & CEO, Lead Full-Stack .Net developer

ALight Technology And Services Limited

ALight Technologies USA Inc

Youtube

Facebook

Phone / SMS / WhatsApp on the following 3 numbers:

+91-789-362-6688, +1-480-347-6849, +44-07718-273-964

+44-33-3303-1284 (Preferred number if calling from U.K, No WhatsApp)

Tags .Net, C#, Security

Logging

Ingesting logs into GrayLog

Post author By Kanti Kalyan Arumilli
Post date 12 January 2023

Graylog is a logs only software, very easy to configure and use. There are several nice features, few of the features I liked:

Easy to setup alerts
Easy to setup processing rules and pipelines
Lighter on system resources
Flexible ways of ingesting logs

Like ELK stack, GrayLog can be easily installed, secured for ingesting logs. ELK stack has manageable ElasticAgents i.e client software running on different systems and the client software can be managed from the web interface. ELK stack has support for metrics, GrayLog does not. GrayLog is for logs only and does well.

Installing and configuring GrayLog consists of installing 3 software:

MongoDB
ElasticSearch
GrayLog

The instructions are very easy to follow and are located in GrayLog’s documentation can be accessed by clicking the appropriate link at: https://www.graylog.org/downloads/.

Some of my favorite Inputs are:

RSysLog
Beats
Beats

–

Mr. Kanti Kalyan Arumilli

Arumilli Kanti Kalyan, Founder & CEO

B.Tech, M.B.A

Founder & CEO, Lead Full-Stack .Net developer

ALight Technology And Services Limited

ALight Technologies USA Inc

Youtube

Facebook

Phone / SMS / WhatsApp on the following 3 numbers:

+91-789-362-6688, +1-480-347-6849, +44-07718-273-964

+44-33-3303-1284 (Preferred number if calling from U.K, No WhatsApp)

Logging Security

Some log management tips and a generic review of ELK Stack, GrayLog and Grafana

Post author By Kanti Kalyan Arumilli
Post date 11 January 2023

Centralized log management is very important for any tech company of any size. For larger companies, entire company logs need not be centralized but can be segmented based on department or product etc…

Background in the context of ALight Technology And Services Limited

ALight Technology And Services Limited is both product and service based company. Currently offers two completely free products – SimplePass and PodDB. With SimplePass, I am not worried much because except for the code there is no data on the server and obviously no customer specific data. With PodDB the risk is slightly higher because there is data but no customer specific data. As of now the AWS account, servers are very highly secured with immediate alerts on login into AWS console or servers, new EC2 instances, instance terminations etc… With the infrastructure, access to infrastructure being secured, the next step is external threats and being able to respond to external threats. These are very important steps prior to developing any products that would possibly contain customer data. What if someone tries to hack by sending malicious payload or DOS (Denial of Service) or DDOS (Distributed Denial of Service)? For identifying, mitigating, preventing such things it’s very important to have proper log management techniques, monitoring of metrics, proper alerts and proper action plan / business continuity plan when such incidents occur. Even if such a thing happened, it’s very important to have logs so that computer forensics can be performed. No company is going to offer free products for ever without generating revenue, in a similar way ALight Technology And Services Limited does have plans of developing revenue generating products or offer services such as architecting, development, hosting etc… Compared with modern days powerful hacking equipment of the anonymous group that calls them the “eyes” (don’t get confused with the intelligence “five eyes”, as a matter of fact the anonymous “eyes” are targeting the five countries that formed the “five eyes” and any whistleblowers like me in this context – I am the whistleblower (but not R&AW) of India’s R&AW equipment capabilities and the atrocities that have been done by the R&AW spies against me), the current state of information security standards are much below.

I have looked into 3 solutions and each of these solutions had strengths and benefits.

What I was looking for:

For example – PodDB has web server logs (NGinx), ASP.Net Core web application logs, and a bunch more of logs from microservice that interacts with the database, microservice that writes some trending data, microservices that queries solr etc… So my log sources are multiple and I want to aggregate all of these along with other logs such as syslog, mariadb audit log etc…

AWS Cloudwatch:

CloudWatch allows easy ingestion, very high availability, metrics, alarms etc… 5GB per month of log ingestion for free. However, live tailing of the logs i.e being able to see logs as they soon as they are ingested is a bit problematic. Even querying / viewing across log groups is a bit problematic. The strength is the definable retention period for each log group. Once ingested the logs cannot be modified, so definitely a great solution if storing logs for compliance reasons. AWS should consider introducing data storage tiers like S3 data storage i.e lifecycle transition – hot logs can be queried and definable period, then lifecycle transition and logs would be stored for archival purpose for some period and then deleted.

2. ELK Stack:

ELK stack consists of ElasticSearch, LogStash and Kibana. ElasticSearch for full-text search capabilities, LogStash for log ingestion, KIbana for visualization. This review is about the self-hosted version. The ELK stack has plenty of features and very easy management if the application and all of it’s components can be properly configured. Built-in support for logs, live tailing of logs, metrics etc… Easier management using ElasticAgents i.e ElasticAgents can be installed on multiple machines and what data should be ingested by each agent can be controlled by the web interface. However, ELK stack seemed a bit heavy in computing resource consumption and for whatever reason, LogStash crashed several times and the system crashed i.e the EC2 instance just hanged, couldn’t even restart. ELK Stack supports, hot and cold log storages i.e the past 15 – 30 days of logs can be kept in the hot storage and the older logs can be automatically moved into cold tier i.e not queried frequently but are kept for various reasons.

3. Graylog:

This is about self hosted version of Graylog. Graylog focuses only on log management. Very easy to setup and ingest logs. Easy querying of logs. No support for metrics. Graylog allows creating snapshots of older data which can be stored elsewhere, restored and searched on a necessity basis.

4. Grafana

This is about the free Grafana account. Grafana offers very generic 50GB log ingestion per month. Logs can be easily ingested into Loki and viewed from Grafana. Metrics can be ingested into Graphite and viewed. Very easy to setup alerts. I have not tried yet but the free tier has 50GB of traces ingestion per month. One of the very best features I liked about Grafana is easy way of tagging logs. If log sources are properly tagged, combining and viewing multiple log sources is very very easy.

Thank you Grafana for such a generous free tier and such a great product.

There seems to be no control of retention period. Grafana paid subscription has control of retention period. The paid version starts at $8 per month. I do have plans about signing up for paid account just before launching commercial products specifically for planning retention i.e either Grafana can store the older logs for few extra months on my behalf or if they can provide a solution to upload into S3 glacier and of course when needed being able to restore from S3 Glacier and being able to search, because storing old logs in S3 Glacier and if there is no way of restoring and searching then the entire purpose of storing old logs would not make sense.

–

Mr. Kanti Kalyan Arumilli

Arumilli Kanti Kalyan, Founder & CEO

B.Tech, M.B.A

Founder & CEO, Lead Full-Stack .Net developer

ALight Technology And Services Limited

ALight Technologies USA Inc

Youtube

Facebook

Phone / SMS / WhatsApp on the following 3 numbers:

+91-789-362-6688, +1-480-347-6849, +44-07718-273-964

+44-33-3303-1284 (Preferred number if calling from U.K, No WhatsApp)

Tags Logging, Security

Linux Security

Some important log management techniques on Linux – AuditD

Post author By Kanti Kalyan Arumilli
Post date 8 January 2023

In my continued pursuit of strengthening the security infrastructure at my own startup – ALight Technology And Services Limited, I have written few blog articles in the past regarding securing web applications, importance of audit, logs – part of the NIST Cyber Security Framework. This blog post talks about some things I have done on AWS infrastructure. While running a company with no other employees and while being the target of state-sponsored / state-trained hackers, I ended up learning a lot and now I have dabbled in pretty much everything in computing (expert at development, learning system administration, infosec etc… as part of running my own startup).

I created a base Ubuntu image by enabling ufw, installed auditd, installed cloudwatch log agent, closing unnecessary ports, some custom alerters as soon as a SSH login happens etc… I call this AMI the golden AMI. I also update the golden AMI every few months. The advantage of using a golden AMI like this is any EC2 instance you would launch would have these in place.
I am using ELK stack along with Cloudwatch logs and S3 for logs. ELK stack for log analysis i.e logs are stored for a shorter period, Cloudwatch logs for various other reasons, (can’t disclose) and finally S3 glacier for longer term retention.
With the above mentioned setup, if an incident happens, all the necessary logs would in place for analysis.

I wanted to give a quick introduction to Cloudwatch log agent, AuditD as part of this blog post.

Cloudwatch log agent:

A small piece of software that ingests logs into AWS Cloudwatch. https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/QuickStartEC2Instance.html

The setup needs IAM role with proper permissions, more details are at the above mentioned link.

On Ubuntu the logs config is stored at:

/var/awslogs/etc/awslogs.conf

The configuration file is very simple and straightforward.

I would suggest ingesting all the ubuntu system logs along with auditd logs and create a golden AMI.

AuditD:

This is a nice audit tool for Linux capable of auditing a lot of things.

Installation:

sudo apt update
sudo apt-get install auditd
sudo systemctl enable auditd
sudo systemctl start auditd

The configuration and rules are stored at /etc/audit. The config file is auditd.conf, rules should be in audit.rules.

The configuration file is self-explanatory.

There are no default rules.

But thankfully there is a github repo with several rule templates for meeting several compliance standards such as PCI. The PCI rules are at: https://github.com/linux-audit/audit-userspace/blob/master/rules/30-pci-dss-v31.rules

Several rule files are located in the same repository:

https://github.com/linux-audit/audit-userspace/tree/master/rules

Stay safe & secure! Stay away from hac#?ers / ransom thieves.

Happy computing!

–

Mr. Kanti Kalyan Arumilli

Arumilli Kanti Kalyan, Founder & CEO

B.Tech, M.B.A

Founder & CEO, Lead Full-Stack .Net developer

ALight Technology And Services Limited

ALight Technologies USA Inc

Youtube

Facebook

Phone / SMS / WhatsApp on the following 3 numbers:

+91-789-362-6688, +1-480-347-6849, +44-07718-273-964

+44-33-3303-1284 (Preferred number if calling from U.K, No WhatsApp)

Tags Security

Cookie	Duration	Description
cookielawinfo-checkbox-advertisement	1 year	Set by the GDPR Cookie Consent plugin, this cookie is used to record the user consent for the cookies in the "Advertisement" category .
cookielawinfo-checkbox-analytics	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional	11 months	The cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
viewed_cookie_policy	11 months	The cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.