Customers who sign-up prior to 30/06/2024 get unlimited access to free features, newer features (with some restrictions), but for free for at least 1 year.Sign up now! https://webveta.alightservices.com/
Categories
.Net AWS C#

How to export logs from cloudwatch to S3.

There might be a business case for log retention whether it’s for compliance or any other reason.

This is the approach being used by ALight Technology And Services Limited for longer term log retention.

I have created a bucket in S3 with the following retention policies:

AWS S3 Object Lock Policy

I personally don’t have to follow compliance yet, but nothing wrong in implementing compliance policies.

I have also defined a life-cycle policy to transition objects into Standard-IA (Infrequent Access) after 30 days.

Now I am developing a Lambda that would create Export tasks in CloudWatch once a week.

Here are some relevant C# code snippets:

var _client = new AmazonCloudWatchLogsClient(RegionEndpoint.EUWest2);
// Initialized AmazonCloudWatchLogsClient

var response = await _client.DescribeLogGroupsAsync();
// Get a list of LogGroups

foreach(var logGroup in response.LogGroups)
{
    var prefix = $"{from}-{to}-{logGroup.LogGroupName}";
    // You can define your own prefix
   
    var exportResult = await _client.CreateExportTaskAsync(new
        CreateExportTaskRequest
{
            Destination = "<NAME_OF_S3_BUCKET>",
            DestinationPrefix = prefix,
            From = GetUnixMilliSeconds(from),
            LogGroupName = logGroup.LogGroupName,
            TaskName = prefix,
            To = GetUnixMilliSeconds(to),
        })
    };

The above code is pretty much self-explantory. Here is a code snippet for getting Unix MilliSeconds from epoch.

long GetUnixMilliSeconds(DateTime dateTime)
{
    var _epoch = new DateTime(1970, 1, 1, 0, 0, 0, 0);
    return (dateTime.Ticks - _epoch.Ticks) / 10000;
}

Happy development!

Stay away from psycoSpies!

I don’t have any fake aliases, nor any virtual aliases like some of the the psycho spy R&AW traitors of India. NOT associated – “ass”, eass, female “es”, “eka”, “ok”, “okay”, “is”, “erra”, yerra, karan, kamalakar, diwakar, kareem, karan, sowmya, zinnabathuni, bojja srinivas (was a friend and batchmate 1998 – 2002), mukesh golla (was a friend and classmate 1998 – 2002), thota veera, uttam’s, bandhavi’s, bhattaru’s, thota’s, bojja’s, bhattaru’s or Arumilli srinivas or Arumilli uttam (may be they are part of a different Arumilli family – not my family).

Mr. Kanti Kalyan Arumilli

Arumilli Kanti Kalyan, Founder & CEO
Arumilli Kanti Kalyan, Founder & CEO

B.Tech, M.B.A

Facebook

LinkedIn

Threads

Instagram

Youtube

Founder & CEO, Lead Full-Stack .Net developer

ALight Technology And Services Limited

ALight Technologies USA Inc

Youtube

Facebook

LinkedIn

Phone / SMS / WhatsApp on the following 3 numbers:

+91-789-362-6688, +1-480-347-6849, +44-07718-273-964

+44-33-3303-1284 (Preferred number if calling from U.K, No WhatsApp)

kantikalyan@gmail.com, kantikalyan@outlook.com, admin@alightservices.com, kantikalyan.arumilli@alightservices.com, KArumilli2020@student.hult.edu, KantiKArumilli@outlook.com and 3 more rarely used email addresses – hardly once or twice a year.