How to export logs from cloudwatch to S3.

There might be a business case for log retention whether it’s for compliance or any other reason.

This is the approach being used by ALight Technology And Services Limited for longer term log retention.

I have created a bucket in S3 with the following retention policies:

I personally don’t have to follow compliance yet, but nothing wrong in implementing compliance policies.

I have also defined a life-cycle policy to transition objects into Standard-IA (Infrequent Access) after 30 days.

Now I am developing a Lambda that would create Export tasks in CloudWatch once a week.

Here are some relevant C# code snippets:

var _client = new AmazonCloudWatchLogsClient(RegionEndpoint.EUWest2);
// Initialized AmazonCloudWatchLogsClient

var response = await _client.DescribeLogGroupsAsync();
// Get a list of LogGroups

foreach(var logGroup in response.LogGroups)
{
    var prefix = $"{from}-{to}-{logGroup.LogGroupName}";
    // You can define your own prefix
   
    var exportResult = await _client.CreateExportTaskAsync(new
        CreateExportTaskRequest
{
            Destination = "<NAME_OF_S3_BUCKET>",
            DestinationPrefix = prefix,
            From = GetUnixMilliSeconds(from),
            LogGroupName = logGroup.LogGroupName,
            TaskName = prefix,
            To = GetUnixMilliSeconds(to),
        })
    };

The above code is pretty much self-explantory. Here is a code snippet for getting Unix MilliSeconds from epoch.

long GetUnixMilliSeconds(DateTime dateTime)
{
    var _epoch = new DateTime(1970, 1, 1, 0, 0, 0, 0);
    return (dateTime.Ticks - _epoch.Ticks) / 10000;
}

Happy development!

Stay away from psycoSpies!

–

Mr. Kanti Kalyan Arumilli

B.Tech, M.B.A

Facebook

LinkedIn

Threads

Instagram

Youtube

Founder & CEO, Lead Full-Stack .Net developer

ALight Technology And Services Limited

ALight Technologies USA Inc

Youtube

Facebook

LinkedIn

Phone / SMS / WhatsApp on the following 3 numbers:

+91-789-362-6688, +1-480-347-6849, +44-07718-273-964

+44-33-3303-1284 (Preferred number if calling from U.K, No WhatsApp)

kantikalyan@gmail.com, kantikalyan@outlook.com, admin@alightservices.com, kantikalyan.arumilli@alightservices.com, KArumilli2020@student.hult.edu, KantiKArumilli@outlook.com and 3 more rarely used email addresses – hardly once or twice a year.