Read Cloudwatch Logs Programatically using C#

In AWS, Cloudwatch is an extremely useful service for ingesting and retrieving logs, metrics, alarms etc… This particular blog post is about how to retrieve logs from cloudwatch.

Cloudwatch logs are organized in the following hierarchy: Region -> LogGroup -> Streams. Inside each stream are log messages which have 3 attributes: Message, Timestamp, Ingestion Time. Currently, I am using Cloudwatch for various logs such as Linux syslogs, web server logs, Cloudtrail events etc… How to programmatically ingest logs of .Net applications directly or using NLog or using Cloudwatch agent would be topics of future blog posts. This blog post assumes that the .Net code is running under an appropriate role that has appropriate permissions.

Install AWSSDK.CloudwatchLogs nuget package.

// Instantiating a Cloudwatch client
AmazonCloudWatchLogsClient client = new AmazonCloudWatchLogsClient(RegionEndpoint.EUWest2);


// Getting Log Groups - Code snippet
var logGroupsresponse = await client.DescribeLogGroupsAsync();

if(logGroupsresponse.HttpStatusCode == HttpStatusCode.OK)
{
   foreach(var logGroup in logGroupsresponse.LogGroups)
   {
         // Process
   }
}


// Getting Streams - Code snippet
var streamResponse = await client.DescribeLogStreamsAsync(
    new DescribeLogStreamsRequest
    {
        LogGroupName = "LogGroupName"
    });

if(streamResponse.HttpStatusCode == HttpStatusCode.OK)
{
   foreach(var stream in streamResponse.LogStreams)
   {
         // Process
   }
}


// Getting Log Messages - Code snippet
var logEventsresponse = await client.GetLogEventsAsync(
    new GetLogEventsRequest
    {
        LogGroupName = "Log Group Name",

        LogStreamName = "Log Stream Name"
    });

if(logEventsresponse.HttpStatusCode == HttpStatusCode.OK)
{
   foreach(var logMessage in logEventsresponse.Events)
   {
         // Process
   }
}

The above code snippets show instantiating, retrieving list of Log Groups, Streams within a particular log group, events within a specified Log Group and Stream.

There are additional parameters that can be specified in the requests. The most important being StartTime, StartFromHead for GetLogEventsRequest, NextToken, LogStreamNamePrefix, OrderBy for DescribeLogStreamsRequest.

Once all the important logs are ingested, an application can be built for monitoring threats or for viewing logs etc… As mentioned above, there will be more posts regarding Cloudwatch, logs, monitoring etc…