AWS architecture for integrating with Splunk

You need to create an IAM user for Splunk to authenticate with your AWS account. Give this IAM user permissions by adding them to a user group with attached relevant permissions. (This is AWS's best practice; it's better than attaching policies directly to users.) You'll have to create an access key for this user, which is then held on Splunk for it to use for authentication. Splunk acts as this user by having access to their access keys. The default policy can be hardened depending on the exact scope of what you want the user to perform. The default policy allows for all functionality that the Splunk app could use, but you probably won't use every feature.

In AWS, your Cloudtrail has an option to send an S3 delivery notification. A message is published to a Simple Notification Service (SNS) topic every time a log file is delivered to the designated S3 bucket. The message sent to SNS contains details of the log file and the bucket, which is how Splunk later knows where and what to retrieve. The setting is enabled by editing a trail and enabling SNS notifications, which will give you the options shown below.

This SNS topic is subscribed to by a Simple Queue Service (SQS) queue. Another SQS queue is configured as the dead-letter queue (DLQ), which ensures any messages that aren't processed correctly don't get lost. The SQS queues are then set as input to your Splunk server. Whenever a notification is received from AWS, the Splunk server goes to the S3 bucket that CloudTrail pushes logs to and retrieves the log file. Splunk allows you to set the batch size and the interval of the log retrieval. This allows you to control how many messages it will receive at once and how often it will go and check for messages.

Searching logs in Splunk

Once your AWS logs are ingested into Splunk, you'll want to be able to search them and find information about what's going on.

To return all logs that have been delivered, enter * in the search bar. The drop-down menu on the right of the screen allows you to specify a time of interest.

You can also search for keywords you expect to feature in the required logs. For example, if you want to know what the user Alice456 has been up to, simply search “Alice456”. Because the username is a field in the logs, all entries attributed to this user will be returned.

You can combine these simple searches with AND, OR, and NOT logic. So, if you wanted to see logs that included the username Alice456 and had the event name ListBuckets, you could search: “Alice456” AND “ListBuckets”. Equally, if you want to see all logs about Alice456 but not console login events, you can search: “Alice456” NOT “ConsoleLogin”.

AWS determines the fields in the logs. Splunk just processes them and displays them. Sometimes, sub-sections will be hidden at first glance. You can expand them by pressing the plus symbol in square brackets.

Navigate to the IAM console and select User Groups on the left-hand side. Click SplunkAccessGroup.

Add SplunkUser to the group.

Look at the attached policy on the group to see the permissions Splunk requires.

Next,

Navigate to the CloudTrail console and select the metrolio-management-trail. Edit the trail and enable SNS notification delivery to a new SNS topic called SplunkTopic.

Navigate to the SQS console and select to create a new queue. Name the queue SplunkQueue and set the visibility timeout to five minutes. Select Create queue, leaving everything else as default.

Click your new SQS queue and select Subscribe to Amazon SNS topic.

Choose the topic you created called SplunkTopic from the dropdown menu and click Save.

Create another new SQS queue called SplunkDLQ. Again, set the visibility timeout to five minutes and leave everything else as default

Select your queue called SplunkQueue and click Edit.

In the dead-letter queue box, select it to be enabled.

Select your queue called SplunkDLQ from the dropdown marked Choose queue.

Set the maximum receives to 3.

Navigate to the IAM console and select the user SplunkUser.

On the Security Credentials panel, select Create access key.

Click to download the key as a .csv file.

In the Splunk web interface, click on the app named Splunk Add-on for AWS.

Select the Configuration tab at the top and, on the Accounts tab, select Add.

Fill in the fields with the name SplunkUser and Key ID and Secret Key from the downloaded .csv file. Leave the Region category as Global.

in the same app, go to the Inputs tab and select Create New Input.

Choose the Cloudtrail option and SQS-Based-S3.

Name the input Metrolio-Trail. Set the AWS account to SplunkUser, set the region to Europe (Ireland), and select the SplunkQueue for the SQS queue name.

Change the SQS batch size to 5 and, under Advanced Settings, set the interval to 10 seconds.

The user Hades12 is repeatedly trying to create an IAM policy. What is the name of the policy?

Policy is named below

What is the username of the user performing GetBucketPolicy against the Metrolio log bucket in the account?

What is the eventSource for the API call that Ares27 is using to list lambda functions in the account?

What is the username of the user trying to stop the metrolio-management-trail from logging? They are getting an AccessDenied exception.