Generated with sparks and insights from 5 sources

Introduction

• Overview: This guide explains how to import log data stored in AWS S3 into Splunk using the Splunk Add-on for AWS.

• Steps: The process involves setting up configurations on both AWS and Splunk platforms.

• AWS Setup: Create an S3 bucket, upload data, and configure IAM user permissions.

• Splunk Setup: Install the Splunk Add-on for AWS, configure the add-on, and set up data inputs.

• Alternative Methods: Using AWS Lambda for filtering and streaming logs to Splunk.

AWS Setup [1]

• Create S3 Bucket: Set up an S3 bucket with public access blocked.

• Upload Data: Upload log data to the S3 bucket, such as access.log from Splunk tutorial data.

• IAM User Creation: Create an IAM user with programmatic access and attach a policy with necessary permissions.

• Policy JSON: Use a policy that allows actions like s3:ListBucket, s3:GetObject, and kms:Decrypt.

• Save Credentials: Save the access key and secret access key for later use in Splunk configuration.

Splunk Setup [1]

• Install Add-on: Install the Splunk Add-on for AWS from Splunkbase.

• Configuration: Open the add-on, navigate to Configuration > Account, and add the IAM user credentials.

• Data Input: Go to Settings > Data Inputs, select AWS S3, and add a new input.

• Input Details: Provide necessary details like name, AWS account, and bucket name.

• Advanced Settings: Configure additional settings like polling interval, sourcetype, host, and index.

Using AWS Lambda [2]

• Purpose: AWS Lambda can be used to filter and stream logs from S3 to Splunk.

• Architecture: Logs are aggregated in an S3 bucket, and S3 sends event notifications to an SQS queue.

• Lambda Function: The function polls SQS, retrieves objects from S3, filters logs, and sends them to Splunk.

• Filtering Example: Filter VPC flow logs where the 'action' field is 'REJECT'.

• Advantages: Lower operational overhead, cost efficiency, and automatic scaling.

Additional Tips [1]

• Health Check: Use the 'Health Check' feature in the Splunk Add-on for AWS to troubleshoot issues.

• Data Filtering: Configure prefix, whitelist, and blacklist settings to filter specific data.

• Compression Support: The add-on supports ZIP, GZIP, TAR, and TAR.GZ formats.

• Handling Large Data: Use multiple S3 inputs or SQS for better performance and reliability.

• Archiving Old Data: Archive old S3 keys to reduce costs and improve performance.

References [1]

• Splunk Add-On for AWS Documentation: Link

• IAM Policy Permissions: Link

• Sizing Information: Link

• AWS Lambda Documentation: Link

• Splunk HTTP Event Collector: Link

<br><br>