End-to-End Multi-Account Amazon VPC Flow Logs Centralization & Analysis with AWS Config, Amazon S3, AWS Glue & Amazon Athena

In multi-account AWS Organizations, it’s surprisingly hard to find a single walkthrough that covers everything from “auto-enable Amazon VPC Flow Logs everywhere” to “build AWS Glue tables and query them via Amazon Athena” across accounts.

While helping one of our DoiT customers deploy AWS Control Tower in their AWS Organization, I realized there is no single document detailing how to 1) automate the enablement of centralized Amazon VPC Flow Logs in an AWS Organization AND 2) configure AWS Glue and Amazon Athena to analyze the collected data.

Existing posts touch on bits of the puzzle:

• Centralizing logs into one bucket:

How to Facilitate Data Analysis and Fulfill Security Requirements by Using Centralized Flow Log Data | Amazon Web Services

I am an AWS Professional Services consultant, which has me working directly with AWS customers on a daily basis. One of my customers recently asked me to provide a solution to help them fulfill their security requirements by having the flow log data from VPC Flow Logs sent to a central AWS account. This is […]

Amazon Web Services

• Automating Amazon VPC Flow Logs enablement with AWS Config rules:

How to enable VPC Flow Logs automatically using AWS Config rules | Amazon Web Services

This post discusses an automated process for enabling Amazon Virtual Private Cloud (Amazon VPC) Flow Logs using AWS Config rule remediation. Customers use Amazon VPC Flow logs to capture information about the IP traffic going to and from network interfaces in an Amazon VPC. You can deploy this solution with the help of AWS Control […]

Amazon Web Services

• Point-and-click Amazon VPC Flow Logs analysis in a single account:

Analyze VPC Flow Logs with point-and-click Amazon Athena integration | Amazon Web Services

Analyzing your VPC Flow Logs using Athena is now easier than ever! The recently introduced VPC Flow Logs integration with Amazon Athena helps you get started with extracting meaningful insights from VPC Flow Logs in just a few clicks. In this blog post, we will walk you through how you can use this recently announced […]

Amazon Web Services

…but none give you an end-to-end, multi-account tutorial, especially around creating and partitioning Amazon Athena tables that span multiple accounts in a central Amazon S3 bucket.

That gap inspired this post. Here you’ll get:

1. Automatic Amazon VPC Flow Logs enablement in every new VPC.
2. Centralized storage in your Log Archive (or equivalent) account.
3. Cross-account AWS Glue & Amazon Athena setup in your Audit (or equivalent) account—complete with Data Definition Language (DDL), crawler configuration, and partition management.

You’ll find the missing links here—packaged in one coherent guide.

NOTE: I'm using the default Log Archive (to store) and Audit accounts (to analyze) the Amazon VPC Flow Logs. They are create automatically when AWS Control Tower is configured in the AWS Organization and they fit the intended purpose, respectively. However, the tutorial would also be valid in an environment where AWS Control Tower is not deployed and you want to follow a similar approach of separating the storage and analysis of the Amazon VPC Flow Logs. The account names are not important.

Prerequisites

• AWS Organization with service-managed AWS CloudFormation permissions.
• Three accounts:
  • Management Account (for AWS Config rules & AWS CloudFormation StackSet deployment)
• Log Archive (to host the central Amazon S3 bucket)
• Audit (to run AWS Glue crawlers and Amazon Athena queries)
• AWS Identity and Access Management (IAM) rights to deploy AWS CloudFormation StackSets at the OU level and configure the involved services.
• Familiarity with AWS CLI or Console.

In the Log Archive Account: Configure the Central Amazon S3 Bucket

Create the Bucket

1. In the Log Archive account, open the Amazon S3 Console.
2. Click Create bucket; name it <central-s3-vpc-flow-logs-bucket>.
3. Enable Block all public access and SSE-KMS (recommended).

Apply the Bucket Policy & Lifecycle Rules

1. Under Permissions > Bucket policy, paste the policy below (fill your <…> values).

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": { "Service": "delivery.logs.amazonaws.com" },
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::<central-s3-vpc-flow-logs-bucket>/*",
            "Condition": {
                "StringEquals": {
                    "s3:x-amz-acl": "bucket-owner-full-control",
                    "aws:SourceOrgID": "<AWS Organization ID>"
                }
            }
        },
        {
            "Effect": "Allow",
            "Principal": { "Service": "delivery.logs.amazonaws.com" },
            "Action": "s3:GetBucketAcl",
            "Resource": "arn:aws:s3:::<central-s3-vpc-flow-logs-bucket>",
            "Condition": {
                "StringEquals": {
                    "aws:SourceOrgID": "<AWS Organization ID>"
                }
            }
        },
        {
            "Sid": "AllowGlueAccess",
            "Effect": "Allow",
            "Principal": { "Service": "glue.amazonaws.com" },
            "Action": [
                "s3:GetBucketLocation","s3:GetObject","s3:ListBucket",
                "s3:ListBucketMultipartUploads","s3:ListMultipartUploadParts",
                "s3:AbortMultipartUpload"
            ],
            "Resource": [
                "arn:aws:s3:::<central-s3-vpc-flow-logs-bucket>",
                "arn:aws:s3:::<central-s3-vpc-flow-logs-bucket>/*"
            ],
            "Condition": {
                "StringEquals": { "aws:SourceAccount": "<Audit Account ID>" }
            }
        },
        {
            "Sid": "AllowAthenaAccess",
            "Effect": "Allow",
            "Principal": { "AWS": "arn:aws:iam::<Audit Account ID>:root" },
            "Action": [
                "s3:GetBucketLocation","s3:GetObject","s3:ListBucket",
                "s3:ListBucketMultipartUploads","s3:ListMultipartUploadParts",
                "s3:AbortMultipartUpload"
            ],
            "Resource": [
                "arn:aws:s3:::<central-s3-vpc-flow-logs-bucket>",
                "arn:aws:s3:::<central-s3-vpc-flow-logs-bucket>/*"
            ]
        }
    ]
}

The preceding bucket policy allows the log delivery service from members of the <AWS Organization ID> (something like “o-1234567890”) to write logs to the bucket and to allow Amazon Athena and AWS Glue services in the Audit account (Account ID: <Audit Account ID>) to access the log data in the centralized S3 bucket.

Lifecycle tip: Transition to Amazon S3 Glacier after 30 days and expire after a year to optimize costs.

Validation Checklist

• Bucket policy applied (aws s3api get-bucket-policy).
• Lifecycle rule visible and enabled, if any.
• Public access blocked and encryption active.

In the Management Account: Deploy Auto-Remediation via AWS CloudFormation StackSet

The following AWS CloudFormation template should be deployed as a AWS CloudFormation StackSet in the Management Account. It automatically checks whether Amazon VPC Flow Logs are enabled on a VPC and, if not, triggers an AWS Systems Manager Automation document (AWS-EnableVPCFlowLogs) for remediation.

AWS CloudFormation Template

Create a template with the following content:

AWSTemplateFormatVersion: "2010-09-09"
Description: "This template will deploy an AWS Config rule to automatically remediate VPC Flow Logs enablement"
Parameters:
  CustomConfigRuleName:
    Description: Name that you want to give to the AWS Config Rule.
    Type: String
    Default: ConfigRuleForEnableVpcFlowLogs
  TrafficType:
    Type: String
    AllowedValues:
      - ACCEPT
      - REJECT
      - ALL
    Description: The value for the VPC Flow Logs traffic type.
    Default: ALL
  MaxExecutionFrequency:
    Type: String
    AllowedValues:
      - One_Hour
      - Three_Hours
      - Six_Hours
      - Twelve_Hours
      - TwentyFour_Hours
    Description: The maximum frequency with which AWS Config runs evaluations for a rule.
    Default: One_Hour
  CentralizedS3LoggingBucket:
    Description: Name of the S3 bucket in the logging account to send VPC Flow Logs.
    Type: String

Resources:
  ConfigRemediationRole:
    Type: "AWS::IAM::Role"
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Principal:
              Service:
                - "ssm.amazonaws.com"
            Action:
              - "sts:AssumeRole"
      Path: "/"
      Policies:
        - PolicyName: aws-config-remediate-vpc-flow-logs-policy
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: "Allow"
                Action:
                  - logs:CreateLogDelivery
                  - logs:DeleteLogDelivery
                  - logs:CreateLogGroup
                Resource: arn:aws:logs:*:*:*
              - Effect: "Allow"
                Action:
                  - ec2:CreateFlowLogs
                  - logs:CreateLogDelivery
                Resource: "*"

  ConfigRuleForEnableVpcFlowLogs:
    Type: AWS::Config::ConfigRule
    Properties:
      ConfigRuleName: !Ref CustomConfigRuleName
      Description: ConfigPermissionToInvokeAnAutomaticRemediation
      InputParameters:
        trafficType: !Ref TrafficType
      MaximumExecutionFrequency: !Ref MaxExecutionFrequency
      Scope:
        ComplianceResourceTypes:
          - AWS::EC2::VPC
      Source:
        Owner: AWS
        SourceIdentifier: VPC_FLOW_LOGS_ENABLED

  VpcFlowLogsRemediationConfiguration:
    DependsOn: ConfigRuleForEnableVpcFlowLogs
    Type: AWS::Config::RemediationConfiguration
    Properties:
      ConfigRuleName: !Ref CustomConfigRuleName
      Automatic: true
      MaximumAutomaticAttempts: 5 # minutes
      RetryAttemptSeconds: 50 # seconds
      ResourceType: AWS::EC2::VPC
      Parameters:
        VPCIds:
          ResourceValue:
            Value: "RESOURCE_ID"
        LogDestinationType:
          StaticValue:
            Values:
              - s3
        LogDestinationArn:
          StaticValue:
            Values:
              - !Sub "arn:aws:s3:::${CentralizedS3LoggingBucket}"
        TrafficType:
          StaticValue:
            Values:
              - !Ref TrafficType
        AutomationAssumeRole:
          StaticValue:
            Values:
              - !GetAtt ConfigRemediationRole.Arn
      TargetId: AWS-EnableVPCFlowLogs
      TargetType: SSM_DOCUMENT
      TargetVersion: 1

Outputs:
  ConfigRuleForEnableVpcFlowLogsArn:
    Description: Arn of the AWS Config Rule to enable VPC Flow Logs
    Value: !GetAtt ConfigRuleForEnableVpcFlowLogs.Arn
  ConfigRemediationRoleArn:
    Description: Arn of the IAM Role to perform auto-remediation
    Value: !GetAtt ConfigRemediationRole.Arn

• Parameters:
  • CustomConfigRuleName: Name of the AWS Config rule. Default is “ConfigRuleForEnableVpcFlowLogs”.
  • TrafficType: Sets the type of traffic to log (ACCEPT, REJECT, or ALL). Default is ALL.
  • MaxExecutionFrequency: How often AWS Config should run evaluations (default One_Hour).
  • CentralizedS3LoggingBucket: The name of the Amazon S3 bucket (in the Log Archive account) where Amazon VPC Flow Logs are delivered.
• Resources:
  • AWS Identity and Access Management (IAM) Role (ConfigRemediationRole): This role (assumed by AWS Systems Manager) has permissions to create log deliveries, log groups, and Amazon VPC Flow Logs.
• AWS Config Rule (ConfigRuleForEnableVpcFlowLogs): This rule continuously checks for Amazon VPC Flow Log enablement on AWS::EC2::VPC resources.
• Remediation Configuration (VpcFlowLogsRemediationConfiguration): When non-compliance is detected, it automatically triggers the remediation using the AWS Systems Manager Automation document (AWS-EnableVPCFlowLogs).
• Outputs:
  • The ARN of the AWS Config rule and the remediation AWS Identity and Access Management (IAM) role for reference and troubleshooting.

NOTE: If you want to avoid the deployment of the CloudFormation Stack instances in all the accounts (including the Control Tower accounts), it is recommended to deploy the StackSet to AWS Organizational Units.

AWS CloudFormation StackSet Deployment Steps

1. In the Management account, open CloudFormation > StackSets.
2. Click Create StackSet, choose Service-managed permissions.
3. Upload the above template, validate.
4. Fill in parameters (use the bucket name from step 1).
5. Under Deployment targets, select Organizational Units, enter your OU IDs.
6. Enable Automatic deployment and set Account removal behavior to Delete stacks.
7. Choose target Regions (e.g. us-east-1, eu-west-1, eu-south-2).
8. Configure concurrency: Max concurrent accounts 6, Failure tolerance 1, Parallel.
9. Review and submit.

NOTE: To add more OUs or change the deployment details, select the StackSet and click on “Add stacks to StackSet” or “Edit StackSet details”. Alternatively, using the AWS CLI, the command could be something like:

aws cloudformation create-stack-instances --stack-set-name EnableVPCFlowLogsStackSet --deployment-targets '{"OrganizationalUnitIds": ["ou-zypz-lsh6c707"]}' --regions eu-west-1 eu-south-2 us-east-1

Validation Checklist

• Member accounts’ VPCs show COMPLIANT in AWS Config.
• AWS Systems Manager Automation runs in each account.
• Amazon VPC Flow Logs appear in the central S3 bucket.

In the Audit Account: Set Up AWS Glue & Amazon Athena

In the Audit Account, if not already configured, configure a query result location for Amazon Athena in Amazon S3. Once this bucket is entered, go to the “Query editor” and run the Athena queries below.

Create AWS Glue Database & Table

1. In Audit, go to Amazon Athena > Query editor; configure query result Amazon S3 location.
2. Run:

CREATE DATABASE IF NOT EXISTS vpc_flow_logs_db
COMMENT 'Database containing the vpc_flow_logs table';

3. Then, use the following Amazon Athena DDL query to create the external AWS Glue table that references the Amazon VPC Flow Logs stored in the Amazon S3 bucket:

CREATE EXTERNAL TABLE IF NOT EXISTS `vpc_flow_logs` (
  version int, account_id string, interface_id string,
  srcaddr string, dstaddr string, srcport int, dstport int,
  protocol bigint, packets bigint, bytes bigint,
  start bigint, `end` bigint, action string, log_status string,
  vpc_id string, subnet_id string, instance_id string,
  tcp_flags int, type string, pkt_srcaddr string, pkt_dstaddr string,
  region string, az_id string, sublocation_type string,
  sublocation_id string, pkt_src_aws_service string,
  pkt_dst_aws_service string, flow_direction string, traffic_path int
)
PARTITIONED BY (
  `account` string, `vpcflowlogs` string,
  `region_id` string, `year` string, `month` string, `day` string
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ' '
LOCATION 's3://<central-s3-vpc-flow-logs-bucket>/AWSLogs/'
TBLPROPERTIES ("skip.header.line.count"="1");

Configure the AWS Glue Crawler

Create a AWS Glue Crawler that updates the vpc_flow_logs table daily at 7:30 AM (for instance). Follow these steps:

1. AWS Glue > Crawlers → Create crawler.
2. Name it (e.g. vpc-flow-logs-crawler).
3. Data source: select your vpc_flow_logs_db.vpc_flow_logs table.
4. AWS Identity and Access Management (IAM) role: read from S3, write to Glue Data Catalog.
5. Advanced options:
   • Ignore schema changes
   • Update all partitions
   • Ignore deletions
6. Schedule: Daily @ 07:30.

Create and Run crawler.

1. Back on the Crawlers page, you’ll now see your newly created crawler.
2. If you set a schedule, it will run automatically. Otherwise, select the crawler and click Run crawler to start it immediately.
3. Once the crawler completes, check the Tables section in AWS Glue to confirm that your data is properly cataloged (or updated).
4. You can now query these tables using Amazon Athena.

NOTE: You can also run the crawler on demand, when needed.

Validation Checklist

• Glue table has today’s partitions.
• SELECT * FROM vpc_flow_logs LIMIT 10; in Amazon Athena returns recent flow logs.

Sample Amazon Athena Queries

Once the AWS Glue table is set up and partitions are updated, you can run queries in Amazon Athena. Here are a few examples:

-- 1) Count Log Entries Per Account
SELECT account, COUNT(*) AS log_entry_count
FROM vpc_flow_logs
GROUP BY account
ORDER BY log_entry_count DESC;

-- 2) Analyze Rejected Traffic Patterns
SELECT region, COUNT(*) AS rejected_logs
FROM vpc_flow_logs
WHERE action = 'REJECT'
GROUP BY region
ORDER BY rejected_logs DESC;

-- 3) Aggregate Data Volume by Region and Date
SELECT region_id, year, month, day, SUM(bytes) AS total_bytes
FROM vpc_flow_logs
GROUP BY region_id, year, month, day
ORDER BY total_bytes DESC;

-- 4) Identify Top Source IPs Sending Traffic
SELECT srcaddr, COUNT(*) AS occurrence
FROM vpc_flow_logs
GROUP BY srcaddr
ORDER BY occurrence DESC
LIMIT 10;

Troubleshooting & Tips

• Force Re-Evaluation: In AWS Config, click Re-evaluate on the rule to trigger immediate remediation.
• Missing Partitions: Manually run the AWS Glue crawler on demand if partitions aren’t appearing.
• AWS Identity and Access Management (IAM) Errors: Verify trust policies on the AWS Systems Manager remediation role and that the StackSet service-managed role exists.

Cost, Security & Extensions

• Cost Considerations
  • Amazon VPC Flow Logs ingestion: ~$0.10/GB.
  • Amazon Athena queries: $5/TB scanned.
  • AWS Glue crawler: ~$0.44 per DPU-hour.
  • Regular Amazon S3 storage and access costs.
• Security
  • Enable Amazon S3 Block Public Access.
  • Audit access via AWS CloudTrail and Amazon GuardDuty.
• Partition Strategy
  • If you have very high log volume, consider hourly partitions (hour field) or Hive-style prefixes.

What’s Next?

• Visualize: Connect Amazon Athena to Amazon QuickSight or build dashboards in Amazon OpenSearch Service.
• Infrastructure as Code: Port this solution to Terraform or AWS CDK for Git-driven deployments.
• Advanced Analysis: Integrate with AWS Lambda for real-time alerts on abnormal traffic patterns.

By following this guide, your AWS Organization will automatically ensure every Amazon VPC (or a selection) is logging traffic, centralize all your logs for security and compliance, and give your security and operations teams powerful query capabilities with very low maintenance efforts.