AWS and its Role in Our System

AWS plays a crucial role in extending the functionality of our system by providing a robust cloud infrastructure for event-driven serverless architecture. Below is an extended explanation of how AWS services integrate with our proprietary tools to handle backend requirements and support dynamic event processing from our Supabase Postgres database.

Lambda

AWS Lambda is used to deploy serverless functions that are triggered by specific events in our system. These events are emitted by our Supabase database and routed to Lambda for processing.

Workflow

Event Emission: Supabase sends an HTTP request to a designated Lambda function with:
- The event type (e.g., insert, update, delete).
- The event body (data payload).
Event Handling: Lambda functions are responsible for:
- Parsing the event payload.
- Executing logic specific to the event type.
- Publishing the processed event to the appropriate SNS topic for further consumption.

Tasks

Each Lambda function is implemented as a task within the aws/tasks folder.

Each task corresponds to a specific Lambda function.
Functions are associated with event topics set by the eventHandler property of the FunctionSettings type.

optional eventHandler: EventHandlerSettings
Defines how the function handles events, including table listeners, custom topics, and cron jobs.
Can take one of the following:
- Topic: Manually written topic that can describe an event that happens to an entity or something specific
- Listener: A table event
- CronJob: A cron expression

ECR (Elastic Container Registry)

AWS ECR is used to store and manage containerized Lambda function images. This enables:

Building custom runtime environments for Lambda.
Easily deploying and updating Lambda functions with new functionality.

Workflow

Container Upload: Function images are built locally or via CI/CD pipelines and pushed to ECR.
Execution: The container image is referenced when deploying Lambda functions, ensuring consistency across environments.

SNS acts as a messaging layer to distribute events processed by Lambda functions.

Workflow

Topic Assignment: Each event type is associated with an SNS topic.
Message Publishing: Lambda functions publish processed events to the relevant SNS topic.
Downstream Consumers: Other systems or services subscribe to these topics and consume the messages.

EventBridge

EventBridge is used for orchestrating scheduled tasks (cron jobs) and routing events between AWS services.

Cron Jobs

Cron expressions are defined within EventBridge to execute tasks at regular intervals.
EventBridge triggers Lambda functions based on these expressions, enabling time-based workflows.

Summary of Key Roles

Event-Driven Architecture:
- Supabase emits database events.
- Lambda functions process and distribute these events.
Orchestration:
- EventBridge schedules periodic tasks and facilitates event routing.
Messaging:
- SNS ensures that processed events are delivered to the appropriate downstream services.
Containerization:
- ECR simplifies deployment and management of custom Lambda runtimes.

This architecture ensures a scalable, maintainable, and efficient backend system for handling dynamic application workflows.

Build and Deployment Process

To effectively manage and deploy your serverless functions, a custom build and deployment process has been implemented. This section outlines how to utilize the provided settings, the impact of these configurations, and how the deployment process interacts with AWS services.

1. Configuration Settings

The deployment process relies on specific configuration settings defined in your project. These settings determine how each Lambda function is built, deployed, and integrated with other AWS services.

FunctionSettings

Located in apps/aws/sdk/types.ts, the FunctionSettings type defines the configuration for each Lambda function.

Type: FunctionSettings
Comprehensive settings for configuring Lambda functions, event handlers, and related resources.

Key Properties:

functionName: The name of the Lambda function.
handler: The entry point for the Lambda function.
runtime: The runtime environment (e.g., nodejs20.x).
memorySize: Allocated memory for the function.
timeoutInSeconds: Function timeout setting.
environment: Environment variables for the function.
imageName: (Optional) Docker image name if using containerization.
eventHandler: Configuration for event handling and SNS integration.

Example Configuration:

export const settings: FunctionSettings = {
  lambda: {
    functionName: "process-tasks-attempt",
    runtime: "nodejs20.x",
    memorySize: 256,
    timeoutInSeconds: 60,
  },
  eventHandler: {
    listener: [
      {
        schema: "system",
        table: "TaskAttempt",
        event: "Created",
      },
      {
        schema: "system",
        table: "TaskAttempt",
        event: "Updated",
      },
    ],
    filter: {
      handlerId: ["project-database-unit-tag-report"],
    },
  },
};

2. Using the Deployment Script

The deploy.ts script orchestrates the build and deployment process. Here’s how to utilize it effectively:

Command-Line Options:

--force or -f: Force deployment regardless of cache status.
--alias or -a: Specify the deployment alias environment (e.g., Production, Development).

Example Usage:

pnpm run deploy

Deployment Steps:

Initialization: The script initializes AWS clients using AWS SSO credentials, ensuring secure and authenticated access to AWS services.
Building Functions: It scans the aws/tasks directory for TypeScript files, compiles them to JavaScript, and packages them as ZIP files or Docker images based on the configuration.
Deployment:
- ZIP Packaging: Non-containerized functions are zipped and uploaded to AWS Lambda.
- Container Deployment: Functions requiring custom runtimes are built as Docker images and pushed to ECR.
Versioning and Aliasing: After deployment, the script publishes new versions of the functions and updates aliases to point to the latest versions.
SNS Subscription Setup: The script configures SNS topics and subscribes Lambda functions to these topics based on the eventHandler settings.
Dead Letter Queue (DLQ) Configuration: Ensures that any failed executions are routed to a designated SQS queue for later analysis.
Cleanup: Removes stale permissions and subscriptions to maintain a clean and secure environment.

3. Effects of Configuration Settings

The settings you define for each Lambda function directly influence how the function behaves and interacts with AWS services:

Function Name (functionName): Determines the identifier used in AWS Lambda. Must be unique across your AWS account.
Handler (handler): Specifies the file and function to execute when the Lambda is invoked.
Runtime (runtime): Defines the language and version environment for the Lambda function (e.g., nodejs20.x).
Memory Size (memorySize): Allocates memory, impacting both performance and cost.
Timeout (timeoutInSeconds): Sets the maximum execution time for the function, preventing runaway processes.
Environment Variables (environment): Passes configuration data and secrets to the function at runtime.
Image Name (imageName): If provided, the function is deployed as a container image from ECR, allowing for custom dependencies and runtimes.
Event Handler (eventHandler): Configures how the function interacts with SNS topics, including filtering policies and event types.

4. Interaction with AWS Services

The deployment process leverages several AWS services to ensure seamless integration and functionality:

AWS Lambda

Creation and Updates: Functions are created or updated based on the provided settings, handling both code and configuration changes.
Versioning and Aliasing: New versions are published, and aliases are updated to point to the latest stable versions, facilitating smooth rollouts and rollbacks.

AWS ECR

Container Management: For functions using container images, ECR stores and manages these images, ensuring they are accessible to Lambda during deployment.

AWS SNS

Messaging: SNS topics are created or utilized to distribute events processed by Lambda functions.
Subscriptions: Lambda functions subscribe to relevant SNS topics, enabling them to receive and process events as defined by the eventHandler settings.

AWS SQS (Dead Letter Queue)

Error Handling: Failed function executions are routed to an SQS queue, allowing for retries and analysis without disrupting the main event flow.

AWS IAM

Permissions Management: Roles and permissions are configured to allow Lambda functions to interact with other AWS services securely.

AWS EventBridge

Scheduling: Manages cron jobs and scheduled tasks, triggering Lambda functions at specified intervals to handle time-based workflows.

5. Practical Workflow for Users

Here’s a step-by-step guide on how users can define settings and deploy their Lambda functions using the provided system:

Define Function Settings:

Navigate to the aws/tasks directory.
Create or modify a TypeScript file representing your Lambda function.
Define the FunctionSettings object with appropriate properties.

Example:

// apps/aws/tasks/processUserEvents.ts
import { FunctionSettings } from "../../sdk/types";
 
export const settings: FunctionSettings = {
  lambda: {
    functionName: "process-tasks-attempt",
    runtime: "nodejs20.x",
    memorySize: 256,
    timeoutInSeconds: 60,
  },
  eventHandler: {
    listener: [
      {
        schema: "system",
        table: "TaskAttempt",
        event: "Created",
      },
      {
        schema: "system",
        table: "TaskAttempt",
        event: "Updated",
      },
    ],
    filter: {
      handlerId: ["project-database-unit-tag-report"],
    },
  },
};
 
export const handler = async (event: any) => {
  // Your event processing logic here
};

Build and Deploy:
- Run the deployment script with desired options.
Example:
```
pnpm run deploy
```
- The script will:
  - Compile TypeScript files to JavaScript.
  - Package functions as ZIP files or Docker images based on the presence of imageName.
  - Upload the packages to AWS Lambda or ECR.
  - Configure SNS subscriptions and permissions.
  - Update function aliases and versions.
Monitor Deployments:
- Check the deployment logs for any errors or confirmations.
- Verify the creation and configuration of AWS resources via the AWS Management Console or AWS CLI.
Handle Events:
- Once deployed, Lambda functions will automatically handle events emitted by Supabase and other integrated services.
- Failed events will be routed to the configured DLQ for later inspection and retrying.

6. Customizing Deployment Behavior

Users can customize various aspects of the deployment process through the FunctionSettings. Here’s how different settings impact the deployment and runtime behavior:

Custom Runtimes:
- By specifying the imageName, users can deploy Lambda functions as container images, allowing for custom dependencies and runtime environments.
Environment Segregation:
- Using the alias option during deployment, users can deploy functions to different environments (e.g., Production, Development) without altering the core codebase.
Resource Allocation:
- Adjusting memorySize and timeoutInSeconds allows users to optimize function performance and cost based on workload requirements.
Event Filtering:
- The filter property within eventHandler enables users to specify which event types should trigger the Lambda function, ensuring that only relevant events are processed.

7. Security Considerations

Proper configuration of permissions and roles is crucial for maintaining a secure deployment environment:

IAM Roles:
- Each Lambda function is associated with an IAM role (BasicLambdaExecution) that grants necessary permissions to interact with other AWS services like SNS, SQS, and ECR.
Resource-Based Policies:
- The deployment script manages resource-based policies for Lambda functions, ensuring that only authorized services (e.g., SNS) can invoke them.
Environment Variables:
- Sensitive data, such as database URLs and API keys, are managed through environment variables, avoiding hard-coded secrets in the codebase.

8. Error Handling and Cleanup

The deployment process includes robust error handling and cleanup mechanisms to maintain system integrity:

Stale Permission Removal:
- The script identifies and removes outdated permissions and SNS subscriptions that are no longer associated with the current deployment, preventing unauthorized access and reducing clutter.
Dead Letter Queues:
- Failed function executions are routed to SQS queues, allowing for post-mortem analysis and retries without disrupting the main event flow.
Deployment Caching:
- By leveraging a deployment cache, the script avoids redeploying functions that haven’t changed, optimizing deployment times and reducing AWS costs.

Conclusion

By meticulously defining configuration settings and leveraging the provided deployment script, users can efficiently manage and deploy Lambda functions within a robust AWS-based event-driven architecture. This process ensures that functions are consistently built, securely deployed, and seamlessly integrated with AWS services, providing a scalable and maintainable backend system.

Key Takeaways:

Configuration Flexibility: Customize Lambda functions through detailed settings to meet specific application requirements.
Automated Deployment: Streamline the build and deployment process with automation, reducing manual effort and potential errors.
Secure Integration: Maintain security through proper permission management and secure handling of environment variables.
Scalability and Maintainability: Ensure that the system can grow with your application needs while remaining easy to manage and extend.

For further assistance or specific use-case implementations, feel free to reach out or consult the provided type definitions and deployment scripts.

User and Auth OpenAI

AWS and its Role in Our System

Lambda

Workflow

Tasks

ECR (Elastic Container Registry)

Workflow

SNS (Simple Notification Service)

Workflow

EventBridge

Cron Jobs

Summary of Key Roles

Build and Deployment Process

1. Configuration Settings

FunctionSettings

2. Using the Deployment Script

Command-Line Options:

Deployment Steps:

3. Effects of Configuration Settings

4. Interaction with AWS Services

AWS Lambda

AWS ECR

AWS SNS

AWS SQS (Dead Letter Queue)

AWS IAM

AWS EventBridge

5. Practical Workflow for Users

6. Customizing Deployment Behavior

7. Security Considerations

8. Error Handling and Cleanup

Conclusion

Key Takeaways: