Message Queue
A message queue is a system that stores messages (tasks or events) sent from producers and delivers them to consumers for processing, decoupling the two and enabling asynchronous, reliable communication between system components.
Understanding Message Queue
Message queues solve a fundamental distributed systems problem: how do you reliably pass work between system components when both sides may be unavailable at the same time? Without a queue, if the worker is busy or down when a job arrives, the job is lost. With a queue, the job is stored until a worker is available to process it.
The producer-consumer model is simple: a producer (web server, webhook handler, user action) puts a message in the queue. A consumer (background worker) takes messages from the queue and processes them. Multiple producers and consumers can work simultaneously, scaling independently.
Popular message queue systems include RabbitMQ (full-featured, supports complex routing), Redis (lightweight, fast, used for simple queues), AWS SQS (managed, serverless), and Apache Kafka (high-throughput streaming). GAIA uses RabbitMQ for complex routing and Redis/ARQ for simpler background jobs.
Message queues enable graceful handling of load spikes. If 1000 webhook events arrive simultaneously, they queue immediately and get processed at the rate the consumers can handle — no events are lost, and the system doesn't collapse under load.
How GAIA Uses Message Queue
GAIA uses RabbitMQ for routing events between system components and ARQ (Async Redis Queue) for background job execution. When an email arrives or an automation trigger fires, the event is queued immediately and processed by background workers.
Related Concepts
Webhook
A webhook is an HTTP callback mechanism where a system sends an automated HTTP request to a specified URL whenever a defined event occurs, enabling real-time notification and integration between services without polling.
Event-Driven Automation
Event-driven automation (also called event-based automation) runs a workflow the moment something happens, such as a new email arriving, an issue being created or a row being added to a spreadsheet, instead of at a fixed time. Each automation pairs a trigger event with the actions to take, and the system where the event happened notifies the automation platform, usually through a webhook.
Workflow Automation
Workflow automation is the use of technology to execute repeatable business processes and tasks automatically, reducing manual effort and human error.
Cron Job
A cron job is a command or script that runs automatically on a repeating schedule, such as every night at 2:00 or every Monday at 9:00. The schedule is written as a cron expression with five fields (minute, hour, day of month, month, day of week), and the cron scheduler on Linux, macOS and other Unix systems checks it every minute and starts the job when the time matches.
API Integration
API integration is the process of connecting different software applications through their Application Programming Interfaces, enabling them to share data and functionality directly.
