In this blog post Azure Event Hubs Large Messages When to Redesign Your Architecture we will explain why oversized events cause cost, reliability and performance problems, even when Azure technically allows them.
The warning signs are familiar. An integration works well during testing, then production data grows. Messages start failing, processing slows down and the proposed fix is to increase limits, add capacity or split payloads into chunks.
Sometimes that fix is reasonable. In other cases, the message-size error is telling you that Event Hubs is being asked to carry something it was not designed to carry.
What Azure Event Hubs actually does
Azure Event Hubs is a real-time data streaming service. Applications and devices publish a continuous stream of small events, while one or more receiving systems read and process that stream independently.
Think of it as a fast-moving conveyor belt. Each event should usually describe something that happened, such as an order being placed, a device changing status or an AI agent completing a task.
Event Hubs divides the stream into partitions, which are separate lanes that allow data to be processed at scale. Events within each partition are read in order, so one unusually large or slow event can delay the work behind it.
What counts as a large Event Hubs message
Message limits depend on the Azure Event Hubs tier. At the time of writing, Basic supports publications up to 256 KB, while Standard and Premium support up to 1 MB. Eligible Dedicated clusters can support messages up to 20 MB, although large-message support is currently a preview feature.
The limit applies to the complete publication, including message content and associated data. A batch of events must also fit within the permitted publication size.
However, the service limit should not be your only design test. A 700 KB event may be technically valid in Standard, but it can still become expensive when sent thousands of times per hour and read by several consumer applications.
Why simply increasing the limit may not solve the problem
1. Large events consume capacity quickly
Event Hubs capacity is affected by both the number of events and the amount of data moving through the service. Larger messages use more network bandwidth, take longer to process and require more memory in receiving applications.
The impact multiplies when several consumers read the same stream. A 5 MB payload read by six systems can create far more data movement than the sending application suggests.
2. Failures become more expensive
Event-driven systems must assume that processing will occasionally fail. When a consumer retries a large message, the application must download, decode and process the entire payload again.
This increases recovery time and can create a backlog during an outage. What begins as one failed integration can delay reporting, customer notifications and other business processes using the same stream.
3. Every consumer receives data it may not need
An event may contain a document, image, AI conversation or detailed business record even though most consumers only need an identifier and status.
Sending the full object forces every system to handle sensitive or unnecessary data. That increases processing costs and creates more places where confidential information must be secured.
4. A tier upgrade can hide the underlying design issue
Moving to an Event Hubs Dedicated cluster may be justified when the business needs very high throughput, predictable performance and dedicated capacity. It is harder to justify when the only reason is that one application sends oversized documents.
The business may end up paying for a larger platform to avoid changing one integration. Before approving that cost, compare it with storing the payload separately and sending a lightweight event.
The better pattern for files and large business objects
When an event contains a large document, image, export file or AI-generated artefact, the usual alternative is the claim-check pattern. Despite the technical name, the idea is simple.
The application stores the large payload in Azure Blob Storage or Azure Data Lake Storage. It then sends a small Event Hubs message containing the location, identifier, version and basic processing instructions.
{
"eventType": "CustomerDocumentReady",
"documentId": "DOC-10482",
"storagePath": "customer-documents/DOC-10482.json",
"contentType": "application/json",
"createdUtc": "2026-08-30T09:30:00Z",
"correlationId": "PROCESS-7281"
}Consumers that need the document retrieve it from storage. Consumers that only need to update a dashboard, create an alert or record an audit entry can use the small event without downloading the file.
Access should be controlled through managed identities, which allow Azure services to authenticate without passwords stored in code. Avoid placing permanent storage credentials or long-lived access links inside the event.
You also need rules for retention, deletion, versioning and failed processing. Otherwise, storage can gradually fill with payloads that are no longer required.
When breaking the event into smaller events makes sense
Sometimes the payload is large because it combines several business events into one object. For example, an application may publish an entire customer record whenever only an address, account status or preference has changed.
Publishing smaller events such as CustomerAddressChanged or AccountStatusUpdated reduces data movement and allows each consumer to respond only to relevant changes.
This is particularly important in AI systems. An AI agent should not send its complete conversation history, retrieved documents and generated files through Event Hubs every time it hands work to another agent.
Send the task status, references and approved output instead. Our guides to hub-and-spoke architecture for AI agents and monitoring agent-to-agent communication in Azure explain how this structure improves control and troubleshooting.
Why message chunking should be a last resort
Splitting a 10 MB payload into smaller pieces can get around a size limit, but it creates new problems. Chunks may arrive late, be processed twice or be read in the wrong order if partitioning is not handled carefully.
The receiving application must track every piece, rebuild the original payload and decide what to do when one piece is missing. This adds code, support effort and failure points.
Chunking can be appropriate when there is no separate storage option and strict streaming is required. For most business documents and AI payloads, the claim-check pattern is simpler and more reliable.
A practical business scenario
Consider a 200-person professional services company sending completed case files through Event Hubs. Each file contains metadata, notes, attachments and an AI-generated summary, producing messages between 2 MB and 8 MB.
Upgrading infrastructure would allow the messages to pass, but every reporting, compliance and notification consumer would still receive the full file. Retries would remain slow, and sensitive data would continue travelling to systems that did not need it.
A redesigned model stores each case file securely in Blob Storage and publishes a small CaseFileCompleted event. Reporting receives the case number and completion time, while the compliance application retrieves the full file using its own controlled identity.
The business outcome is lower data movement, faster processing, simpler retries and tighter control over sensitive information. It also becomes easier to prove who accessed each document during an audit.
Five questions to ask before changing your Event Hubs tier
- Is the payload genuinely an event, or is it a file being transported through an event stream?
- Do all consumers need the complete payload?
- What happens to processing time and cost when the message is retried?
- Are we considering Dedicated because of overall scale, or only because one message exceeds 1 MB?
- Could Blob Storage, Data Lake Storage or smaller business events achieve the same outcome more safely?
Also test with real production-sized data. Compression can help in some cases, but it should not be the only plan because compression rates vary and decompression adds processing work.
Design for the business outcome
Large-message support in Azure Event Hubs is useful when events cannot reasonably be divided or stored elsewhere. It should be treated as an informed architecture choice, not an automatic response to a failed publication.
For Australian organisations, separating large payloads from events can also strengthen access control, monitoring and incident investigation. These measures support broader security and compliance programs alongside Essential Eight, the Australian Government’s recommended cybersecurity framework.
CloudProInc brings more than 20 years of enterprise IT experience to these decisions. As a Melbourne-based Microsoft Partner and Wiz Security Integrator, we help organisations examine Azure integrations across performance, security, support effort and long-term cost rather than treating each limit as an isolated technical problem.
If large Event Hubs messages are causing failures or pushing you towards an expensive tier upgrade, we are happy to review the design and explain the practical options. No strings attached.
Discover more from CPI Consulting
Subscribe to get the latest posts sent to your email.