Azure Storage Blob - Consumer Examples
Consumer Examples
To consume a blob into a file using the file component, this can be done like this:
-
Java
-
XML
-
YAML
from("azure-storage-blob://camelazure/container1?blobName=hello.txt&accountName=yourAccountName&accessKey=yourAccessKey")
.to("file://blobdirectory"); <route>
<from uri="azure-storage-blob://camelazure/container1?blobName=hello.txt&accountName=yourAccountName&accessKey=yourAccessKey"/>
<to uri="file://blobdirectory"/>
</route> - route:
from:
uri: azure-storage-blob://camelazure/container1
parameters:
blobName: hello.txt
accountName: yourAccountName
accessKey: yourAccessKey
steps:
- to:
uri: file://blobdirectory However, you can also write to file directly without using the file component, you will need to specify fileDir folder path to save your blob in your machine.
-
Java
-
XML
-
YAML
from("azure-storage-blob://camelazure/container1?blobName=hello.txt&accountName=yourAccountName&accessKey=yourAccessKey&fileDir=/var/to/awesome/dir")
.to("mock:results"); <route>
<from uri="azure-storage-blob://camelazure/container1?blobName=hello.txt&accountName=yourAccountName&accessKey=yourAccessKey&fileDir=/var/to/awesome/dir"/>
<to uri="mock:results"/>
</route> - route:
from:
uri: azure-storage-blob://camelazure/container1
parameters:
blobName: hello.txt
accountName: yourAccountName
accessKey: yourAccessKey
fileDir: /var/to/awesome/dir
steps:
- to:
uri: mock:results Also, the component supports batch consumer, hence you can consume multiple blobs with only specifying the container name, the consumer will return multiple exchanges depending on the number of the blobs in the container. Example:
-
Java
-
XML
-
YAML
from("azure-storage-blob://camelazure/container1?accountName=yourAccountName&accessKey=yourAccessKey&fileDir=/var/to/awesome/dir")
.to("mock:results"); <route>
<from uri="azure-storage-blob://camelazure/container1?accountName=yourAccountName&accessKey=yourAccessKey&fileDir=/var/to/awesome/dir"/>
<to uri="mock:results"/>
</route> - route:
from:
uri: azure-storage-blob://camelazure/container1
parameters:
accountName: yourAccountName
accessKey: yourAccessKey
fileDir: /var/to/awesome/dir
steps:
- to:
uri: mock:results Limiting the Number of Blobs per Poll
The consumer supports the maxMessagesPerPoll option to limit how many blobs are retrieved in one polling cycle:
-
Java
-
YAML
from("azure-storage-blob://camelazure/container1?maxMessagesPerPoll=10&accessKey=RAW(yourAccessKey)")
.to("direct:processBlob"); - route:
from:
uri: azure-storage-blob://camelazure/container1
parameters:
maxMessagesPerPoll: 10
accessKey: "RAW(yourAccessKey)"
steps:
- to:
uri: direct:processBlob The default value is 0, which means unlimited. A positive value limits the number of successfully retrieved blobs in each poll. Blobs skipped because they are already in progress, no longer exist, or cannot be retrieved do not count toward the limit.
The consumer lists blobs page by page and retains unprocessed names from the current page for subsequent polls. The Azure continuation token advances across pages, and the listing wraps to the beginning after the last page. maxResultsPerPage independently controls the number of items in each Azure listing page.
The consumer prepares the complete batch before routing its first exchange. This bounded prefetch preserves the exact Camel CamelBatchSize, CamelBatchIndex and CamelBatchComplete properties. It is not strict just-in-time retrieval immediately before each route invocation. |
Delete After Read
The consumer supports automatic deletion of blobs after they have been successfully processed. This is useful when you want to ensure that blobs are only processed once.
-
Java
-
XML
-
YAML
from("azure-storage-blob://camelazure/container1?deleteAfterRead=true&accessKey=RAW(yourAccessKey)")
.log("Processing blob: ${header.CamelAzureStorageBlobBlobName}")
.to("direct:processBlob"); <route>
<from uri="azure-storage-blob://camelazure/container1?deleteAfterRead=true&accessKey=RAW(yourAccessKey)"/>
<log message="Processing blob: ${header.CamelAzureStorageBlobBlobName}"/>
<to uri="direct:processBlob"/>
</route> - route:
from:
uri: azure-storage-blob://camelazure/container1
parameters:
deleteAfterRead: true
accessKey: "RAW(yourAccessKey)"
steps:
- log:
message: "Processing blob: ${header.CamelAzureStorageBlobBlobName}"
- to:
uri: direct:processBlob | The delete operation is only performed if the Exchange is successfully committed. If processing fails or a rollback occurs, the blob will not be deleted and can be reprocessed on the next poll. |
When deleteAfterRead is set to false (the default), the same blobs will be retrieved repeatedly in subsequent polls. The Idempotent Consumer EIP can filter duplicate exchanges based on the CamelAzureStorageBlobBlobName header, but it runs after the consumer has retrieved the blob and therefore does not prevent the Azure read or download itself. Use maxMessagesPerPoll to bound the number of blobs retrieved during each poll. |
A blob is not retrieved again while the exchange of a previous poll is still processing it, for example when the route continues asynchronously (such as to("seda:…")). It is retrieved again by a later poll once that exchange is done, unless it was deleted or moved. A blob that was deleted (for example by another consumer) between the listing and its download is skipped. |
Move After Read
The consumer can move blobs to a different container after successful processing. This is useful for archiving processed blobs or implementing a processing pipeline across containers.
-
Java
-
XML
-
YAML
from("azure-storage-blob://camelazure/incoming?moveAfterRead=true&destinationContainer=archive&accessKey=RAW(yourAccessKey)")
.log("Processing blob: ${header.CamelAzureStorageBlobBlobName}")
.to("direct:processBlob"); <route>
<from uri="azure-storage-blob://camelazure/incoming?moveAfterRead=true&destinationContainer=archive&accessKey=RAW(yourAccessKey)"/>
<log message="Processing blob: ${header.CamelAzureStorageBlobBlobName}"/>
<to uri="direct:processBlob"/>
</route> - route:
from:
uri: azure-storage-blob://camelazure/incoming
parameters:
moveAfterRead: true
destinationContainer: archive
accessKey: "RAW(yourAccessKey)"
steps:
- log:
message: "Processing blob: ${header.CamelAzureStorageBlobBlobName}"
- to:
uri: direct:processBlob You can also customize the destination blob name by adding a prefix and/or suffix:
-
Java
-
XML
-
YAML
from("azure-storage-blob://camelazure/source?moveAfterRead=true&destinationContainer=archive&prefix=incoming/&removePrefixOnMove=true&destinationBlobPrefix=processed/&destinationBlobSuffix=.done&accessKey=RAW(yourAccessKey)")
.log("Processing: ${header.CamelAzureStorageBlobBlobName}")
.to("direct:processBlob"); <route>
<from uri="azure-storage-blob://camelazure/source?moveAfterRead=true&destinationContainer=archive&prefix=incoming/&removePrefixOnMove=true&destinationBlobPrefix=processed/&destinationBlobSuffix=.done&accessKey=RAW(yourAccessKey)"/>
<log message="Processing: ${header.CamelAzureStorageBlobBlobName}"/>
<to uri="direct:processBlob"/>
</route> - route:
from:
uri: azure-storage-blob://camelazure/source
parameters:
moveAfterRead: true
destinationContainer: archive
prefix: "incoming/"
removePrefixOnMove: true
destinationBlobPrefix: "processed/"
destinationBlobSuffix: ".done"
accessKey: "RAW(yourAccessKey)"
steps:
- log:
message: "Processing: ${header.CamelAzureStorageBlobBlobName}"
- to:
uri: direct:processBlob The move operation works as follows:
-
The blob is copied to the destination container with the new name
-
The original blob is deleted from the source container
-
Both operations only occur after the Exchange is successfully committed
The following options control the move behavior:
| Option | Description |
|---|---|
| Enable moving blobs after successful processing |
| Target container for moved blobs (required when |
| Prefix to add to the blob name in the destination |
| Suffix to add to the blob name in the destination |
| Remove the source prefix from the blob name before adding destination prefix |