Use Python to create multiple containers with specific content - python-3.x

Question

I have a single container with around 200k images on my blob storage. I want to write a script in Python that copies out batches of 20k of these images to new containers called something like imageset1, imageset2, ..., imageset20 (the last container will have less than 20k images in it which is fine).
I have the following so far:
from azure.storage.blob import BlockBlobService
from io import BytesIO from shutil
import copyfileobj
with BytesIO() as input_blob:
with BytesIO() as output_blob:
block_blob_service = BlockBlobService(account_name='my_account_name', account_key='my_account_key')
# Download as a stream
block_blob_service.get_blob_to_stream('mycontainer', 'myinputfilename', input_blob)
# Here is where I want to chunk up the container contents into batches of 20k
# Then I want to write the above to a set of new containers using, I think, something like this...
block_blob_service.create_blob_from_stream('mycontainer', 'myoutputfilename', output_blob)
It's the chunking up the contents of a container and writing the results out to new containers which I don't know how to do. Can anyone help?

Answer 1

here is my sample code to realize your needs, and it works on my container.
from azure.storage.blob.baseblobservice import BaseBlobService
account_name = '<your account name>'
account_key = '<your account key>'
container_name = '<the source container name>'
blob_service = BaseBlobService(
account_name=account_name,
account_key=account_key
)
blobs = blob_service.list_blobs(container_name)
# The target container index starts with 1
container_index = 1
# The blob number in new container, such as 3 in my testing
num_per_container = 3
count = 0
# The prefix of new container name
prefix_of_new_container = 'imageset'
flag_of_new_container = False
for blob in blobs:
if flag_of_new_container == False:
flag_of_new_container = blob_service.create_container("%s%d" % (prefix_of_new_container, container_index))
print(blob.name, "%s%d" % (prefix_of_new_container,container_index))
blob_service.copy_blob("%s%d" % (prefix_of_new_container, container_index), blob.name, "https://%s.blob.core.windows.net/%s/%s" % (account_name, container_name, blob.name))
count += 1
if count == num_per_container:
container_index += 1
count = 0
flag_of_new_container = False
Note: I only use BaseBlobService because it's enough for your needs, even for AppendBlob or PageBlob. Also, you can use BlockBlobService instead of it.