In the software that I am programming, I am attempting to create a virtual file system over the blobs structure of Azure.
Many times in the process, I get a path from the system and I need to tell whether the path is of a Blob or just a virtual BlobDirectory that azure provides. I did this by casting it from one form to another and handling the error.
But now, if I know that a path points to a virtual directory, how can I check whether this virtual directory exists or not?
I can get the reference to the CloudBlobDirectory with the following code:
var blobDirectory = client.GetBlobDirectoryReference("Path_to_dir");
In blob storage, directories don't exist as an item by themselves. What you can have is a blob that has a name that can be interpreted as being in a directory. If you look at the underlying REST API you'll see that that there's nothing in there about directories. What the storage client library is doing for you is searching for blobs that start with the directory name then the delimiter e.g. "DirectoryA/DirectoryB/FileName.txt". What this means is that for a directory to exist it must contain a blob. To check if the directory exists you can try either:
var blobDirectory = client.GetBlobDirectoryReference("Path_to_dir");
bool directoryExists = blobDirectory.ListBlobs().Count() > 0
or
bool directoryExists = client.ListBlobsWithPrefix("DirectoryA/DirectoryB/").Count() > 0
I'm aware that listing everything in the directory just to get the count isn't that great an idea, I'm sure you can come up with a better method.
Not sure if you can use GetAttributes method and if it raised exception then means no directory exist. I used the similar approach to verify if an blob exist, but didn't tested on a directory yet.
For Java
this can be used:
container.getDirectoryReference(directoryName).listBlobs().iterator().hasNext() == true
means directory exists else no directory exists.
val storageAccountString: String = s"BlobEndpoint=https://$account.$endpoint;SharedAccessSignature=$SASTOKEN"
val client: CloudBlobContainer = CloudStorageAccountparse(storageAccountString).createCloudBlobClient().getContainerReference(container)
val blobPattern: String = s"wasbs://$containerName#$accountName.blob.core.windows.net/$dirPath"
client
.getBlockBlobReference(blobPattern)
.exists
.booleanValue
Related
Is this function not implemented in the java sdk? It appears to always return null. I am copy one page blob to another and want to track the status of the copy.
CloudPageBlob srcBlob = container.getPageBlobReference("source.vhd";
String newname="dst.vhd";
CloudPageBlob dstBlob = container.getPageBlobReference(newname);
dstBlob.startCopyFromBlob(srcBlob);
//Get the blob again for updated state
dstBlob = container.getPageBlobReference(newname);
CopyState state = dstBlob.getCopyState();
Is there any other way to get status? I am using azure-storage-1.2.0.jar
getPageBlobReference() is purely a local operation, it does not communicate with the Azure Storage service. You need to call dstBlob.downloadAttributes() in between calling getPageBlobReference() and getCopyState(). This will make the service call that will populate the blob's properties, including the copy state.
I have a continuous Azure WebJob that is running off of a QueueInput, generating a report, and outputting a file to a BlobOutput. This job will run for differing sets of data, each requiring a unique output file. (The number of inputs is guaranteed to scale significantly over time, so I cannot write a single job per input.) I would like to be able to run this off of a QueueInput, but I cannot find a way to set the output based on the QueueInput value, or any value except for a blob input name.
As an example, this is basically what I want to do, though it is invalid code and will fail.
public static void Job([QueueInput("inputqueue")] InputItem input, [BlobOutput("fileoutput/{input.Name}")] Stream output)
{
//job work here
}
I know I could do something similar if I used BlobInput instead of QueueInput, but I would prefer to use a queue for this job. Am I missing something or is generating a unique output from a QueueInput just not possible?
There are two alternatives:
Use IBInder to generate the blob name. Like shown in these samples
Have an autogenerated in the queue message object and bind the blob name to that property. See here (the BlobNameFromQueueMessage method) how to bind a queue message property to a blob name
Found the solution at Advanced bindings with the Windows Azure Web Jobs SDK via Curah's Complete List of Web Jobs Tutorials and Videos.
Quote for posterity:
One approach is to use the IBinder interface to bind the output blob and specify the name that equals the order id. The better and simpler approach (SimpleBatch) is to bind the blob name placeholder to the queue message properties:
public static void ProcessOrder(
[QueueInput("orders")] Order newOrder,
[BlobOutput("invoices/{OrderId}")] TextWriter invoice)
{
// Code that creates the invoice
}
The {OrderId} placeholder from the blob name gets its value from the OrderId property of the newOrder object. For example, newOrder is (JSON): {"CustomerName":"Victor","OrderId":"abc42"} then the output blob name is “invoices/abc42″. The placeholder is case-sensitive.
So, you can reference individual properties from the QueueInput object in the BlobOutput string and they will be populated correctly.
I am storing image files as blobs in Azure Storage with the following naming convention:
directory/image-name
When trying to retrieve the blobs using BlobService.listBlobs(container, options, callback) in Javascript on the server, I use:
var options = { "prefix":directory }
and it gets back only blobs that start with the directory name, as I expect, but I thought I would also be able to use:
var options = { "delimiter":"/", "prefix":directory }
and get back the same blobs, perhaps without the prefix in their names. Instead I get back nothing at all. What is the correct way to use the delimiter? What's the point in having it if you get the items that you want with only using the prefix?
I've not used the REST APIs from JavaScript, but I think what you are missing is a trailing slash after the directory name, so I suggest:
var options = { "delimiter":"/", "prefix":directory+"/" }
Windows Azure Storage doesn't really have directories, in the underlying implementation all the blobs in a container are just flat list, and blob names (not container names) may contain slashes. The delimiter is an option when calling the ListBlobs REST API that allows you to simulate directory-like behavior. If the delimiter option is enabled, and the part of the blob name past the prefix contains the delimiter, the reply will omit that blob.
To illustrate, lets name some blobs, assuming all of them in the same container https://myaccount.blob.core.windows.net/mycontainer":
a/b/extra.txt
a/bloba.txt
a/blobb.txt
other.txt
So then if you invoke listBlobs on that container with the prefix "a/" and without specifying the delimiter, it will return the first three names, because they all have the "a/" prefix.
If instead you invoke listBlobs with the same "a/" prefix and set the delimiter to "/", you only get the middle two names; the service leaves out a/b/extra.txt because it's in a (simulated) sub-directory "b".
The easy way to add or update files in JGit is like this:
git.add().addFilepattern(file).call()
But that assumes that the file exists in the Git working directory.
If I have a multi-threaded setup (using Scala and Akka), is there a way to work only on a bare repository, writing the data directly to JGit, avoiding having to first write the file in the working directory?
For getting the file, that seems to work with:
git.getRepository().open(objId).getBytes()
Is there something similar for adding or updating files?
"Add" is a high-level abstraction that places a file in the index. In a bare repository, you lack an index, so this is not a 1:1 correspondence between the functionality. Instead, you can create a file in a new commit. To do this, you would use an ObjectInserter to add objects to the repository (one per thread, please). Then you would:
Add the contents of the file to the repository, as a blob, by inserting its bytes (or providing an InputStream).
Create a tree that includes the new file, by using a TreeFormatter.
Create a commit that points to the tree, by using a CommitBuilder.
For example, to create a new commit (with no parents) that contains only your file:
ObjectInserter repoInserter = repository.newObjectInserter();
ObjectId blobId;
try
{
// Add a blob to the repository
ObjectId blobId = repoInserter.insert(OBJ_BLOB, "Hello World!\n".getBytes());
// Create a tree that contains the blob as file "hello.txt"
TreeFormatter treeFormatter = new TreeFormatter();
treeFormatter.append("hello.txt", FileMode.TYPE_FILE, blobId);
ObjectId treeId = treeFormatter.insertTo(repoInserter);
// Create a commit that contains this tree
CommitBuilder commit = new CommitBuilder();
PersonIdent ident = new PersonIdent("Me", "me#example.com");
commit.setCommitter(ident);
commit.setAuthor(ident);
commit.setMessage("This is a new commit!");
commit.setTreeId(treeId);
ObjectId commitId = repositoryInserter.insert(commit);
repoInserter.flush();
}
finally
{
repoInserter.release();
}
Now you can git checkout the commit id returned as commitId.
I am trying to port my LWUIT application to Codename one.
I have used RMS in LWUIT and now obviously I have to transform this to Storage.
I don't understand how the Storage class works in Codename one and the documentation for codename one has nothing about either.
1) What is the structure of a storage file?
--> In J2ME RecordStore , you have records bunched together like a table. Every row, corresponds to a record. Each record has a unique record ID and you can access the record with this record id. Every record can have some data stored in it.
How does this map to Storage class?
2)I wish to store some records in my storage, how do i do it?
The documentation says:
static Storage getInstance()
Returns the storage instance or null if the storage wasn't initialized using a call to init(String) first.
--> In LWUIT it was something like Storage.init(storageName). ; However there is no init in codename one!!!. How do I open a Storage in Codename one??
3)If i try to open a storage file which does not exist, what will happen (RMS gives an exception)?
The easiest way to think about Storage is as a flat file system (without directories/folders).
When running on top of RMS this file system abstraction is mapped to the RMS database seamlessly for you.
Notice that init() for Storage in Codename One is no longer necessary, under LWUIT it only performed basic initialization and the name was usually ignored.
The Storage class has several methods:
InputStream createInputStream(String name)
Creates an input stream to the given storage source file
OutputStream createOutputStream(String name)
Creates an output stream to the storage with the given name
boolean exists(String name)
Returns true if the given storage file exists
String[] listEntries()
Lists the names of the storage files
You can use these to just store and check if data exists. However you can also store complex objects in storage without using input/output streams by using these two methods:
Object readObject(String name)
Reads the object from the storage, returns null if the object isn't there
boolean writeObject(String name, Object o)
Writes the given object to storage assuming it is an externalizable type or one of the supported types
So to simulate something like byte[] storage you can do something like this:
Vector p = new Vector();
byte[] myData = ...;
p.addElement(myData);
p.addElement(additionalData);
Storage.getInstance().writeObject("myStore", p);
Then just read it as:
Vector p = (Vector)Storage.getInstance().read("myStore");
// p will be null if nothing was written