I'm new to GCS. I'm trying to list the files in a "folder" & download them, however my list is return the folder name as index 0.
How do I only list the files name and not the name of the directory? Or should I just shift() the array after I get the list back?
const prefix = 'testFiles/'; //contains jpgs
const delimiter = '/';
const options = {
prefix: prefix,
};
if (delimiter) {
options.delimiter = delimiter;
}
const [files] = await storage.bucket(bucketName).getFiles();
files.forEach((file, i) => {
options.destination = `./downloads/${i}.jpg`
console.log(options);
file.download(options)
});
}
Google Cloud Storage does not have real "folders". Instead, they are emulated using the prefix and delimiter parameters to the List Objects method.
The way you're doing it is correct. But when you list with a prefix, all objects that start with that prefix are returned, up to the delmiter, if specified. That means, like you found, that if have a "directory placeholder" object (an object that ends in /), it will be returned. Note that it also means any "subdirectories" of that prefix will also be returned. For example, if you have the following list of objects in a bucket:
testFiles/
testFiles/a
testFiles/b
testFiles/c/d
testFiles/v
If you call list objects with prefix=testFiles/ and delimiter=/, you'll have the following returned:
items =
testFiles/
testFiles/a
testFiles/b
testFiles/v
prefixes =
testFiles/c/
This is because testFiles/ is actually an object in your bucket in GCS.
Related
I am using Cloudinary to host my media on the cloud for my NodeJS project.
To delete an image from the Clodinary Cloud, I need to pass a Public Id for that image, to the Cloudinary API.
I realised, Public ID is embedded into the url, how to I extract it out from the URL?
Because, I don't want to store my data in this format :
image : {
url : `http://res.cloudinary.com/cloud_name/image/upload/v1647610701/rsorl4rtziefw46fllvh.png`,
publicId : `rsorl4rtziefw46fllvh`
}
Rather, I find it better to store it like this :
image : `http://res.cloudinary.com/cloud_name/image/upload/v1647610701/rsorl4rtziefw46fllvh.png`
The solution to this problem is to implement a funciton which extracts the publicId for every URL passed in as argument.
Here's the function :
const getPublicId = (imageURL) => imageURL.split("/").pop().split(".")[0];
Edited after #loic-vdb 's suggestion
Explanation :
It splits the string in an array using "/" as seperator.
imageURL="http://res.cloudinary.com/cloud_name/image/upload/v1647610701/rsorl4rtziefw46fllvh.png";
becomes,
imageURL = [ 'http:',
'',
'res.cloudinary.com',
'cloud_name',
'image',
'upload',
'v1647610701',
'rsorl4rtziefw46fllvh.png' ]
Next, pop the array (returns the last element of the array)
imageURL = 'rsorl4rtziefw46fllvh.png';
Now, split this string into array using "." as seperator, we get :
imageURL = [ 'rsorl4rtziefw46fllvh', 'png' ]
Finally select the 0th element that is our PublicId return that
imageURL = 'rsorl4rtziefw46fllvh';
Based on the answer by a Cloudinary support team member
... the public_id contains all folders and the last part of the public_id is the filename.
Here is what I tried and worked
const path = require("path");
const getPublicId = (imageURL) => {
const [, publicIdWithExtensionName] = imageURL.split("upload/");
const extensionName = path.extname(publicIdWithExtensionName)
const publicId = publicIdWithExtensionName.replace(extensionName, "")
return publicId
};
especially for cases where you store your assets in folders
I want file names produced using drive.files.list to use as strings for downloading all files in a folder.
I am having trouble accessing an array of filenames. Basically through lack of knowledge, So I am lost by the logic of the mouse over reference of map in files.map() See below code.
My code:
if (files.length) {
// map method calls the callbackfn one time for each element in the array
files.map((file) => {
// there are x elements (a filename/id) called x times
// I want to access one filename at a time. Return a plain string filename for a downloadFile() function
var names = [];
// rough test to produce desired output. Produces 'undefined' for array indexes greater than 0 e.g names[1]
// files.length = 1;
// Without the files.length = 1; filename is outputted x times
var names = [];
files.forEach(function (file, i) {
names[i]= file.name;
})
// first array index (with files.length =1;) first filename and only this filename. Correct result!!!
console.log(names[0]);
I don't have much experience with OOP or Nodejs. But using various tests (code changes) the largest output looked like an array of arrays. I want to narrow it down to an array of filenames that I can JSON.stringify before using for downloading.
Mouse over of 'map'
(method) Array<drive_v3.Schema$File>.map<void>(callbackfn: (value: drive_v3.Schema$File, index: number, array: drive_v3.Schema$File[]) => void, thisArg?: any): void[]
Calls a defined callback function on each element of an array, and returns an array that contains the results.
#param callbackfn — A function that accepts up to three arguments. The map method calls the callbackfn function one time for each element in the array.
#param thisArg — An object to which the this keyword can refer in the callbackfn function. If thisArg is omitted, undefined is used as the this value.
Any suggestions apprecieated.
Try replacing the code with this one:
if (files.length) {
var names = [];
files.forEach(function(file, i) {
names.push(file.name);
});
console.log('filenames: ', names);
}
and then you can read the filenames by looping through names array, and pass it to other functions for processing:
names.forEach((fileName, i)=>{
console.log('filename: ', fileName, ' index', i);
// pass the name to other functions for processing
download(fileName);
});
I have
// file cars.js
var bodyshop = require('./bodyshop')
var connections = [];
many functions which operate on connections. adding them, changing them etc.
code in this file includes things like
bodyshop.meld(blah)
bodyshop.mix(blah)
exports.connections = connections
and then
// file bodyshop.js
let cars = require('./cars');
even more functions which operate on connections. adding them, changing them etc.
code in this file includes things like
cars.connections[3].color = pink
cars.connections.splice(deleteMe, 1)
module.exports = { meld, mix, flatten }
Is it absolutely honestly the case that code in bodyshop such as cars.connections.splice(deleteMe, 1) will indeed delete an item from "the" connections (ie, the one and only connections, declared in cars.js) and code in bodyshop such as cars.connections[3].color = pink will indeed change the color of index 3 of "the" self-same one and only connections?
Is it quite OK / safe / acceptable that I used the syntax "module.exports = { }" at the end of bodyshop, rather than three lines like "exports.meld = meld" ?
Is this sentence indeed to totally correct?? "In Node.js if you export from M an array, when using the array in another module X which requires M, the array will be by reference in X, i.e. not by copy" ... ?
I created two files with the following methods and the array as you mentioned.
First File: test1.js
const testArray = [];
const getArray = () => {
return testArray;
};
module.exports = {
testArray,
getArray
}
Second File: test2.js
const { testArray, getArray } = require('./test1');
console.log('testing the required array before modifying it');
console.log(getArray());
testArray.push('test');
console.log('testing the method result after modifying the required array content');
console.log(getArray());
If you can create the mentioned files and run them locally, you will see the following result.
>node test2.js
testing the required array before modifying it
[]
testing the method result after modifying the required array content
[ 'test' ]
The points observed is,
yes, it's okay if you want to export it with the syntax module.exports = { }, It not much an issue.
If any of the methods modify this array outside of the required file, it will affect here as well, This because require will be a reference, not a copy.
The one possible solution will be creating a JSON copy of it while requiring as below:
const { testArray, getArray } = require('./test1');
const testArrayCopy = JSON.parse(JSON.stringify(testArray));
console.log('testing the required array before modifying it');
console.log(getArray());
testArrayCopy.push('test');
console.log('testing the method result after modifying the required array content');
console.log(getArray());
This is the result:
>node test2.js
testing the required array before modifying it
[]
testing the method result after modifying the required array content
[]
Note: JSON copy will not help you in parsing DateTime properly.
When I am doing the search, the results always contain only five records (in ascending order). I need to get a list of all files in the folder. What am I doing wrong?
FileSearch fileSearch = new FileSearch();
FileSearchBasic fileSearchBasic = new FileSearchBasic();
// Specify the folder in which the search is to be done.
SearchMultiSelectField folderFilter = new SearchMultiSelectField
{
#operator = SearchMultiSelectFieldOperator.anyOf,
operatorSpecified = true
};
RecordRef[] folder = new RecordRef[1];
folder[0] = new RecordRef
{
// Internal id of the folder where pospay files are stored.
internalId = ns.DataCollection["netSuite:PositivePayFolderInternalId"]
};
folderFilter.searchValue = folder;
fileSearchBasic.folder = folderFilter;
fileSearch.basic = fileSearchBasic;
var result = NSBase.Client.Service.search(fileSearch);
var recordList = result.recordList; // this has ony five results, why?
I found the issue, 'PageSize' was set to 5 in the service client's constructor.
My code delete just files which parse to Jenkins name in the file. I would like to delete file based on the author (Jenkins) in the last commit. What is the best solution for that?
def changelogPath = "C:\\test"
def PackID = "test"
def delete(String changelogPath, String PackID) {
String folderPath = "$changelogPath"+ "\\" + "$PackID"
new File(folderPath).eachFile(FileType.FILES) { file ->
if (file.name.contains('Jenkins')) file.delete()
}
delete(changelogPath, PackID)
In order to find all files that have been changed with a certain commit, you need a diff of that commit with its predecessor.
You can let JGit compute a list of DiffEntries like this:
ObjectReader reader = git.getRepository().newObjectReader();
CanonicalTreeParser oldTreeIter = new CanonicalTreeParser();
ObjectId oldTree = git.getRepository().resolve( "HEAD^{tree}" );
oldTreeIter.reset( reader, oldTree );
CanonicalTreeParser newTreeIter = new CanonicalTreeParser();
ObjectId newTree = git.getRepository().resolve( "HEAD~1^{tree}" );
newTreeIter.reset( reader, newTree );
DiffFormatter df = new DiffFormatter( new ByteArrayOutputStream() );
df.setRepository( git.getRepository() );
List<DiffEntry> entries = df.scan( oldTreeIter, newTreeIter );
Each DiffEntry has a path that denotes the file which was added, changed, or deleted. The path is relative to the root of the working directory of the repository. Actually, there is an oldPath and newPath, see the JavaDoc which one to use when.
See also here for a general overview of JGit's diff API: http://www.codeaffine.com/2016/06/16/jgit-diff/