I would like to know how to grant a Google Cloud Platform App Engine project permissions to serve content from Google Cloud Storage without setting the Google Cloud Storage bucket permissions to ‘share publicly'.
My App engine project is running Node JS. Uses Passport-SAML authentication to authenticate users before allowing them to view content, hence I do not want to set access on an individual user level via IAM. Images and videos are currently served from within a private folder of my app, which is only accessible once users are authenticated. I wish to move these assets to Google Cloud Storage and allow the app to read the files, whist not providing global access. How should I go about doing this? I failed to find any documentation on it.
I think this might work for you https://cloud.google.com/storage/docs/access-control/create-signed-urls-program
I can't seem to find the API doc for nodejs (google is really messing around with their doc urls). Here's some sample code:
bucket.upload(filename, options, function(err, file, apiResponse) {
var mil = Date.now()+60000;
var config = {
action: 'read',
expires: mil
};
file.getSignedUrl(config, function(err, url) {
if (err) {
return;
}
console.log(url);
});
As stated in the official documentation:
By default, when you create a bucket for your project, your app has
all the permissions required to read and write to it.
Whenever you create an App Engine application, there is a default bucket that comes with the following perks:
5GB of free storage.
Free quota for Cloud Storage I/O operations
By default it is created automatically with your application, but in any case you can follow the same link I shared previously in other to create the bucket. Should you need more than those 5GB of free storage, you can make it a paid bucket and you will only be charged for the storage that surpasses the first 5 GB.
Then, you can make use of the Cloud Storage Client Libraries for Node.js and have a look at some nice samples (general samples here or even specific operations over files here) for working with the files inside your bucket.
UPDATE:
Here there is a small working example on how to use the Cloud Storage client libraries to retrieve images from your private bucket without making them public, by means of authenticating requests. It works in a Cloud Function, so you should have no issues in reproducing the same behavior in App Engine. It does not perform exactly what you need, as it displays the image in the bucket alone, without any integration inside an HTML file, but you should be able to build something from that (I am not too used to work with Node.js, unfortunately).
I hope this can be of some help too.
'use strict';
const gcs = require('#google-cloud/storage')();
exports.imageServer = function imageSender(req, res) {
let file = gcs.bucket('<YOUR_BUCKET>').file('<YOUR_IMAGE>');
let readStream = file.createReadStream();
res.setHeader("content-type", "image/jpeg");
readStream.pipe(res);
};
Related
I need to copy data from one bucket in Project A in gCloud to another bucket in Project B. Any utility if present in NodeJs to do this?
You might be tempted to get a list of blobs inside the bucket, download them, and upload them to another storage bucket (this is what gsuilt cp -m gs://origin_bucket/** gs://destination_bucket/ does).
The problem with this approach is you will consume CPU (on your side), and will take some time.
If you want to move all data from one bucket to another one, the best way to do this is using the Storage Transfer Service.
With the Storage Transfer Service you just tell the origin and destination buckets, and optionally a schedule... and google will do the operation much faster than you can do it yourself.
Also remember that the source and destination buckets can be GCS buckets, S3, or Azure Storage buckets, too.
Take a look at google-provided Node Sample Code for Storage Transfer Service.
If you just want to transfer some files, the Storage Transfer Service has a (in beta as Feb 2022) feature that allows you to specify a manifest file (a CSV file stored in a gcs bucket). See: Transfer specific files or objects using a Manifest
You can use the Cloud Storage Client Libraries. To copy an object in one of your Cloud Storage buckets, see this sample code:
const srcBucketName = 'your-source-bucket';
const srcFilename = 'your-file-name';
const destBucketName = 'target-file-bucket';
const destFileName = 'target-file-name';
// Imports the Google Cloud client library
const {Storage} = require('#google-cloud/storage');
// Creates a client
const storage = new Storage();
async function copyFile() {
// Copies the file to the other bucket
await storage
.bucket(srcBucketName)
.file(srcFilename)
.copy(storage.bucket(destBucketName).file(destFileName));
console.log(
`gs://${srcBucketName}/${srcFilename} copied to gs://${destBucketName}/${destFileName}`
);
}
copyFile().catch(console.error);
Additionally, you need to ensure that you have been assigned a role with the necessary permissions from the other project.
When I used a bucket a key file was downloaded and it said keep this file safe ?
now I Cannot use .env to encrypt because in the following code you have to link the json file directly to gain access to GCS bucket.
const {Storage} = require('#google-cloud/storage');
const storage = new Storage({
keyFilename:path.join(__dirname,'/<keyfilename>.json'),
projectId:'<project ID>'
});
Now I am concerned when i deploy my app on the app engine this file may be accessed by someone somehow
that is a serious threat because it gives direct access to my GCS bucket
Should I be concerned about that file being accessed by anyone??
Instead of using the Service Account JSON file in AppEngine, You can use the App Engine default service. account to access the GCS buckets or any other service in GCP. By default, the App Engine default service account has the Editor role in the project, Any user account with sufficient permissions to deploy changes to the Cloud project can also run code with read/write access to all resources within that project. However, you can change the service account permissions through the Console.
Open the Cloud Console.
In the Members list, locate the ID of the App Engine default
service account.
The App Engine default service account uses the member ID:
YOUR_PROJECT_ID#appspot.gserviceaccount.com
Use the dropdown menu to modify the roles assigned to the service
account.
I have created some functions that upload and download files from firebase storage with cloud functions using the Firebase SDK and they work.
I would like for the functions to be executed as admin so that they don't need to abide by the storage rules.
I have replaced the firebase SDK with the admin SDK but I found out that my firebase.storage().ref reference doesn't work anymore and by reading around some docs I have realized I now need to use the google cloud services system instead.
So my question is, is there a way to have a cloud function have administrator powers on the entire firebase project without having to switch to google cloud functions and if not, is there a work around to do that so that I can somehow authorize my cloud function to have full read/write powers on the entire storage? I am puzzled!
Here is a snippet of my code:
const firebase = require('firebase-admin');
const functions = require('firebase-functions');
require("firebase-admin")
require("firebase")
require("firebase/storage");
var serviceAccount = require("serviceAccount.json");
var config = {
[...]
credential: firebase.credential.cert(serviceAccount)
};
firebase.initializeApp(config);
var storage = firebase.storage();
var storageRef = storage.ref(); //This returns .ref() is not a function
The Firebase client libraries are not intended to work on backend server environments. The Firebase Admin SDK is meant for backends, but the API to access Cloud Storage is different than the client SDK. The Admin SDK just wraps the Cloud Storage server SDKs. So for node environments, you are actually just going to use the Cloud Storage Node.js client.
When you call:
const admin = require('firebase-admin')
const storage = admin.storage()
you are getting a Storage object from the node SDK. It doesn't have a ref() method. You will need to get a Bucket object and use that instead:
const bucket = storage.bucket()
From here, you should continue to use the API docs I'm linking to.
I have a collection in firebase which looks something like this:
people:
-KuM2GgA5JdH0Inem6lGaddclose
appliedTo: "-KuM1IB5TisBtc34y2Bb"
document: "docs/837c2500-9cbe-11e7-8ac1-17a6c37e2057"
name: "Test Testerson"
the document node contains a path to a file in a storage bucket. Is it possible to download this file to the client using an http firebase function. According to Stream files in node/express to client I should be able to stream to response in express. Will the google-storage readstream work for this?
Thanks,
Ben
The Firebase Admin SDK has a Storage object you can use. It gives you an entry point into the Google Cloud Storage SDK which can interact with storage buckets.
const bucket = admin.storage().bucket
Use this Bucket object to upload and download files. You should be able to use a stream to send contents to the client.
We have a Blob storage container in Azure for uploading application specific documents and we have Azure Sql Db where meta data for particular files are saved during the file upload process. This upload process needs to be consistent so that we should not have files in the storage for which there is no record of meta data in Sql Db and vice versa.
We are uploading list of files which we get from front-end as multi-part HttpContent. From Web Api controller we call the upload service passing the httpContent, file names and a folder path where the files will be uploaded. The Web Api controller, service method, repository, all are asyn.
var files = await this.uploadService.UploadFiles(httpContent, fileNames, pathName);
Here is the service method:
public async Task<List<FileUploadModel>> UploadFiles(HttpContent httpContent, List<string> fileNames, string folderPath)
{
var blobUploadProvider = this.Container.Resolve<UploadProvider>(
new DependencyOverride<UploadProviderModel>(new UploadProviderModel(fileNames, folderPath)));
var list = await httpContent.ReadAsMultipartAsync(blobUploadProvider).ContinueWith(
task =>
{
if (task.IsFaulted || task.IsCanceled)
{
throw task.Exception;
}
var provider = task.Result;
return provider.Uploads.ToList();
});
return list;
}
The service method uses a customized upload provider which is derived from System.Net.Http.MultipartFileStreamProvider and we resolve this using a dependency resolver.
After this, we create the meta deta models for each of those files and then save in the Db using Entity framework. The full process works fine in ideal situation.
The problem is if the upload process is successful but somehow the Db operation fails, then we have files uploaded in Blob storage but there is no corresponding entry in Sql Db, and thus there is data inconsistency.
Following are the different technologies used in the system:
Azure Api App
Azure Blob Storage
Web Api
.Net 4.6.1
Entity framework 6.1.3
Azure MSSql Database (we are not using any VM)
I have tried using TransactionScope for consistency which seems not working for Blob and Db, (works for Db only)
How do we solve this issue?
Is there any built in or supported feature for this?
What are the best practices in this case?
Is there any built in or supported feature for this?
As of today no. Essentially Blob Service and SQL Database are two separate services hence it is not possible to implement "atomic transaction" functionality like you're expecting.
How do we solve this issue?
I could think of two ways to solve this issue (I am sure there would be other as well):
Implement your own transaction functionality: Basically check for the database transaction failure and if that happens delete the blob manually.
Use some background process: Here you would continue to save the data in blob storage and then periodically find out orphaned blobs through some background process and delete those blobs.