I am using Active Storage on Rails 5.2.1
I have followed the tutorial and I am using the credentials to set it up in my storage.yml
amazon:
service: S3
access_key_id: <%= Rails.application.credentials.aws_access_key_id! %>
secret_access_key: <%= Rails.application.credentials.aws_secret_key! %>
region: us-east-1
bucket: bucket-on-aws
company.rb:
has_one_attached :image, dependent: :purge_later
development.rb: config.active_storage.service = :amazon
I can upload and view the images. My issues are coming when I delete the company, it is not deleted from S3. The same goes when I update the image. Do I actually have to do all of that myself? I would have expected Active Storage to handle the deletion at the very least....especially with an option named dependent: :purge_later
I did try the actual purging from the console:
c.image.purge does indeed work. But I worry about never being able to purge images that were left behind after updating.
I generally do updates on the entire model:
companies_controller.rb: #company.update(company_params)
What is the rails way to handle these two situations?
Went into the development logs and saw that Sidekiq hadn't been restarted since the changes....meaning it still thought that it was using local storage! Be sure to restart sidekiq or whatever you use for background jobs
Related
So I have a node js web app, this web app has a folder to store images uploaded by users from a mobile app. How I upload the image to the folder is by using the image's base64 string, and using fs.writeFile to save the image to the folder, like this:
fs.writeFile(__dirname + '/../images/complaintImg/complaintcase_' + data.cID + '.jpg', Buffer.from(data.complaintImage, 'base64'), function (err) {
if (err) {
console.log(err);
} else {
console.log("success");
}
});
The problem is, whenever the application is redeployed to google cloud, the images gets deleted. This is because the image folder of the local version of the application is empty - when the user uploads an image, i don't get a local copy of that image.
How do i prevent the images from getting deleted with every deployment? because the app is constantly updated (changes to js or html files), i can't have the images getting deleted with every deployment. How do i update a deployment to only deploy certain files? the gcloud app deploy command seems to deploy the entire project. or should i upload the images directly to google cloud storage?
please help, currently the mobile app isn't released to the public yet, so having the images deleted with every deployment is still not a big problem now, but it will be once it's released to the public. because the images they upload are very important. thank you in advance!
It appears that your __dirname directory you chose may be under /tmp or, if you use the flexible environment, some other directory local to your instance. If so the images will disappear whenever new instances are started (which always happens at new deployment, but it can happen in between deployments as well). This is expected, the instances are always started "from scratch".
You need to store the files that your app creates and you want to survive instance (re)starts on a persistent storage product, like Cloud Storage, see Using Cloud Storage (or Using Cloud Storage for flexible env). Note that you can't use the regular filesystem calls with Cloud Storage, you need to use the documented client library.
As stated in Dan Cornilescu's answer, for user uploaded files, you should store them in Cloud Storage for GAE Standard or for GAE Flexible.
Just as a reference, there is an alternative for those who are using Python 2.7, Java 8 or PHP 5, which is the BlobStore API
I have three environments is AWS dev/uat/prod, and the same branch(develop) I wanted to develop in all three respective environments using bitbucket-pipelines. As I know we need AWS AWS_ACCESS_KEY_ID to do so.
My question is: how to provide AWS AWS_ACCESS_KEY_ID for all the three environments dynamically?,
I am able to deploy at time on one environment as of now.
Thanks for help in advanced
There's a number of client libraries that allow you to parametrize AWS credentials without having to store them in the environment-specific config files. You didn't specify what AWS service you want to use, but here's and example for S3: s3_website
Their config file looks like this; you can configure multiple sets of variables.
s3_id: <%= ENV['S3_ID'] %>
s3_secret: <%= ENV['S3_SECRET'] %>
If this doesn't work for you, write a shell/python script around AWS CLI and pull the environment-specific variables into AWS config file yourself. Manage that script as part of your source code or a docker image.
I am attempting to set a NodeJS Lambda function to be triggered when an image is uploaded to an Amazon S3 bucket. I have seen multiple tutorials and have the yml file set up as shown. Below is the YML config file:
functions:
image-read:
handler: handler.imageRead
events:
- s3:
bucket: <bucket-name-here>
event: s3:ObjectCreated:*
Is there something I am missing for the configuration? Is there something I need to do in an IAM role to set this up properly?
The YAML that you have here looks good but there may be some other problems.
Just to get you started:
are you deploying the function using the right credentials? (I've seen it many times that people are deploying in some other account etc. than they think - verify in the web console that it's there)
can you invoke the function in some other way? (from the serverless command line, using http trigger etc.)
do you see anything in the logs of that function? (add console.log statements to see if anything is being run)
do you see the trigger installed in the web console?
can you add trigger manually on the web console?
Try to add a simple function that would only print some logs when it is run and try to add a trigger for that function manually. If it works then try to do the same with the serverless command line but start with a simple function with just one log statement and if it works then go from there.
See also this post for more hints - S3 trigger is not registered after deployment:
https://forum.serverless.com/t/s3-trigger-is-not-registered-after-deployment/1858
I'm playing with this PDF To Image converter and I've cloned the repo, run npm install, changed this section:
var s3EventHandler = new S3EventHandler({
region: 'my-region',
outputBucketName: 'my-bucket-name'
s3: s3,
resolution: 72
});
Renamed it exports.js, zipped up the the js, node_modules folder, package.json and event.json (I've also tried with both of these jsons removed) and uploaded it into my Lambda function. The s3 trigger has been created and so far is working fine.
I've had multiple test failures because it couldn't find a either the async module and tmp module, which I've moved to the top level and it seems to fix it (however it doesn't complain about the other modules that it requires and aren't in the top level).
In the test it complains s3 is not defined which I'm sorta lost with as there isn't a lot of details with it. I thought it could be that I'm just running test so the s3 trigger with itself is missing.
When I upload a pdf into the bucket, Lambda reports that it runs but fails. Going into CloudWatch Logs says there is no log stream for it. I've checked the IAM role and it has permissions to CreateLogStream and PutLogEvents (it was the templated IAM policy).
How can I get my logs working to find the problem? Or what can I do to fix the s3 not defined issue which is my only clue atm? It could be related to the top level module requirement however that doesn't seem consistent as only some modules need to be at the top level?
Looks like "CreateLogGroup" Permission is missing from what you have mentioned. The following permissions are required for lambda to write logs to CloudWatch
"logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents"
I'm trying out gun.js I have it installed as a node.js project, I have configured the amazon S3 bucket through the dotenv and I have tried adding a data.json file and still I cant get gun.js to save the file locally or to he S3 bucket.
I know its early days for gun, but I get the feeling I'm missing something obvious.
I'm expecting to find a .json file in he local file system and or in the S3 bucket but I get neither.
require('dotenv').config();
var Gun = require('gun');
var gun = Gun({
file: 'data.json', // local testing and development
s3: {
key: process.env.AWS_KEY, // AWS Access Key
secret: process.env.AWS_SECRET, // AWS Secret Token
bucket: process.env.AWS_BUCKET // The bucket you want to save into
}
});
gun.put({ hello: 'world' }).key('my/first/data');
#bill Just noticed this now, sorry for the late answer. Thanks to #paul-w for notifying me of this and his response earlier today.
This question and answer assumes you are running a version EARLIER than v0.4.x!
If you are in NodeJS and are getting the error “You have no persistence layer to save to”, it means the default storage drivers (S3, file.js) didn't get installed or were deactivated - which is unusual as this happens automatically.
Try installing gun (again?) via npm install gun in your local NodeJS project directory, not a git clone or a copy&paste.
I can only guess, given the context you explain, that you might have copied/moved gun (like the gun.js file) into your project. The browser will work with just the single file, but NodeJS needs more - it needs the S3/file.js modules, which will be included if installed with npm or properly git cloned.
Also unlikely (since your code doesn't show this), if you happen to (this is bad) Gun({wire: {put: null, get: null}}) (or something similar) it would intentionally break the persistence drivers.
If you are in the browser and getting the error (and assuming your not overwriting the persistence drivers like in the previous paragraph), it could be because of some weird situation like you are using an old version of IE or a browser that doesn't have JSON support. Again, all these things are unlikely but I'm just wanting to be comprehensive.
Note: The above applies to the question in your title. However your actual question doesn't ask about the error, it asks about not seeing data in data.json or in S3. Answering that below.
To which #paul-w is more on track. If you are using S3 then the file.js module (data.json) automatically deactivates itself. If you are using the file.js module (data.json) then S3 does not get activated. As #paul-w mentioned, v0.4.x will support easily having multiple storage engines simultaneously. However, you should see your data in at least one or the other - unless you are getting the "no persistence layer" error, in which case you won't see your data anywhere because there isn't any persistence! But again, default persistence layers are included with gun by default (unless installation was incorrect, or you explicitly overwrite them - both unusual things).
I hope this answers your question. Sorry I didn't see it till now. Please let me know if this works, and also join the conversation at https://gitter.im/amark/gun . Thank you for helping start the stackoverflow questions! We need more of these!
I think Mark is going to answer this more officially, but the quick answer is that in gun.js 0.3 (current) there is a single gun server peer or storage target, and when you run gun as a server (e.g. from node.js rather than a browser), S3 is preferred, if S3 credentials are specified. But gun is also saving your data changes in browser memory, or localStorage (up to the browser limit of 5MB), and S3 is there for a more permanent storage.
So in the example above, I think the problem is that the file entry will only be used if there is a problem saving changes to S3, and that's why you don't see the new data going there. Maybe try putting an error in the S3 credentials (e.g. add an 'x' for now) and see if it starts using the file path instead.
In gun.js 0.4 there are plans to make use of all peers specified in the constructor or dynamically, but that feature isn't here yet.
(And I probably butchered that answer, but hopefully Mark can correct any inaccuracies in this. I'm new to gun.js but had the same question.)