I am going to make some modifications to methods and the biosphere3 database. As I might break things (I have before), I would like to create backups.
Thankfully, there exist backup() methods for just this. For example:
myBiosphere = Database('biosphere3')
myBiosphere.backup()
According to the docs, this "Write[s] a backup version of the data to the backups directory." Doing so indeed creates a backup, and the location of this backup is conveniently returned when calling backup().
What I wish to do is to load this backup and replace the database I have broken, if need be. The docs seem to stay silent on this, though the docs on serialize say "filepath (str, optional): Provide an alternate filepath (e.g. for backup)."
How can one restore a database with a saved version?
As a bonus question: how is increment_version(database, number=None) called, and how can one use it to help with database management?
The code to backup is quite simple:
def backup(self):
"""Save a backup to ``backups`` folder.
Returns:
File path of backup.
"""
from bw2io import BW2Package
return BW2Package.export_obj(self)
So you would restore the same as with any BW2Package:
from brightway2 import *
BW2Package.import_file(filepath)
However, if would recommend using backup_project_directory(project) and restore_project_directory(filepath) instead, as they don't go through an (older) intermediate format.
increment_version is only for the single file database backend, and is invoked automatically every time the database is saved. You could add versioning to the sqlite database backend, but this is non-trivial.
Related
I have an existing app which uses SQLAlchemy for DB access. It works well.
Now I want to introduce DB migrations, and I read alembic is the recommended way. Basically I want to start with the "current DB state" (not empty DB!) as revision 0, and then track further revisions going forward.
So I installed alembic (version 1.7.3) and put
from my_project.db_tables import Base
target_metadata = Base.metadata
into my env.py. The Base is just standard SQLAlchemy Base = sqlalchemy.ext.declarative.declarative_base() within my project (again, which works fine).
Then I ran alembic revision --autogenerate -m "revision0", expecting to see an upgrade() method that gets me to the current DB state from an empty DB. Or maybe an empty upgrade(), since it's the first revision, I don't know.
Instead, the upgrade() method is full of op.drop_index and op.drop_table calls, while downgrade() is all op.create_index and op.create_table. Basically the opposite of what I expected.
Any idea what's wrong?
What's the recommended way to "initialize" migrations from an existing DB state?
OK, I figured it out.
The existing production DB has lots of stuff that alembic revision --autogenerate is not picking up on. That's why its generated migration scripts are full of op.drops in upgrade(), and op.creates in downgrade().
So I'll have to manually clean up the generated scripts every time. Or automate this cleanup Python script cleanup somehow programmatically, outside of Alembic.
I am trying to read a file content after executing a class GetContentsAPI, basically this class GetContentsAPI will write into the file /etc/api/token.
class Main{
require GetContentsAPI
file("/etc/api/token")
}
When I did the above steps, its says Evaluation Error: Error while evaluating a Function Call, Could not find any files from /etc/api. Not sure how to make sure the file is already created before trying to read.
Thanks James
The file() function reads the contents of a file during catalog building. You don't present any details of class GetContentsApi, but all of the standard puppet facilities that write to files (especially, but not limited to, File resources) write during catalog application. Unless you've cooked up something highly customized, the file() function will always read before GetContentsApi writes.
Moreover, in a master / agent setup (which is the only kind supported in current Puppet), catalog building happens on the master, whereas catalog application happens on the target node, which is usually a different machine, so you're unlikely even to be able to read what was written during a previous catalog-building run.
Also, file() just returns the file contents as a string, so it's not very useful to call it without using the return value somehow.
It's not at all clear what you're trying to achieve, but from what I can see, you are not going in a fruitful direction. Perhaps you should take a step back and ask a different question about that.
When I run the command in the terminal back to back, it doesn't sync the second time. Which is great! It shouldn't. But, if I run my build process and run aws s3 sync programmatically, back to back, it syncs all the files both times, as if my build process is changing something differently the second time.
Can't figure out what might be happening. Any ideas?
My build process is basically pug source/ --out static-site/ and stylus -c styles/ --out static-site/styles/
According to this - http://docs.aws.amazon.com/cli/latest/reference/s3/sync.html
S3 sync compares the size of the file and the last modified timestamp to see if a file needs to be synced.
In your case, I'd suspect the build system is resulting in a newer timestamp even though the file size hasn't changed?
AWS CLI sync:
A local file will require uploading if the size of the local file is
different than the size of the s3 object, the last modified time of
the local file is newer than the last modified time of the s3 object,
or the local file does not exist under the specified bucket and
prefix.
--size-only (boolean) Makes the size of each key the only criteria used to decide whether to sync from source to destination.
You want the --size-only option which looks only at the file size not the last modified date. This is perfect for an asset build system that will change the last modified date frequently but not the actual contents of the files (I'm running into this with webpack builds where things like fonts kept syncing even though the file contents were identical). If you don't use a build method that incorporates the hash of the contents into the filename it might be possible to run into problems (if build emits same sized file but with different contents) so watch out for that.
I did manually test adding a new file that wasn't on the remote bucket and it is indeed added to the remote bucket with --size-only.
This article is a bit dated but i'll contribute nonetheless for folks arriving here via google.
I agree with checked answer. To add additional context, AWS S3 functionality is different than standard linux s3 in a number of ways. In Linux, an md5hash can be computed to determine if a file has changed. S3 does not do this, so it can only determine based on size and/or timestamp. What's worse, AWS does not preserve timestamp when transferring either way, so timestamp is ignored when syncing to local and only used when syncing to s3.
i have uploaded file(PDF 4mb) on server it is stored in blobdir. and reference in object of MYCLASS with attribute attachment in (zodb data.fs). if i am deleting object of MYCLASS then that object is deleted but the file(PDF 4mb) on blobdir is not deleted. how to delete that blob file after object is deleted?
The file is part of a past ZODB revision. You'll need to pack your ZODB database to remove historical revisions.
How far back you pack your database is up to you. Once you have removed old revisions you can no longer roll back the database to those states anymore.
How you pack the ZODB depends on your setup. If you are using ZEO there is a command-line tool (zeopack) that instructs ZEO server to pack the storage for you.
You can also do it programmatically; from your Pyramid app, for example, with the db.pack() method:
import time
from pyramid_zodbconn import get_connection
db = get_connection(request).db()
db.pack(days=7)
I used the days parameter to pack the ZODB but retain history for the past week. You can also use a timestamp t (UNIX seconds since the epoch) to specify a specific point in time to which to pack, or omit either to remove all old revisions.
Once the revision that references the blob is removed, the blob file is not immediately removed; whenever you pack a backup is created in case you need to revert the operation. A future pack operation replaces the previous backup with the new backup, clearing the blobs for good.
We have a Job which takes a Back-up of VSAM file followed by standard Delete-Define-Repro of the same VSAM file. To handle the scenario of trying to delete a non-existing file we are following a standard practice to set MAXCC/LASTCC to 0 if Delete returns a non-zero return code and then continue the process as if there are no errors.
But sometimes we are facing a situation where Delete is not working because file is opened by some user or being read in some other Job. In this case Job fails because while Defining a new VSAM file because file is already present (Delete could not purge it).
Any work-arounds for this situation? Or can we force delete a file even if it is held by some other process/user?
Thanks for reading!
You should be able to work out that it would not be a good idea to delete a VSAM file (or any other) whilst it is being used by "something else".
Why don't you test for the specific value from the DELETE?
If you are doing a backup, then delete/define, it would be a really, really good idea to get exclusive control of the file, else something is going to get messed-up.
You could put a DD with DSN being the VSAM file in question with DISP=OLD, so that your job would only be selected when nothing is using the file.
How are you doing the backup? Why are other jobs accessing the file at the same time anyway? Is this in a "test" environment? What type of VSAM file is it? Why are you doing the REPRO, and do you feel that that is the best way to do it?
An actual answer is difficult without knowing all this, and more.