I am migrating a system from the old server (Slackware) to the new one (Redhat). The system includes some .gdbm files. I find out that on my new server, when running
WEB_SERVICES = file.gdbm
tie( %webservices, 'GDBM_File', $WEB_SERVICES, O_RDONLY, 0 )
the %webservices turns out to be empty. But this was working fine on my old server.
So my question is, are .gdbm files able to be simply transferred (using scp command) from one server to another (different operating system and different version of gdbm)?
Also I read the documents http://www.gnu.org.ua/software/gdbm/manual/gdbm.html#SEC12, which says .gdbm files need to be converted into flat format before sending over the network. But still I'm not sure how to do it.
Please help, thanks in advance!
On the old system, GDBM-tie to the hash, dump the hash. Move the dump to the new system. Read the dump into a hash, tie to GDBM to write it.
For dumping, use a platform independent serialisation format (Sereal is best), or if the dump needs to be human readable, Data::Dumper or similar for writing and Data::Undump for reading.
Related
I am working on a project where we are migrating from Open VMS to Unix/Linux.
There's a functionality called "FDL" in open vms, which i want to achieve in Unix.
What FDL actually does is , it defines a certain set of attributes for a file or a record, like fixing some block size for a particular file, file organization as sequential, variable or relative, specifying record size in a file beforehand, specifying carriage return(escape sequence) for record etc.
How can i set these attributes before a file gets created in unix.
FDL is merely a syntax/descriptive method to set/view OpenVMS file attributes (metadata) which has no equivalent in typical Linux file systems. Those attributes are implemented by the (Files-11 / ODS) file system an acted on by RMS (the OpenVMS Record management Services) for which again there is no equivalent in Linux although there are packages (sector7).
So much more than an FDL question , this is an RMS question.
RMS offers 'record' access where a record is a blob of byte defined in the file which can be read sequentially, by number or by key (indexed file). The attributes mentioned in the question are to do with simple sequential access, but there Linux just offers a byte-stream method. The application is supposed to know how much to read / when to stop reading. Possibly a (record) terminator like (frequently) (linefeed) is used but that's about it (fscanf).
Other than using a 'parallel' meta file, or reserving an initial byte stream in your files there is no standard way to store metadata on how to use the bytestream in the file, and making them hard to use by other applications.
All this to say: No Can Do.
Sorry.
In a very real sense, my question is actually 'can I skip generating a checksum', but answering that question rests on the above question.
To give you some background, I'm (finally) converting from Paperclip to ActiveStorage, and one of the pains of my particular conversion process is that I'm storing a decent sized number of fairly large files -- in addition to normal sized thumbnail images, I'm also storing large multimedia files, some in excess of 10GBs (currently poking at a 15GB file).
The basic conversion process has me downloading the file to generate a checksum, and a few other minor details that could be done with a head request instead of downloading the full file. We also copy the file from it's old 'home' to its new 'home', but that is done as an S3 to S3 copy, and doesn't take as long as downloading and uploading.
I'd love to skip the download & generate checksum process -- or at least, put it off for another day, as a cleanup step that isn't important to what we're actually doing.
So the question is: does the checksum actually do anything in ActiveStorage, or is it just a 'nice-to-have' feature that would allow me to, for example, publish the checksum if someone wanted to verify their version?
Found in code Rails
Prior to uploading, we compute the checksum, which is sent to the
service for transit integrity validation. If the checksum does not
match what the service receives, an exception will be raised.
You can create your own checksum without downloading the file:
Found in code Rails
def compute_checksum_in_chunks(io)
OpenSSL::Digest::MD5.new.tap do |checksum|
while chunk = io.read(5.megabytes)
checksum << chunk
end
io.rewind
end.base64digest
end
I'm currently working on one Node.js project. I want to have an ability to read, modify and write ZIP file without saving it into FS (we receive it by TCP and send it back after modifications were made), and so far it looks like possible bocause of simple ZIP file structure. Currently I refer to this documentation.
So ZIP file has simple structure:
File header 1
File data 1
File data descriptor 1
File header 2
File data 2
File data descriptor 2
...
[other not important yet]
First we need to read file header, which contains field compressed size, and it could be the perfect way to read file data 1 by it's length. But it's actually not. This field may contain '0' or '0xFFFFFFFF', and those values don't describe its actual length. In that case we have to read file data without information about it's length. But how?..
Compression/Decopression algorithm descriptions looks pretty complex to me, and I plan to use ZLIB for compression itself anyway. So if something useful described there, then I missed the point.
Can someone explain the proper way to read those files?
P.S. Please avoid suggesting npm modules. I do not want to only solve the problem, but also to understand how things work.
Note - I'm assuming you want to read and process the zip file as
it comes off the socket, rather than reading the complete zip file into
memory before processing. Both options are valid.
I'd initially ignore the use cases where the compressed size has a value of '0' or '0xFFFFFFFF'. The former is only present in zip files created in streaming mode, the latter for zip files larger than 4Gig.
Dealing with them adds a lot of complexity - you can add support for them later, if necessary. Whether you ever need to support the 0/0xFFFFFFFF use cases depends on the nature of the zip files you intend to process.
When the compression method is deflated (8), use zlib for compression/decompression. You also need to support compression method stored (0). It gets used for very small files where compression isn't appropriate.
We have a server with a depot that does not allow committing files which are in a client mapping therefore I need a stream configuration.
Now I struggle with a task which I would assume should be simple:
We have a very large stream with lots of different file types and I would like to check out the entire stream but get only a certain file type.
Can this be done with perforce without black-listing every file type in question?
Edit: Sorry that I (for some reason omitted) so many information in my question.
I am already setting up a virtual stream where the UI gives me three nice fields:
Paths – where I can enter import, share isolate paths
Remapping – ignored in my case
Ignored – here I can enter wildcards to ignore directories or files
I was hoping that by creating a virtual stream I actually could define the file types I want, e.g. I could write an import statement like
import RootDir/....txt //Depot/mainline/RootDir/....txt (note the 4 dots, 3 for perforce and the other as a "wildcard"
however the stream definition does not support this and only allows me to write
import RootDir/... //Depot/mainline/RootDir/...
Since I was not able to find a way to white list the files I wanted I only knew a way to blacklist all things I did not want but I would like to avoid that because my Ignored list would be dozens of entries long.
Now I will look into that sync hint because I could use the full stream spec without filter and only sync the files I need on disk, which might be very good.
There are a few different things going on in your question but this seems the most like a statement of what you're trying to do so I'm going to zero in on it:
I would like to check out the entire stream but get only a certain
file type.
If by "check out" you mean you only want to sync that file type to your local workspace:
p4 sync ....TXT
If by "check out" you mean you want to open only that file type for edit:
p4 edit ....TXT
ANY operation in Perforce that operates on files accepts an arbitrary file path, because Perforce tracks all of its state per-file. This is true whether you're using classic clients or streams.
There needs to be some mechanism for telling the Helix (Perforce) server that you only want to retrieve certain files from the stream.
Virtual Streams may be a good fit here, as they allow you to filter the view of an existing stream.
This means you can sync only the files you want and when you submit you will be submitting directly back to the stream your virtual stream is based on.
More information is available here:
https://www.perforce.com/perforce/doc.current/manuals/p4v/p4v_virtual_streams.html
We have SSH-based file transfer scripts currently set up for Linux-to-Linux and we're porting them to z/OS to go z/OS-to-Linux. Note that this is with USS, the UNIX system services within z/OS otherwise known as OMVS, which uses EBCDIC under the covers, not zLinux which uses ASCII.
We've set up all the SSH key files and what-not, and the transfer itself is working fine.
However z/OS, in it's infinite wisdom, insists on converting the files from EBCDIC to ASCII despite the fact that they're binary files - this is screwing up the content of the destination files.
The scp manpage on z/OS states:
scp assumes that files are text. Files copied between EBCDIC and ASCII platforms
are converted.
and I can find nothing useful in the manuals that indicates how to get around this.
It seems a bizarre limitation for anyone wanting to transfer binary files between the two platforms. Does anyone know of a way, using SSH-standard keyfiles (we need this for security, no naked FTP allowed), to effect a binary transfer without translation?
You can use one of the other SSH-based tools such as sftp.
Whereas scp will let you transfer a file (with automatic authentication set up) with something like:
scp -i ident_file zos_file linux_user#linux_box:linux_file
you can do a similar thing with the secure FTP:
sftp IdentityFile=ident_file -b - linux_user#linux_box <<EOF
binary
put zos_file linux_file
EOF