I'm using unison version 2.51.2 (ocaml 4.06.1) to sync files between a Synology NAS (BTRFS filesystem) and a Windows 2016 Server (NTFS filesystem). I'm trying to sync one-way from Linux to Windows using the -force option. I seem to experience some issues with files containing whitespaces like
The name of this Unix file is not allowed under Windows. (File 'Dermapharm/013165/zwi/013165_27556_41955_1_PU_2019-09-02_Fexofenaderm 120 mg Filmtabletten/var/013165_27556_41955_1_PU_1_2019-09-02_Fexofenaderm ')
which indicates a file ending on whitespace which is indeed not allowed in Windows. However, when I descend into the dir, there is NO such file ending with whitespace! A ls -la yields
013165_27556_41955_1_PU_1_2019-09-02_Fexofenaderm 120 mg Filmtabletten.rtf
I cannot determine a clear pattern, because this seem to happen only for a few files... (like 100 compared to 150.000 containing whitespace which work fine). Does anyone have an idea what could be the cause for this?
The problem was not related to Unison. There were indeed some corrupted files with the above mentioned name that did not show via SMB (due to the trailing space). After removing these files via PowerShell everything works fine.
Related
On a Linux volume as part of a NAS with many TB of data some files were created from macOS and some of those files uploaded from macOS seem to include characters in filenames that cannot be reproduced via FTP or SMB file protocol. These files will appear as e.g. "picture_name001.jpg". Where the "" probably stands for a colon or slash.
I can search for "" and found out it applies to 2171 files in distributed locations on the volume. Way too much to manually find and correct each file name.
I thought I can connect to the NAS via SSH and simply loop through each directory doing an automated replace of the "" into "_", but this doesn't work because:
for file in **; do mv -- "$file" "${file///_}"; done
this attempt will throw back an error on the first item matching with:
mv: can't rename '120422_LAXJFK': No such file or directory
So obviously this substitute character displayed as "" is not the way to address the file or directory as it refers to a name that doesn't actually exists in the volume index.
(A) How do I find out if "120422_LAX:JFK" or "120422_LAX/JFK" is meant here, and (B) how do I escape these invalid characters to eventually be able to automatically rename all those names to for example "120422_LAX_JFK"?
Is there for example a way to get a numerical file ID from the name and then instruct to rename the file by number in case its name contains ""?
I think the problem is that behind this "" can be different codes of symbols. When the system can't represent some characters (for example, given encoding is not supported), then it automatically replaced by some default character (in your case it is ""). But actually there is some code of the character, that should be in the name. BUT when you trying to do this for file in **; do mv -- "$file" "${file///_}"; done system can't recognize code, that symbol is "" is stands for.
I think this problem can be solved by changing the encoding of characters (they should be compatible and better the same) on both devices (mac and NAS)
Hope this would help
I'm on a Windows machine, and in my P4V workspace:advanced-settings, I have the following selected:
But I still notice extra spaces in some of my files after submitting and checking on a different machine. Is there a way to force every checkout and submission to be Unix style?
Note, I can't modify server settings, only my local workspaces.
If you set the LineEnd to unix and the files you sync have \r characters in them, it means that somebody using unix submitted them in that form (i.e. the files were checked in with the understanding that the Unix version of them is supposed to have those characters in there distinct from the native line endings). If you submit Windows-style files from a unix-mode workspace, you're saying that all unix workspaces should have Windows-style line endings for those files. This is desirable in some instances, like when you're using Unix machines to build packages for Windows systems, but for source code that's meant to be used cross-platform it's generally a bad thing.
It's not too hard to go through the history, figure out who did that, and cluebat them (or get the admin to force a fix of their LineEnd setting so that it actually matches the contents of their workspace), and I wholeheartedly recommend doing this any time you ask for unix files and get a win-style file that you didn't want. Usually if everyone uses the default setting of local everything works fine.
As far as fixing the files, a pretty easy method is to change your LineEnd to share, open all the files for edit, and then submit them -- the share setting forces all the \r characters to be stripped out on submit, as if you were running dos2unix over all your files every time you submitted.
I've googled this, and saw that someone had a similar problem here -> stackoverflow, but as luck would have it, unresolved.
I have an issue where my P4 client is not able to see or access any submitted files that are newer than 4 months old.
Background - My root directory/db files are on D drive which is undisturbed. I got a nasty virus yesterday, so I checked in all my workspace stuff and recovered to a December drive of my OS. Now that I've successfully booted up in my December version of my OS (where the same version P4V and P4A is already installed), my P4V is only able to see and access files up to 12/15/2015 - nothing after. The drive image I recovered from is dated 12/19/2016. Yet, I can physically inspect all my post Dec P4 checkins in the physical location of my DB on D drive. It's ALL THERE.
Here's really interesting info - out of curiosity, I recovered back to my Virus laden OS from yesterday. Opened up P4V, and it is able to see and access all my files up till my very last submitted file which was checked in yesterday.
Other important into
- The depot, stream, and workspace that my p4 is using is the same between my recovered Dec recovery OS, and yesterday's Virus laden OS. Nothing was ever changed in my P4 settings.
System info:
Windows 10
P4V and P4A - NTX64/2014.3/1007540 (for both my recovered windows image from Dec, and yesterday's virus laden windows)
P4 is my achilless heel. Help appreciated. I will not take offense if you explain things to me like a 3rd grader.
Cheers,
Paul
First off, I used incorrect terminology in my original question. I had my depots on D drive and everything else in their default locations on C.
Where I had it wrong: in the belief that as long as I had my depot files backed up to a safe location, I was protected. There is more to it than that :|
What went wrong: When my PC crashed, I recovered to a 5 month earlier December version of my Windows install which means (obviously) I have December's install of P4 with database files only aware of files I checked in up to December. I did indeed have 1000s of files check in after Dec, all alive and intact, but P4 is simply unaware of it. An assumption (which I had) that P4 simply peeks into the Depots to stay up to date of their contents is an incorrect and dangerous one.
To those who may benefit: For those do-it-yourselfers, these are the 2 things in Perforce you should be backing up if you want to be able to do a full recovery later down the line when things go bad:
Database files. By default, they live in the Perforce application’s “server” directory. For me, there are 67 database files in my C:\Program Files\Perforce\Server. They are all prefixed with db.* and it’s these files that P4 uses to “know” the current state of things, like, your changlists, checked-out files, all your workspaces and their settings, depots and their settings, etc, etc etc. Oh, and what files you even have in your depot! Unfortunately, when backing up database files, you can’t just copy/paste them, but must use command line commands to properly generate 2 other file types (journal, checkpoint) specifically for recovery purposes. It is these files that you'll generate new database files from in a recovery operation.
Depot directories. Depots specify directory trees that your files get stored to, and checked out from when working in your workspace. Files stored here won’t be in their native format, and non-binary files will have their contents tweaked on top of that. Depot directory trees can be manually copied/paste to a backup location.
Proper backup/recovery information can be found here. Like many things, it’s simple once you understand it, but reading this is a bit of a mouth full. It would have helped me to just see slightly fleshed out sample commands along with the reading material, so I’ve included some here. These are the bare minimum command line commands (plus other manual stuff), and will recover a checkpoint, not checkpoint + journal (again, take a look at the link to get a better understanding). Also, I would recommend practicing backup/recovery in a safe throw away environment (or throw away assets) to get the hang of this before doing it for real.
---------- Backup ------------
Despite all the material in the link, it only takes a bare minimum of 3 command line commands (plus other non-command stuff) to do a proper P4 backup, and 1 cmd (among other things) to recover on my end. Again this is only a partially fleshed out sample to illustrate what the command prompt portion of backup/recovery looks like. There is a little more to it than simply running these to get you where you want.
(note: On Windows, you may need to start a windows command prompt as administrator)
>p4 set P4USER=superuser_name
>p4 -q verify //...
>p4d -r "C:\Program Files\Perforce\Server" -jc <yourBackupDirectory>\<prefixOfYourChoosing>
---------- Recovery ------------
>p4d -f -r "C:\Program Files\Perforce\Server" -jr <yourBackupDirectory>\<prefixYouChose>.ckp.1
In my case, I had to use the -f flag due to errors when not using it.
Cheers,
Paul W
For no reason that I can see, I can no longer run a TortoiseSVN Update on a development directory on my portable Windows XP Professional SP3 machine, getting the error:
Previous operation has not finished; run 'cleanup' if it was interrupted
Please execute the 'Cleanup' command.
If I try running cleanup, I get another error,
cannot process the following paths: cannot move $ROOT_DIR/.svn/tmp/tmp-... to $ROOT_DIR/path/where/thing/should/go: no such file or directory
I have verified that both files exist, and actually from CMD.EXE prompt I am able to issue a MOVE with those two filenames and have it work correctly. It's no use because next time SVN tries to repeat the operation itself after creating a different tmp file name, and while CMD succeeded, SVN fails.
UPDATE: the path lengths are in both cases well below PATH_MAX, target file system is NTFS, and permissions are OK. Maybe I'll now try with FileMon to see whatever TortoiseSVN is really up to.
I tried downgrading TortoiseSVN but to no avail. Other repositories work OK between the same machines.
TortoiseSVN 1.7.9, Build 23248 - 32 Bit , 2012/08/30 18:25:37
Subversion 1.7.6,
apr 1.4.6
apr-utils 1.3.12
neon 0.29.6
OpenSSL 1.0.1c 10 May 2012
zlib 1.2.7
Both server (OpenSuSE Linux 12.2) and client now run the latest version of SVN.
On Windows, I also cannot seem to get any more informative logs or information (I'm not very skilled with TortoiseSVN, I have always used the Linux command line version).
I might delete the local copy and run a checkout, but it's about 2 GB of data, and I'm on a slow connection, so it is really more of a "fly physically to server location and hook a copper Ethernet to the local network there" alternative. I'm reserving that as a sort of last ditch, nuclear option; I'd really rather understand what the problem is, for I fear it might happen again.
UPDATE
I've tried to delete remotely the subdirectory involved, committing the deletion on the server; deleting the subdirectory locally, and emptying the .svn/tmp subdirectory where I found sixteen tmp files, all copies of the one PNG causing problems.
I am still not able to perform any SVN subcommand, getting "Run cleanup!" error; on cleanup; I get a failed attempt to copy a tmpfile to the never-sufficiently-damned .PNG file, which no longer exists anywhere, into a directory that no longer exists anywhere.
I tried recreating the directory locally (but not the file!), no changes.
With FileMon, I traced the source PNG to 8e4c2389cf9d85c8b8ee54d49ea053c752a38187.svn-base in .svn/pristine subdirectory, tried removing it and got SVN complaining. I tried copying it to its intended destination (so that the file-as-it-should-be and the file-as-it-is are identical), no joy.
UPDATE
Well, this is weird. I decided to track everything that TortoiseSVN is doing using FileMon. I could see it checking the wc.db and search the item, checking for it in .svn/pristine (and finding it), copying it (unnecessarily if you ask me...) in .svn/tmp, and finally checking $DESTINATION_FILE (with correct case) using Windows Open() API. And getting PATH NOT FOUND. Yet the file is there, I can see it (and the name is less than 8.3 characters). And why PATH not found and not FILE not found?
Okay, it all boiled down to a directory that had been created remotely with a name ending with space. The file in itself was OK; the directory where it stood was not.
When updating, apparently, the directory got created but the name was shortened by Windows to exclude the final space.
To add to the difficulty of diagnosing, while TortoiseSVN did tell me what the problem was, it did so in the dialog box where the Arial font made the space in \path\to\your \file not clearly recognizable (it was, once I knew where to look, and compared that slash with the others. This one stood a little farther from the letter at its left).
Lesson learned: check really carefully the dialog file name, character by character (note to self: find a way of having it in Courier New if at all possible).
You may have two files in the repository that differ only in case. That's a problem on Windows. See this FAQ for details.
Im using Liferay portal server on tomcat and Linux Ubuntu.
Liferay is generating a file that is very long. I've seen those files in windows and its working. But when i tried running it in ubuntu, it doesn't create the file and my server is giving me error. I've also tried to make a file with a very long filename and it really doesn't allow me.
Is there a way for Linux Ubuntu to allow me to do this?
Fix this...
The source of my problem is the encrypted home of my ubuntu OS. It seems that the filename of the file created is also encrypted making my long filename even longer.
When i made a new installation of my Ubuntu, i didn't encrypt my home anymore and it works fine now.. thanks a lot all...
There's a huge slew of reasons it may not be working, probably the least of which is a long file name (unless we're talking about a filename over 255 characters, which I believe is the hard-limit).
Also, file length isn't going to be a big problem unless you've got some truly enormous files (sometimes linux filesystems cap at 2GB, but I don't know what the behaviour is if you went over. You'd probably still see a 2GB file that just doesn't contain everything).
My knee-jerk reaction would be to say you're having a permissions problem where the user the server is running as (say, 'www' or 'www-data', or whatever) doesn't have permission to write in the folder its trying too.
The filename you have given as an example is fine:
kevin#latte:~/miscdev/j$ touch 'everything.jsp_Q_browserId=firefox&themeId=controlpanel&colorSchemeId=01&minifierType=js&minifierBundleId=javascript.everything.files&t=1249034302000'
kevin#latte:~/miscdev/j$ ls -l
total 0
-rw-r--r-- 1 kevin kevin 0 2009-07-30 17:07 everything.jsp_Q_browserId=firefox&themeId=controlpanel&colorSchemeId=01&minifierType=js&minifierBundleId=javascript.everything.files&t=1249034302000
I imagine the problem is that you are passing that filename to a shell un-escaped, and it is interpreting the & character. Put the filename in single-quotes, as I have in my example.
I had the same problem on my Ubuntu 9.10 machine and I think it really was caused by the home-directory encryption. Those "too long" filenames work fine outside my home.