I am trying to copy data from an (azure vm) ftp hosted .csv file. When i execute the data pipeline i can see the ftp log and it initiates the file transfer, but at ~11mb it servers, attempts to reconnects, successfully reconnects, but then immediately disconnects.
Has anyone encountered this?
I can successfully transfer the file from cyberduck to local machine - and if i delete a lot of data from the csv, making it much smaller in size, the pipeline works correctly.
I have gone through passive ftp settings and vm firewall settings but i still cannot successfully get the file to completely transfer.
I realized this is not a data transfer issue but a file format issue. It seem like the FTP transfer protocol doesn't download the entire file and then load - it seems to do partials and there was a piece of it that was breaking (double quotes in a string and double quote identifier")
So it's an issue but a different issue and not related to this question.
Related
We have several servers using shared IIS config stored on a network storage. After access to that storage is down for a few seconds (and then comes back), IIS isn't working until you do iisreset.
The problem seems to be that the local app pool config files become corrupted. To be more precise, the error given out is "Configuration file is not well-formed XML", and if you go to the app pool config, you see that instead of an actual config, it contains the following:
Now, trying to solve this we've come across the "Offline Files" feature and tried it for the shared applicationHost.config, but it wouldn't sync (saying other process is using the file, which is strange - I can easily change and save it).
The shared path starts with an IP (like \1.2.3.4...) - perhaps that's the issue (can't figure why it would be, just out of ideas at this point)?
Basically, I have two questions:
1) If the shared config is unavailable for a bit, how to make IIS recover and not be left with corrupt files till iisreset?
2) Any other idea to prevent this situation altogether.
We did manage to get offline files to work - the problem was the network drive is over Samba, and had to have oplocks on - otherwise was telling it can't sync because file is used by another process.
Now, the IIS does recover - actually, doesn't go down with the drive. However, since our websites are also on that drive, they are not available during network outage (which is predictable), the last strange thing is that it takes IIS about 1 minute to "feel" them again after the drive is back online.
Here is the setup
We are testing Cloudberry for backing up files to a remote FTP server
As a test we are backing up files on a desktop, using Cloudberry FTP to a FTP server (FileZilla server) located on the same desktop. The FileZilla Server in turns is accessing a Synology NAS located on the same network.
The job is set to run every 24 hours
According the Cloudberry interface, it was last run at midnight and latested 1h 31min
There are no running jobs showing in Cloudberry interface
HOWEVER, it is 9AM , FileZilla server is still showing files upload. Filezilla has a counter to keep track on the number connection. The count is currently at 1.2million, but thereare only ~ 70,000 file being backed up.
I deleted the job and created a new one with the same result
So what is going on?
Alex
Found the root cause of this issue.
By looking through the logs in %programdata%\CloudBerryLab\CloudBerry Backup\Logs, I found that a Consistency job was running every hour...
No matter how many times I checked the Backup Job definition, this setting was never shown as it is only displayed in the Welcome tab, not the Backup Plans tab...
Changed the Consistency job to running weekly.
Hope this will help somebody else
Note: Disappointed with the lack of Support from CloudBerry given that Stackoverflow is officially their Support page as per http://www.cloudberrylab.com/support.aspx?page=support
I need to copy a depot from one Perforce server to another. The file revision history needs to be intact but the user information and workspace information can not be copied to the new server.
I've tried a standard checkpoint creation and restore procedure, but if there exist users or workspaces with the same name on both servers, the source server will overwrite this info on the destination server. This is pretty bad if those user accounts and workspaces do not have exactly identical details.
The goal of this sort of operation is to allow two separate, disconnected groups to view a versioned source tree with revision history. Updates would be single directional with one group developing and one just viewing. Each group's network is completely enclosed, no outside connections of any kind.
Any ideas would be appreciated, i've been busting my brains on this one for a while.
EDIT:
Ultimately my solution was to install an intermediate Perforce server on the same machine as my source server. Using that I could do a standard backup/restore from the source server to the intermediate server and then delete all unwanted meta data in the intermediate server before backing up from the intermediate server to the final destination server. Pretty complicated but it got the job done and it can all be done programatically in Windows Power Shell.
There are a few ways, but I think you are going about this one the hard way.
Continue to do what you are doing, but delete the db.user, db.view(I think) and db.group. Then when you start the perforce server, it will create these, but they will be empty, which will make it hard for anyone to log in. So you'll have to create users/groups. I'm not sure if you can take those db files from another server and copy them in, never tried that.
The MUCH easier way, make a replica. http://www.perforce.com/perforce/r10.2/manuals/p4sag/10_replication.html Make sure you look at the p4d -M flag to make sure it's a read only replica. I assume you have a USB drive or something to move between networks, so you can just issue a p4 pull onto the USB drive, then move the drive, and either run it off the USB, or issue another p4 pull, pulling to a final server. Never tried this, but with some work it should be possible, you'll have to run a server off the USB to issue the final p4 pull.
You could take a look at perforce git fusion, and make some git clones.
You could also look at remote depots. Basically you create a new depot on your destination server, and point it at a depot on your source server. This works if you have a fast connection between the 2 servers. Protections are handled by the destination server, as to who has access to that new depot. The source server can be set up to share it out as read only to the destination server. Here is some info
http://answers.perforce.com/articles/KB_Article/Creating-A-Remote-Depot
Just make sure you test it during a slow period, as it can slow down the destination server. I tried it from 2 remote locations, both on the east coast US, and it was acceptable, but not too useful. If both servers are in the same building it would be fine.
I have an issue that my idiot web host support team cannot solve, so here it is:
When I'm working on a site, and I'm uploading many files here and there (small files, most of them a few dozen lines at most, php and js files mostly, with some png and jpg files), after multiple uploads in a very short timeframe, the FTP chokes on me. It cuts me off with a "refused connection" error from the server end as if I am brute-force attacking the server, or trying to overload it. And then after 30 minutes or so it seems to work again.
I have a dedicated server with inmotion hosting (which I do NOT recommend, but that's another story - I have too many accounts to switch over), so I have access to all logs etc. if you want me to look.
Here's what I have as settings so far:
I have my own IP on the whitelist in the firewall.
FTP settings have maximum 2000 connections at a time (Which I am
nowhere near close to hitting - most of the accounts I manage
myself, without client access allowed)
Broken Compatibility ON
Idle time 15 mins
On regular port 21
regular FTP (not SFTP)
access to a sub-domain of a major domain
Anyhow this is very frustrating because I have to pause my web development work in the middle of an update. Restarting FTP on WHM doesn't seem to resolve it right away either - I just have to wait. However when I try to access the website directly through the browser, or use ping/traceroute commands to see if I can reach it, there's no problem - just the FTP is cut off.
The ftp server is configured for such a behavior. If you cannot change its configuration (or switch to another ftp server program on the server), you can't avoid that.
For example vsftpd has many such configuration switches.
Going to something else like scp or ssh should help
(I'm not sure that calling idiot your web support team can help you)
Pardon my outrageous silliness, I don't know if this is even possible.
Here's the situation.
There is an MS Access "database" (yes, I know, believe me, I know) which I'll need to SELECT, UPDATE and INSERT to from a remote location. The catch is that this needs to happen securely.
I have complete control over the remote machine which hosts the MS Access file, so I can put in drivers and software as I please. The server is Microsoft Windows Server 2003.
The approach that I had intended to take was to host a PHP script on an HTTPS server (using either Apache or IIS, doesn't matter), send XML to the PHP script which would then do its thing on the MS Access database and send XML results back. However, due to time restraints, I'm trying to figure out if I can connect directly through ODBC in a secure manner, and have it speak to an MS Access database.
It's my understanding that ODBC is not exactly famous for being secure, but that there are ODBC drivers that support encrypted connections, or that I can somehow tunnel the ODBC connection through SSL. However, all the information I have found so far relies on the database being Microsoft SQL.
In particular I'm interested if there are ways to SSL-ify ODBC connections without regard to the underlying database. I could probably figure that out on a Unix-clone by myself, but the host is a Windows Server 2003 in which case, I don't know how to proceed.
Is this possible at all? Any information highly appreciated!
The problem here is you are not quite understanding how an ODBC connection works with access. We are not talking about a TC/IP or socked based connection here.
If you look at ANY connection string for an JET to access file, you see in the ODBC connection will always, I REPEAT ALWAYS include a fully qualified windows path name. When I say a fully qualified windows path name, I am talking to about a file that sitting on the hard disk.
At the end of the day we are thus talking about opening a plain Jane windows file. A horse is a horse is a horse and a windows file is a windows file, is a windows file.
In other words we are talking about opening a file sitting on the hard disk. So, this whole process is not any different than opening excel file, a text file, a PowerPoint file, or in this case an access file that just also happens to be sitting on the hard disk.
There's no server or particular database software that EVER has to be installed on the computer where this file sets. It is the CLIENT SIDE that must have the software and execute a standard windows file open command to pull the data off the disk drive. Remember when you place a word file on a server and open it, you never had to install word on the server, is the client side that's doing a windows standard file open, and the exact same scenario applies to JET when it opens a access file.
What this means then if you're going to open this file up over an Internet connection, you therefore must extend windows networking over the Internet. HTTP, or even FTP is nothing remotely close to the windows file networking protocol.
However, you can extend windows networking system over the Internet, and this is typically done by which called a VPN (virtual private network). That means you'll have to set up a VPN. This will thus allow you to see this other computer via network neighborhood and browse to the files on that folder on the server, and simply open it. Again your opening a standard windows file, there's not some type of service running on the server that you can connect to like with SQL server.
You can read the following article of mine and I explain why running a VPN over the Internet with windows networking and a JET (access) file simply will not work in an reliable fashion:
http://www.members.shaw.ca/AlbertKallal//Wan/Wans.html
So, just keep in mind that if you look at any JET ODBC connection string, you'll notice it's never a IP based, but must be a FULLY QUALIFIED STANDARD windows file name. I cannot stress and repeat again that we talking about a standard windows file name and location that we going to open.
Remember this is no different than opening word or excel or PowerPoint. The ODBC driver confuses this issue, since the driver is ONLY required to be installed and setup on the client side, there's nothing to connect two on the server side, except the required ability to open a standard plain Jane windows file.
What you thus ask as possible with a VPN, but not practical. You can read the above article and it explains in detail why this cannot reliably work and function.
With the advent of several free editions of SQL server, and so many other choices, the above limitation is likely not going to be an issue for you. These other server database systems are not file based, and your connection strings will NEVER resolve to some file name. And, thus these database servers also do not require the windows networking proto call to open that file, and therefore you can even connect to servers such as running linux etc. that don't even have windows networking installed. For a jet connection, you have to use windows networking to directly open the file .
Usually one puts an intermediary between clients and the database. The intermediary handles authentication, authorization, secure data transmission, etc. You assume that the database is inside your firewall, in a secure area. All the things you want to add to make things secure for clients that are outside your firewall are handled by the intermediary.
Being a Java person, I would automatically think web client talking to one or more servlets. Let the servlet handle authentication and authorization. HTTP means no firewall worries. You can use HTTPS, too.
I think that'd be easier to put in place. Besides, even an SSL-ified ODBC connection still exposes your database to the wider Internet. I wouldn't want my data in such a repository. Would you?
Why does your MS-Access (really MS-Jet) database have only 1 file?
I can't picture that. If it were not an ODBC database, then I can picture it.
Most MS-Jet ODBC databases have 100's of *.MDB files in them,
where each MDB file is acting as either a: single table, group of tables, or partial table that is logically and physically spread (not split, and with no linking) across dozens or 100's of MDB files. No MDB file is considered a database in and of itself.
This is how I have seen ODBC databases built using the MS-Access Driver and
MS-Jet Engine.
Most ODBC MS-Jet/MS-Access Driver databases are around 5 billion rows and 1 Terabyte in size.