I have an Azure VM with Pentaho Data Integration installed, i'm trying to build some ETL which loads a dimensional model from the staging area, but when i start a transformation, the load speed of PDI into any azure database is painfully slow.
It is possible to have PDI working on cloud with Azure Databases? There is some configuration step needed to achieve a reasonable loading speed?
PS:
VM and databases are in the same region
There is a firewall rule to allow port acess
Reading speed is working just fine
PDI 8.1, using table output step
I've been experiencing same speed problem but I will tell you my workarounds with this.
First of all: Download and install latest jdbc driver that lets you gain connection with azure sql database, in documentation the link is here but the way I do is keep it synced from here in GitHub any of this will let you use the latest driver in PDI.
Second workaround: for large files what I've found most powerful is using BCP Utility integrated with PowerShell or Linux Batch. Doesn't care if it files are local or in azure blob storage but you might need credentials for this.
Last but not least: Use Azure Data Factory V2 to move and load files (if you are like me I try to keep it in PDI until I have to load it, the http get step will let you trigger ADF pipeline).
Good luck and let me know if you get it.
I am currently on a project where I have 2 VM (virtual machine), a Windows and Linux one.
I also have an Oracle database where I have a simple table called "Material".
On the 2 VM, I want to connect to my Oracle database without any client or libraries. The thing is I want to create a script which would run on the VM and can connect to my database and insert some datas to my table "Material" but I can't install anything on my VM (like the mysqlclient for exemple).
So is it possible to connect to a database without installing anything on my VM? Or perhaps can I access to an online client to send my SQL to my Oracle Database?
I know it's quite difficult to understand my problem so if you have any question, feel free to ask.
I have a web application (using MongoDB database, AngularJS on front-end and NodeJS on back-end) that deployed on 2 places. First is on static ip so that it can access from anywhere and second is on one local machine so that user can use it when the internet connection is not available. So on both places, data can be inserted by user. My requirement is to sync the both databases, when internet connection is available on local machine i.e. from local system database to remote system database and vice-versa without loosing any data on both places.
One way I am thinking about is provide the sync button in the application and sync the databases using insert/update query. I am not sure is there any better and automated way to do this task so that the databases sync automatically like data copied in replica set.
Please provide the best solution to do this task. Thanks in advance.
I am developing a website that I intend to run within Windows Azure using a single Web Role. The site will make use of the Sphinx Search engine which will need to run as a Windows Service. So, my question is this...is it possible to install the Sphinx Search Windows Service inside of a Web Role.
From my initial research into Azure I am thinking "yes" for the reason that the Web Role is a VM running IIS. Therefore I should be able to remote in, install the service, and it should work. :)
Does this sound right?
Installing software via RDP is not a viable solution with Web/Worker role instances, as these changes won't persist. You need to install it either from a startup script or from OnStart(). Since you want to install as a service, that would imply startup script, since it would need elevated permissions. Note: The installer must support unattended mode, where all parameters are specified via command line with no human interaction.
What about scalability? If you have more than one instance of your web role running, can sphinx run across two instances? From what I read, it supports ODBC-compliant databases, and you might be able to use it against Windows Azure SQL Database. If that's the case, can two sphinx engines run on two different machines accessing the same data store? If so, this sounds like a viable solution.
If installation cannot be automated, or you need something additional like MySQL, you may want to consider placing the sphinx search engine inside a Virtual Machine (new in June 2012). Now you can spin up a Windows 2008 Server, RDP into it, configure it exactly how you want it.
Strictly speaking yes, you could do that. However this makes the assumption that you would be running on one VM instance and also that the instance would never need restarting.
You should consider looking at Azure worker roles for any functionality that would normally exist as a windows service.
After reading your answers, and thinking about it a bit more, I think dropping the idea of installing a service would be the best course of action. I've been looking at the API for Lucene.NET (this may be the same for Sphinx) and it's possible to encapsulate the writing/managing of indexes, etc, within in code and therefore no need for a service.
For the Azure, there is a library for managing index files using both local and Azure storage which could be of use. Scenarios I've read about show that it's then possible to have a Web Role that will process HTTP requests and perform the searches and a Worker Role to accept DB changes via a queue and have it write them to the indexes.
Pardon my outrageous silliness, I don't know if this is even possible.
Here's the situation.
There is an MS Access "database" (yes, I know, believe me, I know) which I'll need to SELECT, UPDATE and INSERT to from a remote location. The catch is that this needs to happen securely.
I have complete control over the remote machine which hosts the MS Access file, so I can put in drivers and software as I please. The server is Microsoft Windows Server 2003.
The approach that I had intended to take was to host a PHP script on an HTTPS server (using either Apache or IIS, doesn't matter), send XML to the PHP script which would then do its thing on the MS Access database and send XML results back. However, due to time restraints, I'm trying to figure out if I can connect directly through ODBC in a secure manner, and have it speak to an MS Access database.
It's my understanding that ODBC is not exactly famous for being secure, but that there are ODBC drivers that support encrypted connections, or that I can somehow tunnel the ODBC connection through SSL. However, all the information I have found so far relies on the database being Microsoft SQL.
In particular I'm interested if there are ways to SSL-ify ODBC connections without regard to the underlying database. I could probably figure that out on a Unix-clone by myself, but the host is a Windows Server 2003 in which case, I don't know how to proceed.
Is this possible at all? Any information highly appreciated!
The problem here is you are not quite understanding how an ODBC connection works with access. We are not talking about a TC/IP or socked based connection here.
If you look at ANY connection string for an JET to access file, you see in the ODBC connection will always, I REPEAT ALWAYS include a fully qualified windows path name. When I say a fully qualified windows path name, I am talking to about a file that sitting on the hard disk.
At the end of the day we are thus talking about opening a plain Jane windows file. A horse is a horse is a horse and a windows file is a windows file, is a windows file.
In other words we are talking about opening a file sitting on the hard disk. So, this whole process is not any different than opening excel file, a text file, a PowerPoint file, or in this case an access file that just also happens to be sitting on the hard disk.
There's no server or particular database software that EVER has to be installed on the computer where this file sets. It is the CLIENT SIDE that must have the software and execute a standard windows file open command to pull the data off the disk drive. Remember when you place a word file on a server and open it, you never had to install word on the server, is the client side that's doing a windows standard file open, and the exact same scenario applies to JET when it opens a access file.
What this means then if you're going to open this file up over an Internet connection, you therefore must extend windows networking over the Internet. HTTP, or even FTP is nothing remotely close to the windows file networking protocol.
However, you can extend windows networking system over the Internet, and this is typically done by which called a VPN (virtual private network). That means you'll have to set up a VPN. This will thus allow you to see this other computer via network neighborhood and browse to the files on that folder on the server, and simply open it. Again your opening a standard windows file, there's not some type of service running on the server that you can connect to like with SQL server.
You can read the following article of mine and I explain why running a VPN over the Internet with windows networking and a JET (access) file simply will not work in an reliable fashion:
http://www.members.shaw.ca/AlbertKallal//Wan/Wans.html
So, just keep in mind that if you look at any JET ODBC connection string, you'll notice it's never a IP based, but must be a FULLY QUALIFIED STANDARD windows file name. I cannot stress and repeat again that we talking about a standard windows file name and location that we going to open.
Remember this is no different than opening word or excel or PowerPoint. The ODBC driver confuses this issue, since the driver is ONLY required to be installed and setup on the client side, there's nothing to connect two on the server side, except the required ability to open a standard plain Jane windows file.
What you thus ask as possible with a VPN, but not practical. You can read the above article and it explains in detail why this cannot reliably work and function.
With the advent of several free editions of SQL server, and so many other choices, the above limitation is likely not going to be an issue for you. These other server database systems are not file based, and your connection strings will NEVER resolve to some file name. And, thus these database servers also do not require the windows networking proto call to open that file, and therefore you can even connect to servers such as running linux etc. that don't even have windows networking installed. For a jet connection, you have to use windows networking to directly open the file .
Usually one puts an intermediary between clients and the database. The intermediary handles authentication, authorization, secure data transmission, etc. You assume that the database is inside your firewall, in a secure area. All the things you want to add to make things secure for clients that are outside your firewall are handled by the intermediary.
Being a Java person, I would automatically think web client talking to one or more servlets. Let the servlet handle authentication and authorization. HTTP means no firewall worries. You can use HTTPS, too.
I think that'd be easier to put in place. Besides, even an SSL-ified ODBC connection still exposes your database to the wider Internet. I wouldn't want my data in such a repository. Would you?
Why does your MS-Access (really MS-Jet) database have only 1 file?
I can't picture that. If it were not an ODBC database, then I can picture it.
Most MS-Jet ODBC databases have 100's of *.MDB files in them,
where each MDB file is acting as either a: single table, group of tables, or partial table that is logically and physically spread (not split, and with no linking) across dozens or 100's of MDB files. No MDB file is considered a database in and of itself.
This is how I have seen ODBC databases built using the MS-Access Driver and
MS-Jet Engine.
Most ODBC MS-Jet/MS-Access Driver databases are around 5 billion rows and 1 Terabyte in size.