My work environment is MS SQL Server 2016 and I'm in need to create a NodeJS ETL tool to capture all inserts and updates for a large scale DB between 2 servers. I was doing my research and found couple ETL tools such as Nextract and Empujar but none of those have examples or connections for MSSQL. But they claim they do support MSSQL still need to make the connections and everything ground up. However I think I can build a simple ETL tool to select all the records from those tables using NodeJS, that's no issue but how would I tackle the updates?
Now you might think why can't you have some INSERT and UPDATE triggers? Well the issue is our ERP system is very fragile and it breaks once we have triggers set up.
All I need the ETL tool to do is constantly checking for new data and if it gets INSERTED or UPDATED then pass it to the other server(As the real meaning of ETL). Appreciate all the help!
We have two applications that use separate databases on SQL Server 2012, however we have several stored procedures that get data from the other using INNER JOINs (7 joins in total). We are looking to see if it is possible to move to Azure and have set up a test using our existing databases, with external tables to get the data from the other database.
The problem is that the performance of these queries goes from 1-15 seconds on our server, to 4+ minutes on Azure. We have tried moving the tables to the same database and it did fix the speed problem, although it isn't ideal to move all the tables over to the same DB.
For the purpose of our test, we are using Azure Standard elastic pool with 50 DTUs.
Cross database queries show good performance when the remote tables are not big. When remote tables are big, this article shows you how to perform joins remotely using table variables and improve performance.
This other article shows you also how to push parameterized operations to remote databases and improve performance.
Hope this helps.
I'm attempting to make my existing SQL Server 2008 database compatible with the Windows Azure platform by using SSDT, however I am getting a whole bunch of errors when I build the project due to TVFs and views looking for an external database that sits in the same instance in SSMS.
I've added the database that its looking for into Azure, which wasn't a problem.
I've found that if I load the offending piece of code I can add the Azure server address to the FROM statement which resolves the error (shown below), however I have a huge number that rely on the external db and hoped there may be a quicker way?
FROM [myAzureserver.database.windows.net.ExternalDBName.dbo.TableName] as ALIAS
I understand that this issue would not exist if I merged the databases, however this isn't possible at present.
Thanks a lot for your help.
Why are you trying to make your local SQL Server Azure compliant? Are you planning to move it at some point in the cloud? If so, you won't be able to use linked servers. Your FROM clause will work as long as the database remains on an on-premise SQL Server instance.
Assuming that's what you want to do, you are asking if there is quicker way to change your references to point to the cloud database, right? I am not sure if this will work for you but I had a similar issue on another project and ended up using synonyms. Check our synonyms here: http://msdn.microsoft.com/en-us/library/ms177544.aspx. Although you can't create a synonym for a server, you can create synonyms for tables/views/procs.
Again, this may not work for you, but let's try this...
Assuming you have your primary database called DB1, the secondary database called DB2, and the cloud database of DB2 called AzureDB2, you could create synonyms in DB2 to point to the cloud database without changing any SQL statement from DB1.
So assume you have this statement today in DB1:
SELECT * FROM DB2.MyTable
You could create a synonym in DB2 called MyTable:
CREATE SYNONYM MyTable FOR [myAzureserver.database.windows.net.ExternalDBName.dbo.TableName]
DB2 becomes a bridge basically. You don't need to change any statement in DB1; just create synonyms in DB2 that point to the cloud database.
Hopefully this works for you. :)
I currently developed an app that connects to SQL Server 2005 database, so my DAL objects where generated using information from that DB.
It will also be possible to connect to an Oracle and MySQL db, all with the same table structures (aside from the normal differences in fields, such as varbinary(max) in SQL Server and BLOB in Oracle, and so on). For this purpose, I already defined multiple connection strings and multiple SubSonic providers for the different DB's the app will run on.
My question is, if I generated my objects using a SQL Server database, should the generated objects work transparently with the other DB's or do I need to generate a different DAL for each database engine I use? Should I be aware of any possible bugs I may encounter while performing these operations?
Thanks in advance for any advice on this issue.
I'm using SubSonic 2.2 by the way....
From what I've been able to test so far, I can't see an easy way to achieve what I'm trying to do.
The ideal situation for me would have been to generate SubSonic objects using SQL Server for example, and just be able to switch dynamically to MySQL by just creating at runtime the correct Provider for it along with its connection string. I got to a point where my app would correctly connect from SQL Server to a MySQL DB, but there's a point where the app fails since SubSonic internally generates queries of the form
SELECT * FROM dbo.MyTable
which MySQL doesn't support obviously. I also noticed queries that enclosed table names with brackets ([]), so it seems that there are a number of factors that would limit the use of one Provider along multiple DB engines.
I guess my only other option is to sort it out with multiple generated providers, although I must admit it does not make me comfortable knowing that I'll have N copies of basically the same classes along my project.
I would really love to hear from anyone else if they've had similar experiences. I'll be sure to post my results once I get everything sorted out and working for my project.
Has any of this changed in 3.0? This would definitely be a worthy reason for me to upgrade if life is any easier on this matter...
I read on the MS site that SQL Azure does not support SQL Profiler. What are people using to profile queries running on this platform?
I haven't got too far playing around with SQL Azure as yet, but from what I understand there isn't anything you can use at the moment.
From MS (probably the article you read):
Because SQL Azure performs the
physical administration, any
statements and options that attempt to
directly manipulate physical resources
will be blocked, such as Resource
Governor, file group references, and
some physical server DDL statements.
It is also not possible to set server
options and SQL trace flags or use the
SQL Server Profiler or the Database
Tuning Advisor utilities.
If there were to be an alernative, I'd imagine it would require the ability to set trace flags which you can't do, hence I don't think there is an option at the moment.
Solution? I can only suggest you have a local development copy of the db so you can run profiler locally on it. I know that won't help with "live" issues/debugging/monitoring but it depends on what you need it for.
Edit:
Quote from MSDN forum:
Q: Is SQL Profiler supported in SQL
Azure?
A: We do not support SQL Profiler in
v1 of SQL Azure.
Now, you could interpret that as a hint that Profiler will be supported in future versions. I think it will be a big requirement to get a lot of people on board, using SQL Azure seriously.
Update as of 9/17/2015:
Microsoft just announced a new feature called Index Advisor:
How does Index Advisor work? Index Advisor continuously monitors your
database workload, performs the analysis and recommends new indexes
that can further improve the DB performance.
Recommendations are always kept up-to-date: As the DB workload and
schema evolves, Index Advisor will monitor the changes and adjust the
recommendations accordingly. Each recommendation comes with the
estimated impact to DB workload performance: You can use this
information to prioritize the most impactful recommendations first. In
addition, Index Advisor provides a very easy and powerful way of
creating the recommended indexes.
Creating new indexes only takes a couple of clicks. Index Advisor
measures the impact of newly created indexes and provides a report on
index impact to users. You can get started with Index Advisor and
improve your database performance with the following simple steps. It
literally takes five minutes to get accustomed with Index Advisor’s
simple and intuitive user interface. Let’s get started!
Original Answer:
SQL Azure now has some native profiling. See http://blogs.msdn.com/b/benko/archive/2012/05/19/cloudtip-14-how-do-i-get-sql-profiler-info-from-sql-azure.aspx for details.
Microsoft's stated position SQL Server Profiler is deprecated. As much as this is a bad idea, that's what they have said.
SQL Profile is already deprecated in SQL Server, and that’s part of
the reason that it doesn’t make sense to bring to SQL DB.
What this means is you are going back 20+ years in database performance monitoring and everyone is going to have to write their own perf monitoring scripts instead of having a standard factory delivered tool that's on every server you will go to. It's tantamount to deprecating "sp_help" and making every DBA write their own. Hope you know all your DMVs inside and out... INNER JOIN, OUTER JOIN, and CROSS APPLY syntax really well.
Update as of 2017/04/14:
Microsoft's Scott Guthrie today announced a lot of new features in SQLAzure(this is called sqlazure managed instance,which is currently in preview),which are expected to be present in SQLAzure in coming months..below are them
1.SQLAgent
2.SQLProfiler
3.SQLCLR
4.Service Broker
5.Logshipping,Transactional Replication
6.Native/Backup restore
7.Additional DMV's and Xevents
8.cross database querying
References:
https://youtu.be/0uT46lpjeQE?t=1415
I have tried today a new tool suggested by Microsoft that is called Azure Data Studio.
In this tool you can download an extension called Profiler and it seems to be working just as expected.
You can use Query store feature, look here for more details: http://azure.microsoft.com/blog/2015/06/08/query-store-a-flight-data-recorder-for-your-database/
The most close to SQL profiler, that I found working in Azure SQL, is SQL Workload Profiler
However note, that it’s beta version of a tool, created but a single person, and it is not too convinient to use.
SQL Azure offers following features to tune performance, profile queries in its own way, identity long running queries and much more
Intelligent Performance
Performance overview
Performance recommendations
Query Performance Insight
Automatic tuning