How to get SQL Profiler working with Azure SQLDb? - azure

We are having a Power BI dataset in the service. The source of this dataset are some Azure SQLdb tables. PaaS (Platform as a Service) setup. The daily refresh of this Power BI dataset takes long. SQL Profiler would be the tool to check the events that are happening. But we can't get it working in this PaaS environment.
We used SQLServer, DAX Studio and Azure Data Studio.
So how can I trace query execution, capture events in a PaaS environment?
Really hope someone has the answer
regards Ron

SQL Server Profiler and SQL Trace are deprecated. For Azure SQL Database you should use Extended Events to capture the queries.
With Extended Events you can create a session, define what events to be captured in this session, and for each of the events to say which fields to be retrieved. You can define filters on these fields too (e.g. capture the events in one specific database only). The last thing when you create a session is to define where to store the data - in a file, ring buffer and so on. In your case, sql_batch_starting event with sql_text field, captured to a ring buffer should be enough (capturing to a file will require setting up Azure Storage).
You can create the event session with a script or with a wizard in SQL Server Management Studio. The script could be something like this:
CREATE EVENT SESSION [Capture queries] ON DATABASE
ADD EVENT sqlserver.sql_batch_starting(
ACTION(sqlserver.sql_text))
ADD TARGET package0.ring_buffer
GO
where [Capture queries] is the name of the session. If you create the session with the wizard, you have the option to start it automatically after it is created, but if you use the script, you must start it manually, like this:
ALTER EVENT SESSION [Capture queries] ON SERVER STATE = START
It is very important to stop the session, when it is not needed anymore, because it has impact on the performance. You can stop a session with the following script:
ALTER EVENT SESSION [Capture queries] ON SERVER STATE = STOP
And eventually drop it when it is no longer needed:
DROP EVENT SESSION [Capture queries] ON SERVER
In SQL Server Management Studio, you can see the result by right-clicking on the ring buffer and select View Target Data...:
which will show you an XML to click on:
Or you can use a query, like this:
select
se.name as session_name,
ev.event_name,
ac.action_name,
st.target_name,
se.session_source,
st.target_data,
CAST(st.target_data AS XML) as target_data_XML
from sys.dm_xe_database_session_event_actions ac
INNER JOIN sys.dm_xe_database_session_events ev on ev.event_name = ac.event_name and cast(ev.event_session_address AS BINARY(8)) = cast(ac.event_session_address AS BINARY(8))
INNER JOIN sys.dm_xe_database_session_object_columns oc on cast(oc.event_session_address AS BINARY(8)) = cast(ac.event_session_address AS BINARY(8))
INNER JOIN sys.dm_xe_database_session_targets st on cast(st.event_session_address AS BINARY(8)) = cast(ac.event_session_address AS BINARY(8))
INNER JOIN sys.dm_xe_database_sessions se on cast(ac.event_session_address AS BINARY(8)) = cast(se.address AS BINARY(8))
The last column is an XML like the one above, where you can see the captured statements:
Of course, it is possible to use XQuery and transform the returned XML to a tabular result, but in your case it is not needed - just look for the queries in the XML itself.

Related

Query my temporary tables outside my java app

I have created a java application starting spark (local[*]) and exploiting it to read a csv file as a Dataset<Row> and to create a temporary view with createOrReplaceTempView.
At this point I am able to exploit SQL to query the view inside my application.
What I would like to do, for development and debugging purposes, is to execute queries in an interactive way from outside my application.
Any hints?
Thanks in advance
You can use spark's DeveloperApi - HiveThriftServer2.
#DeveloperApi
def startWithContext(sqlContext: SQLContext): Unit = {
val server = new HiveThriftServer2(sqlContext)
Only thing you need to do in your application is to get SQLContext and use it as follows:
HiveThriftServer2.startWithContext(sqlContext)
This will start hive thrift server (by default on port 10000) and you can use sql client - e.g. beeline for accessing and querying your data in temp tables.
Also you will need to set --conf spark.sql.hive.thriftServer.singleSession=true which allows you to see temp tables. By default it's set to false so each connection has it's own session and they dont see others temp tables.
"spark.sql.hive.thriftServer.singleSession" - When set to true, Hive Thrift server is running in a single session
mode. All the JDBC/ODBC connections share the temporary views, function registries, SQL configuration and the current database.

Sql query for Data Service Server 3.1.0

I am working on wso2 DSS 3.1.0 and inserting data as array into sql , this is Sql query
INSERT INTO memployeecount (CompanyCode,NoofEmployees) VALUES
('SPS', 1000),
('SPS', 2000),
('SPS', 3000),
('SFS', 500),
('SFS', 600),
('SFS', 700);
it's working fine,
But how can i write the same query for Data Service server.
Some one guide me.
WSO2 DSS supports batch requests, which takes of array of parameters as the input. What you need to do is create a dataservice enabling batch requests[1], with a simple insert query.
Once it is deployed there will be a batch enabled request created from which you can insert multiple records. For more information you can tryout Batch Processing sample available inside DSS[2].
[1]http://docs.wso2.org/display/DSS310/Creating+Using+Various+Data+Sources
[2]http://docs.wso2.org/display/DSS310/Batch+Processing+Sample

How to manage centralized values in a sharded environment

I have an ASP.NET app being developed for Windows Azure. It's been deemed necessary that we use sharding for the DB to improve write times since the app is very write heavy but the data is easily isolated. However, I need to keep track of a few central variables across all instances, and I'm not sure the best place to store that info. What are my options?
Requirements:
Must be durable, can survive instance reboots
Must be synchronized. It's incredibly important to avoid conflicting updates or at least throw an exception in such cases, rather than overwriting values or failing silently.
Must be reasonably fast (2000+ read/writes per second
I thought about writing a separate component to run on a worker role that simply reads/writes the values in memory and flushes them to disk every so often, but I figure there's got to be something already written for that purpose that I can appropriate in Windows Azure.
I think what I'm looking for is a system like Apache ZooKeeper, but I dont' want to have to deal with installing the JRE during the worker role startup and all that jazz.
Edit: Based on the suggestion below, I'm trying to use Azure Table Storage using the following code:
var context = table.ServiceClient.GetTableServiceContext();
var item = context.CreateQuery<OfferDataItemTableEntity>(table.Name)
.Where(x => x.PartitionKey == Name).FirstOrDefault();
if (item == null)
{
item = new OfferDataItemTableEntity(Name);
context.AddObject(table.Name, item);
}
if (item.Allocated < Quantity)
{
allocated = ++item.Allocated;
context.UpdateObject(item);
context.SaveChanges();
return true;
}
However, the context.UpdateObject(item) call fails with The context is not currently tracking the entity. Doesn't querying the context for the item initially add it to the context tracking mechanism?
Have you looked into SQL Azure Federations? It seems like exactly what you're looking for:
sharding for SQL Azure.
Here are a few links to read:
http://msdn.microsoft.com/en-us/library/windowsazure/hh597452.aspx
http://convective.wordpress.com/2012/03/05/introduction-to-sql-azure-federations/
http://searchcloudapplications.techtarget.com/tip/Tips-for-deploying-SQL-Azure-Federations
What you need is Table Storage since it matches all your requirements:
Durable: Yes, Table Storage is part of a Storage Account, which isn't related to a specific Cloud Service or instance.
Synchronized: Yes, Table Storage is part of a Storage Account, which isn't related to a specific Cloud Service or instance.
It's incredibly important to avoid conflicting updates: Yes, this is possible with the use of ETags
Reasonably fast? Very fast, up to 20,000 entities/messages/blobs per second
Update:
Here is some sample code that uses the new storage SDK (2.0):
var storageAccount = CloudStorageAccount.DevelopmentStorageAccount;
var table = storageAccount.CreateCloudTableClient()
.GetTableReference("Records");
table.CreateIfNotExists();
// Add item.
table.Execute(TableOperation.Insert(new MyEntity() { PartitionKey = "", RowKey ="123456", Customer = "Sandrino" }));
var user1record = table.Execute(TableOperation.Retrieve<MyEntity>("", "123456")).Result as MyEntity;
var user2record = table.Execute(TableOperation.Retrieve<MyEntity>("", "123456")).Result as MyEntity;
user1record.Customer = "Steve";
table.Execute(TableOperation.Replace(user1record));
user2record.Customer = "John";
table.Execute(TableOperation.Replace(user2record));
First it adds the item 123456.
Then I'm simulating 2 users getting that same record (imagine they both opened a page displaying the record).
User 1 is fast and updates the item. This works.
User 2 still had the window open. This means he's working on an old version of the item. He updates the old item and tries to save it. This causes the following exception (this is possible because the SDK matches the ETag):
The remote server returned an error: (412) Precondition Failed.
I ended up with a hybrid cache / table storage solution. All instances track the variable via Azure caching, while the first instance spins up a timer that saves the value to table storage once per second. On startup, the cache variable is initialized with the value saved to table storage, if available.

How to configure Quartz.net to use an Azure SQL database to store ADOJobStore details

I am using quartz.net as a scheduler in a Microsoft Azure Web Role. I can get Quartz.net to work just fine if I use the RamDataStore. However, I want to break this into two components: the first will allow scheduling of jobs through a web interface and the second will execute the jobs through a worker role. To have this distributed processing, I will need to use an ADOJobStore.
Everything works fine with the RamDataStore but it breaks when I try to switch over to the ADOJobStore. So this leads me to believe that there is something in my properties that I'm missing. I am using Azure SQL database and while this is similar to SQL Server, there are some gotchas that sometimes cause problems.
I am using Quartz.net 2.0 (from nuGet) in VS2010, the database is Azure SQL.
When I call .GetScheduler(), I get the following exception:
{"JobStore type 'Quartz.Impl.AdoJobStore.JobStoreTX' props could not
be configured."}
with the details:
{"Could not parse property 'default.connectionString' into correct
data type: No writable property 'Default.connectionString' found"}
My connection code (including programatically set properties):
NameValueCollection properties = new NameValueCollection();
properties["quartz.scheduler.instanceName"] = "SchedulingServer";
properties["quartz.threadPool.type"] = "Quartz.Simpl.ZeroSizeThreadPool, Quartz";
properties["quartz.jobStore.type"] = "Quartz.Impl.AdoJobStore.JobStoreTX, Quartz";
properties["quartz.jobStore.tablePrefix"] = "QRTZ_";
properties["quartz.jobStore.clustered"] = "false";
properties["quartz.jobStore.driverDelegateType"] = "Quartz.Impl.AdoJobStore.SqlServerDelegate, Quartz";
properties["quartz.jobStore.dataSource"] = "default";
properties["quartz.jobStore.default.connectionString"] = "Server=tcp:serverName.database.windows.net;Database=scheduler;User ID=scheduler#serverName;Password=***;Trusted_Connection=False;Encrypt=True;";
properties["quartz.jobStore.default.provider"] = "SqlServer-20";
properties["quartz.jobStore.useProperties"] = "true";
ISchedulerFactory sf = new StdSchedulerFactory(properties);
_scheduler = sf.GetScheduler();
Any help or suggestions would be appreciated.
You have small but subtle error in your data source property naming, it should read:
properties["quartz.dataSource.default.connectionString"] = "Server=tcp:serverName.database.windows.net;Database=scheduler;User ID=scheduler#serverName;Password=***;Trusted_Connection=False;Encrypt=True;";
Also there is a property connectionStringName if you want to use the connection string section of configuration file.

SQL Azure unexpected database deletion/recreation

I've been scratching my head on this for hours, but can't seem to figure out what's wrong.
Here's our project basic setup:
MVC 3.0 Project with ASP.NET Membership
Entity Framework 4.3, Code First approach
Local environment: local SQL Server with 2 MDF database files attached (aspnet.mdf + entities.mdf)
Server environment: Windows Azure + 2 SQL Azure databases (aspnet and entities)
Here's what we did:
Created local and remote databases, modified web.config to use SQLEXPRESS connection strings in debug mode and SQL Azure connection strings in release mode
Created a SampleData class extending DropCreateDatabaseAlways<Entities> with a Seed method to seed data.
Used System.Data.Entity.Database.SetInitializer(new Models.SampleData()); in Application_Start to seed data to our databases.
Ran app locally - tables were created and seeded, all OK.
Deployed, ran remote app - tables were created and seeded, all OK.
Added pre-processor directives to stop destroying the Entity database at each application start on our remote Azure environment:
#if DEBUG
System.Data.Entity.Database.SetInitializer(new Models.SampleData());
#else
System.Data.Entity.Database.SetInitializer<Entities>(null);
#endif
Here's where it got ugly
We enabled Migrations using NuGet, with AutomaticMigrationsEnabled = true;
Everything was running smooth and nice. We left it cooking for a couple days
Today, we noticed an unknown bug on the Azure environment:
we have several classes deriving from a superclass SuperClass
the corresponding Entity table stores all of these objects in the same SuperClass table, using a discriminator to know which column to feed from when loading the various classes
While the loading went just fine before today, it doesn't anymore. We get the following error message:
The 'Foo' property on 'SubClass1' could not be set to a 'null' value. You must set this property to a non-null value of type 'Int32'.
After a quick check, our SuperClass table has columns Foo and Foo1. Logical enough, since SuperClass has 2 subclasses SubClass1 and SubClass2, each with a Foo property. In our case, Foo is NULL but Foo1 has an int32 value. So the problem is not with the database - rather, it would seem that the link between our Model and Database has been lost. The discriminator logic was corrupted.
Trying to find indications on what could've gone wrong, we noticed several things:
Even though we never performed any migration on the SQL Azure Entity database, the database now has a _MigrationHistory table
The _MigrationHistory table has one record:
MigrationID: 201204102350574_InitialCreate
CreatedOn: 4/10/2012 11:50:57 PM
Model: <Binary data>
ProductVersion: 4.3.1
Looking at other tables, most of them were emptied when this migration happened. Only the tables that were initially seeded with SampleData remained untouched.
Checking in with the SQL Azure Management portal, our Entity database shows the following creation date: 4/10/2012 23:50:55.
Here is our understanding
For some reason, SQL Azure deleted and recreated our database
The _MigrationHistory table was created in the process, registering a starting point to test the model against for future migrations
Here are our Questions
Who / What triggered the database deletion / recreation?
How could EF re-seed our sample data since Application_Start has System.Data.Entity.Database.SetInitializer<Entities>(null);?
EDIT: Looking at what could've gone wrong, we noticed one thing we didn't respect in this SQL Azure tutorial: we didn't remove PersistSecurityInfo from our SQL Azure Entity database connection string after the database was created. Can't see why on Earth it could have caused the problem, but still worth mentioning...
Nevermind, found the cause of our problem. In case anybody wonders: we hadn't made any Azure deployment since the addition of the pre-processor directives. MS must have restarted the machine our VM resided on, and the new VM recreated the database using see data.
Lesson learned: always do frequent Azure deployments.

Resources