I am trying to use multi thread for connecting CData drivers. Whether is it possible for parallel processing of data in CData.
OdbcConnection conn = new OdbcConnection();
conn.ConnectionString = "xxxx";
Task task1 = Task.Factory.StartNew(() => ReadData(conn));
Task task2 = Task.Factory.StartNew(() => ReadData(conn));
Task task3 = Task.Factory.StartNew(() => ReadData(conn));
Task task4 = Task.Factory.StartNew(() => ReadData(conn));
Task task5 = Task.Factory.StartNew(() => ReadData(conn));
public static void ReadData(OdbcConnection con)
{
con.Open();
// code
}
Please let me know it there is any solution. Thanks in advance.
This is possible. If you set the OAuthSettingsLocation property using the Other property, you can perform parallel processing of data. You can set the property in the DSN or dynamically in a DSN-less connection (see the code sample below).
OAuthSettingsLocation - The location of the settings file where OAuth values are saved. This can be any location on disk for which the driver has read/write permissions.
Performing OAuth Using the DSN Manager
You will need to authenticate with the service before using the driver. To do so, you can use the Test Connection in the DSN Wizard.
Open the ODBC Data Source Administrator (from the Start Menu, type "ODBC")
Select or create a new DSN using the driver.
Set the OAuthSettingsLocation property in the Other property and click Test Connection. (You will be prompted to authenticate with the service in a new browser window.)
Code Sample
string driver = "CData ODBC Driver for QuickBooksOnline";
string oauthSettingsLocation = "C:/users//AppData/Roaming/CData/QuickBooksOnline ODBC Driver";
string connString = "DRIVER={" + driver + "};Other=OAuthSettingsLocation=" + oauthSettingsLocation;
Task task1 = Task.Factory.StartNew(() => ReadData(new OdbcConnection(connString)));
Task task2 = Task.Factory.StartNew(() => ReadData(new OdbcConnection(connString)));
Task task3 = Task.Factory.StartNew(() => ReadData(new OdbcConnection(connString)));
Task task4 = Task.Factory.StartNew(() => ReadData(new OdbcConnection(connString)));
Task task5 = Task.Factory.StartNew(() => ReadData(new OdbcConnection(connString)));
NOTE
CData Software also makes ADO.NET Providers that might provide a more native experience in .NET applications.
Related
I want to run each TPL task in a separate thread (the idea is having TPL advantages in the same time with working with a separate threads). It looks like this task scheduler is exactly what I'm looking for: ThreadPerTaskScheduler .
I made several local tests and I see that it works as I expected including the ability to call Task.WaitAll.
var task = Task.Factory.StartNew(() =>
{
Thread.Sleep(10000);
}, CancellationToken.None, TaskCreationOptions.None, new ThreadPerTaskScheduler());
Task.WaitAll(task);
But, I have a question about this line from the task scheduler implementation:
protected override void QueueTask(Task task)
{
new Thread(() => TryExecuteTask(task)) { IsBackground = true }.Start();
}
as I see, we just create a new thread without saving any reference on this thread anywhere. If so, how does Task.WaitAll work?
I attempt to update the task status progress (Status Assignment) using project server 2013 CSOM, but I got "Unknown Error" Exception and the following StackTrace:
at
Microsoft.SharePoint.Client.ClientRequest.ProcessResponseStream(Stream
responseStream) at
Microsoft.SharePoint.Client.ClientRequest.ProcessResponse() at
Limitless.Components.Project2013.ProjectComponent.UpdateTask(ProjectServerConnection
connection, Guid ProjectUID, List`1 data)
ProjectContext context = GetProjectContext(connection);
// Get the user name and their assignments
EnterpriseResource self = EnterpriseResource.GetSelf(context);
context.Load(self, r => r.Name, r => r.Assignments
.IncludeWithDefaultProperties(a => a.Project, a=>a.Comments));
context.ExecuteQuery();
foreach(var item in self.Assignments)
{
Entities.Task task = data.Where(t => t.ID == item.Id).SingleOrDefault();
if(task!= null)
{
item.PercentComplete = (short)task.PercentComplete;
item.Comments = "comment";
}
}
// Update the assignments and submit the status updates.
self.Assignments.Update();
self.Assignments.SubmitAllStatusUpdates("By PS Web App");
context.ExecuteQuery();
I couldn't find any error in the sharepoint logs. its a fresh installation Project Server 2013, and it retrieves Tasks (StatusAssignments) successfully, but after self.Assignments.Update(); and self.Assignments.SubmitAllStatusUpdates("By PS Web App"); the exception is thrown on context.ExecuteQuery();.
any one came cross such scenario ?
I hope it's not that late for anyone who face the same issue, this was solved and working properly but I'm not sure if it is the right way.
instead of updating the TaskAssignment by the current user context, I get it using elevated context (admin context for example), and get resource's assignments, and update them.
//private function to get admin context
ProjectContext elevatedContext = GetProjectElevatedContext(connection);
//private function to get current user context
ProjectContext context = GetProjectContext(connection);
// Get the user name and their assignments
EnterpriseResource self = EnterpriseResource.GetSelf(context);
context.Load(self, r => r.Id, r => r.Assignments.IncludeWithDefaultProperties(a => a.ActualFinish));
context.ExecuteQuery();
var resource = elevatedContext.EnterpriseResources.GetById(self.Id.ToString());
elevatedContext.Load(resource, r => r.Assignments
.IncludeWithDefaultProperties(a => a.Project, a => a.Comments, a => a.PercentComplete));
elevatedContext.ExecuteQuery();
int count = 0;
foreach (var item in resource.Assignments)
{
Entities.Task task = data.Where(t => t.ID == item.Id).SingleOrDefault();
if (task != null)
{
item.PercentComplete = (short)task.PercentComplete;
item.Comments = "comment";
count++;
}
}
resource.Assignments.Update();
resource.Assignments.SubmitAllStatusUpdates($"comment");
elevatedContext.ExecuteQuery();
I couldn't find a permission that is granted for such functionality from CSOM.
If anyone has a better tested solution please contribute :)
Using the Transient Fault Handling Application Block with SQL Azure.
In this sample is a specific retry logic for oCmd.ExecuteReader() is mandatory or does the ReliableSqlConnection takes care of it ?
Using oCmd As New SqlCommand()
strSQL = "SELECT xxxxxx.* FROM xxxxxx"
oCmd.CommandText = strSQL
Using oConn As New ReliableSqlConnection(Cs, retryPolicy)
oConn.Open()
oCmd.Connection = oConn
Using oDR As SqlDataReader = oCmd.ExecuteReader()
If oDR.Read() Then
sb1.Append(oDR("xxxxxx").ToString)
End If
End Using
End Using
End Using
* UPDATE *
From response below, If I create the SqlCommand object from the context of the ReliableSqlConnect object i can expect to have the retry behaviour extended to the command also, as stated in this page
http://geekswithblogs.net/ScottKlein/archive/2012/01/27/understanding-sql-azure-throttling-and-implementing-retry-logic.aspx
"This next code example below illustrates creating a retry policy using the RetryPolicy class, specifying the retry attempts and the fixed time between retries. This policy is then applied to the ReliablSqlConnection as both a policy to the connection as well as the policy to the command."
RetryPolicy myretrypolicy = new RetryPolicy<SqlAzureTransientErrorDetectionStrategy>(3, TimeSpan.FromSeconds(30));
using (ReliableSqlConnection cnn = new ReliableSqlConnection(connString, myretrypolicy, myretrypolicy))
{
try
{
cnn.Open();
using (var cmd = cnn.CreateCommand())
{
cmd.CommandText = "SELECT * FROM HumanResources.Employee";
using (var rdr = cnn.ExecuteCommand<IDataReader>(cmd))
{
//
}
}
}
catch (Exception ex)
{
MessageBox.Show(ex.Message.ToString());
}
}
ReliableSqlConnection applies retry logic only to the process of establishing a connection. Try using something like this:
Using oDR As SqlDataReader = oCmd.ExecuteReaderWithRetry(retryPolicy)
If oDR.Read() Then
sb1.Append(oDR("xxxxxx").ToString)
End If
End Using
For further information on usage your can refer to MSDN or there are some additional examples here.
I have following code which will fire query on each database in SQL SERVER 2008R2,
public DataTable GetResultsOfAllDB(string query)
{
SqlConnection con = new SqlConnection(_ConnectionString);
string locleQuery = "select name from [master].sys.sysdatabases";
DataTable dtResult = new DataTable("Result");
SqlCommand cmdData = new SqlCommand(locleQuery, con);
cmdData.CommandTimeout = 0;
SqlDataAdapter adapter = new SqlDataAdapter(cmdData);
DataTable dtDataBases = new DataTable("DataBase");
adapter.Fill(dtDataBases);
// This is implemented for sequential
foreach (DataRow drDB in dtDataBases.Rows)
{
locleQuery = " Use [" + Convert.ToString(drDB[0]) + "]; " + query;
cmdData = new SqlCommand(locleQuery, con);
adapter = new SqlDataAdapter(cmdData);
DataTable dtTemp = new DataTable();
adapter.Fill(dtTemp);
dtResult.Merge(dtTemp);
}
//Parallel Implementation
Parallel.ForEach(dtDataBases.AsEnumerable(), drDB =>
{
locleQuery = " Use [" + Convert.ToString(drDB[0]) + "]; " + query;
con = new SqlConnection(_ConnectionString);
cmdData = new SqlCommand(locleQuery, con);
cmdData.CommandTimeout = 0;
adapter = new SqlDataAdapter(cmdData);
DataTable dtTemp = new DataTable();
adapter.Fill(dtTemp);
dtResult.Merge(dtTemp);
}
);
return dtResult;
}
Now the problem is when i use the second loop i.e. Parallel ForEach loop it gives me different errors at the line adapter.Fill(dtTemp); as following,
Yes of-course these are expected errors.
Connection is closed
Connection is opening,
Data reader is closed
reader is connecting..
Blha Blha ... All connection related errors.
Note : Some time it works like charm i mean no errors.
And absolutely the first loop i.e. Sequential foreach loop works fine but the performance is not that good looking which i fall in Love with it :)
Now my question is if i want to use parallel foreach loop for the same, then How should i do this? Is there any cosmetics which will help the Parallel Foreach loop good looking ;)
Thanks in Advance.
The database connection can only run one query at a time, so when a thread tries to run a query while the connection is busy, you get an error. If you want to run queries in parallel each thread needs its own database connection.
Though you are creating a new SqlConnection object for each iteration, all the objects are using the same physical database connection. This is because of connection pooling used by the .NET framework. In your case you need to manually configure the behavior of the connection pool. For example you can disable connection pooling; this will have performance implications. Read about connection pooling at MSDN.
trying to grasp the TPL.
Just for fun I tried to create some Tasks with a random sleep to see how it was processed. I was targeting a fire and forget pattern..
static void Main(string[] args)
{
Console.WriteLine("Demonstrating a successful transaction");
Random d = new Random();
for (int i = 0; i < 10; i++)
{
var sleep = d.Next(100, 2000);
Action<int> succes = (int x) =>
{
Thread.Sleep(x);
Console.WriteLine("sleep={2}, Task={0}, Thread={1}: Begin successful transaction",
Task.CurrentId, Thread.CurrentThread.ManagedThreadId, x);
};
Task t1 = Task.Factory.StartNew(() => succes(sleep));
}
Console.ReadLine();
}
But I don't understand why it outputs all lines to the Console ignoring the Sleep(random)
Can someone explain that to me?
Important:
The TPL default TaskScheduler does not guarantee Thread per Task - one thread can be used for processing several tasks.
Calling Thread.Sleep might impact other tasks performance.
You can construct your task with the TaskCreationOptions.LongRunning hint this way the TaskScheduler will assign a dedicated thread for the task and it will be safe to block on it.
Your code uses the value of i instead of the generated random number. It does not ignore the sleep but rather sleeps between 0 and 10ms each iteration.
Try:
Thread.Sleep(sleep);
The sentence
Task t1 = Task.Factory.StartNew(() => succes(sleep));
Will create the Task and automatically start it, then will iterate again inside the for, without waiting the task to end its process. So when the second task is created and executed, the first one may be finished. I mean you are not waiting for the tasks to end:
You should try
Task t1 = Task.Factory.StartNew(() => succes(sleep));
t1.Wait();