I have an application that runs multiple concurrent background processes to insert data into the database using the Enterprise Library's Data Access application block.
Each of the background thread uses the DatabaseFactory.CreateDatabase passing in the same database instance name. The following is the snippet of code that retrieves the database and command object:
Microsoft.Practices.EnterpriseLibrary.Data.Database database = DatabaseFactory.CreateDatabase(this.DatabaseInstanceName);
DbCommand commandObj = database.GetSqlStringCommand(statement);
I'm finding that this is not thread safe and I'm getting errors due to the values getting mixed up across the threads. How should I handle this to ensure that it is thread safe?
thanks in advance!
I found my issue. The values that were getting mixed up across threads were not due to the Enterprise Library Data Access objects but another object I used to store parameters. I had accidentally made it global instead of a local resource within each thread.
Related
I have a backend NodeJS API and I am trying to setting trace id. What I have been thinking is that I would generate a UUID through a Singleton module and then use it across for logging. But since NodeJS is single-threaded, would that mean that UUID will always remain the same for all clients?
For eg: If the API gets a request from https://www.example.com/client-1 and https://www.example-two.com/client-2, would it spin a new process and thereby generate separate UUIDs? or it's just one process that would be running with a single thread? If it's just one process with one thread then I think both the client apps will get the same UUID assigned.
Is this understanding correct?
Nodejs uses only one single thread to run all your Javascript (unless you specifically create a WorkerThread or child_process). Nodejs uses some threads internally for use in some of the library functions, but those aren't used for running your Javascript and are transparent to you.
So, unlike some other environments, each new request runs in the same thread. There is no new process or thread created for an incoming request.
If you use some singleton, it will have the same value for every request.
But since NodeJS is single threaded, would that mean that UUID will always remains the same for all clients?
Yes, the UUID would be the same for all requests.
For eg: If the API gets a request from https://www.example.com/client-1 and https://www.example-two.com/client-2, would it spin a new process and thereby generate separate UUIDs?
No, it would not spin a new process and would not generate a new UUID.
or it's just one process that would be running with a single thread? If it's just one process with one thread then I think both the client apps will get the same UUID assigned.
One process. One thread. Same UUID from a singleton.
If you're trying to put some request-specific UUID in every log statement, then there aren't many options. The usual option is to coin a new UUID for each new request in some middleware and attach it to the req object as a property such as req.uuid and then pass the req object or the uuid itself as a function argument to all code that might want to have access to it.
There is also a technology that has been called "async local storage" that could serve you here. Here's the doc. It can be used kind of like "thread local storage" works in other environments that do use a thread for each new request. It provides some local storage that is tied to an execution context which each incoming request that is still being processed will have, even as it goes through various asynchronous operations and even when it returns control temporarily back to the event loop.
As best I know, the async local storage interface has undergone several different implementations and is still considered experimental.
See this diagram to understand ,how node js server handles requests as compared to other language servers
So in your case there won't be a separate thread
And unless you are creating a separate process by using pm2 to run your app or explicitly creating the process using internal modules ,it won't be a separate process
Node.js is a single thread run-time environment provided that internally it does assign threads for requests that block the event loop.
What I have been thinking is that I would generate a UUID through a
Singleton module
Yes, it will generate UUID only once and every time you have new request it will reuse the same UUID, this is the main aim of using the Singleton design pattern.
would it spin a new process and thereby generate separate UUIDs? or
it's just one process that would be running with a single thread?
The process is the instance of any computer program that can have one or multiple threads in this case it is Node.js(the process), the event loop and execution context or stack are two threads part of this process. Every time the request is received, it will go to the event loop and then be passed to the stack for its execution.
You can create a separate process in Node.js using child modules.
Is this understanding correct?
Yes, your understanding is correct about the UUID Singleton pattern. I would recommend you to see how Node.js processes the request. This video helps you understand how the event loop works.
I'm facing this issue in test server. but not in production. I tried some of the solutions like python manage.py runserver --noreload and edit
/lib/python3.6/site-packages/django/utils/autoreload.py this file.
Mentioned in the document.
https://github.com/django/django/commit/5bf2c87ece216b00a55a6ec0d6c824c9edabf188
This the error message look like,
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140000522213120 and this is thread id 140000744696768.
Please suggest me a solution to rectify this problem, Anyone faced this issue before. Help me to solve this issue.
The problem here is that SQLite has to deal with conflicts arising due to concurrent access by multiple threads i.e., SQLite database created and accessed by one thread cannot allow another thread to access it. This may result from following scenarios:
global connection objects are created which are then accessed later by different threads
connection objects are not closed properly between different connections
Its always recommended that an ORM is used to deal with databases and efficiently manage their connection lifecycles. For Sqlite, the most widely used ORM is SqlAlchemy. Using an ORM can probably fix the issue.
However, for very simple applications, where using an ORM is just an overkill, you can tweak the way connection is created to the Sqlite database by allowing concurrent access. This can be done by setting check_same_thread parameter to False while establishing the connection:
def initDB(self, file_path):
self.file_path = file_path
self.cx = sqlite3.connect(file_path, check_same_thread=False)
self.cx.execute(self.create_table_str)
self.cx.execute(self.create_detail_table_str)
print("init the table strucutre successfully")
Having said that, setting up Sqlite connection this way lays responsibility to handle concurrency on the application instead of the database and user should ensure that write operations to the database are serialized in order to avoid any dirty writes/updates.
Note: When using sqlalchemy, its important to use the right libraries and code segregation. I have particularly found this post helpful as well.
I am trying to turn my app multithreading. What I want to achieve is:
- Receive command via TidHTTPServer
- Execute local action (might involve using tidHTTP to send/receive data to other services)
- return execution result to the original caller
since I am pretty new to multi-threading I would like to know if my design-idea is correct
TMsgHandler=Class(TThread)
in TidHTTPServer.OnCommandGet I create a new instance of TMsgHandler and pass ARequestInfo and AResponseInfo
TMsgHandler.Excecute interprest the data
Can TMsgHandler.Execeute use Objects (descendants of TidHTTP) in my Main to communicate with other services?
TMsgHandler sends answer through AResponseInfo and terminates.
will this work?
This is not the correct design.
THTTPServer is a multi-threaded component. Its OnCommand... events are fired in the context of worker threads that Indy creates for you.
As such, you do not need to derive your TMsgHandler from TThread. Do your TIdHTTP directly in the context of the OnCommand... thread instead. A response will not be sent back to the client until your event handler exits (unless you send one manually). However, you should not share a single TIdHTTP from the main thread (unless you absolute need to, in which case you would need to synchronize access to it). You should create a new TIdHTTP dynamically directly in your OnCommand.../TMsgHandler code as needed.
I am writing an application to serve facebook APIs; share, like etc.. I am keeping all those shared objects from my appliction in a database and I do not want to share the same object if it already been shared.
Considering I will deploy application on different servers there could be a case where both instance tries to insert the same object to table.
How can I manage this concurrency problem with blocking the applications fully ? I mean two threads will try to insert same object and they must sync but they should not block a 3rd thread where it is inserting totally different object.
If there's a way to derive primary key of data entry from data itself, database will resolve such concurrency issue by itself -- 2nd insert will fail with 'Primary Key constraint violation'. Perhaps, data supplied by Facebook API already have some unique ID?
Or, you can consider some distributed lock solution, for example, based on Hazelcast or on similar data grid. This would allow to have record state shared by different JVMs, so it will be possible to avoid unneeded INSERTS.
In our ColdFusion application we have stateless model objects.
All the data I want I can get with one method call (it calls other internally without saving the state).
Methods usually ask the database for the data. All methods are read only, so I don't have to worry about thread safety (please correct me if I'm wrong).
So there is no need to instantiate objects at all. I could call them statically, but ColdFusion doesn't have static methods - calling the method would mean instantiating the object first.
To improve performance I have created singletons for every Model object.
So far it works great - each object is created once and then accessed as needed.
Now my worry is that all requests for data would go through only 1 model object.
Should I? I mean if on my object I have a method getOfferData() and it's time-consuming.
What if a couple of clients want to access it?
Will second one wait for the first request to finish or is it executed in a separate thread?
It's the same object after all.
Should I implement some kind of object pool for this?
The singleton pattern you are using won't cause the problem you are describing. If getOfferData() is still running when another call to that function gets called on a different request then this will not cause it to queue unless you do one of the following:-
Use cflock to grant an exclusive lock
Get queueing connecting to your database because of locking / transactions
You have too many things running and you use all the available concurrent threads available to ColdFusion
So the way you are going about it is fine.
Hope that helps.