I would like to allow two threads to write in a table at the same time (I know the problem of updating the same row, but this would be a story apart). I need that in behalf of speed up the operations in my aplication (one thread could write in row X while another could do the same in row X+n instead of waiting the first to finalize).
So, can I block rows instead of tables with Linq to SQL?
Thanks.
LINQ to SQL does not block any tables. It simply sends SQL queries to your SQL Server database and it is up to SQL Server to deside how to block. However, when your database has the proper design (right indexes etc) SQL server will not lock your whole table when you've queried a single row and update that row. Therefore, you can expect that scenario to just work.
Related
Azure external tables between two azure sql databases on the same server don't perform well. This is known. I've been able to improve performance by defining a view from which the external table is defined. This works if the view can limit the data set returned. But this partial solution isn't enough. I'd love a way to at least nightly, move all the data that has been inserted or updated from the full set of tables from the one database (dbo schema) to the second database (pushing into the altdbo schema). I think Azure data factory will let me do this, but I haven't figured out how. Any thoughts / guidance? The copy option doesn't copy over table schemas or updates
Data Factory Mapping Data Flow can help you achieve that.
Using the AlterRow active and select an uptade method in Sink:
This can help you copy the new inserted or updated data to the another Azure SQL database based on the Key Column.
Alter Row: Use the Alter Row transformation to set insert, delete, update, and
upsert policies on rows.
Update method: Determines what operations are allowed on your
database destination. The default is to only allow inserts. To
update, upsert, or delete rows, an alter-row transformation is
required to tag rows for those actions. For updates, upserts and
deletes, a key column or columns must be set to determine which row
to alter.
Hope this helps.
I have currently setup multiple Azure DBs on a single server primarily with different schemas.
External Queries work perfectly fine on small tables. I am currently using an INNER JOIN on multiple tables within 2 DBs.
This works great for small tables with limited data sets since it appears to be physically copying the tables over to a temp table then performing the query.
However when I do a join on a large table ~500K rows the query will fail as the size of the table causes a timeout while it tries to copy the table to the temp directory.
Is there a way to execute the query without copying the JOIN table to a temp directory?
I have previously tried to create Stored procedures on the DB with the Large Table I am trying to join, however that DB will eventually be sunset and I will be back where I am now so I would like a longer term solution.
Alternately consider consolidating your separate databases into one single database, eg using schemas to provide separation. Ask yourself the question "Why are my databases split?" or is having to join across them an occasional thing? Do you need them to be split. If having to join across them is a regular task then consolidating them makes sense.
Alternately consider Managed Instance. This is a PaaS service but gives you an experience closer to traditional SQL Server. First off, you can have multiple databases in once instance and cross-database joins are as easy as they are in box product SQL Server.
I ended up using a CROSS APPLY (Inner Join) and OUTER CROSS APPLY (Left Join) then adding the logic to a where statement within
Select c.Id, a.Quantity, b.Name
From Table1 a CROSS APPLY
(Select * FROM Table2 WHERE x = y) c
INNER JOIN Table3 b ON c.x = b.x
Add Join
Add Join
This executed the joins on the target without having to bring the whole table into a TempTable.
I am supposed to optimize the performance of an old Access DB in my company. It contains several tables with about 20 columns and 50000 rows. The speed is very slow, because the people work with the whole table and set the filters afterwards.
Now I want to compose a query to reduce the amount of data in Excel before transfering the complete rows, but the speed is still very slow.
First I tried the new power query editor from Excel. I first reduced the rows by selecting only the last few ones (by date). Then I made an inner join with the 2nd table.
Finally I got less than 20 rows returned, and I thought I was fine.
But when I started Excel to perform the query, it took 10 - 20 seconds to read the data. I could see, Excel loads the complete tables, before setting the filters.
My next try was to create the same query direcly inside the Access DB, same setting. Then I opened this query in Excel, and the time to load the rows is nearly zero. You select "refresh", and the result is shown instantly.
My question is: Is there any way to perform a query in Excel only (without touching the Access file), that is nearly as fast as a query in Access itself?
Best regards,
Stefan
Of course.
Just run an SQL query from MS Query in Excel. You can create the query in Access, and copy-paste the SQL in MS Query. They're executed by the same database engine, and should run at exactly the same speed.
See this support page on how to run queries using MS Query in Excel.
More complex solutions using VBA are available, but shouldn't be needed.
Suppose I have two tables, TableA(embedded data), TableB(external data).
Scenario 1:
TableB is set On-Demand based on the markings from TableA. When you mark something from TableA, it take some "n" seconds to populate the data in TableB. On-Demand setting on external table is like screenshot named LOD.png
Scenario 2:
On-Demand settings have not been induced on TableB(please note TableB still is External). There has been a relationship created between TableA and TableB. TableB is now limited based on marking from TableA by the option"Limit data using Markings".screenshot named ss2
Questions:
1. Which scenario fetches data quicker.
2. From the debug log, the query passed in both the scenario is the same.Does that mean both scenarios are same or are they different?
Scenario 1 is good if Table B is really large, or records take a long time to fetch from the database. In this case, Table B is only returning rows that are based on what you marked in Table A. This means that the number of rows could be significantly less, but this also means that every time the marking changes, those rows have to be fetched at that time. If the database takes a long time, this can become frustrating. On the flip side, if the database is really fast and you are limiting rows down enough, this can be almost seamless. At the end of the day, you are pulling this data into memory after the query runs, so all Spotfire functionality is available.
Scenario 2 is good if calculations are highly complex and need to take advantage of the power of the external DB to perform. This means that any change to the report, a change of visualization etc., will require a new query to be sent to the external data source resulting in a new table of aggregated data. This means that no changes to a visualization using an in-db data table can be made when you are not connected to the external data source. Please note, there is functionality in Spotfire that is available to in memory data like data on demand that is not available to external data.
I personally like to keep data close to Spotfire to take advantage of all Spotfire functionality, but I cannot tell you exactly which is the correct method in your case. Perhaps these TIBCO links on the difference between in memory data and external data can help:
https://docs.tibco.com/pub/spotfire/6.5.1/doc/html/data/data_overview.htm
https://docs.tibco.com/pub/spotfire/6.5.1/doc/html/data/data_working_with_in-database_data.htm
Business Case:
I have a list of key IDs in an excel spreadsheet. I want to use Power Query to join these IDs with a details table in a SQL Server database.
Problem
Currently using Power Query I only know how to import the entire table, which is greater than 1 million records, then do a left join on it against an existing query that targets a local table of IDs.
What I want to do is send that set of IDs in the original query so I'm not pulling back the entire table and then filtering it.
Question
Is there an example of placing an IN clause targeting a local table similar to what is shown below?
= Sql.Database("SQLServer001", "SQLDatabase001",
[Query="SELECT * FROM DTree WHERE ParentID
IN(Excel.CurrentWorkbook(){[Name="tbl_IDs"]}[Content])"])
I would first build a "Connection only" Query on the excel spreadsheet key IDs.
Then I would start a new Query by connecting to the SQL table. In that query I would add a Merge step to apply the key IDs query as an Inner Join (filter).
This will download the 1m rows to apply the filter, but it is surprisingly quick as this is mostly done in memory. It will only write the filtered result to an Excel table.
To improve performance, filter the rows and columns as much as you can before the Merge step.