I'm testing Azure SQL Serverless and from SSMS it seems to work fine, but from my ASP.NET Core application it never wakes up.
Using SSMS I can open a connection to a sleeping Serverless SQL database and after a delay the connection will go through.
Using my ASP.NET Core application I tried the same. From the login page I tried to login, which opens a connection to the database. After 10 or 11 seconds (I looked up the default timeout and its supposed to be 15 seconds but in this case it always seems to be about 10.5 seconds +/-0.5s). According to the docs, the first connection attempt may fail but subsequent ones should succeed, but I can send multiple queries to the database and it always fails with the following error:
Microsoft.Data.SqlClient.SqlException (0x80131904): Database 'myDb' on server
'MyDbSvr.database.windows.net' is not currently available. Please retry the connection later. If the
problem persists, contact customer support, and provide them the session tracing ID of
'{XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX}'.
If I wake the database up using SSMS then the login web page can connect to the database and succeeds.
I have added Connect Timeout=120; to the connection string.
The connection does happen during an HTTP request that is marked async on the Controller, thought I don't know if that makes any difference.
Am I doing something wrong or is there something additional I need to do to get the DB to wake?
[updte]
as an extra test wrote the following test
void Main()
{
SqlConnection con = new SqlConnection("Server=mydbsvr.database.windows.net;Database=mydb;User Id=abc;Password=xyz;Connect Timeout=120;");
Console.WriteLine(con.ConnectionTimeout);
con.Open();
var cmd = con.CreateCommand();
cmd.CommandText = "select getdate();";
Console.WriteLine(cmd.ExecuteScalar());
}
and got the same error.
I figured it out and its the dumbest thing.
This Azure SQL Server instance was migrated from another subscription and the group that migrated it gave it a new name, but they did something that allowed the use of the old name also. I'm researching to figure out how that was done. I will update this answer when I find out what that was.
As it turns out, using the old name with an Serverless Database won't wake up the db. Don't know why. But if you change to use the new/real server name it works. you do have to add a retry to the connection as it may fail the first few times.
[Update]
The new server allows logins using the old name by using a Azure SQL Database Alias https://learn.microsoft.com/en-us/azure/sql-database/dns-alias-overview
Related
I have looked through various posts related to this problem, but none provide an answer. I created a .Net 5.0 app that accesses an Azure SQL DB using EF 6.4.4 which works with .Net standard libraries. I modified the EF by adding a function that creates the connection string from appsettings.json since .Net 5 apps don't use a web.config file. This also works well in Azure with the configuration settings in an app service.
The connection string looks like this:
metadata=res://*/EF.myDB.csdl|res://*/EF.myDB.ssdl|res://*/EF.myDB.msl;provider=System.Data.SqlClient;provider connection string='Data Source=tcp:mydb.database.windows.net,1433;Initial Catalog=myDB;Integrated Security=False;Persist Security Info=False;User ID=myuserid#mydb;Password="password";MultipleActiveResultSets=True;Connect Timeout=120;Encrypt=True;TrustServerCertificate=True'
I also have a deployment pipeline that will deploy the code after a check-in instead of using the Visual Studio publish feature, but the pipeline deployed code has the same problem.
When I first created the app and published it to the app service, it worked. Recently I updated the app with no changes to the EF connection. Now I get the "Invalid Object name when I reference any table in the model. If I run the same code locally and connect to the Azure SQL DB, the DB is accessed as expected. This problem only occurs when running in the Azure app service. Note that there are no connection strings configured for the app service since the EF string is built from the config settings. I saw this post, but I don't think it applies:
Local works. Azure give error: Invalid object name 'dbo.AspNetUsers'. Why?
even though the problem is the same. I have also read various posts about the format of the EF connection string. Since my model is database first, (and the connection used to work), I'm confident the string has the correct format. I don't think the problem is in the code since it works when running locally with a connection to the Azure SQL DB. It must have something to do with the Azure app service configuration, but I'm not sure what to look for at this point. Unfortunately I don't have a copy of the code and publish files that did work to compare to, but it the pipeline build doesn't work either and that it how the code would normally be deployed. Thanks for any insight you might have!
UPDATE
metadata=res://*/EF.myDB.csdl|res://*/EF.myDB.ssdl|res://*/EF.myDB.msl;provider=System.Data.SqlClient;provider connection string='Data Source=tcp:yourdbsqlserver.database.windows.net,1433;Initial Catalog=yourdb;Persist Security Info=False;User ID=userid;Password=your_password;MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30'
When the troubleshooting problem is not on the string, our easiest way is to use vs2019 to re-use the generated string.
Your connection string should be like below.
<connectionStrings>
<add name="SchoolDBEntities" connectionString="metadata=res://*/SchoolDB.csdl|res://*/SchoolDB.ssdl|res://*/SchoolDB.msl;provider=System.Data.SqlClient;provider connection string="data source=.\sqlexpress;initial catalog=SchoolDB;integrated security=True;multipleactiveresultsets=True;application name=EntityFramework"" providerName="System.Data.EntityClient"/>
</connectionStrings>
For more details, you can refer my answer in the post and the tutorial.
1. Timeout period elasped prior to obtaining a connection from the pool - Entity Framework
2. Entity Framework Tutorial
The problem was one of my config settings in Azure. The catalog parameter was missing. A simple fix, but the error message was misleading, so I thought I would note that here in case anyone else gets the same "Invalid object name" message when referencing an Azure SQL DB with EF. It would have been more helpful if the message was "catalog name invalid" or "unable to connect to database".
For those who are curious about building an EF connection string, here is example code:
public string BuildEFConnectionString(SqlConnectionStringModel sqlModel, EntityConnectionStringModel entityModel)
{
SqlConnectionStringBuilder sqlString = new SqlConnectionStringBuilder()
{
DataSource = sqlModel.DataSource,
InitialCatalog = sqlModel.InitialCatalog,
PersistSecurityInfo = sqlModel.PersistSecurityInfo,
UserID = sqlModel.UserID, // Blank if using Windows authentication
Password = sqlModel.Password, // Blank if using Windows authentication
MultipleActiveResultSets = sqlModel.MultipleActiveResultSets,
Encrypt = sqlModel.Encrypt,
TrustServerCertificate = sqlModel.TrustServerCertificate,
IntegratedSecurity = sqlModel.IntegratedSecurity,
ConnectTimeout = sqlModel.ConnectTimeout
};
//Build an Entity Framework connection string
EntityConnectionStringBuilder entityString = new EntityConnectionStringBuilder()
{
Provider = entityModel.Provider, // "System.Data.SqlClient",
Metadata = entityModel.Metadata,
ProviderConnectionString = sqlString.ToString()
};
return entityString.ConnectionString;
}
Given what I have learned, the properties should be validated before the string is returned. If the string is created this way, all of the connection string properties can be added to the config settings in the app service. I used the options pattern to get them at runtime. Thanks to everyone for your suggestions.
I have trouble connecting to the Azure postgres database from python. I am following the guide here - https://learn.microsoft.com/cs-cz/azure/postgresql/connect-python
I have basically the same code for setting up the connection.
But the psycopg2 and SQLalchemy throw me the same error:
OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
I am able to connect to the instance by other client tools like dbeaver but from python it does not work.
When I investigate in Postgres logs I can see that the server actually authorized the connection but the next line says
could not receive data from client: An existing connection was forcibly closed by the remote host.
Python is 3.7
psycopg's version is 2.8.5
Azure Postgres region is in West Europe
Does someone has any suggestion on what should I try to make it work?
Thank you!
EDIT:
The issue resolved itself. I tried the same setup a few days later and it started working. Might have been something wrong with the Azure West Europe.
I had this issue too. I think I read somewhere (I forget where) that Azure has an issue with the # you have to for the username (user#serverName).
I created variables and an f-string and then it worked OK.
import sqlalchemy
username = 'user#server_name'
password = 'PassWord!'
host = 'server_name.postgres.database.azure.com'
database = 'your_database'
conn_str = f'postgresql+psycopg2://{username}:{password}#{host}/{database}'
After that:
engine = sqlalchemy.create_engine(conn_str, pool_pre_ping=True)
conn = engine.connect()
Test it with a simple SQL statement.
sql = 'SELECT * FROM public.some_table;'
results = conn.engine.execute(sql)
This was a connection in UK South. Before that it did complain about the format of the username having to use #, although the username was correct, as tested from the command line with PSQL and another SQL client.
I have a Glue job script that does this (not showing imports and setup here) and it inserts the row into SQL Server RDS just fine:
columns = ['test']
vals = [("test")]
df = sqlContext.createDataFrame(vals, columns)
test = DynamicFrame.fromDF(df, glueContext, "test")
datasink = glueContext.write_dynamic_frame.from_catalog(frame = test,
database = "database-name", table_name = "table-name")
job.commit()
When I run with this same connection but for a larger test load (ends up being about 100 rows) I get this error:
An error occurred while calling o596.pyWriteDynamicFrame. The TCP/IP connection to the host , port 1433 has failed. Error: "Connection timed out: no further information. Verify the connection properties. Make sure that an instance of SQL Server is running on the host and accepting TCP/IP connections at the port. Make sure that TCP connections to the port are not blocked by a firewall
The thing is that I know there's no firewall or security group issue since one row inserts just fine. I've tried adding a loginTimeout parameter to the JDBC connection like so:
jdbc:sqlserver://<host>:<port>;databaseName=dbName;loginTimeout=600;
As it indicates you can do so here. But the connection fails with Glue when I do that but succeeds when I remove the loginTimeout parameter.
I've also checked the remote timeout configuration on my SQL Server instance and it shows as 600 seconds which is longer than any of my failed jobs so it couldn't be that.
How can I get around this connection timeout error? It seems to be a limitation built into Glue.
In order to do a JDBC connection with Glue you need to follow the steps in this documentation: https://docs.aws.amazon.com/glue/latest/dg/setup-vpc-for-glue-access.html
We had done that but it turns out that our self-referencing sec group wasn't actually self-referencing. Once we changed that it got resolved
I also had to create the connection as an Amazon RDS connection and not as a JDBC connection even though it's doing the same thing under the hood.
Even after doing all that I still had issues. Turns out that you need to add the sql connection specifically to the job outside of the script. If you hit "Edit Job" you'll see a list of sql connections there. If the connection you're trying to hit isn't on the list of required connections you will always timeout
I can't get SqlDataProvider to work when executed in a fsx script which is running in an Azure Web Site.
I have started from the samples that Tomas Petrecek has here: https://github.com/tpetricek/Dojo-Suave-FsHome.
In short it is a FSX script that is executed using the IIS httpPlatformHandler so that all http requests to my Azure Web site is forwarded to my F# script.
The F# Script use Suave to handle the requests.
When I tried adding some database access to my HTTP handlers I got into problems.
The problematic code looks like this:
[<Literal>]
let connStr = "Server=(localdb)\\v11.0;Initial Catalog=My_Database;Integrated Security=true;"
[<Literal>]
let resolutionFolder = __SOURCE_DIRECTORY__
FSharp.Data.Sql.Common.QueryEvents.SqlQueryEvent |> Event.add (printfn "Executing SQL: %s")
// the following line fails when executing in azure
type db = SqlDataProvider<connStr, Common.DatabaseProviderTypes.MSSQLSERVER, ResolutionPath = resolutionFolder>
let saveData someDataToSave =
let ctx = db.GetDataContext(Environment.GetEnvironmentVariable("SQLAZURECONNSTR_QUERIES"))
.....
/// code using the context here
This works just fine when I run it locally, but when I deploy it to the azure site it will fail at the line where the type dbis created.
The error message is (line 70 is the line that has the type db = ...:
D:\home\site\wwwroot\app.fsx(70,11): error FS3033: The type provider
'FSharp.Data.Sql.SqlTypeProvider' reported an error: A network-related
or instance-specific error occurred while establishing a connection to
SQL Server. The server was not found or was not accessible. Verify
that the instance name is correct and that SQL Server is configured to
allow remote connections. (provider: SQL Network Interfaces, error: 52
- Unable to locate a Local Database Runtime installation. Verify that SQL Server Express is properly installed and that the Local Database
Runtime feature is enabled.)
The design-time database in the connStr is not available in the azure site, but I thought this is why we have the GetDataContext overload that takes a connection string to be used at run-time?
Is it because it is running as a script and not as compiled code that it is trying to access the database when creating the TypeProvider?
If yes, does it mean that my only option is to compile and provide the database code as a compiled assembly that I load and use in my Suave FSX script?
Reading the connection string from a config file does not work very well as this is in a azure site. I really need to get the connection string from an environment variable (which is set in the azure management interface).
Hmm, this is a bit unfortunate - as #Fyodor mentioned in the comments, the problem is that the script-based deployment to Azure actually compiles the script on the Azure machine - and so you need to have a statically-resolved connection string that works on Azure.
There are two options:
Use compiled project instead. If you compile your F# code locally and deploy the compiled code to Azure it will work. Sadly, there are no good samples for that.
Do some clever trick to make the connection string accessible to the script at compile time.
Send a PR to the SQL provider so that you can give it the name of an environment variable and it reads the connection string from there.
I think (3) would actually be quite nice and useful feature.
I'm not necessarily sure what the best way to do (2) would be. But I think you might be able to modify app.azure.fsx so that it creates a file (say connection.fsx) that contains something like:
module Connection
let [<Literal>] ConnString = "<Contents of SQLAZURECONNSTR_QUERIES>"
Then app.fsx could load this script and use Connection.ConnString in the argument of SQL type provider.
I am attempting to obtain a data feed from yahoo finance. I am doing this with the following code:
System.Net.WebRequest request = System.Net.WebRequest.Create(http://download.finance.yahoo.com/download/quotes.csv?format=sl&ext=.csv&symbols=^ftse,^ftmc,^ftas,^ftt1x,^dJA);
request.UseDefaultCredentials = true;
// set properties of the request
using (System.Net.WebResponse response = request.GetResponse())
{
using (System.IO.StreamReader reader = new System.IO.StreamReader(response.GetResponseStream()))
{
return reader.ReadToEnd();
}
}
I have placed this code into a console application and, using Console.WriteLine on the output I receive the information I require. I have used the 'Run as..' command to execute this using a specific domain account.
When I use this code from within a Page load I receive the following error message "No connection could be made because the target machine actively refused it 76.13.114.90:80".
This seems to suggest that the call is reaching yahoo (is this true?) and that there is something missing.
This would suggest there is an identity difference in the calls between the console application and application pool.
Environment is: Windows Server 2003, IIS 6.0, .net 4.0
"Target machine actively refused it" indicates that the TCP connection itself is not succeeding. This could be due to the fact that the Proxy settings when run under IIS are not the same as those that apply when you run in the console.
You can fix this by setting a WebProxy on your request, that points to the proxy server being used in the environment.
Yes, an active refusal is indication that the target machine is receiving the request and the information in the headers is either incorrect or insufficient to process the request. It is entirely possible that if you had to run this call using a "run as" command in console that the application pool's identity user does not have the appropriate permission or username. You can attempt to change the identity user to this specific domain account to see if that alleviates the problem, but you may have to isolate this particular function into its own application pool in order to protect the rest of the website from having this specification.