SharePoint Online multi-tenant REST calls return 404 on resources that definitely exist - sharepoint

Question

I am attempting to access the SharePoint Online REST API (this is hand coded REST calls, no library being used).
Access tokens are acquired using authorization grant flow as follows:
I send the browser https://login.microsoftonline.com/common/oauth2/authorize?...
This redirects to a handler endpoint that we extract the access code from
I obtain the tenant ID by:
GET https://{tenantname}.sharepoint.com/_vti_bin/client.svc
Then extracting the tenant ID from the WWW-Authenticate header
I then POST https://login.microsoftonline.com/{tenantid}/oauth2/authorize to obtain the access token
When I use that access token, I am able to do queries using:
GET https://{tenantname}.sharepoint.com/_api/search/query?querytext=....
This works and returns documents.
But when I attempt to retrieve information about one of those documents:
GET https://{tenantname}.sharepoint.com/_api/web/getfilebyserverrelativeurl('/TestFiles/test.pdf')
I get a 404 response with the following body:
{"odata.error":{"code":"-2130575338, Microsoft.SharePoint.SPException","message":{"lang":"en-US","value":"The file /TestFiles/test.pdf does not exist."}}}
If I navigate to the URL in a browser (https://{tenantname}.sharepoint.com/TestFiles/test.pdf), it accesses the file without issue.
This makes me think that I'm running into some sort of permission issue.
I have tried setting the following scopes in the authorize redirect:
Attempt 1: scope = Web.Write AllSites.Write Site.Write
Attempt 2: scope = https://{tenantname}.sharepoint.com/.default
Attempt 3: scope = https://{tenantname}.sharepoint.com/Web.Write https://{tenantname}.sharepoint.com/AllSites.Write https://{tenantname}.sharepoint.com/Site.Write
No matter what I set as the scope parameter of the authorize URL, the JWT details of the access token show (I can post the entire decoded JWT if anyone needs it):
"scp": "User.Read"
Nothing I do has any impact on the scp in the token - I have no idea if that's the issue or not. If it is, I would appreciate hearing how to properly request scope.
The application registration in Azure Active Directory has desired permissions (plus more):
What am I doing wrong?
UPDATE: Switching to OAuth endpoint v2.0:
https://login.microsoftonline.com/common/oauth2/v2.0/authorize
With query parameters:
response_type = code
client_id = my app id
redirect_uri = my redirect uri
scope = <varying - I'll explain what happens under different scenarios below>
Here's what I've tried for scopes:
AllSites.Write Site.Write - the redirect has invalid_client with error_description = AADSTS650053: The application '' asked for scope 'AllSites.Write' that doesn't exist on the resource '00000003-0000-0000-c000-000000000000'. Contact the app vendor.
https://{tenantname}.sharepoint.com/AllSites.Write https://.sharepoint.com/Site.Write - the redirect has invalid_client with error description = AADSTS650053: The application '' asked for scope 'Site.Write' that doesn't exist on the resource '00000003-0000-0ff1-ce00-000000000000'. Contact the app vendor.
https://{tenantname}.sharepoint.com/.default - this goes through
But the resulting JWT has only scp=User.Read
The following works: GET https://{tenantname}.sharepoint.com/_api/search/query?querytext=
But the following returns a 404: GET https://{tenantname}.sharepoint.com/_api/web/getfilebyserverrelativeurl('/TestFiles/test.pdf')
I don't understand how Scope=.Default isn't including the allowed permissions from the application registration. And I definitely don't understand why the AllSites.Write scope is failing when it's explicitly specified.
If it helps, I have also tried all of the above using a tenant specific authorize endpoint instead of 'common':
https://login.microsoftonline.com/{tenantid}/oauth2/v2.0/authorize
UPDATE2: More scope changes:
I finally found a magical combination that works:
Use a tenant based URI for the /authorize and /token endpoint and use {tenanturl}\AllSites.Write for the scope (do NOT specify the Site.Write scope):
https://login.microsoftonline.com/{tenantid}/oauth2/v2.0/authorize?response_type=code&client_id={clientid}&redirect_uri={redirecturi}&scope=https%3A%2F%2F{tenantname}.sharepoint.com%2FAllSites.Write
The resulting JWT has the following:
"scp": "AllSites.Write User.Read"
I am completely perplexed about why Site.Write wasn't allowed. I suppose that AllSites.Write is a superset of Site.Write, so maybe not needed?
All of my testing so far has been on my own tenant, next step is to test on a different tenant and make sure it actually works there as well.

Answer 1

It seems you use v1.0 endpoint https://login.microsoftonline.com/common/oauth2/authorize but not v2.0 endpoint https://login.microsoftonline.com/common/oauth2/v2.0/authorize. If we use v1.0 endpoint, we should use resource instead of scope. So that is why the scp claim in your access token always the same no matter you modify the scope.
You should use resource with https://{tenant-name}.sharepoint.com and the parameter scope is useless when you use v1.0 endpoint.
If you still want to use scope parameter, you can also change the endpoint to v2.0. Just add v2.0 into your endpoint, like: https://login.microsoftonline.com/common/oauth2/v2.0/authorize

Answer 2

I finally found a magical combination that works:
use the https://login.microsoftonline.com/{tenantid}/oauth2/v2.0/authorize and https://login.microsoftonline.com/{tenantid}/oauth2/v2.0/token endpoints
specify {tenanturl}\AllSites.Write for the scope (do NOT specify the Site.Write scope - that was the primary problem):
https://login.microsoftonline.com/{tenantid}/oauth2/v2.0/authorize?response_type=code&client_id={clientid}&redirect_uri={redirecturi}&scope=https%3A%2F%2F{tenantname}.sharepoint.com%2FAllSites.Write
The resulting JWT has the following: "scp": "AllSites.Write User.Read"
This works across tenants and gets us the access we need.
For thoroughness, we also specify offline_access scope so we get a refresh_token in addition to the access_token.