MSAL access token getting unauthorized with MicrosoftIdentityWebApi Authentication - azure

Question

I set up my Startup Class below:
public void ConfigureServices(IServiceCollection services)
{
services.AddControllers();
services.AddMicrosoftIdentityWebApiAuthentication(Configuration);
}
// This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
if (env.IsDevelopment())
{
app.UseDeveloperExceptionPage();
}
app.UseHttpsRedirection();
app.UseRouting();
app.UseAuthentication();
app.UseAuthorization();
app.UseEndpoints(endpoints =>
{
endpoints.MapControllers();
});
}
My AzureAd appsettings.json is below:
"AzureAd": {
"Instance": "https://login.microsoftonline.com/",
"ClientId": "2f0d9252-e207-4d7f-b4da-*******",
"TenantId": "de1ef02c-7cfa-46b8-a02b-*******",
"Audience": "2f0d9252-e207-4d7f-b4da-*******"
}
Now my controller below:
[HttpGet]
public async Task<string> Get()
{
//To be put on appsettings.json
string clientId = "2f0d9252-e207-4d7f-b4da-0cc618e77c93";
string tenantId = "de1ef02c-7cfa-46b8-a02b-61ab78bc602b";
var app = PublicClientApplicationBuilder.Create(clientId)
.WithRedirectUri("http://localhost:5000")
.WithTenantId(tenantId)
.Build();
string[] scopes = new string[] { };
//Azure Login Success here
var result = await app.AcquireTokenInteractive(scopes).ExecuteAsync();
//Azure service graph get success.
string graphResult = await "https://graph.microsoft.com/beta/me"
.WithOAuthBearerToken(result.AccessToken)
.GetStringAsync();
//I pass the AccessToken from the result. But I'm getting UnAuthorized.
string authorizeResult = await "https://localhost:44328/weatherforecast/AuthorizeGet"
.WithOAuthBearerToken(result.AccessToken)
.GetStringAsync();
return graphResult;
}
[Authorize]
[HttpGet]
public async Task<string> AuthorizeGet()
{
return "Authorize";
}
The Azure Login will success here. But after I get the token and request to AuthorizeGet Api. It will give me Unauthorized 401.
Anything I missed out for the configuration?
Below is my Directory Authentication config.