QnA maker - Different Results between REST API and Preview Page - azure

I'm using the Azure QnA Version 4. I'm posting using the REST API.
If I'm posting against the Live-Database using the parameter isTest=true I'm getting an answer score of around 80% which is very reasonable as my question almost matches the database. I'm getting exactly the same result using the Webinterface on qnamaker.ai.
Using the same POST against the published version (without isTest=true) I'm getting a score of only around 13% (which is very odd for entering almost a question which matches the database).
I've found some hints within the FAQs that slight differences are normal but I don't think 67% difference is normal. Is there anything I can do, so that the published version gets scores closer to the test version?

Pursang has a good point on his answer.
A good way to solve this problem is adding "isTest: true" on QnAMaker post request body. It has worked for me.
Its a qnaMaker bug when we have to add multiples knowledge bases...
{"question":"your question here", "top":3, "isTest": true }
Good Luck!

The test version and the published version are two different knowledge bases. This allows you to make changes and test them without affecting the live knowledge base that your customers are using. If you're getting worse results with your published knowledge base than your test version, that seems to indicate that you've trained your test knowledge base after you've published. Publishing again may fix the issue.
If you publish again and your published version still doesn't seem to behave the same as the test version, consider this entry in the FAQ:
The updates that I made to my knowledge base are not reflected on publish. Why not?
Every edit operation, whether in a table update, test, or settings,
needs to be saved before it can be published. Be sure to click the
Save and train button after every edit operation.

I had the same exact problem. It was related to something going wrong when I created the QnA Service in Azzure. The Language of your QnA Knowldege Base is automatically detected. You can see your Language in your Azure Search Ressource=>testkb=>Fields=>question/awnser MSDN
Mine was set to Standard-Lucene instead of German-Microsoft. I did not find any way to change that, so I had to recreate the QnA Service and move all Knowledge Bases there. Example picture wrong language Example picture correct language

I'm using a QnA service created in February this year. There are discrepancies between the test (QnA portal) & the published version (api). A correct answer would drop 10%, while a bad answer rises 10%, which ultimately converts good matches in test into bad ones in the bot application. Try to explain that to your customer.
It appears that you can run into this trouble if you use multiple KBs (= knowledge bases) on a single search service. The test index is a single index that covers all your KBs for that search service, while production KBs, when published, are indexed separately per KB. The QnA Maker help bot on the QnA portal mentions this:
"The top answer can sometimes vary because of small score variations between the test and production indexes. The test chat in portal hits the test index, and the generateAnswer API hits the production index. This typically happens when you have multiple knowledge bases in the same QnA Maker service. Learn more about confidence score differences.
This happens because all test knowledge bases are combined into a single index, while prod knowledge bases are on separate indexes. We can help you by separating all test and prod into separate indexes for your service."
So we need to contact Microsoft to also split up the test index per KB ? So that will rectify any discrepancies between test & published version ? Did not try this yet, anyone else?
Or do we limit ourselves to a single KB per search service (= multiple search services = expensive).
Or do we put all in a single KB, and use metadata to logically separate the answers and pray that this single massive KB produces good enough results ?

Related

Why are the download numbers for #azure/identity, #azure/msal-node, auth0, and passport currently declining? What is replacing those?

All of those packages have a declining number of downloads lately? Is there a new way to authorize to Azure AD? Or is it related to JavaScript? What is the currently preferred way to authorize and authenticate with OAuth2.0 against Azure?
Source of graph
If you change the "Downloads in past:" UI input field to "All time", you will see that there are these dramatic drops every year at around the same time of year. It's the holiday season. Lots of people are on work holiday or off school or just taking a break from working on personal projects. That's why there are fewer package downloads. You'll also see that the overall trend for the packages you listed is upward growth. You'll also see similar patterns for pretty much every package on the main NPM registry.
I.e. the data does not indicate that their usage is on an overall decline (it's the opposite). I'm pretty sure your misinterpretation of the data invalidates your other questions.

Does Clarity Actually Push Data to Google Analytics?

Does Clarity Actually Push Data to Google Analytics?
Context A:
I started a project for our company for internal intranet analytics so they can learn more about which pages work, which ones don't, popular documents, etc. I found Clarity (and thought this is great) it says that it pushes information to Google Analytics, but that doesn't seem to be the case.
Additional context at the bottom to ensure question clarity, pun intended.
Confusing / Misleading Documentation:
Microsoft can you answer this question or reply to my other emails that you and I were working back and forth on? Your documentation says one thing (the quote below), but that doesn't seem to be accurate.
How does Clarity integration work?
Clarity will create a custom dimension called the 'Clarity Playback URL'. Data will start flowing into Google Analytics from the Clarity JavaScript you've added to your site without any more changes.
My Question:
So is this accurate or not, does Clarity pass information over or is this incorrectly describing Clarity behavior and that the two need to independently gather data for it to work correctly?
Initial Microsoft Response:
Sometimes it takes about 1 - 2 days to populate the initial data, if you still don’t see the data by then, you can reach out again!
Second Microsoft Response:
The account/web property that is connected has no rows in it for any query. Could you please double check and connect an account which contains data.
Final Thoughts:
So which one is it? There is no documentation on this and the project I am working on is at a standstill because of this, please advise so I can either continue forward with this or move on to a different solution because ms-clarity doesn't seem to be working as documented.
Context B:
I originally started in on this project because the previous Google Analytics that were linked into the intranet stopped working after modern / online updates (not sure which one did it, that all happened before I got here) and I had to get special approval to use Clarity and we only went with Clarity because of this piece in the documentation that essentially told us we could re-make the link between the intranet and Google Analytics as well as get more functionality coupled with Clarity.
We also did not want to do a weird patch job to get Google Analytics injected into pages and I told them a SPPKG of Clarity would do the trick... however, here we are now.
Clarity DOES NOT push data to Google Analytics. You have to gather data using Clarity and Google Analytics SEPERATLEY, and then when you integrate Clarity will work with Google Analytics to bring the data together.
So you CANNOT only install Clarity, and then expect to integrate with Google Analytics to push Clarity gathered data to Google Analytics.

OpenAI API and GPT-3, not clear how can I access or set up a learning/dev?

I am reading tons of GPT-3 samples, and came cross many code samples.
None of them mentions that how and where I can run and play with the code myself... and especially not mentioning I can not.
So I did my research, and concluded, I can not, but I may be wrong:
There is no way to run the "thing" on-premises on a dev machine, it is a hosted service by definition (?)
As of now (Oct. 11th 2020) the OpenAI API is in invite only beta (?)
Did I miss something?
As of now (Oct. 11th 2020) the OpenAI API is in invite only beta (?)
Yes, that is correct! Check out their API introduction and application form. On there you'll find the necessary information to apply (in case you want that).
There is no way to run the "thing" on-premises on a dev machine, it is a hosted service by definition
Well, you have access to their API. You either can use the inbuilt playground or access the API via a HTTP request (and therefore via most programming languages). But there isn't much coding to be done as you only have a few parameters to pass into the request - e.g. the amount of tokens, temperature, etc.
They even have a function which writes the code for you (though that's probably not necessary):
As you can see, you're able to write (and test) your scripts with the different settings and engines. Once you set it up properly, you may simply export the code. That obviously shouldn't be used in production as your software is (presumably) more in-depth then just one standardized call.
As of Nov 18 2021, the API is now available publicly with no waitlist. You just need to sign up for an account and get an API key (instant access). You need to pay per usage, but you get some free usage to start with.
Pricing for Da Vinci (the main/best GPT-3 model) is $0.006 per 1000 tokens (approx 750 words). So for about 1 cent, you can get it to read and/or write about 1500 words.
https://openai.com/blog/api-no-waitlist/

Data lost after a while of adding it in Dialogflow Is it a storage or server problem?

We are working on Dialogflow for a year now, and lately, we experienced some issues regarding stabilization, new behavior and new features causing problems. Among those problems is that when adding a synonym to an entity or a training phrases and hit save, wait for it to train then refresh again, all the newly added items are gone. It seems that Dialogflow is experiencing storage issues. And we are losing time trying to retrain again and add those items again and this is frustrating.
We have tried to troubleshoot this issue for more than a week and it seems that those issues still there:
Example 1: Synonyms already exist but Dialogflow treats them like they don't exist when visiting the "Validation" option.
Example 2: Adding new synonyms, saving and training; After a while, they disappear.
Example 3: The DF server is most of the time unavailable.
Please, Dialogflow Support Team helps us check those issues.
Thank you.
If you are an Enterprise customer I would suggest to contact them directly, if not probably you should switch to enterprise for better customer support. Since you have been using DF for a year now, you guys might have reaches some limits. For example, a single agent cannot have more than 1 million entity reference values and synonyms. For more information on quotas and limits check the following link: Doc

google analytics a/b testing with 2 site instances.

I am getting ready to release a new web site in the coming weeks, and would like the ability to run multivariate or a/b tests between two version of the site.
The site is hosted on azure, and I am using the Service Gateway to split traffic between the instances of the site, both of which are deployed from Visual Studio Online. One from the main branch and the other from an "experimental" branch.
Can I configure Google analytics to assist me in tracking the success of my tests. From what I have read Google analytics seems to focus on multiple versions of a page within the same site for running its experiments.
I have though of perhaps using 2 separate tracking codes, but my customers are not overly technically savvy, so I would like to keep things as simple as possible. I have also considered collecting my own metrics inside the application, but I would prefer to use an existing tool as I don't really have the time to implement something like that.
can this be done? are there better options? is there a good nugget package that might fulfil my needs? any advice welcome.
I'd suggest setting a custom dimension that tells you which version of the site the user is on. Then in the reports you can segment and compare the data.

Resources