I'm trying to get data from fbo.gov, which is a government website where they post contracts that vendors can bid in. They have a document containing ways of accessing information on the site through SOAP requests, which is what I'm trying to do. Although all of the examples in that document are in PHP, I am trying to make my requests in Python, because I've never done anything with PHP before.
To make the SOAP requests in Python, I'm using zeep.
Right now, I can successfully authenticate myself through HTTP, but no matter what method I try to call, I always get the same error: This user has an inactive agency.
Here is the code I'm using to send the request
from requests import Session
from requests.auth import HTTPBasicAuth
import zeep
from zeep.transports import Transport
test = "https://fbo-test.symplicity.com"
prod = "https://fbo.gov"
session = Session()
session.auth = HTTPBasicAuth("sample_username", "sample_password")
client = zeep.Client(f"{test}/ws/fbo_api.php?wsdl", transport=Transport(session=session))
dictionary = {"notice_type": "PRESOL"}
print(client.service.getList(data=dictionary))
I realize this is a long shot, but what could be causing this error? I can't find anything even remotely related to the error anywhere on the internet.
Per the Federal Service Desk:
The FBO API is only available for government user accounts.
Some of the FBO data is available at: ftp://ftp.fbo.gov
Currently, FBO is in the process of moving to SAM, and will have a public API once the move is complete. The new API is under development, with the latest specification at: https://open.gsa.gov/api/get-opportunities-public-api/
FBO.GOV has been retired as of 11/12/2019 along with the ftp.fbo.gov bulk download, use the following instead,
https://open.gsa.gov/api/sam-entity-extracts-api/
Related
I'm trying to download some files from behind a SSO (Single Sign-On) site. It seems to be SAML authenticated, that's where I'm stuck. Once authenticated I'll be able to perform API requests that return JSON, so no need to interpret/scrape.
Not really sure how to deal with that in mechanicalsoup (and relatively unfamiliar with web-programming in general), help would be much appreciated.
Here's what I've got so far:
import mechanicalsoup
from getpass import getpass
import json
login_url = ...
br = mechanicalsoup.StatefulBrowser()
response = br.open(login_url)
if verbose: print(response)
# provide the username + password
br.select_form('form[id="loginForm"]')
print(br.get_current_form().print_summary()) # Just to see what's there.
br['UserName'] = input('Email: ')
br['Password'] = getpass()
response = br.submit_selected().text
if verbose: print(response)
At this point I get a page telling me javascript is disabled and that I must click submit to continue. So I do:
br.select_form()
response = br.submit_selected().text
if verbose: print(response)
That's where I get a complaint about state information being lost.
Output:
<h2>State information lost</h2>
State information lost, and no way to restart the request<h3>Suggestions for resolving this problem:</h3><ul><li>Go back to the previous page and try again.</li><li>Close the web browser, and try again.</li></ul><h3>This error may be caused by:</h3><ul><li>Using the back and forward buttons in the web browser.</li><li>Opened the web browser with tabs saved from the previous session.</li><li>Cookies may be disabled in the web browser.</li></ul>
The only hits I've found on scraping behind SAML logins are all going with a selenium approach (and sometimes dropping down to requests).
Is this possible with mechanicalsoup?
My situation turned out to require Javascript for login. My original question about getting into SAML auth was not the true environment. So this question has not truly been answered.
Thanks to #Daniel Hemberger for helping me figure that out in the comments.
In this situation MechanicalSoup is not the correct tool (due to Javascript) and I ended up using selenium to get through authenication then using requests.
Am using python googleapi client library to get instance data for a project
Am getting instances like this:
from googleapiclient import discovery
from google.oauth2 import service_account
scopes = ['https://www.googleapis.com/auth/cloud-platform']
service_cred_file = 'service-credentials.json'
zone = 'us-central1-c'
project_id = 'my_project_id'
credentials = service_account.Credentials.from_service_account_file(service_cred_file, scopes=scopes)
service = discovery.build('compute', 'v1', credentials=credentials)
request = service.instances().list(project=project_id, zone=zone)
while request is not None:
response = request.execute()
for instance in response['items']:
print(instance)
Instance response does not contain instance price, uptime and user data (data of user that created instance)
How can i get these attributes?
The shortest answer to your question is that, currently, it is not possible to get this information from the “compute” API.
However, there are other APIs which can give you this information, even if not as easy as just retrieving a property from an instance.
For uptime and user data you could use the Monitoring Client Library.
For user data you can use this client library to look for the logs of the “insert” protoPayload.methodName for a specific instance, and get the information about the user from the protoPayload.authenticationInfo property.
To get information about the uptime, you would need to set uptime checks and calculate the uptime from the logs generated by the check you created.
For information about pricing however, it’s not possible to do the same.
I was looking through different possible solutions and I even found the page for the Cloud Billing API, which has a skus() method, however since the documentation is scarce, and there is no filter for specific instances as resources, this would probably be even harder to implement.
I want to use twitter api for poll votes using tweepy is there any methods to implement this?
I tried doing api.update_status("poll_1"[1112227775552223333]) but its not working.
No. The poll API is private and is not available to anyone apart from Twitter's own apps.
There are no plans to open this to the public - https://twittercommunity.com/t/poll-support/78235
The other answer is incorrect. There is a way, it's just the JSON data you get back is quite unwieldy (I have plans to build a tweepy addon to make this easier). The way to get it with Python 3.x is through requests. Don't worry though, it's only a line or so of code. As you can see in this link, we can indeed get back twitter poll data. Here's a code snippet:
import requests, json, OAuth1
# provided by a Twitter dev account
auth = OAuth1(my_key, my_secret_key, my_access_token, my_secret_access_token)
tweet_ids_string = "12345678,123456790,09886654333"
# for more expansions see the link above
data_url = f'https://api.twitter.com/2/tweets?ids={tweet_ids_string}&expansions=attachments.poll_ids&poll.fields=duration_minutes,end_datetime,options,voting_status'
response = requests.get(get_data_url, auth=auth)
response_json = response.json() # then you can get all sorts of data from here
Beware though, the Twitter polls API isn't lightning fast, so there might be several seconds of a delay between a poll getting voted on and the request JSON changing.
I have created my own REST service based on the examples from "Domino Sample REST Service Feature" from 901v00_11.20141217-1000 version of XPages Extension Library.
As far as I understand the design of the library each REST request will be run in its own thread on the server. This approach does not allow to handle parallel POST requests to the same document.
I have not found any examples in XPages Extension Library which would handle post requests as transactions on the server, i.e. which would block the server resource for the whole request processing time and would put put next requests in the queue?
Can anybody point to the source code of the service which would allow to handle parallel requests?
The skeleton for my post request processing function is this
#POST
#Path(PATH_SEPARATOR + MyURL)
#Consumes(MediaType.APPLICATION_JSON)
#Produces(MediaType.APPLICATION_JSON)
public Response myPost(
String requestEntity,
#Context final UriInfo uriInfo)
{
LogMgr.traceEntry(this, "myPost");
RestContext.verifyUserContext();
String myJson = ... // Process post
Response response = buildResponse(myJson);
LogMgr.traceExit(this, "myPost", "OK");
return response;
}
And I would like to implement something like this
// Start transaction
String myJson = ... // Process post
// Stop transaction
Is there a way to do it in Java?
I suppose you could use document locking in traditional Notes/Domino context - and synchronized in Java :-)
Have you tried any of these? I cannot see why they should not work.
/John
I agree with John. You can use document locking to prevent simultaneous updates to the same document. You might also want to consider some changes to the definition of your REST API.
First, you imply you are using POST to update an existing document. Usually, POST is used to create a new resource. Consider using PUT instead of POST.
Second, even with document locking, you still might want to check for version conflicts. For example, let's say a client reads version 2 of a document and then attempts to update it. Meanwhile, another client has already updated the document to version 3. Many REST APIs use HTTP ETags to handle such version conflicts.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 2 years ago.
Improve this question
I want to know does Netsuite provides REST ful api? Currently i am doing integration with my application(java) with soap based web services.i have done some research but didn't get useful information.IF it does where can i found api?
Avoid the SuiteTalk SOAP web services API like the plague; it will do nothing but waste your time to. Usage of Netsuite SOAP API is viable only when you are okay with the SOAP API being non performant, don't mind interacting with gross buggy SOAP API, have much time to implement robust error handling to account for the random SOAP errors, concurrency errors. You'll need much time to develop robust fault tolerance. All that time will be wasted time; because no amount of time will make the SOAP API performance acceptable.
RESTlet's are preferred over SOAP API usage for writing data; RESTlets tend to be slightly more performant for writes (although responses are still extremely slow and not suitable for a customer facing app).
RESTlet's are a viable short term solution for writing data to Netsuite. Its essentially a JS script that allows you to set up a token based auth poor man's JSON endpoint; in which you can send JSON request bodies and get back JSON response bodies. Usage is reasonable in cases in which not much data needs to be written via the Restlet's (for instance for SalesOrders). A queue based system and background jobs with retry capabilities will mitigate the random Netsuite error issues (concurrency errors, timeouts etc).
If you must write to a bunch of Netsuite entities frequently and are using Netsuite as the source of truth for your data rather than attempting to build an entire REST like JSON API on top of Netsuite; I'd recommend implementing a pub/sub service in which Netsuite publishes events to an external service subscribed to by your app/API. Your app could also publish mutations to a channel subscribed to by Netsuite. This way data mutations sent to Netsuite can occur in a middle layer with reduced complexity.
To fetch Netsuite data for outside apps the most efficient means available appears to be the Netsuite ODBC database driver; it provides a direct connection to Netsuite database read only table views. Simple select queries for a set of Items that with same schema in Postgres or MySQL typically take 0.5 ms or less; typically take between 15 seconds to slightly over 100 seconds to return.
Connection timeouts and other errors from Netsuite are still common using NS ODBC driver. Despite slow query results retrieval of all data needed for a set of 5000 items in 14 seconds is far better than the hours it would take to get the same via Netsuite's SOAP API.
Yes. That is in Customization/Scripts section. You will find "RestLet" there. Doc is here.
However you said your application is soap based, I suggest you take a look Netsuite's WebServices aka SuiteTalk.
The SuiteTalk Platform provides programmatic access to your NetSuite data and business processes through an XML-based application programming interface (API).
I think you do need to access to your Netsuite data, right?
You can download their sample for test and learning.
In NetSuite, you can build RESTlet scripts which provide a REST-based interface. You can essentially use them to build your own JSON API. Recommend researching RESTlets in the NetSuite Help.
SOAP is easier to configure and use, but only allows 1 connection per
Netsuite account (you use your login credentials as authentication)
and is relatively slow.
That's not quite true, as you can extend it with suite cloud plus program. Check help for:
- Understanding Web Services Governance
- Enabling Web Services Concurrent Users with SuiteCloud Plus
UPDATE: There are two types of governance in NetSuite since approx July 2016 - user governance (also known as a legacy governance model, implicitly used when sessions / SOAP Login method are utilized) and account governance. In the account governance there is a shared pool for all incoming concurrent requests (no sessions should be used, authentication via user credentials or Token-Based Authentication).
This is the proper REST API provided by NetSuite for integration purposes.
https://system.netsuite.com/help/helpcenter/en_US/APIs/REST_API_Browser/record/v1/2020.1/index.html
The REST API can be invoked either via Token-based authentication or OAuth 2.0 enabled HTTP client.
First you need to login to NetSuite account and enable the SuiteTalk Webservice features of the account (Setup->Company->Enable Features).
Then obtain the SuiteTalk Base URL, which contains the account ID under the company URLs (Setup->Company->Company Information). E.g., https://<ACCOUNT_ID>.suitetalk.api.netsuite.com
After that create an integration application (Setup->Integration->New), enable OAuth 2.0 or TBA. This blog contains the process of enabling features and obtaining tokens.
Then use the BaseUrl + API resource path to as the HTTP client path to invoke each record API. Operations such as CRUD, search and filter can be done via this REST API. For more information See NetSuite Documentation
Yes, Netsuite supports REST web services.
Here's a working Java example, that uses the open source scribe library.
Note that an Accept (and for Posts, a Content-Type) header of application/json is needed for Netsuite to accept the requests, otherwise you'll get a "Request media type is not valid." error. Also getSignatureType method must be implemented for API class (NetSuiteApi.java).
Change all the string constants to suit your setup. Note that this code will also work with Netsuite RESTlets.
REST documentation is available here:
https://[your-netsuite-ID].app.netsuite.com/help/helpcenter/en_US/PDF/REST_Web_Services.pdf
File #1: NetSuiteApi.java
package com.scribe.api;
import com.github.scribejava.core.builder.api.DefaultApi10a;
import com.github.scribejava.core.model.OAuth1RequestToken;
public class NetSuiteApi extends DefaultApi10a {
private static class InstanceHolder {
private static final NetSuiteApi INSTANCE = new NetSuiteApi();
}
public static NetSuiteApi instance() {
return InstanceHolder.INSTANCE;
}
#Override
public String getAccessTokenEndpoint() {
return null;
}
#Override
public String getRequestTokenEndpoint() {
return null;
}
#Override
public String getAuthorizationUrl(OAuth1RequestToken requestToken) {
return null;
}
#Override
protected String getAuthorizationBaseUrl() {
return null;
}
#Override
public OAuth1SignatureType getSignatureType() {
return OAuth1SignatureType.HEADER;
}
}
File #2: NetSuiteApiCallExample.java
package com.scribe.api;
import com.github.scribejava.core.builder.ServiceBuilder;
import com.github.scribejava.core.model.OAuth1AccessToken;
import com.github.scribejava.core.model.OAuthRequest;
import com.github.scribejava.core.model.Response;
import com.github.scribejava.core.model.Verb;
import com.github.scribejava.core.oauth.OAuth10aService;
public final class NetSuiteRestExample {
private String CONSUMER_KEY = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx";
private String CONSUMER_SECRET = "yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy";
private String TOKEN_ID = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
private String TOKEN_SECRET = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb";
private String REST_URL = "https://1234567-sb1.suitetalk.api.netsuite.com/rest/platform/v1/record/inventoryitem/";
private String REALM = "1234567_SB1";
private String POSTBODY = "{\"type\": \"SIMPLE\",\"authorId\": -5}";
public static void main(String[] args) {
final OAuth10aService service = new ServiceBuilder(CONSUMER_KEY).apiSecret(CONSUMER_SECRET))
.build(NetSuiteApi.instance());
OAuth1AccessToken accessToken = new OAuth1AccessToken(TOKEN_ID, TOKEN_SECRET);
// This is POST method call
// OAuthRequest request = new OAuthRequest(Verb.POST, REST_URL);
// request.addHeader("Content-Type", "application/json");
// // Without next line, you'll get a "Request media type is not valid." error, even though this is not needed with Postman
// request.addHeader("Accept", "application/json");
// request.setRealm(REALM);
// request.setPayload(POSTBODY);
// This is GET method call
OAuthRequest request = new OAuthRequest(Verb.GET, params.get("REST_URL"));
// Without next line, you'll get a "Request media type is not valid." error, even though this is not needed with Postman
request.addHeader("Accept", "application/json");
request.setRealm(params.get("REALM"));
service.signRequest(accessToken, request);
System.out.println("Sending this request...");
System.out.println(request.getHeaders());
System.out.println(request.getCompleteUrl());
// System.out.println(request.getPayload());
final Response response = service.execute(request);
System.out.println("Got this response...");
System.out.println(response.getCode() + "\n" + response.getHeaders());
System.out.println(response.getBody());
return response.getBody();
}
}
Add this to you Maven dependencies (pom.xml):
...
<dependencies>
...
<dependency>
<groupId>com.github.scribejava</groupId>
<artifactId>scribejava-apis</artifactId>
<version>6.9.0</version>
</dependency>
</dependencies>