Prefect2.0 How to trigger a flow using just curl? - prefect

Here is my dead simple flow:
from prefect import flow
import datetime
#flow
def firstflow(inreq):
retval={}
retval['type']=str(type(retval))
retval['datetime']=str(datetime.datetime.now())
print(retval)
return retval
I run prefect orion and prefect agent.
Make a trigger using web ui (deployments run) ... the agent succesfully pull and do the job.
My question is how to do the trigger using just curl?
Note : I already read http://127.0.0.1:4200/docs.
but my lame brain couldn't find how to do it.
note:
Lets say my flow id is : 7ca8a456-94d7-4aa1-80b9-64894fdca93b
Parameters I want to be processed is {'msg':'Hello world'}
blindly Tried with
curl -X POST -H 'Content-Type: application/json' http://127.0.0.1:4200/api/flow_runs \
-d '{"flow_id": "7ca8a456-94d7-4aa1-80b9-64894fdca93b", "parameters": {"msg": "Hello World"}, "tags": ["test"]}'
but prefect orion say
INFO: 127.0.0.1:53482 - "POST /flow_runs HTTP/1.1" 307 Temporary Redirect
Sincerely
-bino-

It's certainly possible to do it via curl but it might be painful especially if your flow has parameters. There's much easier way to trigger a flow that will be tracked by the backend API - run the flow Python script and it will have exactly the same effect. This is because the (ephemeral) backend API of Prefect 2.0 is always active in the background and all flow runs, even those started from a terminal, are tracked in the backend.
Regarding curl, it looks like you are missing the trailing slash after flow_runs. Changing your command to this one should work:
curl -X POST -H 'Content-Type: application/json' http://127.0.0.1:4200/api/flow_runs/ \
-d '{"flow_id": "7ca8a456-94d7-4aa1-80b9-64894fdca93b", "parameters": {"msg": "Hello World"}, "tags": ["test"]}'
The route which might be more helpful, though, is this one - it will create a flow run from a deployment and set it into a scheduled state - the default state is pending, which would cause the flow run to be stuck. This should work directly:
curl -X POST -H 'Content-Type: application/json' \
http://127.0.0.1:4200/api/deployments/your-uuid/create_flow_run \
-d '{"name": "curl", "state": {"type": "SCHEDULED"}}'

Related

Using wiremock to proxy/record calls to AWS Cloudfront service

We have a container-based service running in AWS ECS with the front end hosted by AWS Cloudfront, and authorization handled by AWS Cognito. I'm trying to configure Wiremock to be a proxy for this service so I can record the calls and mappings to later use in unit tests for a client app I'm writing in python.
I'm running the Wiremock server in standalone mode, and have it proxying to calls to the url of our service. However, Cloudfront keeps returning either a 403-Bad Request error or 403-Forbidden error when I connect via Wiremock.
When I use curl, and pass all the correct headers (Content-Type: application/json, Authentication: Bearer ) it works just fine when I use https://myservice.example.com/api/foo. But as soon as I swap out "myservice.example.com" for "localhost:8000", I get the Cloudfront generated errors.
I'm guessing I have some mis-configuration where, despite passing the headers to Wiremock, I haven't properly told Wiremock to pass those headers on to "the service", which is really Cloudfront.
Not being a Java guy, I'm finding the Wiremock docs a little difficult to understand, and am trying to use the command-line arguments to configure Wiremock like this:
/usr/bin/java -jar \
./wiremock-jre8-standalone-2.35.0.jar \
--port=8000 \
--verbose \
--root-dir=test_data/wiremock \
--enable-browser-proxying \
--preserve-host-header \
--print-all-network-traffic \
--record-mappings \
--trust-proxy-target=https://myservice.example.com \
--proxy-all=https://myservice.example.com
Request:
$ curl -k -X GET -H "Content-Type: application/json" \
-H "Authorization: Bearer ${JWT}" \
http://127.0.0.1:8000/api/foo
Response:
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>CloudFront</center>
</body>
</html>
When using exactly the same curl command, but changing the URL to point directly at my service instead of the proxy, I get the response I expected (hoped for?) through the proxy:
curl -k -X GET -H "Content-Type: application/json" \
-H "Authorization: Bearer ${JWT}" \
https://myservice.example.com/api/foo
[
{
"id": "09d91ea0-7cb0-4786-b3fc-145fc88a1a3b",
"name": "foo",
"created": "2022-06-09T02:32:11Z",
"updated": "2022-06-09T20:08:43Z",
},
{
"id": "fb2b6454-4336-421a-bc2f-f1d588a78d12",
"name": "bar",
"created": "2022-10-05T06:23:24Z",
"updated": "2022-10-05T18:34:32Z",
}
]
Any help would be greatly appreciated.
Thanks.

Get à 401 Unauthorized when trying to lint my gitlab-ci.yml (version 13.8)

Since monday, I'm not able to use the lint CI API from gitlab, which is documented here https://docs.gitlab.com/ee/api/lint.html#validate-the-ci-yaml-configuration
I'm working on a self hosted gitlab, and we updated gitlab to the last version (13.8.4).
I've noticed that the documentation has changed between 13.8 and 13.9, they mentioned
Access to this endpoint requires authentication.
So I tried to generate a personal access token with full access (I'm admin), but I still get a 401.
Here is my try :
$ curl --header "Content-Type: application/json" --header "PRIVATE-TOKEN: P3r50Na1t0k3N" "https://my-domain.artips.fr/api/v4/ci/lint" --data '{"content": "{ \"image\": \"ruby:2.6\", \"services\": [\"postgres\"], \"before_script\": [\"bundle install\", \"bundle exec rake db:create\"], \"variables\": {\"DB_NAME\": \"postgres\"}, \"types\": [\"test\", \"deploy\", \"notify\"], \"rspec\": { \"script\": \"rake spec\", \"tags\": [\"ruby\", \"postgres\"], \"only\": [\"branches\"]}}"}'
# Result : {"message":"401 Unauthorized"}
# Other try
$ curl -X POST --header "Content-Type: application/json" --header "PRIVATE-TOKEN: P3r50Na1t0k3N" "https:///my-domain.artips.fr/api/v4/ci/lint" --data '{"content": "{ \"image\": \"ruby:2.6\", \"services\": [\"postgres\"], \"before_script\": [\"bundle install\", \"bundle exec rake db:create\"], \"variables\": {\"DB_NAME\": \"postgres\"}, \"types\": [\"test\", \"deploy\", \"notify\"], \"rspec\": { \"script\": \"rake spec\", \"tags\": [\"ruby\", \"postgres\"], \"only\": [\"branches\"]}}"}'
# same result: {"message":"401 Unauthorized"}
Has anyone run into the same problem ?
Thanks in advance
Takeshi
There is an ongoing issue regarding the CI Lint API endpoint. It seems that authentication for this endpoint doesn't work when "signing up" is disabled on on-premise Gitlab instance. I guess the issue will be fixed in future releases.
The user token's scope must grant read/write permission to yout gitlab instance's API (as you may see in the picture). The user hasn't to be admin.
import requests
import urllib.parse
from requests.auth import HTTPBasicAuth
url = 'https://gitlab.example.com/'
api = urllib.parse.urljoin(url, '/api/v4/projects')
private_token = 'xxx'
params = dict(private_token=private_token)
auth = HTTPBasicAuth('username', 'password') # username and password, different from login account, generally provided from ops
response = requests.get(api, params, auth=auth)
print(response)
print(response.json())

How can I use the GitHub Rest API to update a repository that is part of an organization?

I have a large number of GitHub repositories that have been sorted into various organizations. I wish to change the visibility of the repos by batch to public using the following API call:
curl -u {user}:{pat} -H "Accept: application/vnd.github.v3+json" -X PATCH https://api.github.com/repos/{user}/angular.powershifter -d '{"private":false}'
Which as best as I can tell is the API call (see github docs). The {user] and {pat} in the sample above are real values, and the {pat} has every option selected. The {user} is both the owner of the repo and the organization.
What I get in response is is a 404 with the following body.
{
"message": "Not Found",
"documentation_url": "https://docs.github.com/rest/reference/repos#update-a-repository"
}
Is it possible to set the properties of repositories under an organization using the API?
What is the API call?
Bonus points if you can point me to the documentation!
Thanks.
Use Bearer Token based authentication. Github is deprecating password based authentications.
curl \
-H 'Authorization: Bearer ${GITHUB-TOKEN}' \
-X PATCH \
-H "Accept: application/vnd.github.v3+json" \
https://api.github.com/repos/${OWNER}/${repo} \
-d '{"private":"false"}'
Link for the github documentation
https://docs.github.com/en/free-pro-team#latest/rest/overview/other-authentication-methods
Yes it is possible.
I've had some issues with this too.
Short answer use bearer token (As #shek said), and change false to "false":
curl -v -H "Authorization: Bearer ${GITHUB_TOKEN}"
-X PATCH https://api.github.com/repos/${ORGANIZATION}/${REPO}
-H "Accept: application/vnd.github.v3+json" -d '{"private": "false"}';
Notice that your token has full control of private repositories when creating the token:
full control of private repositories image
The link is quite confusing, because if for example you need to get all the repositories from GitHub you'll have to use:
https://api.github.com/orgs/${ORGANIZATION}/repos?${parameters}
Also when you need to change permissions you'll need to use:
https://api.github.com/orgs/${ORGANIZATION}/teams/${TEAM}/repos/${ORGANIZATION}/${REPO}
(Notice the orgs, which is missing in the PATCH command).
GitHub repositories documentation:
https://docs.github.com/en/rest/reference/repos#update-a-repository
GitHub auth methods (From #shek):
https://docs.github.com/en/free-pro-team#latest/rest/overview/other-authentication-methods

create repo using github api (only curl and no oauth)

I'm trying to create a github repo using the v3 api, but I always get a Not Found error. I want to do it only with curl and without oauth. This is what I'm trying:
curl -u myusername https://api.github.com/users/repos -d '{"name": "reponame"}'
What am I missing?
You can't do it without an Access Token.
Also, please feel free to look at my GitHub open source project Git-Captain.
I created a web-application with a Node.js back-end and HTML/JS front-end that you can setup to have an API do many of these calls for you. It has a step-by-step for a windows server and I'll be adding a Linux step-by-step soon.
It would only take a slight tweak to the project to add a new end-point to the source to do this for you.
To answer your question,
The GitHub API documentation explains exactly how to do what you are requesting on this link.
Giving this example:
as you requested in CURL and obviously replace the token "5199..." with your own:
curl -i -H "Authorization: token 5199831f4dd3b79e7c5b7e0ebe75d67aa66e79d4" \
-d '{ \
"name": "blog", \
"auto_init": true, \
"private": true, \
"gitignore_template": "nanoc" \
}' \
https://api.github.com/user/repos
OR
Not in CURL and according to this StackOverflow question you can do the following:
https://api.github.com/orgs/<organisation_name>/repos?access_token=<generated token>
or
https://api.github.com/users/<username>/repos?access_token=<generated token>
In body, pass this as a payload:
{
<br/>"name": "<Repo Name>",<br/>
"description": "<Whateveryour description is>",<br/>
"homepage": "https://github.com",<br/>
"private": false,<br/>
}
You can get a "personal access token in GitHub" by going to Settings->Developer Settings-> Personal Access Tokens->Generate new token
OR do all of the following
Write a script (let's call this script #1) that takes the username,password, and repoName as a parameter.
That script will call script #2, which is curl -u ' USER-NAME-HERE' https://api.github.com/user/repos -d '{"name": "REPO-NAME-HERE"}' which will prompt for your user password,
have your script #1 listen for script #2's response and then have it enter in the password which the user passed in as a parameter in script#1
Finally programmatically hit enter which fires off the curl to create your repo.
UPDATE*
So for some reason, the CURL won't work at all, but the Git-Hub API end point https://api.github.com/user/repos does indeed work. Using POSTMAN, I was able to create a new POST with the URL being https://api.github.com/user/repos and the BODY set to:
{
"name": "Hello-World",
"description": "This is your first repository",
"homepage": "https://github.com",
"private": false,
"has_issues": true,
"has_projects": true,
"has_wiki": true
}
Then I went to the 'Authorization' section of postman and under 'Type' I selected "Basic Auth" entered my username and password.
Clicked update request and then send and my repo was created!

Internal Server Error after updating custom Language Model for Watson's speech2text service

We've had success using a custom Language Model for Watson's speech2text service. However, recently, we decided to extend the model (extended vocabulary, utterances, etc.). To that end, we reset the model:
curl -X POST -u ${credentials}
"https://stream.watsonplatform.net/speech-to-text/api/v1/customizations/${customization_id}/reset"
added our extended corpus:
curl -X POST -u ${credentials} \
--data-binary #corpus.txt \
"https://stream.watsonplatform.net/speech-to-text/api/v1/customizations/${customization_id}/corpora/corpus_test01"
verified the process completed, and re-trained the model:
curl -X POST -u ${cred} \
"https://stream.watsonplatform.net/speech-to-text/api/v1/customizations/${customization_id}/train"
Once the process completed ("status": "available"), we were initially able to get responses, and the recognition quality did improve. However, within a few minutes, we started getting {"error": "Internal Server Error"}. Can't reset or query. The base model works fine though. Is our only option to create a new model (we would prefer not to)?
Any help would be appreciated.
Revised: don't seem to be able to create a new LM either:
curl -X POST -u ${cred} \
--header "Content-Type: application/json" \
--data "{\"name\": \"Test model\", \
\"base_model_name\": \"en-US_BroadbandModel\", \
\"description\": \"language model 02\"}" \
"https://stream.watsonplatform.net/speech-to-text/api/v1/customizations"
returns:
{
"code": 500,
"code_description": "Internal Server Error",
"error": "Internal Server Error"
}
But base model still works fine ...
New update: LM came back several hours later, out of a blue, and kept working fine through the day. However, this evening became intermittent, returning "Internal Server Error" with increasing frequency. Not very usable at this point.
-rg

Resources