I have been trying to externally log data to my home server with a little GSM modem and InfluxDB2 HTTP API- it is far away and needs to be external, just checking water levels and other system stuff.
I am struggeling to understand the correct payload and keys to give it in order for it to accept my data.
I am using an ESP32 and the requests module on Micro Python, using MicroPython_ESP32_psRAM_LoBo.
The GSM library I am using makes everything work as if I was connected to Wifi, I am pretty certian that my problem has nothing to do with the GSM side of things.
The system uses InfluxDB2 and Python3 ( Micro Python to be exact )
Current Setup
payload = {"header":"Authorization: Token CrAzYlOnG_ToKeN", "data-raw":"WaterWatcher1 Level=532,Volume=752.22,BattV=3.768,Temperature=25.36 1621434110"}
response = requests.post("http://xxx.xxx.xxx.xxx:8086/api/v2/write?org=Home_Org&bucket=BackYardInfo&precision=s", params=payload)
( xxx Replaced with my correct IP - Yes I have made sure this is correct and got my ISP to set it to static )
This gives me a constant Auth error :
W (3390149) HTTP_CLIENT: This request requires authentication, but does not provide header information for that
I have read over the InfluxDB2 docs and I still cant seem to get it correct, I seem to think that if I did this in InfluxDB Ver 1 it would have been easier but now I just want to try learn it this way.
I have entered the exact same data manually and that works fine ( changed the time stamps though )
I am pretty sure it boils down to me just not knowing how to send the data correctly with the requests module, any pointers and explanations would be a great help.
I managed to find a solution after days of searching and trying new things.
Basically I used the mrequests lib
( https://github.com/SpotlightKid/mrequests )
With this I managed to get the whole thing working as expected using the following code - Done in Micropython
I did make a free account with No-IP to set my ISP external IP to a URL, I dont think this has a massive change but it is worth noting.
I did try the older methods with this new URL and they still did not work.
Here is the new code with the mrequest lib.
import mrequests
url = "http://ni-ip-address:8086/api/v2/write?org=Home_org&bucket=backYardData&precision=s"
token = "Token CrazyLongToken"
DataString = "WaterWatcher1 Level=888,Volume=888.88,BattV=8.888,Temperature=25.36 1621694039"
res = mrequests.request("POST", url, headers={"Authorization":token}, data=DataString)
This works perfectly for me.
On a side note, I can also upload multiple points if I read them from a file but I had to put the newline char "\n" at the front of each of my lines not the end or Influx would not pars the data.
I think it boiled down to the urequest lib not being able to do headers as I needed them in post requests.
I may have just been using it wrong but I feel that I did give it more than a fair go at getting it right where as this mrequests lib just worked perfectly.
Hope this helps anyone else out there one day.
Related
Bots are amazing, unless you're Google Analytics
After many months of learning to host my own Discord bot, I finally figured it out! I now have a node server running on my localhost that sends and receives data from my Discord server; it works great. I can do all kinds of the things I want to with my Discord bot.
Given that I work with analytics everyday, one project I want to figure out is how to send data to Google Analytics (specifically GA4) from this node server.
NOTE: I have had success in sending data to my Universal Analytics property. However, as awesome as that was to finally see pageviews coming into, it was equally heartbreaking to recall that Google will be getting rid of Universal Analytics in July of this year.
I have tried the following options:
GET/POST requests to the collect endpoint
This option presented itself as impossible from the get-go. In order to send a request to the collection endpoint, a client_id must be sent along with the request itself. And this client_id is something that must be generated using Google's client id algorithm. So, I can't just make one up.
If you consider this option possible, please let me know why.
Install googleapis npm package
At first, I thought I could just install the googleapis package and be ready to go, but that idea fell on its face immediately too. With this package, I can't send data to GA, I can only read with it.
Find and install a GTM npm package
There are GTM npm packages out there, but I quickly found out that they all require there to be a window object, which is something my node server would not have because it isn't a browser.
How I did this for Universal Analytics
My biggest goal is to do this without using Python, Java, C++ or any other low level languages. Because, that route would require me to learn new languages. Surely it's possible with NodeJS alone... no?
I eventually stumbled upon the idea of actually hosting a webpage as some sort of pseudo-proxy that would send data from the page to GA when accessed by something like a page scraper. It was simple. I created an HTML file that has Google Tag Manager installed on it, and all I had to do was use the puppeteer npm package.
It isn't perfect, but it works and I can use Google Tag Manager to handle and manipulate input, which is wonderful.
Unfortunately, this same method will not work for GA4 because GA4 automatically excludes all identified bot traffic automatically, and there is no way to turn that setting off. It is a very useful feature for GA4, giving it quite a bit more integrity than UA, and I'm not trying to get around that fact, but it is now the Bane of my entire goal.
https://support.google.com/analytics/answer/9888366?hl=en
Where to go from here?
I'm nearly at the end of my wits on figuring this one out. So, either an npm package exists out there that I haven't found yet, or this is a futile project.
Does anyone have any experience in sending data from NodeJS to GA4? (or even GTM?) How did you do it?
...and this client_id is something that must be generated using Google's client id algorithm. So, I can't just make one up...
Why, of course you can. GA4 generates it pretty much the same as UA does. You don't need anything from google to do it.
Besides, instead of mimicking just requests to the collect endpoint, you may just wanna go the MP route right away: https://developers.google.com/analytics/devguides/collection/protocol/ga4 The links #dockeryZ gave, work perfectly fine. Maybe try opening them in incognito, or in a different browser? Maybe you have a plugin blocking analytics urls.
Moreover, you don't really need to reinvent the bicycle. Node already has a few packages to send events to GA4, here's one looking good: https://www.npmjs.com/package/ga4-mp?activeTab=readme
Or you can just use gtag directly to send events. I see a lot of people doing it even on the front-end: https://www.npmjs.com/package/ga-gtag Gtag has a whole api not described in there. Here's more on gtag: https://developers.google.com/tag-platform/gtagjs/reference Note how the library allows you to set the client id there.
The only caveat there is that you'll have to track client ids and session ids manually. Shouldn't be too bad though. Oh, and you will have to redefine the concept of a pageview, I guess. Well, the obvious one is whenever people post in the chan that is different from the previous post in a session. Still, this will have to be defined in the code.
Don't worry about google's bot traffic detection. It's really primitive. Just make sure your useragent doesn't scream "bot" in it. Make something better up.
I want to save the data column from this url to an array in python. I tried it with, for instance, pandas.save_table:
import pandas as pd
pd.read_table('https://adventofcode.com/2019/day/1/input', sep='')
but I get HTTPError: HTTP Error 400: Bad Request and I think this is not the right way to do that.
Can someone help me with that?
If you try to open the link in your question (in a browser using incognito mode or something similar i.e. delete your cookies) you'll see that you need login into the website to access the page. This is why the you're getting a 400 Bad Request error as a response from the server.
From the FAQ section of the website that you're trying to access:
How does authentication work? Advent of Code uses OAuth to confirm
your identity through other services. When you log in, you only ever
give your credentials to that service - never to Advent of Code. Then,
the service you use tells the Advent of Code servers that you're
really you. In general, this reveals no information about you beyond
what is already public; here are examples from Reddit and GitHub.
Advent of Code will remember your unique ID, names, URL, and image
from the service you use to authenticate.
The website uses OAuth to handle logins to the url that you create will need these access tokens. You can use a library like python-oauth2 to help you with this (there are others so you can read around and decide which you'd like to use). Creating and understanding how to make http requests is beyond the scope of this answer. I'd suggest you have a look around on the internet for some explanations and try again, if you have get stuck please ask another question. Otherwise it'll probably be easier to save the file from your browser...But I'll leave this answer here for the next person who runs into the same problem.
I was trying to do some simple authorization for ameritrade's developer platform. I was attempting.
According to the platform, the Endpoint I need to access is is:
https://auth.tdameritrade.com/auth?response_type=code&redirect_uri={uri}&client_id={client_id}}%40AMER.OAUTHAP
https://developer.tdameritrade.com/content/simple-auth-local-apps
When looking at the client_id, for the dev application, I was noticing that they may actually be referencing the Applications, Consumer Key instead? So i did just that, but when attempting to query the information, it returns: A third-party application may be attempting to make unauthorized access to your account. The reason why i think it is the consumer key, is listed at: https://developer.tdameritrade.com/content/getting-started
So I ended up doing something like:
from urllib.parse import urlencode, quote_plus
url = "https://auth.tdameritrade.com/auth?response_type=code&redirect_uri={uri}&client_id={client_id}}%40AMER.OAUTHAP".format(
uri=urlencode("http://localhost", quote_via=quote_plus),
client_id="JHBDFGJH45OOUDFHGJKSDBNG" #Sample
)
I dont think this is because I am currently in a different country currently, I think that something else is wrong here.
It doesnt follow through with it, but instead returns a 400 error with that information. Im not sure whats wrong though.
This happens when you copied the callback URI incorrectly. Imagine if this were a client application, and TD detected that the application is trying to send the user to a different URL than the app is configured with. If they send the callback request to that application, it will receive the token and gain full control over your account.
Have you double and triple checked that you're copying the callback URL correctly, protocol name, ports, and trailing slashes and everything? Also, consider using an API library instead of writing your own. You can find documentation about this specific error here.
I had this issue and I solved it using simply using http://127.0.0.1 on the call back URI of the App.
I then used below URL and it worked as expected.
https://auth.tdameritrade.com/auth?response_type=code&redirect_uri=http%3A%2F%2F127.0.0.1&client_id={MyConsumerKey}%40AMER.OAUTHAP
Just in case anyone is still having this problem, make sure the callback URI is spelled EXACTLY the same as you specified when creating the app. I was having this problem because I set the callback on the TD developer website to "https://localhost/" and used "https://localhost" in the URL instead (missing the slash at the end). As soon as I added the slash at the end, it worked.
I found out that the issue is caused by the way the callback URL is set. It have to be exactly the same as the callback URL you have typed in at the apps details on the TD developer API page. I tried several permutations and indeed to get the authorization to work both have to be the same. eg. https or http.. end with '/' or does not, it matters. There is also no need to URL encode it.
I'm still new to AWS and just following the documentation and asking questions here when I get stuck. Please excuse me if this question sounds really noobish.
So far, I've deployed the following:
EB to deploy my REST API
RDS to deploy my psql database
Lambda functions to handle things like authentication & sending JWTs, uploading images to S3, etc.
I have got my basic back end (no caching (just started learning about redis), etc. set up yet, just the bare bones so far) deployed.
I'm still developing my front end, and have not even thought about how I will be deploying it yet (probably another deployment on EB, since I am using universal react). I am just developing it locally but using my production env variables now so I am hitting my deployed API, etc.
One of the MAJOR things I have no idea on how to do is detecting incoming requests from client side to get the client's location by IP address. This is so that I can return the INITIAL results in your general location just like yelp, foursquare, etc. do when you go to to their sites.
For now, I am just building a web app on desktop so I just want to worry about getting the IP address to get the general area of the user. My use case is something similar to other sites you might have used which provides an INITIAL result set for things in your area (think foursquare or yelp).
Here are my questions:
What would be a good way to do this? I'm thinking of handling this in my front end react universal deployment since it will be a node server with rendered page caching. Is this a terrible idea? It would work something like
(1) request from client comes in
(2) get IP from request and lookup the IP location using some service (still not sure what I'm going to use, have found a few plus a nodejs library called node-geoip). Preferably, I can get the zip code since I am trying to save having to do so many queries by unique locations in my database, and instead return results in the zip code and the front end will show an initial map with the initial results in that zip code.
(3) return to client the rendered page with those location params if it exists, otherwise create it, send it, and cache it.
Is the above a really dumb idea? Maybe you have already done something like this, and could share your wisdom :)
Is there an AWS service which can already handle something like this for me? Perhaps there's some functionality which can already do this.
Thanks.
AGAIN - I apologize if this is long winded. I don't know anyone in real life who can help me and I feel alone :(. I appreciate the help you guys can provide.
There are two parts to this:
Getting the user's IP address. You mentioned you're using 'EB' - I presume you mean AWS ELB (Elastic Load Balancer)? If so, then you need to read the X-Forwarded-For HTTP header in your app code, since otherwise what you'll really detect is the ELB's IP address. X-Forwarded-For contains the user's real IP - or rather, the IP of the end-connection being made (there's no telling if this is really a VPN, Proxy or something else-- but it's as far as you can get with an IP.)
Querying an IP DB that can turn the addr into a location object. There are tons of libraries for you. Assuming you're using Node, you can use node-geoip as you mentioned. Or you can just search 'geoip service' on Google and find managed services, like Telize on Mashape. If you don't want to manage the DB lookup yourself or keep the thing up to date, then a managed service would help.
In either case, it's likely that you'll be doing asynchronous look-ups. In that case, you might want to use async/await to get the user's full object before injecting that into your React props and ultimately rendering it as a HTML string that's sent down to the client.
You could also use a library like redial to decorate your components with data requirements, and return a Promise you can await on to know when you're okay to render.
Since you probably want to enable client routing too (i.e. where the user can click on a route in their browser, and the server isn't touched at all), then you will probably need some way to retrieve the IP address/results based on that IP even when the server isn't involved in the initial render.
For that, you could write a REST service that retrieves the results. Or write a GraphQL back-end that gets the data. It doesn't matter how you write it, since the server will have access to the X-Forwarded-For header and can use that to retrieve the results and send back location-aware data.
FYI, I'm writing a React starter kit (called ReactNow) that uses rxjs for handling async streams. It's not ready yet, but it might help you figure out the code layout that would offer a balanced mix between rendering on the server, and writing universal code that requires some heavy lifting from the server.
I'm using Play 2.1.2 and I'm having some trouble with an Enumerator and I'm seeking ideas on how to debug this.
I'm trying to stream some S3 data through my server. I can get an InputStream from the Amazon SDK for my S3 file (getObject(bucket, key).getObjectContent()). I then turn that InputStream into an Enumerator[Array[Byte]] using Enumerator.fromStream.
All this type checks and on my local development machine it all works perfectly. When I formulate my Result in Play, I just return Ok.stream(enum).
The problem comes when I deploy this to a production server. The very first time I request the file, it works just fine and I get the whole file. But subsequent times it frequently gets part way through (different amounts each time) and then gets "stuck". I wrapped the Enumerator as follows to be able to log whether the enumeration completed:
val wrapped = enum.onDoneEnumerating { println("Contents fully enumerated"); }
Ok.stream(wrapped);
As expected, on my development machine (and the first time through on the production machines), I get the message "Contents fully enumerated". But after that, the production machine will start the download of the file, but it doesn't finish (in both the HTTP sense and in the Enumerator sense).
I'm not sure how to debug this. Obviously, fromStream does some magic and I don't know how to figure out what is happening between chunks. I thought this might be a thread pool issue so I wrapped the whole response in a future { blocking { ... } } block, but it didn't appear to make any difference.
I'm trying to avoid the hassle of creating a local temporary file from S3 and then building my Enumerator from that. Using the fromStream to create an Enumerator seemed like the elegant way to do this...if it worked.
Suggestions?
OK, so I think I figured this out. It turns out that on the Play side, things appear to be working. I tried all kinds of variations (different ways of constructing the enumerator, creating temporary files, etc). It didn't really matter.
What did matter was the proxy I was using. I'm using node-http-proxy and if I make a request on the server behind the proxy, I get the correct response (directly from Play). If I make the request on the server outside the proxy, I get an incorrect (empty) response. So it looks like the proxy is "dropping" the response.
It appears the issue is that the response is chunked by the stream call and this is causing the problem with the proxy. If I reformulate my response to be:
SimpleResult( header = ResponseHeader(200), body = enum)
Then play uses the Enumerator to construct a complete response (not streamed) and things work again. Of course, it is stupid to have to form the complete response in this case, but it does work. Hopefully I can find a better solution than this in the long term, but this seems to work for now.