making go http client work with non-standard http servers - audio

Shoutcast servers basically speak http, with one important difference: they respond to GET requests with ICY 200 OK instead of HTTP/1.1 200 OK.
Go won't have a bar of it, and correctly fails with the error malformed HTTP version "ICY".
However I would like to make things work and am wondering what the best approach is. My ideas so far:
use a custom http.Transport.Proxy to change ICY to HTTP/1.1 in-flight
an out of process proxy that does the same thing
overload http.ParseHTTPVersion (but golang doesn't have function overloading)
duplicate the entire http package, just to modify ParseHTTPVersion
Number 1. seems the most attractive attractive, but I have no idea how to respect the http "scope" and actually modify all responses on a given http version. Is this the kind of thing http.Transport.Proxy can handle?
Can anyone give me any pointers?

I got this working by creating a custom Dial function that returns a wrapped connection. My wrapper intercepts the first read on the connection and replaces ICY with HTTP/1.1. Not super robust, but proves the concept:
package main
import (
"fmt"
"net"
"net/http"
)
type IcyConnWrapper struct {
net.Conn
haveReadAny bool
}
func (i *IcyConnWrapper) Read(b []byte) (int, error) {
if i.haveReadAny {
return i.Conn.Read(b)
}
i.haveReadAny = true
//bounds checking ommitted. There are a few ways this can go wrong.
//always check array sizes and returned n.
n, err := i.Conn.Read(b[:3])
if err != nil {
return n, err
}
if string(b[:3]) == "ICY" {
//write Correct http response into buffer
copy(b, []byte("HTTP/1.1"))
return 8, nil
}
return n, nil
}
func main() {
tr := &http.Transport{
Dial: func(network, a string) (net.Conn, error) {
realConn, err := net.Dial(network, a)
if err != nil {
return nil, err
}
return &IcyConnWrapper{Conn: realConn}, nil
},
}
client := &http.Client{Transport: tr}
http.DefaultClient = client
resp, err := http.Get("http://178.33.230.189:8100") //random url I found on the internet
fmt.Println(err)
fmt.Println(resp.StatusCode)
}

Related

Azure sdk for go - code can't make it passed compute.VirtualMachinesClient.ListAllComplete()

I'm testing a function I have that gets all azure vms under a specific subscription.
It uses azure sdk for go's compute.VirtualMachinesClient to do so.
My problem is with the vm clients ListAllComplete function.
It's not returning an error. The code just doesn't seem to be able to make it passed that line.
Any suggestions on the source of the problem would be appreciated.
This is the code, I've used the fmt package to follow how far it gets:
func GetAllAzureVms() ([]compute.VirtualMachine, error) {
fmt.Printf("In getAllAzureVm\n")
var vmList []compute.VirtualMachine
vmClient, err := GetAzureVmClient()
fmt.Printf("Out of GetAzureVmClient\n")
if err != nil {
return nil, err
}
fmt.Print("No error from getazurevmclient\n")
vmListComplete, err := vmClient.ListAllComplete(context.Background(), "statusOnly=false")
fmt.Print("vmClient.ListAllComplete done")
if err != nil {
fmt.Print("vmClient.ListAllComplete error")
return nil, err
}
fmt.Print("here")
for vmListComplete.NotDone() {
vmList = append(vmList, vmListComplete.Value())
err := vmListComplete.NextWithContext(context.Background())
if err != nil {
return nil, err
}
}
fmt.Print("here2")
return vmList, nil
}
It cant make it passed the line:
vmListComplete, err := vmClient.ListAllComplete(context.Background(), "statusOnly=false")
No error is returned.
I have a similar piece of code, not for vmClient unluckily, but for securityCustomRules.
What I've found useful was to use ListComplete() instead of ListAll() and the print values through JSON marhsalling.
I hope you can find it useful anyway.
import (
"context"
"fmt"
"os"
"testing"
"github.com/gruntwork-io/terratest/modules/azure"
"github.com/gruntwork-io/terratest/modules/terraform"
)
securityCustomRulesList, err := securityCustomRulesClient.ListComplete(context.Background(), tfResourceGroupName, tfVnetName+"-"+fmt.Sprint(vnetIndex)+"-subnet-02-nsg")
// - Scan iterator items [https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/services/network/mgmt/2019-09-01/network#ApplicationGatewayListResultIterator]
if err != nil {
fmt.Fprintf(os.Stderr, ">>>> Error parsing securityCustomRulesClient:: %s", err)
return
}
for securityCustomRulesList.NotDone() {
// securityCustomRulesList.Value() -> securityRule [https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/services/network/mgmt/2019-09-01/network#SecurityRule]
v := securityCustomRulesList.Value()
vJson, _ := v.MarshalJSON()
fmt.Printf(">> Network securityCustomRulesList JSON %s \n", string(vJson))
//fmt.Printf(">> Network securityCustomRulesList %s - %s\n", *v.Name, *v.SecurityRulePropertiesFormat.Description)
securityCustomRulesList.NextWithContext(context.Background())
}
Which gives me an output like:
>> Creating security custom rules list instance 'securityCustomRulesList'..
>> Network securityCustomRulesList JSON {"id":"/subscriptions/8f7d6be2/resourceGroups/unit-tests-tfm-azure-network-resource-group/providers/Microsoft.Network/networkSecurityGroups/unit-tests-tfm-azure-network-vnet-0-subnet-02-nsg/securityRules/test-02","name":"test-02","properties":{"access":"Deny","description":"Deny access","destinationAddressPrefix":"10.0.2.1","destinationAddressPrefixes":[],"destinationPortRange":"*","destinationPortRanges":[],"direction":"Inbound","priority":111,"protocol":"*","sourceAddressPrefix":"10.0.1.0/24","sourceAddressPrefixes":[],"sourcePortRange":"*","sourcePortRanges":[]}}
Useful links:
Here you can find a complete usage example from the official terratest/azure module: https://github.com/gruntwork-io/terratest/blob/dae956eb39e91dfb00f3ba85060a6dbf58c6782b/modules/azure/nsg.go
https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/services/network/mgmt/2019-09-01/network#SecurityRule.MarshalJSON

how to gzip a template in golang web

I have already set the value of contest-encoding, but how can I gzip the template, since the file is still to big.
func indexPageHandler(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Encoding", "gzip")
r.Header.Set("Accept-Encoding", "gzip")
tmpl, err := template.New("index.html").ParseGlob("./templates/*")
if err != nil {
log.Println(err)
return
}
err = tmpl.Execute(w, nil)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
}
Is there any function that can gzip the response?]
Followed by the advise, I changed my code as this:
func indexPageHandler(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Encoding", "gzip")
r.Header.Set("Accept-Encoding", "gzip")
gz := gzip.NewWriter(w)
defer gz.Close()
tmpl, err := template.New("index.html").ParseGlob("./templates/*")
if err != nil {
log.Println(err)
return
}
err = tmpl.Execute(gz, nil)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
}
But when I query the url, I only download the gzip file. What's wrong with it?
You can use package compress/gzip:
func NewWriter(w io.Writer) *Writer
NewWriter returns a new Writer.
Writes to the returned writer are compressed and written to w.
It is the caller's responsibility to call Close on the WriteCloser
when done. Writes may be buffered and not flushed until Close.
In your code, it might look like this:
gz := gzip.NewWriter(w)
defer gz.Close()
err = tmpl.Execute(gz, nil)
if err != nil {
http.Error(gz, err.Error(), http.StatusInternalServerError)
return
}
Ps.
You might want to check the Accept-Encoding header in the request to see if the browser accepts gzip encoded content.
r.Header.Set("Accept-Encoding", "gzip")
Setting the headers on the request has no effect. You would want to read this header to determine whether to use gzip.
The reason the browser downloads the file instead of rendering it is that it's missing a content type. When you write to an http.ResponseWriter without setting the Content-Type there's an algorithm that determines the Content-Type and sets the correct header. But with your body gzipped, it can't determine the correct content type. Add this before you write any data back to the client:
w.Header().Set("Content-Type", "text/html")
This is a bit to late but i hope someone else finds this usefull.
The code works but you need to set the header to be properly interpreted by the browser:
w.Header().Add("Content-Type","text/html")
w.Header().Add("Content-Encoding","gzip")
Anyway, I have not solved this problem. As I mentioned, when I add this code, restart the server and query this url, I actually download the zip file!?
In the end, I used Nginx to make a reverse proxy to compress the template. It works.
Still thanks to #Anisus

What's the optimal way to execute a nodejs script from golang, that returns a string, and then communicate that string back to a golang variable?

I'm currently doing it with os/exec and Stdout on golang's side, and console.log("string") on nodejs's side.
Basically I need to generate a string but can only do so within nodejs but the majority of my code is in golang, so I'm trying to make this little blip in my code as seamless, secure, and reliable as possible and I'm a little uneasy about resting such an important part of my program on "console.log" and reading from shell output.
In short: I'm wondering if there exists a better and more standard communication line between my node and go code then console.log + shell output, or is that perhaps optimal enough?
Oh and the function of this particular part of my program is to take a markdown text file and convert it to HTML using markdown-it.
Some ideas:
Communicate through HTTP (send the data/string to a golang http listener)
Communicate through the filesystem (write the string to a temporary file and read it with golang)
Communicate through "something similiar to HTTP but specific to local application data sharing"
P.S.
I can't use otto, since markdown-it doesn't run there.
Actual code:
parser.go
package main
import (
"os"
"os/exec"
"fmt"
"bytes"
)
func main() {
cmd := "node"
args := []string{"parser.js", "/home/user1/dev/current/wikis/Bob's Pain/markup/index.md"}
process := exec.Command(cmd, args...)
stdin, err := process.StdinPipe()
if err != nil {
fmt.Println(err)
}
defer stdin.Close()
buf := new(bytes.Buffer) // THIS STORES THE NODEJS OUTPUT
process.Stdout = buf
process.Stderr = os.Stderr
if err = process.Start(); err != nil {
fmt.Println("An error occured: ", err)
}
process.Wait()
fmt.Println("Generated string:", buf)
}
parser.js
var md = require('markdown-it')();
var yaml = require('js-yaml');
var fs = require('fs');
if (process.argv.length < 3) {
console.log('Usage: node ' + process.argv[1] + ' FILENAME');
process.exit(1);
}
var filename = process.argv[2];
fs.readFile(filename, 'utf8', function(err, data) {
if (err) {
throw err;
}
parse(data)
});
function parse(data) {
data = data.split("---")
yamlData = data[1];
markData = data[2];
y = yamlProcess(yamlData);
markData = "# "+y.title+"\n\n"+markData
html = markdownToHTML(markData);
console.log(html) // SEND THE DATA BACK TO GOLANG
}
function yamlProcess(data) {
try {
var doc = yaml.safeLoad(data);
return doc;
} catch (e) {
console.log(e);
return {};
}
}
function markdownToHTML(data) {
return md.render(data);
}
The easiest way to do this with os/exec:
command := "node parser.js /path/to/some/file.md"
parts := strings.Fields(command)
data, err := exec.Command(parts[0], parts[1:]...).Output()
if err != nil {
panic(err)
}
output := string(data)
"output" is the data that is printed from your NodeJS script. "command" is any command as a string.
I've approached similar requirement both ways.
For a build pipeline extension, I'd write a Python script that takes arguments from command line and outputs results to stdout. It's a simple interface, for a "run once", "everything succeeds otherwise fail fast" usage. If that's the same in your case, I'd keep the implementation as-is.
For a web application, I had Java service for just a specific function (in my case, Natty date recognition from a natural language string). This has the benefit that the application is already "warmed up" at the time of the call, and will definitely respond faster rather than booting up each time the request comes in. Going with a rest interface will probably reveal more benefits over time - e.g. simpler client implementation, monitoring, deployment options, switch implementation without changing clients, etc. It's just more conventional this way.

Reading from a Reader multiple times

I'm building a simple caching proxy that intercepts HTTP requests, grabs the content in response.Body, then writes it back to the client. The problem is, as soon as I read from response.Body, the write back to the client contains an empty body (everything else, like the headers, are written as expected).
Here's the current code:
func requestHandler(w http.ResponseWriter, r *http.Request) {
client := &http.Client{}
r.RequestURI = ""
response, err := client.Do(r)
defer response.Body.Close()
if err != nil {
log.Fatal(err)
}
content, _ := ioutil.ReadAll(response.Body)
cachePage(response.Request.URL.String(), content)
response.Write(w)
}
If I remove the content, _ and cachePage lines, it works fine. With the lines included, requests return and empty body. Any idea how I can get just the Body of the http.Response and still write out the response in full to the http.ResponseWriter?
As in my comment you could implement the io.ReadCloser
As per Dewy Broto (Thanks) you can do this much simpler with:
content, _ := ioutil.ReadAll(response.Body)
response.Body = ioutil.NopCloser(bytes.NewReader(content))
response.Write(w)
As you have discovered, you can only read once from a request's Body.
Go has a reverse proxy that will facilitate what you are trying to do. Check out httputil.ReverseProxy and httputil.DumpResponse
You do not need to read from the response a second time. You already have the data in hand and can write it directly to the response writer.
The call
response.Write(w)
writes the response in wire format to the server's response body. This is not what you want for a proxy. You need to copy the headers, status and body to the server response individually.
I have noted other issues in the code comments below.
I recommend using the standard library's ReverseProxy or copying it and modifying it to meet your needs.
func requestHandler(w http.ResponseWriter, r *http.Request) {
// No need to make a client, use the default
// client := &http.Client{}
r.RequestURI = ""
response, err := http.DefaultClient.Do(r)
// response can be nil, close after error check
// defer response.Body.Close()
if err != nil {
log.Fatal(err)
}
defer response.Body.Close()
// Check errors! Always.
// content, _ := ioutil.ReadAll(response.Body)
content, err := ioutil.ReadAll(response.Body)
if err != nil {
// handle error
}
cachePage(response.Request.URL.String(), content)
// The Write method writes the response in wire format to w.
// Because the server handles the wire format, you need to do
// copy the individual pieces.
// response.Write(w)
// Copy headers
for k, v := range response.Header {
w.Header()[k] = v
}
// Copy status code
w.WriteHeader(response.StatusCode)
// Write the response body.
w.Write(content)
}

go websockets eof

I'm trying to make a simple command forwarder to connect my home computer to a server I own, so that I can push commands to my server and my home pc gets it. Those commands are simple pause/resume for my downloader. My design is, that on a server, I run a hub instance, which creates a window for passing commands and a window for backend to pass those commands to my pc. I'm bounding those two "windows" with a channel, they run a server. When a client connects and sends a message to the hub, it gets streamed through a channel to backend window and then to the real backend (on my home pc). When backend responds to the backend window on the hub, the hub prints the result back to the client.
With this approach, only the first message passes and works with my downloader. I have to reconnect the backend from my home pc with the hub each time I get a message to get this working properly. I don't think that's the proper way with websockets, so here I am. After one successful request (when the backend finishes it's work and replies the result), it gets looped forever with EOF error.
The important parts of the code are:
main executable
hub handlers
backend connector
If you put the source in your GOPATH (i'm developing it for the tip version of go to support modern websockets), to compile it:
go build gosab/cmd, to run it:
./cmd -mode="hub" hub
./cmd -mode="backend" --address="localhost:8082" backend
To pass messages to the hub, use this javascript:
var s = new WebSocket("ws://localhost:8082")
s.send("1 5")
So how do I handle it? Are channels a good way to communicate between two different requests?
I'm surprised you haven't received an answer to this.
What you need to do is something like the code below. When you receive an incoming websocket connection, a new goroutine is spawned for that connection. If you let that goroutine end, it'll disconnect the websocket client.
I'm making an assumption that you're not necessarily going to be running the client and server on the same computer. If you always are, then it'd be better to do the communication internally via channels or such instead of using websockets or a network port. I only mention this because I'm not completely sure what you're using this for. I just hope I answered the right part of your question.
package main
import (
"code.google.com/p/go.net/websocket"
"flag"
"fmt"
"net/http"
"os"
"time"
)
type Message struct {
RequestID int
Command string
SomeOtherThing string
Success bool
}
var mode *string = flag.String("mode", "<nil>", "Mode: server or client")
var address *string = flag.String("address", "localhost:8080", "Bind address:port")
func main() {
flag.Parse()
switch *mode {
case "server":
RunServer()
case "client":
RunClient()
default:
flag.Usage()
}
}
func RunServer() {
http.Handle("/", http.FileServer(http.Dir("www")))
http.Handle("/server", websocket.Handler(WSHandler))
fmt.Println("Starting Server")
err := http.ListenAndServe(*address, nil)
if err != nil {
fmt.Printf("HTTP failed: %s\n", err.Error())
os.Exit(1)
}
}
func WSHandler(ws *websocket.Conn) {
defer ws.Close()
fmt.Println("Client Connected")
for {
var message Message
err := websocket.JSON.Receive(ws, &message)
if err != nil {
fmt.Printf("Error: %s\n", err.Error())
return
}
fmt.Println(message)
// do something useful here...
response := new(Message)
response.RequestID = message.RequestID
response.Success = true
response.SomeOtherThing = "The hot dog left the castle as requested."
err = websocket.JSON.Send(ws, response)
if err != nil {
fmt.Printf("Send failed: %s\n", err.Error())
os.Exit(1)
}
}
}
func RunClient() {
fmt.Println("Starting Client")
ws, err := websocket.Dial(fmt.Sprintf("ws://%s/server", *address), "", fmt.Sprintf("http://%s/", *address))
if err != nil {
fmt.Printf("Dial failed: %s\n", err.Error())
os.Exit(1)
}
incomingMessages := make(chan Message)
go readClientMessages(ws, incomingMessages)
i := 0
for {
select {
case <-time.After(time.Duration(2e9)):
i++
response := new(Message)
response.RequestID = i
response.Command = "Eject the hot dog."
err = websocket.JSON.Send(ws, response)
if err != nil {
fmt.Printf("Send failed: %s\n", err.Error())
os.Exit(1)
}
case message := <-incomingMessages:
fmt.Println(message)
}
}
}
func readClientMessages(ws *websocket.Conn, incomingMessages chan Message) {
for {
var message Message
err := websocket.JSON.Receive(ws, &message)
if err != nil {
fmt.Printf("Error: %s\n", err.Error())
return
}
incomingMessages <- message
}
}

Resources