how to gzip a template in golang web - web

I have already set the value of contest-encoding, but how can I gzip the template, since the file is still to big.
func indexPageHandler(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Encoding", "gzip")
r.Header.Set("Accept-Encoding", "gzip")
tmpl, err := template.New("index.html").ParseGlob("./templates/*")
if err != nil {
log.Println(err)
return
}
err = tmpl.Execute(w, nil)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
}
Is there any function that can gzip the response?]
Followed by the advise, I changed my code as this:
func indexPageHandler(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Encoding", "gzip")
r.Header.Set("Accept-Encoding", "gzip")
gz := gzip.NewWriter(w)
defer gz.Close()
tmpl, err := template.New("index.html").ParseGlob("./templates/*")
if err != nil {
log.Println(err)
return
}
err = tmpl.Execute(gz, nil)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
}
But when I query the url, I only download the gzip file. What's wrong with it?

You can use package compress/gzip:
func NewWriter(w io.Writer) *Writer
NewWriter returns a new Writer.
Writes to the returned writer are compressed and written to w.
It is the caller's responsibility to call Close on the WriteCloser
when done. Writes may be buffered and not flushed until Close.
In your code, it might look like this:
gz := gzip.NewWriter(w)
defer gz.Close()
err = tmpl.Execute(gz, nil)
if err != nil {
http.Error(gz, err.Error(), http.StatusInternalServerError)
return
}
Ps.
You might want to check the Accept-Encoding header in the request to see if the browser accepts gzip encoded content.

r.Header.Set("Accept-Encoding", "gzip")
Setting the headers on the request has no effect. You would want to read this header to determine whether to use gzip.
The reason the browser downloads the file instead of rendering it is that it's missing a content type. When you write to an http.ResponseWriter without setting the Content-Type there's an algorithm that determines the Content-Type and sets the correct header. But with your body gzipped, it can't determine the correct content type. Add this before you write any data back to the client:
w.Header().Set("Content-Type", "text/html")

This is a bit to late but i hope someone else finds this usefull.
The code works but you need to set the header to be properly interpreted by the browser:
w.Header().Add("Content-Type","text/html")
w.Header().Add("Content-Encoding","gzip")

Anyway, I have not solved this problem. As I mentioned, when I add this code, restart the server and query this url, I actually download the zip file!?
In the end, I used Nginx to make a reverse proxy to compress the template. It works.
Still thanks to #Anisus

Related

Azure sdk for go - code can't make it passed compute.VirtualMachinesClient.ListAllComplete()

I'm testing a function I have that gets all azure vms under a specific subscription.
It uses azure sdk for go's compute.VirtualMachinesClient to do so.
My problem is with the vm clients ListAllComplete function.
It's not returning an error. The code just doesn't seem to be able to make it passed that line.
Any suggestions on the source of the problem would be appreciated.
This is the code, I've used the fmt package to follow how far it gets:
func GetAllAzureVms() ([]compute.VirtualMachine, error) {
fmt.Printf("In getAllAzureVm\n")
var vmList []compute.VirtualMachine
vmClient, err := GetAzureVmClient()
fmt.Printf("Out of GetAzureVmClient\n")
if err != nil {
return nil, err
}
fmt.Print("No error from getazurevmclient\n")
vmListComplete, err := vmClient.ListAllComplete(context.Background(), "statusOnly=false")
fmt.Print("vmClient.ListAllComplete done")
if err != nil {
fmt.Print("vmClient.ListAllComplete error")
return nil, err
}
fmt.Print("here")
for vmListComplete.NotDone() {
vmList = append(vmList, vmListComplete.Value())
err := vmListComplete.NextWithContext(context.Background())
if err != nil {
return nil, err
}
}
fmt.Print("here2")
return vmList, nil
}
It cant make it passed the line:
vmListComplete, err := vmClient.ListAllComplete(context.Background(), "statusOnly=false")
No error is returned.
I have a similar piece of code, not for vmClient unluckily, but for securityCustomRules.
What I've found useful was to use ListComplete() instead of ListAll() and the print values through JSON marhsalling.
I hope you can find it useful anyway.
import (
"context"
"fmt"
"os"
"testing"
"github.com/gruntwork-io/terratest/modules/azure"
"github.com/gruntwork-io/terratest/modules/terraform"
)
securityCustomRulesList, err := securityCustomRulesClient.ListComplete(context.Background(), tfResourceGroupName, tfVnetName+"-"+fmt.Sprint(vnetIndex)+"-subnet-02-nsg")
// - Scan iterator items [https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/services/network/mgmt/2019-09-01/network#ApplicationGatewayListResultIterator]
if err != nil {
fmt.Fprintf(os.Stderr, ">>>> Error parsing securityCustomRulesClient:: %s", err)
return
}
for securityCustomRulesList.NotDone() {
// securityCustomRulesList.Value() -> securityRule [https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/services/network/mgmt/2019-09-01/network#SecurityRule]
v := securityCustomRulesList.Value()
vJson, _ := v.MarshalJSON()
fmt.Printf(">> Network securityCustomRulesList JSON %s \n", string(vJson))
//fmt.Printf(">> Network securityCustomRulesList %s - %s\n", *v.Name, *v.SecurityRulePropertiesFormat.Description)
securityCustomRulesList.NextWithContext(context.Background())
}
Which gives me an output like:
>> Creating security custom rules list instance 'securityCustomRulesList'..
>> Network securityCustomRulesList JSON {"id":"/subscriptions/8f7d6be2/resourceGroups/unit-tests-tfm-azure-network-resource-group/providers/Microsoft.Network/networkSecurityGroups/unit-tests-tfm-azure-network-vnet-0-subnet-02-nsg/securityRules/test-02","name":"test-02","properties":{"access":"Deny","description":"Deny access","destinationAddressPrefix":"10.0.2.1","destinationAddressPrefixes":[],"destinationPortRange":"*","destinationPortRanges":[],"direction":"Inbound","priority":111,"protocol":"*","sourceAddressPrefix":"10.0.1.0/24","sourceAddressPrefixes":[],"sourcePortRange":"*","sourcePortRanges":[]}}
Useful links:
Here you can find a complete usage example from the official terratest/azure module: https://github.com/gruntwork-io/terratest/blob/dae956eb39e91dfb00f3ba85060a6dbf58c6782b/modules/azure/nsg.go
https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/services/network/mgmt/2019-09-01/network#SecurityRule.MarshalJSON

While downloading file from Azure Blob Storage using Golang getting " curl Empty reply from server" , but file is downloaded in background

I am trying to download a file from Azure Blob Storage using http request. I am able to download the file but on a terminal curl returns "Empty reply from server". I tried to increase the timeout, but it didn't fix it. I referred other questions related to this response from curl, but it didn't help. For small files this code is working flawlessly but for big files say 75 MB it is not working.
containerURL := azblob.NewContainerURL(*URL, pipeline)
blobURL := containerURL.NewBlockBlobURL(splitArray[1])
ctx := context.Background()
downloadResponse, err := blobURL.Download(ctx, 0, azblob.CountToEnd, azblob.BlobAccessConditions{}, false)
if err != nil {
.
.
.
}
bodyStream := downloadResponse.Body(azblob.RetryReaderOptions{MaxRetryRequests: 20})
// read the body into a buffer
downloadedData := bytes.Buffer{}
_, err = downloadedData.ReadFrom(bodyStream)
file, err := os.OpenFile(
"/tmp/"+fileName,
os.O_RDWR|os.O_TRUNC|os.O_CREATE,
0777,
)
file.Write(downloadedData.Bytes())
file.Close()
filePath := "/tmp/" + fileName
file, err = os.Open(filePath)
return middleware.ResponderFunc(func(w http.ResponseWriter, r runtime.Producer) {
fn := filepath.Base(filePath)
w.Header().Set(CONTENTTYPE, "application/octet-stream")
w.Header().Set("Content-Disposition", fmt.Sprintf("attachment; filename=%q", fn))
io.Copy(w, file)
err := defer os.Remove(filePath)
file.Close()
})
I am thinking of implementing the above logic using goroutines. Is there even a need of using goroutines?
Any constructive feedback will be helpful.
After analyzing packets from wireshark got to know it was getting disconnected from my side due to timeout as I am using go-swagger , I increased the timeout , in configure.go . GoSwagger provides in-built function for handling these scenarios like TLS , Timeout. Below is code for reference.
// As soon as server is initialized but not run yet, this function will be called.
// If you need to modify a config, store server instance to stop it individually later, this is the place.
// This function can be called multiple times, depending on the number of serving schemes.
// scheme value will be set accordingly: "http", "https" or "unix"
func configureServer(s *http.Server, scheme, addr string) {
s.WriteTimeout(time.Minute * 5)
}

Garbled string while do http request in Go

I met a Garbled string question when do Get request in Go, the code is:
req , err:= http.NewRequest(httpMethod, url,strings.NewReader(""))
req.Header.Add("Accept","application/json")
resp, err := http.DefaultClient.Do(req)
body,err := ioutil.ReadAll(resp.Body)
ret := string(body)
log.Warningf("ret: %+v", ret)
if the ret contains only english, it's correct, if contains Chinese, it has garbled string, how to solve this problem, thanks all!
Go strings can hold any type of characters, but when printing them the chars are interpreted as utf-8.
You can try adding:
req.Header.Add("Accept-Charset","utf-8")
If that does not work, you can try using this package to convert from whatever charset it is to utf-8:
https://godoc.org/golang.org/x/text/encoding
The charset depends on the page you are requesting. If it is html, the charset is sometimes specified like this in the response headers:
Content-Type: text/html; charset=utf-8
So you need to figure out what the charset is.
In my case website did not respond with the charset within a Content-Type and did not replied with requested: req.Header.Add("Accept-Charset", "utf-8")
I opended up file in Visual Studio Code and have been switching encoding to find out which one works best "Reopen With Encoding".
Once i figured out which encoding it was i simply used function:
dec := charmap.Windows1250.NewDecoder()
output, _ := dec.Bytes(body)
from: "golang.org/x/text/encoding/charmap"
Full code example:
package main
import (
"fmt"
"net/http"
"io/ioutil"
"golang.org/x/text/encoding/charmap"
)
func main() {
client := &http.Client{}
req, err := http.NewRequest("GET", "example.com", nil)
if err != nil {
fmt.Println(err)
return
}
resp, err := client.Do(req)
if err != nil {
fmt.Println(err)
return
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Println(err)
return
}
dec := charmap.Windows1250.NewDecoder()
output, err:= dec.Bytes(body)
if err != nil {
fmt.Println(err)
return
}
// do something with output
}

making go http client work with non-standard http servers

Shoutcast servers basically speak http, with one important difference: they respond to GET requests with ICY 200 OK instead of HTTP/1.1 200 OK.
Go won't have a bar of it, and correctly fails with the error malformed HTTP version "ICY".
However I would like to make things work and am wondering what the best approach is. My ideas so far:
use a custom http.Transport.Proxy to change ICY to HTTP/1.1 in-flight
an out of process proxy that does the same thing
overload http.ParseHTTPVersion (but golang doesn't have function overloading)
duplicate the entire http package, just to modify ParseHTTPVersion
Number 1. seems the most attractive attractive, but I have no idea how to respect the http "scope" and actually modify all responses on a given http version. Is this the kind of thing http.Transport.Proxy can handle?
Can anyone give me any pointers?
I got this working by creating a custom Dial function that returns a wrapped connection. My wrapper intercepts the first read on the connection and replaces ICY with HTTP/1.1. Not super robust, but proves the concept:
package main
import (
"fmt"
"net"
"net/http"
)
type IcyConnWrapper struct {
net.Conn
haveReadAny bool
}
func (i *IcyConnWrapper) Read(b []byte) (int, error) {
if i.haveReadAny {
return i.Conn.Read(b)
}
i.haveReadAny = true
//bounds checking ommitted. There are a few ways this can go wrong.
//always check array sizes and returned n.
n, err := i.Conn.Read(b[:3])
if err != nil {
return n, err
}
if string(b[:3]) == "ICY" {
//write Correct http response into buffer
copy(b, []byte("HTTP/1.1"))
return 8, nil
}
return n, nil
}
func main() {
tr := &http.Transport{
Dial: func(network, a string) (net.Conn, error) {
realConn, err := net.Dial(network, a)
if err != nil {
return nil, err
}
return &IcyConnWrapper{Conn: realConn}, nil
},
}
client := &http.Client{Transport: tr}
http.DefaultClient = client
resp, err := http.Get("http://178.33.230.189:8100") //random url I found on the internet
fmt.Println(err)
fmt.Println(resp.StatusCode)
}

Reading from a Reader multiple times

I'm building a simple caching proxy that intercepts HTTP requests, grabs the content in response.Body, then writes it back to the client. The problem is, as soon as I read from response.Body, the write back to the client contains an empty body (everything else, like the headers, are written as expected).
Here's the current code:
func requestHandler(w http.ResponseWriter, r *http.Request) {
client := &http.Client{}
r.RequestURI = ""
response, err := client.Do(r)
defer response.Body.Close()
if err != nil {
log.Fatal(err)
}
content, _ := ioutil.ReadAll(response.Body)
cachePage(response.Request.URL.String(), content)
response.Write(w)
}
If I remove the content, _ and cachePage lines, it works fine. With the lines included, requests return and empty body. Any idea how I can get just the Body of the http.Response and still write out the response in full to the http.ResponseWriter?
As in my comment you could implement the io.ReadCloser
As per Dewy Broto (Thanks) you can do this much simpler with:
content, _ := ioutil.ReadAll(response.Body)
response.Body = ioutil.NopCloser(bytes.NewReader(content))
response.Write(w)
As you have discovered, you can only read once from a request's Body.
Go has a reverse proxy that will facilitate what you are trying to do. Check out httputil.ReverseProxy and httputil.DumpResponse
You do not need to read from the response a second time. You already have the data in hand and can write it directly to the response writer.
The call
response.Write(w)
writes the response in wire format to the server's response body. This is not what you want for a proxy. You need to copy the headers, status and body to the server response individually.
I have noted other issues in the code comments below.
I recommend using the standard library's ReverseProxy or copying it and modifying it to meet your needs.
func requestHandler(w http.ResponseWriter, r *http.Request) {
// No need to make a client, use the default
// client := &http.Client{}
r.RequestURI = ""
response, err := http.DefaultClient.Do(r)
// response can be nil, close after error check
// defer response.Body.Close()
if err != nil {
log.Fatal(err)
}
defer response.Body.Close()
// Check errors! Always.
// content, _ := ioutil.ReadAll(response.Body)
content, err := ioutil.ReadAll(response.Body)
if err != nil {
// handle error
}
cachePage(response.Request.URL.String(), content)
// The Write method writes the response in wire format to w.
// Because the server handles the wire format, you need to do
// copy the individual pieces.
// response.Write(w)
// Copy headers
for k, v := range response.Header {
w.Header()[k] = v
}
// Copy status code
w.WriteHeader(response.StatusCode)
// Write the response body.
w.Write(content)
}

Resources