Skip to content

http client POST io.ReadCloser,需要显示指定Request.TransferEncoding为chunked #16264

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
yuehai1117 opened this issue Jul 5, 2016 · 6 comments
Labels
FrozenDueToAge WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.

Comments

@yuehai1117
Copy link

  1. What version of Go are you using (go version)?
    go1.5
  2. What operating system and processor architecture are you using (go env)?
    Linux xxxxxx 3.10.0-327.el7.x86_64 cmd/cgo: fails with gcc 4.4.1 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

http client POST io.ReadCloser流,如果不显示指定Request.TransferEncoding为chunked,这时io.ReadCloser数据将被分成多个chunked,并且第一个chunked的大小总是为1.
下面是一个例子:
package main
import (
"io/ioutil"
"net/http"
"strings"
)
func main() {
http.Post("http://172.18.1.218/hehe/test.php", "application/x-www-form-urlencoded", ioutil.NopCloser(strings.NewReader("teststring")))
}
tcpdump抓包,request请求包如下:
Flags [P.], seq 1:191, ack 1, win 229, options [nop,nop,TS val 3434923483 ecr 1719817619], length 190
0x0000: 4500 00f2 aa2d 4000 4006 3409 ac12 01d1 E....-@.@.4.....
0x0010: ac12 01da d166 0050 33bd 6ff2 4f4a afc1 .....f.P3.o.OJ..
0x0020: 8018 00e5 5cb4 0000 0101 080a ccbc c5db ...............
0x0030: 6682 5593 504f 5354 202f 6865 6865 2f74 f.U.POST./hehe/t
0x0040: 6573 742e 7068 7020 4854 5450 2f31 2e31 est.php.HTTP/1.1
0x0050: 0d0a 486f 7374 3a20 3137 322e 3138 2e31 ..Host:.172.18.1
0x0060: 2e32 3138 0d0a 5573 6572 2d41 6765 6e74 .218..User-Agent
0x0070: 3a20 476f 2d68 7474 702d 636c 6965 6e74 :.Go-http-client
0x0080: 2f31 2e31 0d0a 5472 616e 7366 6572 2d45 /1.1..Transfer-E
0x0090: 6e63 6f64 696e 673a 2063 6875 6e6b 6564 ncoding:.chunked
0x00a0: 0d0a 436f 6e74 656e 742d 5479 7065 3a20 ..Content-Type:.
0x00b0: 6170 706c 6963 6174 696f 6e2f 782d 7777 application/x-ww
0x00c0: 772d 666f 726d 2d75 726c 656e 636f 6465 w-form-urlencode
0x00d0: 640d 0a41 6363 6570 742d 456e 636f 6469 d..Accept-Encodi
0x00e0: 6e67 3a20 677a 6970 0d0a 0d0a 310d 0a74 ng:.gzip....1..t
0x00f0: 0d0a 0000 0000 0000 0000 0000 0000 0000 ................
0x0100: 0000 ..
Flags [P.], seq 191:205, ack 1, win 229, options [nop,nop,TS val 3434923483 ecr 1719817619], length 14
0x0000: 4500 0042 aa2e 4000 4006 34b8 ac12 01d1 E..B..@.@.4.....
0x0010: ac12 01da d166 0050 33bd 70b0 4f4a afc1 .....f.P3.p.OJ..
0x0020: 8018 00e5 5c04 0000 0101 080a ccbc c5db ...............
0x0030: 6682 5593 390d 0a65 7374 7374 7269 6e67 f.U.9..eststring
0x0040: 0d0a 0000 0000 0000 0000 0000 0000 0000 ................
0x0050: 0000 ..
从上面的抓包数据可以看到,"teststring"被分成两个chunked,第一个chunked长度为1,chunked为"t",第二个chunked长度为9,chunked为"eststring"

如果POST io.Reader流,Request.TransferEncoding并不会设置为chunked,而是Request.ContentLength被设置为POST数据的长度,如下面这个例子:
package main
import (
"io/ioutil"
"net/http"
"strings"
)
func main() {
http.Post("http://172.18.1.218/hehe/test.php", "application/x-www-form-urlencoded", strings.NewReader("teststring"))
}
tcpdump抓包,request请求包如下:
Flags [P.], seq 1:187, ack 1, win 229, options [nop,nop,TS val 3434923486 ecr 1719817622], length 186
0x0000: 4500 00ee 7c21 4000 4006 6219 ac12 01d1 E...|!@.@.b.....
0x0010: ac12 01da d167 0050 c2a4 b9c3 ba98 6644 .....g.P......fD
0x0020: 8018 00e5 5cb0 0000 0101 080a ccbc c5de ...............
0x0030: 6682 5596 504f 5354 202f 6865 6865 2f74 f.U.POST./hehe/t
0x0040: 6573 742e 7068 7020 4854 5450 2f31 2e31 est.php.HTTP/1.1
0x0050: 0d0a 486f 7374 3a20 3137 322e 3138 2e31 ..Host:.172.18.1
0x0060: 2e32 3138 0d0a 5573 6572 2d41 6765 6e74 .218..User-Agent
0x0070: 3a20 476f 2d68 7474 702d 636c 6965 6e74 :.Go-http-client
0x0080: 2f31 2e31 0d0a 436f 6e74 656e 742d 4c65 /1.1..Content-Le
0x0090: 6e67 7468 3a20 3130 0d0a 436f 6e74 656e ngth:.10..Conten
0x00a0: 742d 5479 7065 3a20 6170 706c 6963 6174 t-Type:.applicat
0x00b0: 696f 6e2f 782d 7777 772d 666f 726d 2d75 ion/x-www-form-u
0x00c0: 726c 656e 636f 6465 640d 0a41 6363 6570 rlencoded..Accep
0x00d0: 742d 456e 636f 6469 6e67 3a20 677a 6970 t-Encoding:.gzip
0x00e0: 0d0a 0d0a 7465 7374 7374 7269 6e67 0000 ....teststring..
0x00f0: 0000 0000 0000 0000 0000 0000 0000 ..............
可以看出,request请求包中有Content-Length header,数据并不是以chunked方式传输的。

如果POST io.ReadCloser流,并显示指定Request.TransferEncoding为chunked,POST的数据并不会被分为两个chunked。如下面例子:
package main
import (
"fmt"
"io/ioutil"
"net/http"
"strings"
)
func main() {
client := &http.Client{
Transport: &http.Transport{
DisableCompression: true,
},
}
req, err := http.NewRequest("POST", "http://172.18.1.218/hehe/test.php", ioutil.NopCloser(strings.NewReader("teststring")))
if err != nil {
fmt.Println(err)
return
}
req.TransferEncoding = []string{"chunked"}
client.Do(req)
}
tcpdump抓包,request请求包如下:
Flags [P.], seq 1:128, ack 1, win 229, options [nop,nop,TS val 3437830591 ecr 154725], length 127
0x0000: 4500 00b3 f410 4000 4006 ea64 ac12 01d1 E.....@.@..d....
0x0010: ac12 01da ee2f 0050 6ebc d5b1 8c5e 75d2 ...../.Pn....^u.
0x0020: 8018 00e5 5c75 0000 0101 080a cce9 21bf ....\u........!.
0x0030: 0002 5c65 504f 5354 202f 6865 6865 2f74 ..\ePOST./hehe/t
0x0040: 6573 742e 7068 7020 4854 5450 2f31 2e31 est.php.HTTP/1.1
0x0050: 0d0a 486f 7374 3a20 3137 322e 3138 2e31 ..Host:.172.18.1
0x0060: 2e32 3138 0d0a 5573 6572 2d41 6765 6e74 .218..User-Agent
0x0070: 3a20 476f 2d68 7474 702d 636c 6965 6e74 :.Go-http-client
0x0080: 2f31 2e31 0d0a 5472 616e 7366 6572 2d45 /1.1..Transfer-E
0x0090: 6e63 6f64 696e 673a 2063 6875 6e6b 6564 ncoding:.chunked
0x00a0: 0d0a 0d0a 610d 0a74 6573 7473 7472 696e ....a..teststrin
0x00b0: 670d 0a00 0000 0000 0000 0000 0000 0000 g...............
0x00c0: 0000 00 ...
可以看出,request POST是以chunked方式发出去,并且chunked没有分片。

@bradfitz
Copy link
Contributor

bradfitz commented Jul 5, 2016

I maintain the net/http package, but unfortunately I don't know Chinese.

Can you or somebody else translate? Thanks!

@bradfitz bradfitz added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Jul 5, 2016
@tevino
Copy link

tevino commented Jul 5, 2016

Translation:

http client POST io.ReadCloser流,如果不显示指定Request.TransferEncoding为chunked,这时io.ReadCloser数据将被分成多个chunked,并且第一个chunked的大小总是为1.下面是一个例子:

If you do http.Post() without setting req.TransferEncoding = []string{"chunked"} the data will be sent in chunks, but the length of the first chunk is always 1. For example:


从上面的抓包数据可以看到,"teststring"被分成两个chunked,第一个chunked长度为1,chunked为"t",第二个chunked长度为9,chunked为"eststring"

As we can see in the output of tcpdump, teststring is divided into two chunks, the first contains only t, the second contains eststring


可以看出,request请求包中有Content-Length header,数据并不是以chunked方式传输的。

The Content-Length is in the request packet, data are not sent chunked.


如果POST io.ReadCloser流,并显示指定Request.TransferEncoding为chunked,POST的数据并不会被分为两个chunked。如下面例子:
If you set Request.TransferEncoding to "chunked", the data posted are not sent chunked, for example:

@noblehng
Copy link

noblehng commented Jul 5, 2016

Only when he wrapping a *strings.Reader with a ioutil.NopCloser as the post body.

I think that is because the Content-Length will not be set here if body is a ioutil.nopCloser, so the body will be chunked to read one byte first to test its length here. If the Transfer-Encoding is manually set to chunked, then the test len(t.TransferEncoding) == 0 one line above will failed, thus skip the read one byte code path.

@bradfitz
Copy link
Contributor

bradfitz commented Jul 7, 2016

Yes, the 1 byte chunk is to determine whether the Request.Body's io.Reader is actually ContentLength 0, or whether it just wasn't populated.

I don't see an example where Go generates invalid HTTP.

What is the bug here? Or is this just a question?

@tomheng
Copy link
Contributor

tomheng commented Jul 8, 2016

@bradfitz I wonder why the Read function of MultiReader did not supply required length of bytes, by reading across sub readers.

@bradfitz
Copy link
Contributor

bradfitz commented Jul 8, 2016

It could, but nothing in Go requires it for correctness, and there's been no other motivation to do so.

I'm going to close this bug if there's nothing to fix here. I assume this bug report was just asked out of curiosity.

Otherwise we can take discussion to the mailing list. See https://golang.org/wiki/Questions

@bradfitz bradfitz closed this as completed Jul 8, 2016
@golang golang locked and limited conversation to collaborators Jul 8, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Development

No branches or pull requests

6 participants