Skip to content

Issue with "Connection: close" set by client #8525

New issue
New issue
Closed
#8561
Closed
Issue with "Connection: close" set by client#8525
#8561
Assignees
jsternberg
@pavel-odintsov

Description

@pavel-odintsov
pavel-odintsov
opened on Jun 25, 2017

Hello!

I use a big number of independent connections to push ~10 lines of data to InfluxDB. With my data rate, I have about 2500 of TIME_WAIT connections in any period of time.

It's an issue for me because our systems have a pretty small amount of resources. I started to look for options to reduce number of TIME_WAIT connections.

I started from "Conneciton: close" http option to terminate connections immediately because I have never re-use them.

I implemented my own (pretty complicated code based on https://github.com/vinniefalco/Beast/blob/master/example/http-crawl/http_crawl.cpp) implementation and got an issue from InfluxDB side.

This code is complicated to isolated and I moved to simple example with curl.

Curl example:

curl -d "networks_traffic,network=185.86.148.0/22 bits_incoming=0,bits_outgoing=0,packets_incoming=0,packets_outgoing=0 1498413453000000000" -v "http://127.0.0.1:8086/write?db=fastnetmon" -sL -H 'Connection: close'

curl debug below:

curl -d "networks_traffic,network=185.86.148.0/22 bits_incoming=0,bits_outgoing=0,packets_incoming=0,packets_outgoing=0 1498413453000000000" -v "http://127.0.0.1:8086/write?db=fastnetmon" -sL -H 'Connection: close'
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 8086 (#0)
> POST /write?db=fastnetmon HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 127.0.0.1:8086
> Accept: */*
> Connection: close
> Content-Length: 130
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 130 out of 130 bytes
< HTTP/1.1 204 No Content
< Content-Type: application/json
< Request-Id: cfd9829a-59da-11e7-9a97-000000000000
< X-Influxdb-Version: 1.2.4
< Date: Sun, 25 Jun 2017 19:16:38 GMT
< Connection: close
< 
* Closing connection 0

According to this output connection from client side (cURL) was closed immediately.

Then I captured data with TCPDUMP:

20:16:38.296716 IP localhost.44252 > localhost.8086: Flags [S], seq 1370079652, win 43690, options [mss 65495,sackOK,TS val 1326659 ecr 0,nop,wscale 9], length 0
20:16:38.296739 IP localhost.8086 > localhost.44252: Flags [S.], seq 2874189319, ack 1370079653, win 43690, options [mss 65495,sackOK,TS val 1326659 ecr 1326659,nop,wscale 9], length 0
20:16:38.296758 IP localhost.44252 > localhost.8086: Flags [.], ack 1, win 86, options [nop,nop,TS val 1326659 ecr 1326659], length 0
20:16:38.296904 IP localhost.44252 > localhost.8086: Flags [P.], seq 1:318, ack 1, win 86, options [nop,nop,TS val 1326659 ecr 1326659], length 317
20:16:38.296915 IP localhost.8086 > localhost.44252: Flags [.], ack 318, win 88, options [nop,nop,TS val 1326659 ecr 1326659], length 0
20:16:38.301835 IP localhost.8086 > localhost.44252: Flags [P.], seq 1:193, ack 318, win 88, options [nop,nop,TS val 1326660 ecr 1326659], length 192
20:16:38.301849 IP localhost.44252 > localhost.8086: Flags [.], ack 193, win 88, options [nop,nop,TS val 1326660 ecr 1326660], length 0
20:16:38.301889 IP localhost.8086 > localhost.44252: Flags [F.], seq 193, ack 318, win 88, options [nop,nop,TS val 1326660 ecr 1326660], length 0
20:16:38.302299 IP localhost.44252 > localhost.8086: Flags [F.], seq 318, ack 194, win 88, options [nop,nop,TS val 1326661 ecr 1326660], length 0
20:16:38.302318 IP localhost.8086 > localhost.44252: Flags [.], ack 319, win 88, options [nop,nop,TS val 1326661 ecr 1326661], length 0

Well, as I understand here both sides sent FIN packet to close any conversations between them.

But in few seconds after this log messages I still see TIME_WAIT connections on my system:

sudo netstat -apnt|grep 44252
tcp        0      0 127.0.0.1:8086          127.0.0.1:44252         TIME_WAIT   -

And that is incorrect behaviour. If we closed connection from both sides it should disappear immediately from netstat.

Could you confirm that InfluxDB supports "Connection: close" from client side properly?

Thank you for a response!

Activity

pavel-odintsov

pavel-odintsov commented on Jun 25, 2017

@pavel-odintsov
pavel-odintsov
on Jun 25, 2017
Author

Also, I tried to use http 1.0 which does not have any pipelining at all and should not wait for consequent connections:

curl -d "networks_traffic,network=185.86.148.0/22 bits_incoming=0,bits_outgoing=0,packets_incoming=0,packets_outgoing=0 1498413453000000000" -v "http://127.0.0.1:8086/write?db=fastnetmon" -sL -H 'Connection: close' --http1.0
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 8086 (#0)
> POST /write?db=fastnetmon HTTP/1.0
> User-Agent: curl/7.35.0
> Host: 127.0.0.1:8086
> Accept: */*
> Connection: close
> Content-Length: 130
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 130 out of 130 bytes
* HTTP 1.0, assume close after body
< HTTP/1.0 204 No Content
< Content-Type: application/json
< Request-Id: b0b238f1-59dd-11e7-b38a-000000000000
< X-Influxdb-Version: 1.2.4
< Date: Sun, 25 Jun 2017 19:37:14 GMT
< 
* Closing connection 0

tcpdump:

20:37:14.520053 IP localhost.50683 > localhost.8086: Flags [S], seq 559913449, win 43690, options [mss 65495,sackOK,TS val 1635715 ecr 0,nop,wscale 9], length 0
20:37:14.520073 IP localhost.8086 > localhost.50683: Flags [S.], seq 1472424819, ack 559913450, win 43690, options [mss 65495,sackOK,TS val 1635715 ecr 1635715,nop,wscale 9], length 0
20:37:14.520092 IP localhost.50683 > localhost.8086: Flags [.], ack 1, win 86, options [nop,nop,TS val 1635715 ecr 1635715], length 0
20:37:14.520218 IP localhost.50683 > localhost.8086: Flags [P.], seq 1:318, ack 1, win 86, options [nop,nop,TS val 1635715 ecr 1635715], length 317
20:37:14.525446 IP localhost.8086 > localhost.50683: Flags [P.], seq 1:174, ack 318, win 88, options [nop,nop,TS val 1635716 ecr 1635715], length 173
20:37:14.525459 IP localhost.50683 > localhost.8086: Flags [.], ack 174, win 88, options [nop,nop,TS val 1635716 ecr 1635716], length 0
20:37:14.525503 IP localhost.8086 > localhost.50683: Flags [F.], seq 174, ack 318, win 88, options [nop,nop,TS val 1635716 ecr 1635716], length 0
20:37:14.525806 IP localhost.50683 > localhost.8086: Flags [F.], seq 318, ack 175, win 88, options [nop,nop,TS val 1635716 ecr 1635716], length 0
20:37:14.525824 IP localhost.8086 > localhost.50683: Flags [.], ack 319, win 88, options [nop,nop,TS val 1635716 ecr 1635716], length 0

Still have TIME_WAIT:

sudo netstat -apnt|grep 50683
tcp        0      0 127.0.0.1:8086          127.0.0.1:50683         TIME_WAIT   -
lwhile

lwhile commented on Jun 26, 2017

@lwhile
lwhile
on Jun 26, 2017 · edited by lwhile

I have a same problem. My client query data from influxdb every 2s, and I found influxdb take up 6k+ tcp connections which in TIME_WAIT status

jsternberg

jsternberg commented on Jun 29, 2017

@jsternberg
jsternberg
on Jun 29, 2017
Contributor

I don't think there's anything we can do about this from the looks of it: https://serverfault.com/questions/478691/avoid-time-wait-connections

For writes, you should try to pool connections or use something like UDP. But, UDP won't give you a guarantee that your data actually reaches the server. For queries (which automatically close the connection each time), we likely need to start using HTTP/2.

pavel-odintsov

pavel-odintsov commented on Jun 30, 2017

@pavel-odintsov
pavel-odintsov
on Jun 30, 2017
Author

Yes, I could not use UDP because I use pretty big batches of data :(

lwhile

lwhile commented on Jul 3, 2017

@lwhile
lwhile
on Jul 3, 2017 · edited by lwhile

@jsternberg
Does influxdb has a method to reuse connection now? In the other word I want influxdb don't add CLOSE to a http connection header .My server is openning too many connection now which in TIME_WAIT status

https://github.com/influxdata/influxdb/blob/master/services/httpd/handler.go#L429

jsternberg

jsternberg commented on Jul 3, 2017

@jsternberg
jsternberg
on Jul 3, 2017
Contributor

For /write, yes. The same way you could with any other HTTP connection. For /query, no. Unfortunately, the portion fo code that detects if a connection has been abandoned by the client also makes it impossible to reuse the connection.

lwhile

lwhile commented on Jul 4, 2017

@lwhile
lwhile
on Jul 4, 2017 · edited by lwhile

@jsternberg
But Influxdb always add connection-token "close" to header when querys were finished.It cause every query connection be abandoned although I did not closed it in client.

https://github.com/influxdata/influxdb/blob/master/services/httpd/handler.go#L429

jsternberg

jsternberg commented on Jul 4, 2017

@jsternberg
jsternberg
on Jul 4, 2017
Contributor

Yes. I'm looking into this to see if anything has changed since I originally added that code and there's a possibility we can remove the Connection: close being set on /query, but that's preliminary until I determine how CloseNotify() works when using pipelining. The docs state you're not supposed to and the behavior is a bit undefined.

jsternberg
mentioned this on Jul 4, 2017
jsternberg
self-assigned this
on Jul 4, 2017
jsternberg
added
review
on Jul 4, 2017
jsternberg
closed this as completedin #8561on Jul 21, 2017
ghost removed
review
on Jul 21, 2017
lwhile
mentioned this on Aug 9, 2017
GatewayJ

GatewayJ commented on Jul 29, 2019

@GatewayJ
GatewayJ
on Jul 29, 2019 · edited by GatewayJ

Hasn't the problem been solved? Or is there an alternative

1 remaining item

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Development

    Participants

    @jsternberg@pavel-odintsov@lwhile@GatewayJ

    Issue actions
      Issue with "Connection: close" set by client · Issue #8525 · influxdata/influxdb