[alibaba/tengine]proxy_request_buffering的疑问

按文档上说

当设置成off时，每当tengine接收到大于client_body_postpone_size的数据时，就发送这部分数据到后端服务器。

情况一但是我在抓http code为400的包时，即客户端已经发了header，但是未发送body，(header里有content_length,但是客户端没发送body)，但是它的大小也没有大于client_body_postpone_size，但是它还是往后端发送数据。配置为

  client_header_buffer_size 4k;
    client_body_buffer_size 4m;
    client_max_body_size 4m;
    large_client_header_buffers 4 32k;
    proxy_request_buffering off;
    client_body_postpone_size 4m;
    client_body_buffers 128 32k;
    sendfile        on;
    tcp_nopush     on;
    tcp_nodelay    on;

抓包文件 upstream-receive.pcap为接收客户端抓包文件，upstream-proxypass.pcap为转发抓包文件 https://pan.baidu.com/s/1hs2vceo

情况二转发到后端的nginx的http 400包，没有设置fastcgi_request_buffering，按文档上应该是，转发到FastCGI，但是并没有把数据发送给FastCGI，这是为什么呢配置为

  client_header_buffer_size 4k;
    client_body_buffer_size 4m;
    client_max_body_size 4m;
    large_client_header_buffers 4 32k;
    proxy_request_buffering off;
    client_body_postpone_size 4m;
    client_body_buffers 128 32k;

    fastcgi_buffer_size 128k;
    fastcgi_buffers 8 128k;
    fastcgi_busy_buffers_size 256k;
    fastcgi_temp_file_write_size 256k;
    fastcgi_keep_conn on;
    fastcgi_connect_timeout 10s;
    fastcgi_read_timeout 15s;
    fastcgi_send_timeout 10s;
    fastcgi_request_buffering未配置

    sendfile        on;
    tcp_nopush     on;
    tcp_nodelay    on;

抓包文件 https://pan.baidu.com/s/1skFt37B

两个抓包时间是不一样

wu0407

情况一但是我在抓http code为400的包时

我并没有在抓包文件里面看到有400的响应？

并且我也没看到服务器有给客户端发啥响应啊？

不太理解你的抓包文件描述的场景...

aholic

不好意思抓包漏了一个，情况一，客户端连接反向代理服务器，发送了header，但是在客户端一直没发body，反向代理直接把header发往后端。由于后端的tengine，没有往fastcgi，发任何包，所以客户端没有收到任何回应。更新链接，包含去后端请求的抓包 https://pan.baidu.com/s/1hs2vceo

wu0407

client----->tengine------>ups

你的意思是抓包发现，client给tengine发了header，还没发body的时候，tengine已经开始给ups发送header了是吗？

能提供一个完整的能复现问题的最小配置吗？我使用如下配置没法复现

    server {
        listen 44231;
        location / {
            content_by_lua "ngx.print('44231')";
        }
    }

    server {
        listen 34512;
        server_name local_host;

        client_header_buffer_size 4k;
        client_body_buffer_size 4m;
        client_max_body_size 4m;
        large_client_header_buffers 4 32k;
        proxy_request_buffering off;
        client_body_postpone_size 4m;
        client_body_buffers 128 32k;
        sendfile       on;
        tcp_nopush     on;
        tcp_nodelay    on;

        location / {
            proxy_pass http://127.0.0.1:44231;
        }
    }

使用nc一行一行的发送http报文，在发送body之前，我停顿了几秒

➜ /home/ruochen.xuruochen/software/tengine > nc -v 127.0.0.1 34512
Connection to 127.0.0.1 34512 port [tcp/*] succeeded!
POST /1000m HTTP/1.1
Host: 127.0.0.1:34512
Content-Length: 10

xxxxxxxxxx
HTTP/1.1 200 OK
Server: Tengine/2.0.0
Date: Tue, 24 Jan 2017 14:14:16 GMT
Content-Type: application/octet-stream
Content-Length: 5
Connection: keep-alive

44231

抓包内容如下，可以看到，tengine是在收完了请求body才给upstream发送数据的。（client端口为41079，tengine接收请求的端口为34512，tengine向upstream发送数据时端口为33249，upstream接收请求的端口为44231）

22:14:00.261917 IP localhost.41079 > localhost.34512: Flags [P.], seq 1:22, ack 1, win 257, options [nop,nop,TS val 397743599 ecr 397739326], length 21
E..I..@.@.C..........w..N...])r......=.....
.......>POST /1000m HTTP/1.1

22:14:00.261943 IP localhost.34512 > localhost.41079: Flags [.], ack 22, win 256, options [nop,nop,TS val 397743599 ecr 397743599], length 0
E..4w.@.@..<...........w])r.N........(.....
........
22:14:04.655930 IP localhost.41079 > localhost.34512: Flags [P.], seq 22:44, ack 1, win 257, options [nop,nop,TS val 397747993 ecr 397743599], length 22
E..J..@.@.C..........w..N...])r......>.....
..'.....Host: 127.0.0.1:34512

22:14:04.655955 IP localhost.34512 > localhost.41079: Flags [.], ack 44, win 256, options [nop,nop,TS val 397747993 ecr 397747993], length 0
E..4w.@.@..;...........w])r.N........(.....
..'...'.
22:14:09.671922 IP localhost.41079 > localhost.34512: Flags [P.], seq 44:63, ack 1, win 257, options [nop,nop,TS val 397753009 ecr 397747993], length 19
E..G..@.@.C..........w..N...])r......;.....
..:...'.Content-Length: 10

22:14:09.671938 IP localhost.34512 > localhost.41079: Flags [.], ack 63, win 256, options [nop,nop,TS val 397753009 ecr 397753009], length 0
E..4w.@.@..:...........w])r.N........(.....
..:...:.
22:14:10.598937 IP localhost.41079 > localhost.34512: Flags [P.], seq 63:64, ack 1, win 257, options [nop,nop,TS val 397753936 ecr 397753009], length 1
E..5..@.@.C..........w..N...])r......).....
..>P..:.

22:14:10.598973 IP localhost.34512 > localhost.41079: Flags [.], ack 64, win 256, options [nop,nop,TS val 397753936 ecr 397753936], length 0
E..4w.@.@..9...........w])r.N........(.....
..>P..>P
22:14:16.079896 IP localhost.41079 > localhost.34512: Flags [P.], seq 64:75, ack 1, win 257, options [nop,nop,TS val 397759417 ecr 397753936], length 11
E..?..@.@.C#.........w..N...])r......3.....
..S...>Pxxxxxxxxxx

22:14:16.079920 IP localhost.34512 > localhost.41079: Flags [.], ack 75, win 256, options [nop,nop,TS val 397759417 ecr 397759417], length 0
E..4w.@.@..8...........w])r.N........(.....
..S...S.
22:14:16.080044 IP localhost.33249 > localhost.44231: Flags [S], seq 1229565053, win 32792, options [mss 16396,sackOK,TS val 397759417 ecr 0,nop,wscale 7], length 0
E..<..@.@.<.............II.}.........0....@....
..S.........
22:14:16.080064 IP localhost.44231 > localhost.33249: Flags [S.], seq 2813920726, ack 1229565054, win 32768, options [mss 16396,sackOK,TS val 397759417 ecr 397759417,nop,wscale 7], length 0
E..<..@.@.<.................II.~.....0....@....
..S...S.....
22:14:16.080076 IP localhost.33249 > localhost.44231: Flags [.], ack 1, win 257, options [nop,nop,TS val 397759417 ecr 397759417], length 0
E..4..@.@.<.............II.~.........(.....
..S...S.
22:14:16.080178 IP localhost.33249 > localhost.44231: Flags [P.], seq 1:97, ack 1, win 257, options [nop,nop,TS val 397759417 ecr 397759417], length 96
E.....@.@.<.............II.~...............
..S...S.POST /1000m HTTP/1.0
Host: 127.0.0.1:44231
Connection: close
Content-Length: 10

xxxxxxxxxx

aholic

我看你抓包的内容，好像过程是这样的啊：

1. client发送了请求头，告知tengine，接下来的body长度为8
2. client发送了FIN包，告知tengine他不会再发数据了
3. tengine觉得“我擦，这哥们还有8字节的body咋不发了，行吧，那就断了吧。而且body没收够，给你个BAD_REQUEST吧”

代码片段如下：

---> ngx_http_proxy_handler
    ----> ngx_http_read_client_request_body
        ----> ngx_http_do_read_client_request_body

                  n = c->recv(c, rb->buf->last, size);

                  .....

                  if (n == 0 || n == NGX_ERROR) {
                      c->error = 1;
                      return NGX_HTTP_BAD_REQUEST;
                  }

从代码来看，此时tengine是不会继续往ups发送请求的。我测试如下，使用nc发送http报文头，但是不发送body直接断掉：

➜  /home/ruochen.xuruochen/software/tengine > nc -v 127.0.0.1 34512
Connection to 127.0.0.1 34512 port [tcp/*] succeeded!
POST / HTTP/1.1
Host: 127.0.0.1:34512
Content-Length: 10

^C

抓包情况如下，tengine确实没有往ups发数据：

➜  /home/ruochen.xuruochen/gitlab/tengine-cdn git:(master) ✗ > sudo tcpdump -ni any port 34512 or port 44231
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
00:04:46.940145 IP 127.0.0.1.60349 > 127.0.0.1.34512: Flags [S], seq 2897442226, win 32792, options [mss 16396,sackOK,TS val 404390277 ecr 0,nop,wscale 7], length 0
00:04:46.940171 IP 127.0.0.1.34512 > 127.0.0.1.60349: Flags [S.], seq 2922945086, ack 2897442227, win 32768, options [mss 16396,sackOK,TS val 404390277 ecr 404390277,nop,wscale 7], length 0
00:04:46.940185 IP 127.0.0.1.60349 > 127.0.0.1.34512: Flags [.], ack 1, win 257, options [nop,nop,TS val 404390277 ecr 404390277], length 0
00:04:52.046970 IP 127.0.0.1.60349 > 127.0.0.1.34512: Flags [P.], seq 1:17, ack 1, win 257, options [nop,nop,TS val 404395384 ecr 404390277], length 16
00:04:52.047017 IP 127.0.0.1.34512 > 127.0.0.1.60349: Flags [.], ack 17, win 256, options [nop,nop,TS val 404395384 ecr 404395384], length 0
00:04:55.224831 IP 127.0.0.1.60349 > 127.0.0.1.34512: Flags [P.], seq 17:39, ack 1, win 257, options [nop,nop,TS val 404398561 ecr 404395384], length 22
00:04:55.224852 IP 127.0.0.1.34512 > 127.0.0.1.60349: Flags [.], ack 39, win 256, options [nop,nop,TS val 404398561 ecr 404398561], length 0
00:05:02.258919 IP 127.0.0.1.60349 > 127.0.0.1.34512: Flags [P.], seq 39:58, ack 1, win 257, options [nop,nop,TS val 404405596 ecr 404398561], length 19
00:05:02.258950 IP 127.0.0.1.34512 > 127.0.0.1.60349: Flags [.], ack 58, win 256, options [nop,nop,TS val 404405596 ecr 404405596], length 0
00:05:02.988915 IP 127.0.0.1.60349 > 127.0.0.1.34512: Flags [P.], seq 58:59, ack 1, win 257, options [nop,nop,TS val 404406326 ecr 404405596], length 1
00:05:02.988934 IP 127.0.0.1.34512 > 127.0.0.1.60349: Flags [.], ack 59, win 256, options [nop,nop,TS val 404406326 ecr 404406326], length 0
00:05:04.003889 IP 127.0.0.1.60349 > 127.0.0.1.34512: Flags [P.], seq 59:60, ack 1, win 257, options [nop,nop,TS val 404407341 ecr 404406326], length 1
00:05:04.003911 IP 127.0.0.1.34512 > 127.0.0.1.60349: Flags [.], ack 60, win 256, options [nop,nop,TS val 404407341 ecr 404407341], length 0
00:05:04.754089 IP 127.0.0.1.60349 > 127.0.0.1.34512: Flags [F.], seq 60, ack 1, win 257, options [nop,nop,TS val 404408091 ecr 404407341], length 0
00:05:04.754339 IP 127.0.0.1.34512 > 127.0.0.1.60349: Flags [F.], seq 1, ack 61, win 256, options [nop,nop,TS val 404408091 ecr 404408091], length 0
00:05:04.754359 IP 127.0.0.1.60349 > 127.0.0.1.34512: Flags [.], ack 2, win 257, options [nop,nop,TS val 404408091 ecr 404408091], length 0
^C
16 packets captured
33 packets received by filter
0 packets dropped by kernel

所以你能否提供一个能复现你的问题的最小配置，已经请求，这样我好查下。

aholic

不好意思，过年没来回复，

   sendfile        on;
    tcp_nopush     on;
    tcp_nodelay    on;

    #keepalive_timeout  0;
    keepalive_timeout  60;
    keepalive_requests 200;

    ####gzip###########
    gzip  on;
    gzip_disable "msie6";
    # gzip_static on;
    gzip_proxied any;
    gzip_min_length 1000;
    gzip_comp_level 6;
    gzip_types text/plain text/css application/json application/javascript application/x-javascript text/xml application/xml application/xml+rss text/javascript;
    gzip_buffers 64 4k;
    gzip_http_version 1.0;
    gzip_vary on;

    client_header_timeout 10;
    client_body_timeout 10;
    client_header_buffer_size 4k;
    client_body_buffer_size 4m;
    client_max_body_size 4m;
    large_client_header_buffers 4 32k;
    proxy_request_buffering off;
    client_body_postpone_size 4m;
    client_body_buffers 128 32k;

    reset_timedout_connection on;
    send_timeout 20s;

    #set default charset
    charset UTF-8;

    #http proxy
    proxy_connect_timeout 10;
    proxy_read_timeout 10;
    proxy_send_timeout 5;
    proxy_buffer_size 8k;
    proxy_buffers 128 32k;
    proxy_busy_buffers_size 512k;
    #proxy_temp_file_write_size 128k;
    proxy_http_version 1.1;
    proxy_upstream_tries 2;
    proxy_next_upstream http_500 http_502 http_504 error timeout invalid_header;
    proxy_cache_use_stale http_500 http_502 http_504 error timeout invalid_header updating;
    proxy_ignore_headers Expires Cache-Control Set-Cookie X-Accel-Expires;
    proxy_cache_lock on;
    proxy_cache_lock_age 5s;
    proxy_cache_lock_timeout 5s;
    proxy_set_header Connection "";

    location / {
        proxy_pass http://live_backend;
    }

upstream live_backend {
    server 10.2.95.84:80 max_fails=0;
    server 10.2.95.87:80 max_fails=0;
    keepalive 100;
    keepalive_timeout 60s;
    check interval=10000 fall=2 rise=3 timeout=5000 default_down=false type=tcp;
}

wu0407

@aholic 基本配置都列出来了，使用tengine2.2.0

wu0407

@wu0407 好的。。。下周看下。。。最近比较忙。。:-)

aholic

thanks

wu0407

@aholic 我用curl http://url -H 'Content-Length: 12' 回车，然后ctrl+c，验证这个是2.2.0合并官方1.8.1产生的bug，tengine 2.2.0中只要设置proxy_request_buffering off;就会有这种现象

wu0407

官方版本1.8.1，设置proxy_request_buffering off;就，用curl http://url -H 'Content-Length: 12' 回车，直接返回400，并没有往upstream发送请求

wu0407

@yaoweibin 这个是已知的bug

wu0407

[alibaba/tengine]proxy_request_buffering的疑问

回答