青岛商业网站建设,基础软件开发平台,网站开发要多久,网站域名301是什么意思golang http transport源码分析
前言
Golang http库在日常开发中使用会很多。这里通过一个demo例子出发#xff0c;从源码角度梳理golang http库底层的数据结构以及大致的调用流程
例子
package mainimport (fmtnet/httpnet/url从源码角度梳理golang http库底层的数据结构以及大致的调用流程
例子
package mainimport (fmtnet/httpnet/urltimenet
)func send_http_request(addr string, port int) error {client : http.Client{Transport: http.Transport{Proxy: http.ProxyFromEnvironment,DialContext: (net.Dialer{Timeout: 30 * time.Second,KeepAlive: 30 * time.Second,DualStack: true,}).DialContext,MaxIdleConns: 100,MaxIdleConnsPerHost: 100,IdleConnTimeout: 90 * time.Second,},}// construct encoded endpointUrl, err : url.Parse(fmt.Sprintf(http://%s:%d, addr, port))if err ! nil {return err}Url.Path /indexendpoint : Url.String()req, err : http.NewRequest(GET, endpoint, nil)if err ! nil {return err}// use httpClient to send requestrsp, err : client.Do(req)if err ! nil {return err}// close the connection to reuse itdefer rsp.Body.Close()// check status codeif rsp.StatusCode ! http.StatusOK {return fmt.Errorf(get rsp error: %v, rsp)}return err
}func main() {send_http_request(xxx, 8080)
}源码分析
先看看http.Client结构体如下
// A Client is an HTTP client. Its zero value (DefaultClient) is a
// usable client that uses DefaultTransport.
//
// The Clients Transport typically has internal state (cached TCP
// connections), so Clients should be reused instead of created as
// needed. Clients are safe for concurrent use by multiple goroutines.
//
// A Client is higher-level than a RoundTripper (such as Transport)
// and additionally handles HTTP details such as cookies and
// redirects.
//
// When following redirects, the Client will forward all headers set on the
// initial Request except:
//
// • when forwarding sensitive headers like Authorization,
// WWW-Authenticate, and Cookie to untrusted targets.
// These headers will be ignored when following a redirect to a domain
// that is not a subdomain match or exact match of the initial domain.
// For example, a redirect from foo.com to either foo.com or sub.foo.com
// will forward the sensitive headers, but a redirect to bar.com will not.
//
// • when forwarding the Cookie header with a non-nil cookie Jar.
// Since each redirect may mutate the state of the cookie jar,
// a redirect may possibly alter a cookie set in the initial request.
// When forwarding the Cookie header, any mutated cookies will be omitted,
// with the expectation that the Jar will insert those mutated cookies
// with the updated values (assuming the origin matches).
// If Jar is nil, the initial cookies are forwarded without change.
//
type Client struct {// Transport specifies the mechanism by which individual// HTTP requests are made.// If nil, DefaultTransport is used.Transport RoundTripper// CheckRedirect specifies the policy for handling redirects.// If CheckRedirect is not nil, the client calls it before// following an HTTP redirect. The arguments req and via are// the upcoming request and the requests made already, oldest// first. If CheckRedirect returns an error, the Clients Get// method returns both the previous Response (with its Body// closed) and CheckRedirects error (wrapped in a url.Error)// instead of issuing the Request req.// As a special case, if CheckRedirect returns ErrUseLastResponse,// then the most recent response is returned with its body// unclosed, along with a nil error.//// If CheckRedirect is nil, the Client uses its default policy,// which is to stop after 10 consecutive requests.CheckRedirect func(req *Request, via []*Request) error// Jar specifies the cookie jar.//// The Jar is used to insert relevant cookies into every// outbound Request and is updated with the cookie values// of every inbound Response. The Jar is consulted for every// redirect that the Client follows.//// If Jar is nil, cookies are only sent if they are explicitly// set on the Request.Jar CookieJar// Timeout specifies a time limit for requests made by this// Client. The timeout includes connection time, any// redirects, and reading the response body. The timer remains// running after Get, Head, Post, or Do return and will// interrupt reading of the Response.Body.//// A Timeout of zero means no timeout.//// The Client cancels requests to the underlying Transport// as if the Requests Context ended.//// For compatibility, the Client will also use the deprecated// CancelRequest method on Transport if found. New// RoundTripper implementations should use the Requests Context// for cancellation instead of implementing CancelRequest.Timeout time.Duration
}Transporthttp client实际发送请求结构体Timeouthttp client请求超时设置(后续在详细分析)
在看Transport结构体(net/http/client.go)
// RoundTripper is an interface representing the ability to execute a
// single HTTP transaction, obtaining the Response for a given Request.
//
// A RoundTripper must be safe for concurrent use by multiple
// goroutines.
type RoundTripper interface {// RoundTrip executes a single HTTP transaction, returning// a Response for the provided Request.//// RoundTrip should not attempt to interpret the response. In// particular, RoundTrip must return err nil if it obtained// a response, regardless of the responses HTTP status code.// A non-nil err should be reserved for failure to obtain a// response. Similarly, RoundTrip should not attempt to// handle higher-level protocol details such as redirects,// authentication, or cookies.//// RoundTrip should not modify the request, except for// consuming and closing the Requests Body. RoundTrip may// read fields of the request in a separate goroutine. Callers// should not mutate or reuse the request until the Responses// Body has been closed.//// RoundTrip must always close the body, including on errors,// but depending on the implementation may do so in a separate// goroutine even after RoundTrip returns. This means that// callers wanting to reuse the body for subsequent requests// must arrange to wait for the Close call before doing so.//// The Requests URL and Header fields must be initialized.RoundTrip(*Request) (*Response, error)
}从注释可以看到RoundTripper负责HTTP请求的建立发送接收HTTP应答以及关闭但是不应该对HTTP应答进行额外处理例如redirects, authentication, or cookies等上层协议细节。另外RoundTripper也是goroutines-safe的
其中RoundTrip方法的Request和Response参数如下(net/http/request.go)
// A Request represents an HTTP request received by a server
// or to be sent by a client.
//
// The field semantics differ slightly between client and server
// usage. In addition to the notes on the fields below, see the
// documentation for Request.Write and RoundTripper.
type Request struct {// Method specifies the HTTP method (GET, POST, PUT, etc.).// For client requests, an empty string means GET.//// Gos HTTP client does not support sending a request with// the CONNECT method. See the documentation on Transport for// details.Method string// URL specifies either the URI being requested (for server// requests) or the URL to access (for client requests).//// For server requests, the URL is parsed from the URI// supplied on the Request-Line as stored in RequestURI. For// most requests, fields other than Path and RawQuery will be// empty. (See RFC 7230, Section 5.3)//// For client requests, the URLs Host specifies the server to// connect to, while the Requests Host field optionally// specifies the Host header value to send in the HTTP// request.URL *url.URL...// Body is the requests body.//// For client requests, a nil body means the request has no// body, such as a GET request. The HTTP Clients Transport// is responsible for calling the Close method.//// For server requests, the Request Body is always non-nil// but will return EOF immediately when no body is present.// The Server will close the request body. The ServeHTTP// Handler does not need to.Body io.ReadCloser...
}...
// Response represents the response from an HTTP request.
//
// The Client and Transport return Responses from servers once
// the response headers have been received. The response body
// is streamed on demand as the Body field is read.
type Response struct {Status string // e.g. 200 OKStatusCode int // e.g. 200Proto string // e.g. HTTP/1.0ProtoMajor int // e.g. 1ProtoMinor int // e.g. 0...
}这里Request和Response就是我们熟悉的HTTP请求和应答结构体
这里我们看demo对Transport的使用如下(net/http/transport.go)
client : http.Client{Transport: http.Transport{Proxy: http.ProxyFromEnvironment,DialContext: (net.Dialer{Timeout: 30 * time.Second,KeepAlive: 30 * time.Second,DualStack: true,}).DialContext,MaxIdleConns: 100,MaxIdleConnsPerHost: 100,IdleConnTimeout: 90 * time.Second,},
}...
// Transport is an implementation of RoundTripper that supports HTTP,
// HTTPS, and HTTP proxies (for either HTTP or HTTPS with CONNECT).
//
// By default, Transport caches connections for future re-use.
// This may leave many open connections when accessing many hosts.
// This behavior can be managed using Transports CloseIdleConnections method
// and the MaxIdleConnsPerHost and DisableKeepAlives fields.
//
// Transports should be reused instead of created as needed.
// Transports are safe for concurrent use by multiple goroutines.
//
// A Transport is a low-level primitive for making HTTP and HTTPS requests.
// For high-level functionality, such as cookies and redirects, see Client.
//
// Transport uses HTTP/1.1 for HTTP URLs and either HTTP/1.1 or HTTP/2
// for HTTPS URLs, depending on whether the server supports HTTP/2,
// and how the Transport is configured. The DefaultTransport supports HTTP/2.
// To explicitly enable HTTP/2 on a transport, use golang.org/x/net/http2
// and call ConfigureTransport. See the package docs for more about HTTP/2.
//
// Responses with status codes in the 1xx range are either handled
// automatically (100 expect-continue) or ignored. The one
// exception is HTTP status code 101 (Switching Protocols), which is
// considered a terminal status and returned by RoundTrip. To see the
// ignored 1xx responses, use the httptrace trace packages
// ClientTrace.Got1xxResponse.
//
// Transport only retries a request upon encountering a network error
// if the request is idempotent and either has no body or has its
// Request.GetBody defined. HTTP requests are considered idempotent if
// they have HTTP methods GET, HEAD, OPTIONS, or TRACE; or if their
// Header map contains an Idempotency-Key or X-Idempotency-Key
// entry. If the idempotency key value is an zero-length slice, the
// request is treated as idempotent but the header is not sent on the
// wire.
type Transport struct {idleMu sync.MutexcloseIdle bool // user has requested to close all idle connsidleConn map[connectMethodKey][]*persistConn // most recently used at endidleConnWait map[connectMethodKey]wantConnQueue // waiting getConnsidleLRU connLRUreqMu sync.MutexreqCanceler map[*Request]func(error)altMu sync.Mutex // guards changing altProto onlyaltProto atomic.Value // of nil or map[string]RoundTripper, key is URI schemeconnsPerHostMu sync.MutexconnsPerHost map[connectMethodKey]intconnsPerHostWait map[connectMethodKey]wantConnQueue // waiting getConns// Proxy specifies a function to return a proxy for a given// Request. If the function returns a non-nil error, the// request is aborted with the provided error.//// The proxy type is determined by the URL scheme. http,// https, and socks5 are supported. If the scheme is empty,// http is assumed.//// If Proxy is nil or returns a nil *URL, no proxy is used.Proxy func(*Request) (*url.URL, error)// DialContext specifies the dial function for creating unencrypted TCP connections.// If DialContext is nil (and the deprecated Dial below is also nil),// then the transport dials using package net.//// DialContext runs concurrently with calls to RoundTrip.// A RoundTrip call that initiates a dial may end up using// a connection dialed previously when the earlier connection// becomes idle before the later DialContext completes.DialContext func(ctx context.Context, network, addr string) (net.Conn, error)// Dial specifies the dial function for creating unencrypted TCP connections.//// Dial runs concurrently with calls to RoundTrip.// A RoundTrip call that initiates a dial may end up using// a connection dialed previously when the earlier connection// becomes idle before the later Dial completes.//// Deprecated: Use DialContext instead, which allows the transport// to cancel dials as soon as they are no longer needed.// If both are set, DialContext takes priority.Dial func(network, addr string) (net.Conn, error)// DialTLS specifies an optional dial function for creating// TLS connections for non-proxied HTTPS requests.//// If DialTLS is nil, Dial and TLSClientConfig are used.//// If DialTLS is set, the Dial hook is not used for HTTPS// requests and the TLSClientConfig and TLSHandshakeTimeout// are ignored. The returned net.Conn is assumed to already be// past the TLS handshake.DialTLS func(network, addr string) (net.Conn, error)// TLSClientConfig specifies the TLS configuration to use with// tls.Client.// If nil, the default configuration is used.// If non-nil, HTTP/2 support may not be enabled by default.TLSClientConfig *tls.Config// TLSHandshakeTimeout specifies the maximum amount of time waiting to// wait for a TLS handshake. Zero means no timeout.TLSHandshakeTimeout time.Duration// DisableKeepAlives, if true, disables HTTP keep-alives and// will only use the connection to the server for a single// HTTP request.//// This is unrelated to the similarly named TCP keep-alives.DisableKeepAlives bool// DisableCompression, if true, prevents the Transport from// requesting compression with an Accept-Encoding: gzip// request header when the Request contains no existing// Accept-Encoding value. If the Transport requests gzip on// its own and gets a gzipped response, its transparently// decoded in the Response.Body. However, if the user// explicitly requested gzip it is not automatically// uncompressed.DisableCompression bool// MaxIdleConns controls the maximum number of idle (keep-alive)// connections across all hosts. Zero means no limit.MaxIdleConns int// MaxIdleConnsPerHost, if non-zero, controls the maximum idle// (keep-alive) connections to keep per-host. If zero,// DefaultMaxIdleConnsPerHost is used.MaxIdleConnsPerHost int// MaxConnsPerHost optionally limits the total number of// connections per host, including connections in the dialing,// active, and idle states. On limit violation, dials will block.//// Zero means no limit.MaxConnsPerHost int// IdleConnTimeout is the maximum amount of time an idle// (keep-alive) connection will remain idle before closing// itself.// Zero means no limit.IdleConnTimeout time.Duration// ResponseHeaderTimeout, if non-zero, specifies the amount of// time to wait for a servers response headers after fully// writing the request (including its body, if any). This// time does not include the time to read the response body.ResponseHeaderTimeout time.Duration// ExpectContinueTimeout, if non-zero, specifies the amount of// time to wait for a servers first response headers after fully// writing the request headers if the request has an// Expect: 100-continue header. Zero means no timeout and// causes the body to be sent immediately, without// waiting for the server to approve.// This time does not include the time to send the request header.ExpectContinueTimeout time.Duration// TLSNextProto specifies how the Transport switches to an// alternate protocol (such as HTTP/2) after a TLS NPN/ALPN// protocol negotiation. If Transport dials an TLS connection// with a non-empty protocol name and TLSNextProto contains a// map entry for that key (such as h2), then the func is// called with the requests authority (such as example.com// or example.com:1234) and the TLS connection. The function// must return a RoundTripper that then handles the request.// If TLSNextProto is not nil, HTTP/2 support is not enabled// automatically.TLSNextProto map[string]func(authority string, c *tls.Conn) RoundTripper// ProxyConnectHeader optionally specifies headers to send to// proxies during CONNECT requests.ProxyConnectHeader Header// MaxResponseHeaderBytes specifies a limit on how many// response bytes are allowed in the servers response// header.//// Zero means to use a default limit.MaxResponseHeaderBytes int64// WriteBufferSize specifies the size of the write buffer used// when writing to the transport.// If zero, a default (currently 4KB) is used.WriteBufferSize int// ReadBufferSize specifies the size of the read buffer used// when reading from the transport.// If zero, a default (currently 4KB) is used.ReadBufferSize int// nextProtoOnce guards initialization of TLSNextProto and// h2transport (via onceSetNextProtoDefaults)nextProtoOnce sync.Onceh2transport h2Transport // non-nil if http2 wired uptlsNextProtoWasNil bool // whether TLSNextProto was nil when the Once fired// ForceAttemptHTTP2 controls whether HTTP/2 is enabled when a non-zero// Dial, DialTLS, or DialContext func or TLSClientConfig is provided.// By default, use of any those fields conservatively disables HTTP/2.// To use a custom dialer or TLS config and still attempt HTTP/2// upgrades, set this to true.ForceAttemptHTTP2 bool
}http.Transport实现了http.RoundTripper支持HTTPHTTPS以及HTTP Proxy(for either HTTP or HTTPS with CONNECT)
http.Transport默认实现了TCP连接池会复用底层TCP连接。所以推荐的做法是初始化一次http.Transport然后重复使用
http Transport对HTTP URLs使用HTTP/1.1协议对HTTPS URLs使用HTTP/1.1 or HTTP/2具体使用哪种协议要取决于Transport的配置以及服务端是否支持HTTP/2协议
Transport默认支持HTTP/2如果要强制开启HTTP/2使用ConfigureTransport详细参考golang.org/x/net/http2
Transport只有在遇到网络故障的情况下会重试幂等的请求。另外http Transport实现了goroutine-safe
接下来我们分析一下Transport各字段
closeIdle关闭所有空闲的连接idleConn空闲连接池Proxy对特定的请求返回代理DialContext指定底层TCP连接的创建函数Dial(Deprecated)和DialContext一样的功能区别在于DialContext多了一个context.Context参数用于取消dials功能DialTLS为non-proxied HTTPS requests创建TLS connectionsTLSHandshakeTimeoutTLS handshake超时时间DisableKeepAlives禁止HTTP keep-alives一个连接只用于一次请求(注意区分TCP keep-alives。HTTP keep-alives用于连接复用TCP keep-alives用于连接保活)MaxIdleConns控制所有hosts的空闲连接最大数目(Zero means no limit.)MaxIdleConnsPerHost控制某个host的空闲连接(keep-alives)最大数目(设置为0则默认DefaultMaxIdleConnsPerHost2)MaxConnsPerHost控制某个host的所有连接包括创建中正在使用的以及空闲的连接(including connections in the dialing, active, and idle states)。一旦超过限制dial会阻塞(Zero means no limit)IdleConnTimeout空闲连接(keep-alives)超时时间h2transportHTTP/2协议对应的transportForceAttemptHTTP2当Dial, DialTLS, or DialContext func or TLSClientConfig提供时默认情况下会禁止HTTP/2协议。当使用自定义的这些配置时需要设置ForceAttemptHTTP2字段开启HTTP2
我们看一下DefaultTransport配置
// DefaultTransport is the default implementation of Transport and is
// used by DefaultClient. It establishes network connections as needed
// and caches them for reuse by subsequent calls. It uses HTTP proxies
// as directed by the $HTTP_PROXY and $NO_PROXY (or $http_proxy and
// $no_proxy) environment variables.
var DefaultTransport RoundTripper Transport{Proxy: ProxyFromEnvironment,DialContext: (net.Dialer{Timeout: 30 * time.Second,KeepAlive: 30 * time.Second,DualStack: true,}).DialContext,ForceAttemptHTTP2: true,MaxIdleConns: 100,IdleConnTimeout: 90 * time.Second,TLSHandshakeTimeout: 10 * time.Second,ExpectContinueTimeout: 1 * time.Second,
}重点看DialContext实例展开net.Dialer结构体(net/dial.go)
// A Dialer contains options for connecting to an address.
//
// The zero value for each field is equivalent to dialing
// without that option. Dialing with the zero value of Dialer
// is therefore equivalent to just calling the Dial function.
type Dialer struct {// Timeout is the maximum amount of time a dial will wait for// a connect to complete. If Deadline is also set, it may fail// earlier.//// The default is no timeout.//// When using TCP and dialing a host name with multiple IP// addresses, the timeout may be divided between them.//// With or without a timeout, the operating system may impose// its own earlier timeout. For instance, TCP timeouts are// often around 3 minutes.Timeout time.Duration// Deadline is the absolute point in time after which dials// will fail. If Timeout is set, it may fail earlier.// Zero means no deadline, or dependent on the operating system// as with the Timeout option.Deadline time.Time// LocalAddr is the local address to use when dialing an// address. The address must be of a compatible type for the// network being dialed.// If nil, a local address is automatically chosen.LocalAddr Addr// DualStack previously enabled RFC 6555 Fast Fallback// support, also known as Happy Eyeballs, in which IPv4 is// tried soon if IPv6 appears to be misconfigured and// hanging.//// Deprecated: Fast Fallback is enabled by default. To// disable, set FallbackDelay to a negative value.DualStack bool// FallbackDelay specifies the length of time to wait before// spawning a RFC 6555 Fast Fallback connection. That is, this// is the amount of time to wait for IPv6 to succeed before// assuming that IPv6 is misconfigured and falling back to// IPv4.//// If zero, a default delay of 300ms is used.// A negative value disables Fast Fallback support.FallbackDelay time.Duration// KeepAlive specifies the interval between keep-alive// probes for an active network connection.// If zero, keep-alive probes are sent with a default value// (currently 15 seconds), if supported by the protocol and operating// system. Network protocols or operating systems that do// not support keep-alives ignore this field.// If negative, keep-alive probes are disabled.KeepAlive time.Duration// Resolver optionally specifies an alternate resolver to use.Resolver *Resolver// Cancel is an optional channel whose closure indicates that// the dial should be canceled. Not all types of dials support// cancellation.//// Deprecated: Use DialContext instead.Cancel -chan struct{}// If Control is not nil, it is called after creating the network// connection but before actually dialing.//// Network and address parameters passed to Control method are not// necessarily the ones passed to Dial. For example, passing tcp to Dial// will cause the Control function to be called with tcp4 or tcp6.Control func(network, address string, c syscall.RawConn) error
}net.Dialer包含了创建TCP连接的各种选项
TimeoutTCP连接建立的超时时间(也即三次握手的超时时间)操作系统的超时时间一般为3 minutesDeadline与Timeout作用类似只不过限制了确定的超时时刻LocalAddr本地地址TCP四元组的原始IP地址DualStack(Deprecated)enabled RFC 6555 Fast Fallback FeatureFallbackDelayIPv6连接建立的等待时间如果超时则会切换到IPv4(A negative value disables Fast Fallback support.)KeepAlive设置了活跃连接的TCP keep-alive探针间隔需要协议层以及操作系统支持(If zero, keep-alive probes are sent with a default value(currently 15 seconds))Controlit is called after creating the network connection but before actually dialing
接下来我们分析net.Dialer.DialContext如下
// Dial connects to the address on the named network.
//
// Known networks are tcp, tcp4 (IPv4-only), tcp6 (IPv6-only),
// udp, udp4 (IPv4-only), udp6 (IPv6-only), ip, ip4
// (IPv4-only), ip6 (IPv6-only), unix, unixgram and
// unixpacket.
//
// For TCP and UDP networks, the address has the form host:port.
// The host must be a literal IP address, or a host name that can be
// resolved to IP addresses.
// The port must be a literal port number or a service name.
// If the host is a literal IPv6 address it must be enclosed in square
// brackets, as in [2001:db8::1]:80 or [fe80::1%zone]:80.
// The zone specifies the scope of the literal IPv6 address as defined
// in RFC 4007.
// The functions JoinHostPort and SplitHostPort manipulate a pair of
// host and port in this form.
// When using TCP, and the host resolves to multiple IP addresses,
// Dial will try each IP address in order until one succeeds.
//
// Examples:
// Dial(tcp, golang.org:http)
// Dial(tcp, 192.0.2.1:http)
// Dial(tcp, 198.51.100.1:80)
// Dial(udp, [2001:db8::1]:domain)
// Dial(udp, [fe80::1%lo0]:53)
// Dial(tcp, :80)
//
// For IP networks, the network must be ip, ip4 or ip6 followed
// by a colon and a literal protocol number or a protocol name, and
// the address has the form host. The host must be a literal IP
// address or a literal IPv6 address with zone.
// It depends on each operating system how the operating system
// behaves with a non-well known protocol number such as 0 or 255.
//
// Examples:
// Dial(ip4:1, 192.0.2.1)
// Dial(ip6:ipv6-icmp, 2001:db8::1)
// Dial(ip6:58, fe80::1%lo0)
//
// For TCP, UDP and IP networks, if the host is empty or a literal
// unspecified IP address, as in :80, 0.0.0.0:80 or [::]:80 for
// TCP and UDP, , 0.0.0.0 or :: for IP, the local system is
// assumed.
//
// For Unix networks, the address must be a file system path.
func Dial(network, address string) (Conn, error) {var d Dialerreturn d.Dial(network, address)
}...// DialContext connects to the address on the named network using
// the provided context.
//
// The provided Context must be non-nil. If the context expires before
// the connection is complete, an error is returned. Once successfully
// connected, any expiration of the context will not affect the
// connection.
//
// When using TCP, and the host in the address parameter resolves to multiple
// network addresses, any dial timeout (from d.Timeout or ctx) is spread
// over each consecutive dial, such that each is given an appropriate
// fraction of the time to connect.
// For example, if a host has 4 IP addresses and the timeout is 1 minute,
// the connect to each single address will be given 15 seconds to complete
// before trying the next one.
//
// See func Dial for a description of the network and address
// parameters.
func (d *Dialer) DialContext(ctx context.Context, network, address string) (Conn, error) {if ctx nil {panic(nil context)}deadline : d.deadline(ctx, time.Now())if !deadline.IsZero() {if d, ok : ctx.Deadline(); !ok || deadline.Before(d) {subCtx, cancel : context.WithDeadline(ctx, deadline)defer cancel()ctx subCtx}}if oldCancel : d.Cancel; oldCancel ! nil {subCtx, cancel : context.WithCancel(ctx)defer cancel()go func() {select {case -oldCancel:cancel()case -subCtx.Done():}}()ctx subCtx}// Shadow the nettrace (if any) during resolve so Connect events dont fire for DNS lookups.resolveCtx : ctxif trace, _ : ctx.Value(nettrace.TraceKey{}).(*nettrace.Trace); trace ! nil {shadow : *traceshadow.ConnectStart nilshadow.ConnectDone nilresolveCtx context.WithValue(resolveCtx, nettrace.TraceKey{}, shadow)}addrs, err : d.resolver().resolveAddrList(resolveCtx, dial, network, address, d.LocalAddr)if err ! nil {return nil, OpError{Op: dial, Net: network, Source: nil, Addr: nil, Err: err}}sd : sysDialer{Dialer: *d,network: network,address: address,}var primaries, fallbacks addrListif d.dualStack() network tcp {primaries, fallbacks addrs.partition(isIPv4)} else {primaries addrs}var c Connif len(fallbacks) 0 {c, err sd.dialParallel(ctx, primaries, fallbacks)} else {c, err sd.dialSerial(ctx, primaries)}if err ! nil {return nil, err}if tc, ok : c.(*TCPConn); ok d.KeepAlive 0 {setKeepAlive(tc.fd, true)ka : d.KeepAliveif d.KeepAlive 0 {ka defaultTCPKeepAlive}setKeepAlivePeriod(tc.fd, ka)testHookSetKeepAlive(ka)}return c, nil
}DialContext用于创建指定网络协议(例如tcpudp)以及指定地址的连接例如
// Examples:
// Dial(tcp, golang.org:http)
// Dial(tcp, 192.0.2.1:http)
// Dial(tcp, 198.51.100.1:80)
// Dial(udp, [2001:db8::1]:domain)
// Dial(udp, [fe80::1%lo0]:53)
// Dial(tcp, :80)分为三个阶段如下
1、resolveAddrList
// resolveAddrList resolves addr using hint and returns a list of
// addresses. The result contains at least one address when error is
// nil.
func (r *Resolver) resolveAddrList(ctx context.Context, op, network, addr string, hint Addr) (addrList, error) {afnet, _, err : parseNetwork(ctx, network, true)if err ! nil {return nil, err}if op dial addr {return nil, errMissingAddress}switch afnet {case unix, unixgram, unixpacket:addr, err : ResolveUnixAddr(afnet, addr)if err ! nil {return nil, err}if op dial hint ! nil addr.Network() ! hint.Network() {return nil, AddrError{Err: mismatched local address type, Addr: hint.String()}}return addrList{addr}, nil}addrs, err : r.internetAddrList(ctx, afnet, addr)if err ! nil || op ! dial || hint nil {return addrs, err}var (tcp *TCPAddrudp *UDPAddrip *IPAddrwildcard bool)switch hint : hint.(type) {case *TCPAddr:tcp hintwildcard tcp.isWildcard()case *UDPAddr:udp hintwildcard udp.isWildcard()case *IPAddr:ip hintwildcard ip.isWildcard()}naddrs : addrs[:0]for _, addr : range addrs {if addr.Network() ! hint.Network() {return nil, AddrError{Err: mismatched local address type, Addr: hint.String()}}switch addr : addr.(type) {case *TCPAddr:if !wildcard !addr.isWildcard() !addr.IP.matchAddrFamily(tcp.IP) {continue}naddrs append(naddrs, addr)case *UDPAddr:if !wildcard !addr.isWildcard() !addr.IP.matchAddrFamily(udp.IP) {continue}naddrs append(naddrs, addr)case *IPAddr:if !wildcard !addr.isWildcard() !addr.IP.matchAddrFamily(ip.IP) {continue}naddrs append(naddrs, addr)}}if len(naddrs) 0 {return nil, AddrError{Err: errNoSuitableAddress.Error(), Addr: hint.String()}}return naddrs, nil
}// An addrList represents a list of network endpoint addresses.
type addrList []Addr// Addr represents a network end point address.
//
// The two methods Network and String conventionally return strings
// that can be passed as the arguments to Dial, but the exact form
// and meaning of the strings is up to the implementation.
type Addr interface {Network() string // name of the network (for example, tcp, udp)String() string // string form of address (for example, 192.0.2.1:25, [2001:db8::1]:80)
}根据本地地址网路类型以及地址族分拆目标地址返回地址列表
2、dialSerial
// isIPv4 reports whether addr contains an IPv4 address.
func isIPv4(addr Addr) bool {switch addr : addr.(type) {case *TCPAddr:return addr.IP.To4() ! nilcase *UDPAddr:return addr.IP.To4() ! nilcase *IPAddr:return addr.IP.To4() ! nil}return false
}// partition divides an address list into two categories, using a
// strategy function to assign a boolean label to each address.
// The first address, and any with a matching label, are returned as
// primaries, while addresses with the opposite label are returned
// as fallbacks. For non-empty inputs, primaries is guaranteed to be
// non-empty.
func (addrs addrList) partition(strategy func(Addr) bool) (primaries, fallbacks addrList) {var primaryLabel boolfor i, addr : range addrs {label : strategy(addr)if i 0 || label primaryLabel {primaryLabel labelprimaries append(primaries, addr)} else {fallbacks append(fallbacks, addr)}}return
}...
sd : sysDialer{Dialer: *d,network: network,address: address,
}...
// dialSerial connects to a list of addresses in sequence, returning
// either the first successful connection, or the first error.
func (sd *sysDialer) dialSerial(ctx context.Context, ras addrList) (Conn, error) {var firstErr error // The error from the first address is most relevant.for i, ra : range ras {select {case -ctx.Done():return nil, OpError{Op: dial, Net: sd.network, Source: sd.LocalAddr, Addr: ra, Err: mapErr(ctx.Err())}default:}deadline, _ : ctx.Deadline()partialDeadline, err : partialDeadline(time.Now(), deadline, len(ras)-i)if err ! nil {// Ran out of time.if firstErr nil {firstErr OpError{Op: dial, Net: sd.network, Source: sd.LocalAddr, Addr: ra, Err: err}}break}dialCtx : ctxif partialDeadline.Before(deadline) {var cancel context.CancelFuncdialCtx, cancel context.WithDeadline(ctx, partialDeadline)defer cancel()}c, err : sd.dialSingle(dialCtx, ra)if err nil {return c, nil}if firstErr nil {firstErr err}}if firstErr nil {firstErr OpError{Op: dial, Net: sd.network, Source: nil, Addr: nil, Err: errMissingAddress}}return nil, firstErr
}// dialSingle attempts to establish and returns a single connection to
// the destination address.
func (sd *sysDialer) dialSingle(ctx context.Context, ra Addr) (c Conn, err error) {trace, _ : ctx.Value(nettrace.TraceKey{}).(*nettrace.Trace)if trace ! nil {raStr : ra.String()if trace.ConnectStart ! nil {trace.ConnectStart(sd.network, raStr)}if trace.ConnectDone ! nil {defer func() { trace.ConnectDone(sd.network, raStr, err) }()}}la : sd.LocalAddrswitch ra : ra.(type) {case *TCPAddr:la, _ : la.(*TCPAddr)c, err sd.dialTCP(ctx, la, ra)case *UDPAddr:la, _ : la.(*UDPAddr)c, err sd.dialUDP(ctx, la, ra)case *IPAddr:la, _ : la.(*IPAddr)c, err sd.dialIP(ctx, la, ra)case *UnixAddr:la, _ : la.(*UnixAddr)c, err sd.dialUnix(ctx, la, ra)default:return nil, OpError{Op: dial, Net: sd.network, Source: la, Addr: ra, Err: AddrError{Err: unexpected address type, Addr: sd.address}}}if err ! nil {return nil, OpError{Op: dial, Net: sd.network, Source: la, Addr: ra, Err: err} // c is non-nil interface containing nil pointer}return c, nil
}func (sd *sysDialer) dialTCP(ctx context.Context, laddr, raddr *TCPAddr) (*TCPConn, error) {if testHookDialTCP ! nil {return testHookDialTCP(ctx, sd.network, laddr, raddr)}return sd.doDialTCP(ctx, laddr, raddr)
}// net/tcpsock_posix.go
func (sd *sysDialer) doDialTCP(ctx context.Context, laddr, raddr *TCPAddr) (*TCPConn, error) {fd, err : internetSocket(ctx, sd.network, laddr, raddr, syscall.SOCK_STREAM, 0, dial, sd.Dialer.Control)// TCP has a rarely used mechanism called a simultaneous connection in// which Dial(tcp, addr1, addr2) run on the machine at addr1 can// connect to a simultaneous Dial(tcp, addr2, addr1) run on the machine// at addr2, without either machine executing Listen. If laddr nil,// it means we want the kernel to pick an appropriate originating local// address. Some Linux kernels cycle blindly through a fixed range of// local ports, regardless of destination port. If a kernel happens to// pick local port 50001 as the source for a Dial(tcp, , localhost:50001),// then the Dial will succeed, having simultaneously connected to itself.// This can only happen when we are letting the kernel pick a port (laddr nil)// and when there is no listener for the destination address.// Its hard to argue this is anything other than a kernel bug. If we// see this happen, rather than expose the buggy effect to users, we// close the fd and try again. If it happens twice more, we relent and// use the result. See also:// https://golang.org/issue/2690// linux - How can you have a TCP connection back to the same port? - Stack Overflow//// The opposite can also happen: if we ask the kernel to pick an appropriate// originating local address, sometimes it picks one that is already in use.// So if the error is EADDRNOTAVAIL, we have to try again too, just for// a different reason.//// The kernel socket code is no doubt enjoying watching us squirm.for i : 0; i 2 (laddr nil || laddr.Port 0) (selfConnect(fd, err) || spuriousENOTAVAIL(err)); i {if err nil {fd.Close()}fd, err internetSocket(ctx, sd.network, laddr, raddr, syscall.SOCK_STREAM, 0, dial, sd.Dialer.Control)}if err ! nil {return nil, err}return newTCPConn(fd), nil
}func internetSocket(ctx context.Context, net string, laddr, raddr sockaddr, sotype, proto int, mode string, ctrlFn func(string, string, syscall.RawConn) error) (fd *netFD, err error) {if (runtime.GOOS aix || runtime.GOOS windows || runtime.GOOS openbsd || runtime.GOOS nacl) mode dial raddr.isWildcard() {raddr raddr.toLocal(net)}family, ipv6only : favoriteAddrFamily(net, laddr, raddr, mode)return socket(ctx, net, family, sotype, proto, ipv6only, laddr, raddr, ctrlFn)
}// socket returns a network file descriptor that is ready for
// asynchronous I/O using the network poller.
func socket(ctx context.Context, net string, family, sotype, proto int, ipv6only bool, laddr, raddr sockaddr, ctrlFn func(string, string, syscall.RawConn) error) (fd *netFD, err error) {s, err : sysSocket(family, sotype, proto)if err ! nil {return nil, err}if err setDefaultSockopts(s, family, sotype, ipv6only); err ! nil {poll.CloseFunc(s)return nil, err}if fd, err newFD(s, family, sotype, net); err ! nil {poll.CloseFunc(s)return nil, err}// This function makes a network file descriptor for the// following applications://// - An endpoint holder that opens a passive stream// connection, known as a stream listener//// - An endpoint holder that opens a destination-unspecific// datagram connection, known as a datagram listener//// - An endpoint holder that opens an active stream or a// destination-specific datagram connection, known as a// dialer//// - An endpoint holder that opens the other connection, such// as talking to the protocol stack inside the kernel//// For stream and datagram listeners, they will only require// named sockets, so we can assume that its just a request// from stream or datagram listeners when laddr is not nil but// raddr is nil. Otherwise we assume its just for dialers or// the other connection holders.if laddr ! nil raddr nil {switch sotype {case syscall.SOCK_STREAM, syscall.SOCK_SEQPACKET:if err : fd.listenStream(laddr, listenerBacklog(), ctrlFn); err ! nil {fd.Close()return nil, err}return fd, nilcase syscall.SOCK_DGRAM:if err : fd.listenDatagram(laddr, ctrlFn); err ! nil {fd.Close()return nil, err}return fd, nil}}if err : fd.dial(ctx, laddr, raddr, ctrlFn); err ! nil {fd.Close()return nil, err}return fd, nil
}func newFD(sysfd, family, sotype int, net string) (*netFD, error) {ret : netFD{pfd: poll.FD{Sysfd: sysfd,IsStream: sotype syscall.SOCK_STREAM,ZeroReadIsEOF: sotype ! syscall.SOCK_DGRAM sotype ! syscall.SOCK_RAW,},family: family,sotype: sotype,net: net,}return ret, nil
}// Network file descriptor.
type netFD struct {pfd poll.FD// immutable until Closefamily intsotype intisConnected bool // handshake completed or use of association with peernet stringladdr Addrraddr Addr
}func newTCPConn(fd *netFD) *TCPConn {c : TCPConn{conn{fd}}setNoDelay(c.fd, true)return c
}// TCPConn is an implementation of the Conn interface for TCP network
// connections.
type TCPConn struct {conn
}type conn struct {fd *netFD
}对resolveAddrList返回的地址依次尝试创建连接返回第一个创建成功的连接否则返回第一个错误
3、setKeepAlive
创建完连接后检查该连接是否为TCP连接如果是TCP连接则设置KeepAlive时间如下(net/sockopt_posix.go)
...
if tc, ok : c.(*TCPConn); ok d.KeepAlive 0 {setKeepAlive(tc.fd, true)ka : d.KeepAliveif d.KeepAlive 0 {ka defaultTCPKeepAlive}setKeepAlivePeriod(tc.fd, ka)testHookSetKeepAlive(ka)
}...
func setKeepAlive(fd *netFD, keepalive bool) error {err : fd.pfd.SetsockoptInt(syscall.SOL_SOCKET, syscall.SO_KEEPALIVE, boolint(keepalive))runtime.KeepAlive(fd)return wrapSyscallError(setsockopt, err)
}...
// net/tcpsockopt_darwin.go
func setKeepAlivePeriod(fd *netFD, d time.Duration) error {// The kernel expects seconds so round to next highest second.d (time.Second - time.Nanosecond)secs : int(d.Seconds())if err : fd.pfd.SetsockoptInt(syscall.IPPROTO_TCP, sysTCP_KEEPINTVL, secs); err ! nil {return wrapSyscallError(setsockopt, err)}err : fd.pfd.SetsockoptInt(syscall.IPPROTO_TCP, syscall.TCP_KEEPALIVE, secs)runtime.KeepAlive(fd)return wrapSyscallError(setsockopt, err)
}// defaultTCPKeepAlive is a default constant value for TCPKeepAlive times
// See golang.org/issue/31510
const (defaultTCPKeepAlive 15 * time.Second
)Transport.IdleConnTimeout与net.Dialer.KeepAlive有什么关系哪一个是所谓的HTTP keep-alives
回到最开始的http.Client的使用
// construct encoded endpoint
Url, err : url.Parse(fmt.Sprintf(http://%s:%d, addr, port))
if err ! nil {return err
}
Url.Path /index
endpoint : Url.String()
req, err : http.NewRequest(GET, endpoint, nil)
if err ! nil {return err
}
// use httpClient to send request
rsp, err : client.Do(req)
if err ! nil {return err
}
// close the connection to reuse it
defer rsp.Body.Close()
// check status code
if rsp.StatusCode ! http.StatusOK {return fmt.Errorf(get rsp error: %v, rsp)
}我们分析http.Client是如何使用http.Transport的
一、创建http.Request
根据methodurlbody创建http.Request
// NewRequest wraps NewRequestWithContext using the background context.
func NewRequest(method, url string, body io.Reader) (*Request, error) {return NewRequestWithContext(context.Background(), method, url, body)
}// NewRequestWithContext returns a new Request given a method, URL, and
// optional body.
//
// If the provided body is also an io.Closer, the returned
// Request.Body is set to body and will be closed by the Client
// methods Do, Post, and PostForm, and Transport.RoundTrip.
//
// NewRequestWithContext returns a Request suitable for use with
// Client.Do or Transport.RoundTrip. To create a request for use with
// testing a Server Handler, either use the NewRequest function in the
// net/http/httptest package, use ReadRequest, or manually update the
// Request fields. For an outgoing client request, the context
// controls the entire lifetime of a request and its response:
// obtaining a connection, sending the request, and reading the
// response headers and body. See the Request types documentation for
// the difference between inbound and outbound request fields.
//
// If body is of type *bytes.Buffer, *bytes.Reader, or
// *strings.Reader, the returned requests ContentLength is set to its
// exact value (instead of -1), GetBody is populated (so 307 and 308
// redirects can replay the body), and Body is set to NoBody if the
// ContentLength is 0.
func NewRequestWithContext(ctx context.Context, method, url string, body io.Reader) (*Request, error) {if method {// We document that means GET for Request.Method, and people have// relied on that from NewRequest, so keep that working.// We still enforce validMethod for non-empty methods.method GET}if !validMethod(method) {return nil, fmt.Errorf(net/http: invalid method %q, method)}if ctx nil {return nil, errors.New(net/http: nil Context)}u, err : parseURL(url) // Just url.Parse (url is shadowed for godoc).if err ! nil {return nil, err}rc, ok : body.(io.ReadCloser)if !ok body ! nil {rc ioutil.NopCloser(body)}// The hosts colon:port should be normalized. See Issue 14836.u.Host removeEmptyPort(u.Host)req : Request{ctx: ctx,Method: method,URL: u,Proto: HTTP/1.1,ProtoMajor: 1,ProtoMinor: 1,Header: make(Header),Body: rc,Host: u.Host,}if body ! nil {switch v : body.(type) {case *bytes.Buffer:req.ContentLength int64(v.Len())buf : v.Bytes()req.GetBody func() (io.ReadCloser, error) {r : bytes.NewReader(buf)return ioutil.NopCloser(r), nil}case *bytes.Reader:req.ContentLength int64(v.Len())snapshot : *vreq.GetBody func() (io.ReadCloser, error) {r : snapshotreturn ioutil.NopCloser(r), nil}case *strings.Reader:req.ContentLength int64(v.Len())snapshot : *vreq.GetBody func() (io.ReadCloser, error) {r : snapshotreturn ioutil.NopCloser(r), nil}default:// This is where wed set it to -1 (at least// if body ! NoBody) to mean unknown, but// that broke people during the Go 1.8 testing// period. People depend on it being 0 I// guess. Maybe retry later. See Issue 18117.}// For client requests, Request.ContentLength of 0// means either actually 0, or unknown. The only way// to explicitly say that the ContentLength is zero is// to set the Body to nil. But turns out too much code// depends on NewRequest returning a non-nil Body,// so we use a well-known ReadCloser variable instead// and have the http package also treat that sentinel// variable to mean explicitly zero.if req.GetBody ! nil req.ContentLength 0 {req.Body NoBodyreq.GetBody func() (io.ReadCloser, error) { return NoBody, nil }}}return req, nil
}// A Request represents an HTTP request received by a server
// or to be sent by a client.
//
// The field semantics differ slightly between client and server
// usage. In addition to the notes on the fields below, see the
// documentation for Request.Write and RoundTripper.
type Request struct {// Method specifies the HTTP method (GET, POST, PUT, etc.).// For client requests, an empty string means GET.//// Gos HTTP client does not support sending a request with// the CONNECT method. See the documentation on Transport for// details.Method string// URL specifies either the URI being requested (for server// requests) or the URL to access (for client requests).//// For server requests, the URL is parsed from the URI// supplied on the Request-Line as stored in RequestURI. For// most requests, fields other than Path and RawQuery will be// empty. (See RFC 7230, Section 5.3)//// For client requests, the URLs Host specifies the server to// connect to, while the Requests Host field optionally// specifies the Host header value to send in the HTTP// request.URL *url.URL// The protocol version for incoming server requests.//// For client requests, these fields are ignored. The HTTP// client code always uses either HTTP/1.1 or HTTP/2.// See the docs on Transport for details.Proto string // HTTP/1.0ProtoMajor int // 1ProtoMinor int // 0// Header contains the request header fields either received// by the server or to be sent by the client.//// If a server received a request with header lines,//// Host: example.com// accept-encoding: gzip, deflate// Accept-Language: en-us// fOO: Bar// foo: two//// then//// Header map[string][]string{// Accept-Encoding: {gzip, deflate},// Accept-Language: {en-us},// Foo: {Bar, two},// }//// For incoming requests, the Host header is promoted to the// Request.Host field and removed from the Header map.//// HTTP defines that header names are case-insensitive. The// request parser implements this by using CanonicalHeaderKey,// making the first character and any characters following a// hyphen uppercase and the rest lowercase.//// For client requests, certain headers such as Content-Length// and Connection are automatically written when needed and// values in Header may be ignored. See the documentation// for the Request.Write method.Header Header// Body is the requests body.//// For client requests, a nil body means the request has no// body, such as a GET request. The HTTP Clients Transport// is responsible for calling the Close method.//// For server requests, the Request Body is always non-nil// but will return EOF immediately when no body is present.// The Server will close the request body. The ServeHTTP// Handler does not need to.Body io.ReadCloser// GetBody defines an optional func to return a new copy of// Body. It is used for client requests when a redirect requires// reading the body more than once. Use of GetBody still// requires setting Body.//// For server requests, it is unused.GetBody func() (io.ReadCloser, error)// ContentLength records the length of the associated content.// The value -1 indicates that the length is unknown.// Values 0 indicate that the given number of bytes may// be read from Body.//// For client requests, a value of 0 with a non-nil Body is// also treated as unknown.ContentLength int64// TransferEncoding lists the transfer encodings from outermost to// innermost. An empty list denotes the identity encoding.// TransferEncoding can usually be ignored; chunked encoding is// automatically added and removed as necessary when sending and// receiving requests.TransferEncoding []string// Close indicates whether to close the connection after// replying to this request (for servers) or after sending this// request and reading its response (for clients).//// For server requests, the HTTP server handles this automatically// and this field is not needed by Handlers.//// For client requests, setting this field prevents re-use of// TCP connections between requests to the same hosts, as if// Transport.DisableKeepAlives were set.Close bool// For server requests, Host specifies the host on which the URL// is sought. Per RFC 7230, section 5.4, this is either the value// of the Host header or the host name given in the URL itself.// It may be of the form host:port. For international domain// names, Host may be in Punycode or Unicode form. Use// golang.org/x/net/idna to convert it to either format if// needed.// To prevent DNS rebinding attacks, server Handlers should// validate that the Host header has a value for which the// Handler considers itself authoritative. The included// ServeMux supports patterns registered to particular host// names and thus protects its registered Handlers.//// For client requests, Host optionally overrides the Host// header to send. If empty, the Request.Write method uses// the value of URL.Host. Host may contain an international// domain name.Host string// Form contains the parsed form data, including both the URL// fields query parameters and the PATCH, POST, or PUT form data.// This field is only available after ParseForm is called.// The HTTP client ignores Form and uses Body instead.Form url.Values// PostForm contains the parsed form data from PATCH, POST// or PUT body parameters.//// This field is only available after ParseForm is called.// The HTTP client ignores PostForm and uses Body instead.PostForm url.Values// MultipartForm is the parsed multipart form, including file uploads.// This field is only available after ParseMultipartForm is called.// The HTTP client ignores MultipartForm and uses Body instead.MultipartForm *multipart.Form// Trailer specifies additional headers that are sent after the request// body.//// For server requests, the Trailer map initially contains only the// trailer keys, with nil values. (The client declares which trailers it// will later send.) While the handler is reading from Body, it must// not reference Trailer. After reading from Body returns EOF, Trailer// can be read again and will contain non-nil values, if they were sent// by the client.//// For client requests, Trailer must be initialized to a map containing// the trailer keys to later send. The values may be nil or their final// values. The ContentLength must be 0 or -1, to send a chunked request.// After the HTTP request is sent the map values can be updated while// the request body is read. Once the body returns EOF, the caller must// not mutate Trailer.//// Few HTTP clients, servers, or proxies support HTTP trailers.Trailer Header// RemoteAddr allows HTTP servers and other software to record// the network address that sent the request, usually for// logging. This field is not filled in by ReadRequest and// has no defined format. The HTTP server in this package// sets RemoteAddr to an IP:port address before invoking a// handler.// This field is ignored by the HTTP client.RemoteAddr string// RequestURI is the unmodified request-target of the// Request-Line (RFC 7230, Section 3.1.1) as sent by the client// to a server. Usually the URL field should be used instead.// It is an error to set this field in an HTTP client request.RequestURI string// TLS allows HTTP servers and other software to record// information about the TLS connection on which the request// was received. This field is not filled in by ReadRequest.// The HTTP server in this package sets the field for// TLS-enabled connections before invoking a handler;// otherwise it leaves the field nil.// This field is ignored by the HTTP client.TLS *tls.ConnectionState// Cancel is an optional channel whose closure indicates that the client// request should be regarded as canceled. Not all implementations of// RoundTripper may support Cancel.//// For server requests, this field is not applicable.//// Deprecated: Set the Requests context with NewRequestWithContext// instead. If a Requests Cancel field and context are both// set, it is undefined whether Cancel is respected.Cancel -chan struct{}// Response is the redirect response which caused this request// to be created. This field is only populated during client// redirects.Response *Response// ctx is either the client or server context. It should only// be modified via copying the whole Request using WithContext.// It is unexported to prevent people from using Context wrong// and mutating the contexts held by callers of the same request.ctx context.Context
}二、client.Do(req)
使用http.Client发送请求如下
// Do sends an HTTP request and returns an HTTP response, following
// policy (such as redirects, cookies, auth) as configured on the
// client.
//
// An error is returned if caused by client policy (such as
// CheckRedirect), or failure to speak HTTP (such as a network
// connectivity problem). A non-2xx status code doesnt cause an
// error.
//
// If the returned error is nil, the Response will contain a non-nil
// Body which the user is expected to close. If the Body is not both
// read to EOF and closed, the Clients underlying RoundTripper
// (typically Transport) may not be able to re-use a persistent TCP
// connection to the server for a subsequent keep-alive request.
//
// The request Body, if non-nil, will be closed by the underlying
// Transport, even on errors.
//
// On error, any Response can be ignored. A non-nil Response with a
// non-nil error only occurs when CheckRedirect fails, and even then
// the returned Response.Body is already closed.
//
// Generally Get, Post, or PostForm will be used instead of Do.
//
// If the server replies with a redirect, the Client first uses the
// CheckRedirect function to determine whether the redirect should be
// followed. If permitted, a 301, 302, or 303 redirect causes
// subsequent requests to use HTTP method GET
// (or HEAD if the original request was HEAD), with no body.
// A 307 or 308 redirect preserves the original HTTP method and body,
// provided that the Request.GetBody function is defined.
// The NewRequest function automatically sets GetBody for common
// standard library body types.
//
// Any returned error will be of type *url.Error. The url.Error
// values Timeout method will report true if request timed out or was
// canceled.
func (c *Client) Do(req *Request) (*Response, error) {return c.do(req)
}func (c *Client) do(req *Request) (retres *Response, reterr error) {if testHookClientDoResult ! nil {defer func() { testHookClientDoResult(retres, reterr) }()}if req.URL nil {req.closeBody()return nil, url.Error{Op: urlErrorOp(req.Method),Err: errors.New(http: nil Request.URL),}}var (deadline c.deadline()reqs []*Requestresp *ResponsecopyHeaders c.makeHeadersCopier(req)reqBodyClosed false // have we closed the current req.Body?// Redirect behavior:redirectMethod stringincludeBody bool)uerr : func(err error) error {// the body may have been closed already by c.send()if !reqBodyClosed {req.closeBody()}var urlStr stringif resp ! nil resp.Request ! nil {urlStr stripPassword(resp.Request.URL)} else {urlStr stripPassword(req.URL)}return url.Error{Op: urlErrorOp(reqs[0].Method),URL: urlStr,Err: err,}}for {// For all but the first request, create the next// request hop and replace req.if len(reqs) 0 {loc : resp.Header.Get(Location)if loc {resp.closeBody()return nil, uerr(fmt.Errorf(%d response missing Location header, resp.StatusCode))}u, err : req.URL.Parse(loc)if err ! nil {resp.closeBody()return nil, uerr(fmt.Errorf(failed to parse Location header %q: %v, loc, err))}host : if req.Host ! req.Host ! req.URL.Host {// If the caller specified a custom Host header and the// redirect location is relative, preserve the Host header// through the redirect. See issue #22233.if u, _ : url.Parse(loc); u ! nil !u.IsAbs() {host req.Host}}ireq : reqs[0]req Request{Method: redirectMethod,Response: resp,URL: u,Header: make(Header),Host: host,Cancel: ireq.Cancel,ctx: ireq.ctx,}if includeBody ireq.GetBody ! nil {req.Body, err ireq.GetBody()if err ! nil {resp.closeBody()return nil, uerr(err)}req.ContentLength ireq.ContentLength}// Copy original headers before setting the Referer,// in case the user set Referer on their first request.// If they really want to override, they can do it in// their CheckRedirect func.copyHeaders(req)// Add the Referer header from the most recent// request URL to the new one, if its not https-http:if ref : refererForURL(reqs[len(reqs)-1].URL, req.URL); ref ! {req.Header.Set(Referer, ref)}err c.checkRedirect(req, reqs)// Sentinel error to let users select the// previous response, without closing its// body. See Issue 10069.if err ErrUseLastResponse {return resp, nil}// Close the previous responses body. But// read at least some of the body so if its// small the underlying TCP connection will be// re-used. No need to check for errors: if it// fails, the Transport wont reuse it anyway.const maxBodySlurpSize 2 10if resp.ContentLength -1 || resp.ContentLength maxBodySlurpSize {io.CopyN(ioutil.Discard, resp.Body, maxBodySlurpSize)}resp.Body.Close()if err ! nil {// Special case for Go 1 compatibility: return both the response// and an error if the CheckRedirect function failed.// See https://golang.org/issue/3795// The resp.Body has already been closed.ue : uerr(err)ue.(*url.Error).URL locreturn resp, ue}}reqs append(reqs, req)var err errorvar didTimeout func() boolif resp, didTimeout, err c.send(req, deadline); err ! nil {// c.send() always closes req.BodyreqBodyClosed trueif !deadline.IsZero() didTimeout() {err httpError{// TODO: early in cycle: s/Client.Timeout exceeded/timeout or context cancellation/err: err.Error() (Client.Timeout exceeded while awaiting headers),timeout: true,}}return nil, uerr(err)}var shouldRedirect boolredirectMethod, shouldRedirect, includeBody redirectBehavior(req.Method, resp, reqs[0])if !shouldRedirect {return resp, nil}req.closeBody()}
}整个http.Client.Do逻辑分为两道第一道执行send发送请求接收Response关闭Req.Body第二层对请求执行重定向等操作(若需要redirect)并关闭Resp.Body
其中send执行请求的发送和接收动作展开如下
// didTimeout is non-nil only if err ! nil.
func (c *Client) send(req *Request, deadline time.Time) (resp *Response, didTimeout func() bool, err error) {if c.Jar ! nil {for _, cookie : range c.Jar.Cookies(req.URL) {req.AddCookie(cookie)}}resp, didTimeout, err send(req, c.transport(), deadline)if err ! nil {return nil, didTimeout, err}if c.Jar ! nil {if rc : resp.Cookies(); len(rc) 0 {c.Jar.SetCookies(req.URL, rc)}}return resp, nil, nil
}// send issues an HTTP request.
// Caller should close resp.Body when done reading from it.
func send(ireq *Request, rt RoundTripper, deadline time.Time) (resp *Response, didTimeout func() bool, err error) {req : ireq // req is either the original request, or a modified forkif rt nil {req.closeBody()return nil, alwaysFalse, errors.New(http: no Client.Transport or DefaultTransport)}if req.URL nil {req.closeBody()return nil, alwaysFalse, errors.New(http: nil Request.URL)}if req.RequestURI ! {req.closeBody()return nil, alwaysFalse, errors.New(http: Request.RequestURI cant be set in client requests.)}// forkReq forks req into a shallow clone of ireq the first// time its called.forkReq : func() {if ireq req {req new(Request)*req *ireq // shallow clone}}// Most the callers of send (Get, Post, et al) dont need// Headers, leaving it uninitialized. We guarantee to the// Transport that this has been initialized, though.if req.Header nil {forkReq()req.Header make(Header)}if u : req.URL.User; u ! nil req.Header.Get(Authorization) {username : u.Username()password, _ : u.Password()forkReq()req.Header cloneOrMakeHeader(ireq.Header)req.Header.Set(Authorization, Basic basicAuth(username, password))}if !deadline.IsZero() {forkReq()}stopTimer, didTimeout : setRequestCancel(req, rt, deadline)resp, err rt.RoundTrip(req)if err ! nil {stopTimer()if resp ! nil {log.Printf(RoundTripper returned a response error; ignoring response)}if tlsErr, ok : err.(tls.RecordHeaderError); ok {// If we get a bad TLS record header, check to see if the// response looks like HTTP and give a more helpful error.// See golang.org/issue/11111.if string(tlsErr.RecordHeader[:]) HTTP/ {err errors.New(http: server gave HTTP response to HTTPS client)}}return nil, didTimeout, err}if !deadline.IsZero() {resp.Body cancelTimerBody{stop: stopTimer,rc: resp.Body,reqDidTimeout: didTimeout,}}return resp, nil, nil
}func (c *Client) transport() RoundTripper {if c.Transport ! nil {return c.Transport}return DefaultTransport
}入参为http.Requesthttp.Transport以及deadline(http.Client.Timeout)。函数处理逻辑具体分为两个步骤如下
1.setRequestCancel
设置请求超时时间为http.Client.Timeout
// setRequestCancel sets the Cancel field of req, if deadline is
// non-zero. The RoundTrippers type is used to determine whether the legacy
// CancelRequest behavior should be used.
//
// As background, there are three ways to cancel a request:
// First was Transport.CancelRequest. (deprecated)
// Second was Request.Cancel (this mechanism).
// Third was Request.Context.
func setRequestCancel(req *Request, rt RoundTripper, deadline time.Time) (stopTimer func(), didTimeout func() bool) {if deadline.IsZero() {return nop, alwaysFalse}initialReqCancel : req.Cancel // the users original Request.Cancel, if anycancel : make(chan struct{})req.Cancel canceldoCancel : func() {// The newer way (the second way in the func comment):close(cancel)// The legacy compatibility way, used only// for RoundTripper implementations written// before Go 1.5 or Go 1.6.type canceler interface {CancelRequest(*Request)}switch v : rt.(type) {case *Transport, *http2Transport:// Do nothing. The net/http packages transports// support the new Request.Cancel channelcase canceler:v.CancelRequest(req)}}stopTimerCh : make(chan struct{})var once sync.OncestopTimer func() { once.Do(func() { close(stopTimerCh) }) }timer : time.NewTimer(time.Until(deadline))var timedOut atomicBoolgo func() {select {case -initialReqCancel:doCancel()timer.Stop()case -timer.C:timedOut.setTrue()doCancel()case -stopTimerCh:timer.Stop()}}()return stopTimer, timedOut.isSet
}// A Request represents an HTTP request received by a server
// or to be sent by a client.
//
// The field semantics differ slightly between client and server
// usage. In addition to the notes on the fields below, see the
// documentation for Request.Write and RoundTripper.
type Request struct {...// Cancel is an optional channel whose closure indicates that the client// request should be regarded as canceled. Not all implementations of// RoundTripper may support Cancel.//// For server requests, this field is not applicable.//// Deprecated: Set the Requests context with NewRequestWithContext// instead. If a Requests Cancel field and context are both// set, it is undefined whether Cancel is respected.Cancel -chan struct{}// Response is the redirect response which caused this request// to be created. This field is only populated during client// redirects.Response *Response// ctx is either the client or server context. It should only// be modified via copying the whole Request using WithContext.// It is unexported to prevent people from using Context wrong// and mutating the contexts held by callers of the same request.ctx context.Context
}有三种方式取消一个请求如下
First was Transport.CancelRequest. (deprecated)Second was Request.Cancel (this mechanism).Third was Request.Context.
setRequestCancel执行了前面两种方式
2.resp, err rt.RoundTrip(req)
在设置了超时timer后执行具体的请求操作
回到http.Transport(net/http/transport.go)Transport实现了RoundTripper接口如下
client : http.Client{Transport: http.Transport{Proxy: http.ProxyFromEnvironment,DialContext: (net.Dialer{Timeout: 30 * time.Second,KeepAlive: 30 * time.Second,DualStack: true,}).DialContext,MaxIdleConns: 100,MaxIdleConnsPerHost: 100,IdleConnTimeout: 90 * time.Second,},
}
...// RoundTrip implements the RoundTripper interface.
//
// For higher-level HTTP client support (such as handling of cookies
// and redirects), see Get, Post, and the Client type.
//
// Like the RoundTripper interface, the error types returned
// by RoundTrip are unspecified.
func (t *Transport) RoundTrip(req *Request) (*Response, error) {return t.roundTrip(req)
}// roundTrip implements a RoundTripper over HTTP.
func (t *Transport) roundTrip(req *Request) (*Response, error) {t.nextProtoOnce.Do(t.onceSetNextProtoDefaults)ctx : req.Context()trace : httptrace.ContextClientTrace(ctx)if req.URL nil {req.closeBody()return nil, errors.New(http: nil Request.URL)}if req.Header nil {req.closeBody()return nil, errors.New(http: nil Request.Header)}scheme : req.URL.SchemeisHTTP : scheme http || scheme httpsif isHTTP {for k, vv : range req.Header {if !httpguts.ValidHeaderFieldName(k) {return nil, fmt.Errorf(net/http: invalid header field name %q, k)}for _, v : range vv {if !httpguts.ValidHeaderFieldValue(v) {return nil, fmt.Errorf(net/http: invalid header field value %q for key %v, v, k)}}}}if t.useRegisteredProtocol(req) {altProto, _ : t.altProto.Load().(map[string]RoundTripper)if altRT : altProto[scheme]; altRT ! nil {if resp, err : altRT.RoundTrip(req); err ! ErrSkipAltProtocol {return resp, err}}}if !isHTTP {req.closeBody()return nil, badStringError{unsupported protocol scheme, scheme}}if req.Method ! !validMethod(req.Method) {return nil, fmt.Errorf(net/http: invalid method %q, req.Method)}if req.URL.Host {req.closeBody()return nil, errors.New(http: no Host in request URL)}for {select {case -ctx.Done():req.closeBody()return nil, ctx.Err()default:}// treq gets modified by roundTrip, so we need to recreate for each retry.treq : transportRequest{Request: req, trace: trace}cm, err : t.connectMethodForRequest(treq)if err ! nil {req.closeBody()return nil, err}// Get the cached or newly-created connection to either the// host (for http or https), the http proxy, or the http proxy// pre-CONNECTed to https server. In any case, well be ready// to send it requests.pconn, err : t.getConn(treq, cm)if err ! nil {t.setReqCanceler(req, nil)req.closeBody()return nil, err}var resp *Responseif pconn.alt ! nil {// HTTP/2 path.t.setReqCanceler(req, nil) // not cancelable with CancelRequestresp, err pconn.alt.RoundTrip(req)} else {resp, err pconn.roundTrip(treq)}if err nil {return resp, nil}// Failed. Clean up and determine whether to retry._, isH2DialError : pconn.alt.(http2erringRoundTripper)if http2isNoCachedConnError(err) || isH2DialError {t.removeIdleConn(pconn)t.decConnsPerHost(pconn.cacheKey)}if !pconn.shouldRetryRequest(req, err) {// Issue 16465: return underlying net.Conn.Read error from peek,// as weve historically done.if e, ok : err.(transportReadFromServerError); ok {err e.err}return nil, err}testHookRoundTripRetried()// Rewind the body if were able to.if req.GetBody ! nil {newReq : *reqvar err errornewReq.Body, err req.GetBody()if err ! nil {return nil, err}req newReq}}
}拆分步骤如下
获取连接(Transport.getConn)
// treq gets modified by roundTrip, so we need to recreate for each retry.
treq : transportRequest{Request: req, trace: trace}
cm, err : t.connectMethodForRequest(treq)
if err ! nil {req.closeBody()return nil, err
}// Get the cached or newly-created connection to either the
// host (for http or https), the http proxy, or the http proxy
// pre-CONNECTed to https server. In any case, well be ready
// to send it requests.
pconn, err : t.getConn(treq, cm)
if err ! nil {t.setReqCanceler(req, nil)req.closeBody()return nil, err
}...
// transportRequest is a wrapper around a *Request that adds
// optional extra headers to write and stores any error to return
// from roundTrip.
type transportRequest struct {*Request // original request, not to be mutatedextra Header // extra headers to write, or niltrace *httptrace.ClientTrace // optionalmu sync.Mutex // guards errerr error // first setError value for mapRoundTripError to consider
}func (t *Transport) connectMethodForRequest(treq *transportRequest) (cm connectMethod, err error) {// TODO: the validPort check is redundant after CL 189258, as url.URL.Port// only returns valid ports now. golang.org/issue/33600if port : treq.URL.Port(); !validPort(port) {return cm, fmt.Errorf(invalid URL port %q, port)}cm.targetScheme treq.URL.Schemecm.targetAddr canonicalAddr(treq.URL)if t.Proxy ! nil {cm.proxyURL, err t.Proxy(treq.Request)if err nil cm.proxyURL ! nil {if port : cm.proxyURL.Port(); !validPort(port) {return cm, fmt.Errorf(invalid proxy URL port %q, port)}}}cm.onlyH1 treq.requiresHTTP1()return cm, err
}// connectMethod is the map key (in its String form) for keeping persistent
// TCP connections alive for subsequent HTTP requests.
//
// A connect method may be of the following types:
//
// connectMethod.key().String() Description
// ------------------------------ -------------------------
// |http|foo.com http directly to server, no proxy
// |https|foo.com https directly to server, no proxy
// |https,h1|foo.com https directly to server w/o HTTP/2, no proxy
// http://proxy.com|https|foo.com http to proxy, then CONNECT to foo.com
// http://proxy.com|http http to proxy, http to anywhere after that
// socks5://proxy.com|http|foo.com socks5 to proxy, then http to foo.com
// socks5://proxy.com|https|foo.com socks5 to proxy, then https to foo.com
// https://proxy.com|https|foo.com https to proxy, then CONNECT to foo.com
// https://proxy.com|http https to proxy, http to anywhere after that
//
type connectMethod struct {proxyURL *url.URL // nil for no proxy, else full proxy URLtargetScheme string // http or https// If proxyURL specifies an http or https proxy, and targetScheme is http (not https),// then targetAddr is not included in the connect method key, because the socket can// be reused for different targetAddr values.targetAddr stringonlyH1 bool // whether to disable HTTP/2 and force HTTP/1
}...
// getConn dials and creates a new persistConn to the target as
// specified in the connectMethod. This includes doing a proxy CONNECT
// and/or setting up TLS. If this doesnt return an error, the persistConn
// is ready to write requests to.
func (t *Transport) getConn(treq *transportRequest, cm connectMethod) (pc *persistConn, err error) {req : treq.Requesttrace : treq.tracectx : req.Context()if trace ! nil trace.GetConn ! nil {trace.GetConn(cm.addr())}w : wantConn{cm: cm,key: cm.key(),ctx: ctx,ready: make(chan struct{}, 1),beforeDial: testHookPrePendingDial,afterDial: testHookPostPendingDial,}defer func() {if err ! nil {w.cancel(t, err)}}()// Queue for idle connection.if delivered : t.queueForIdleConn(w); delivered {pc : w.pc// Trace only for HTTP/1.// HTTP/2 calls trace.GotConn itself.if pc.alt nil trace ! nil trace.GotConn ! nil {trace.GotConn(pc.gotIdleConnTrace(pc.idleAt))}// set request canceler to some non-nil function so we// can detect whether it was cleared between now and when// we enter roundTript.setReqCanceler(req, func(error) {})return pc, nil}cancelc : make(chan error, 1)t.setReqCanceler(req, func(err error) { cancelc - err })// Queue for permission to dial.t.queueForDial(w)// Wait for completion or cancellation.select {case -w.ready:// Trace success but only for HTTP/1.// HTTP/2 calls trace.GotConn itself.if w.pc ! nil w.pc.alt nil trace ! nil trace.GotConn ! nil {trace.GotConn(httptrace.GotConnInfo{Conn: w.pc.conn, Reused: w.pc.isReused()})}if w.err ! nil {// If the request has been cancelled, thats probably// what caused w.err; if so, prefer to return the// cancellation error (see golang.org/issue/16049).select {case -req.Cancel:return nil, errRequestCanceledConncase -req.Context().Done():return nil, req.Context().Err()case err : -cancelc:if err errRequestCanceled {err errRequestCanceledConn}return nil, errdefault:// return below}}return w.pc, w.errcase -req.Cancel:return nil, errRequestCanceledConncase -req.Context().Done():return nil, req.Context().Err()case err : -cancelc:if err errRequestCanceled {err errRequestCanceledConn}return nil, err}
}其中getConn是重点分析的内容。首先调用queueForIdleConn获取空闲连接
// queueForIdleConn queues w to receive the next idle connection for w.cm.
// As an optimization hint to the caller, queueForIdleConn reports whether
// it successfully delivered an already-idle connection.
func (t *Transport) queueForIdleConn(w *wantConn) (delivered bool) {if t.DisableKeepAlives {return false}t.idleMu.Lock()defer t.idleMu.Unlock()// Stop closing connections that become idle - we might want one.// (That is, undo the effect of t.CloseIdleConnections.)t.closeIdle falseif w nil {// Happens in test hook.return false}// Look for most recently-used idle connection.if list, ok : t.idleConn[w.key]; ok {stop : falsedelivered : falsefor len(list) 0 !stop {pconn : list[len(list)-1]if pconn.isBroken() {// persistConn.readLoop has marked the connection broken,// but Transport.removeIdleConn has not yet removed it from the idle list.// Drop on floor on behalf of Transport.removeIdleConn.list list[:len(list)-1]continue}delivered w.tryDeliver(pconn, nil)if delivered {if pconn.alt ! nil {// HTTP/2: multiple clients can share pconn.// Leave it in the list.} else {// HTTP/1: only one client can use pconn.// Remove it from the list.t.idleLRU.remove(pconn)list list[:len(list)-1]}}stop true}if len(list) 0 {t.idleConn[w.key] list} else {delete(t.idleConn, w.key)}if stop {return delivered}}// Register to receive next connection that becomes idle.if t.idleConnWait nil {t.idleConnWait make(map[connectMethodKey]wantConnQueue)}q : t.idleConnWait[w.key]q.cleanFront()q.pushBack(w)t.idleConnWait[w.key] qreturn false
}// A wantConn records state about a wanted connection
// (that is, an active call to getConn).
// The conn may be gotten by dialing or by finding an idle connection,
// or a cancellation may make the conn no longer wanted.
// These three options are racing against each other and use
// wantConn to coordinate and agree about the winning outcome.
type wantConn struct {cm connectMethodkey connectMethodKey // cm.key()ctx context.Context // context for dialready chan struct{} // closed when pc, err pair is delivered// hooks for testing to know when dials are done// beforeDial is called in the getConn goroutine when the dial is queued.// afterDial is called when the dial is completed or cancelled.beforeDial func()afterDial func()mu sync.Mutex // protects pc, err, close(ready)pc *persistConnerr error
}// tryDeliver attempts to deliver pc, err to w and reports whether it succeeded.
func (w *wantConn) tryDeliver(pc *persistConn, err error) bool {w.mu.Lock()defer w.mu.Unlock()if w.pc ! nil || w.err ! nil {return false}w.pc pcw.err errif w.pc nil w.err nil {panic(net/http: internal error: misuse of tryDeliver)}close(w.ready)return true
}idleConn(map[connectMethodKey][]*persistConn)表示Transport的空闲连接池。其中connectMethodKey是对url关键字段的分解代表连接的目标方(代理协议目的地址)而values是persistConn的列表
而persistConn是对连接的一层封装通常表示keep-alive连接也可以表示non-keep-alive连接
type Transport struct {idleMu sync.MutexcloseIdle bool // user has requested to close all idle connsidleConn map[connectMethodKey][]*persistConn // most recently used at endidleConnWait map[connectMethodKey]wantConnQueue // waiting getConnsidleLRU connLRU...
}// connectMethodKey is the map key version of connectMethod, with a
// stringified proxy URL (or the empty string) instead of a pointer to
// a URL.
type connectMethodKey struct {proxy, scheme, addr stringonlyH1 bool
}// persistConn wraps a connection, usually a persistent one
// (but may be used for non-keep-alive requests as well)
type persistConn struct {// alt optionally specifies the TLS NextProto RoundTripper.// This is used for HTTP/2 today and future protocols later.// If its non-nil, the rest of the fields are unused.alt RoundTrippert *TransportcacheKey connectMethodKeyconn net.ConntlsState *tls.ConnectionStatebr *bufio.Reader // from connbw *bufio.Writer // to connnwrite int64 // bytes writtenreqch chan requestAndChan // written by roundTrip; read by readLoopwritech chan writeRequest // written by roundTrip; read by writeLoopclosech chan struct{} // closed when conn closedisProxy boolsawEOF bool // whether weve seen EOF from conn; owned by readLoopreadLimit int64 // bytes allowed to be read; owned by readLoop// writeErrCh passes the request write error (usually nil)// from the writeLoop goroutine to the readLoop which passes// it off to the res.Body reader, which then uses it to decide// whether or not a connection can be reused. Issue 7569.writeErrCh chan errorwriteLoopDone chan struct{} // closed when write loop ends// Both guarded by Transport.idleMu:idleAt time.Time // time it last become idleidleTimer *time.Timer // holding an AfterFunc to close itmu sync.Mutex // guards following fieldsnumExpectedResponses intclosed error // set non-nil when conn is closed, before closech is closedcanceledErr error // set non-nil if conn is canceledbroken bool // an error has happened on this connection; marked broken so its not reused.reused bool // whether conn has had successful request/response and is being reused.// mutateHeaderFunc is an optional func to modify extra// headers on each outbound request before its written. (the// original Request given to RoundTrip is not modified)mutateHeaderFunc func(Header)
}queueForIdleConn首先尝试从idleConn中获取最近使用的空闲连接(Look for most recently-used idle connection)。如果找到则从连接列表中删除掉该连接(非HTTP/2情况因为HTTP/2协议多个请求可以同时共用底层连接)
如果发现没有对应空闲连接则将wantConn赋值给idleConnWait(waiting getConns)代表此时多了一个等待获取连接的请求
回到getConn内容。在执行queueForIdleConn获取空闲连接成功后直接返回该连接否则执行queueForDial尝试创建连接等待创建结束并返回连接
// getConn dials and creates a new persistConn to the target as
// specified in the connectMethod. This includes doing a proxy CONNECT
// and/or setting up TLS. If this doesnt return an error, the persistConn
// is ready to write requests to.
func (t *Transport) getConn(treq *transportRequest, cm connectMethod) (pc *persistConn, err error) {if delivered : t.queueForIdleConn(w); delivered {...}// Queue for permission to dial.t.queueForDial(w)// Wait for completion or cancellation.select {case -w.ready:// Trace success but only for HTTP/1.// HTTP/2 calls trace.GotConn itself.if w.pc ! nil w.pc.alt nil trace ! nil trace.GotConn ! nil {trace.GotConn(httptrace.GotConnInfo{Conn: w.pc.conn, Reused: w.pc.isReused()})}if w.err ! nil {// If the request has been cancelled, thats probably// what caused w.err; if so, prefer to return the// cancellation error (see golang.org/issue/16049).select {case -req.Cancel:return nil, errRequestCanceledConncase -req.Context().Done():return nil, req.Context().Err()case err : -cancelc:if err errRequestCanceled {err errRequestCanceledConn}return nil, errdefault:// return below}}return w.pc, w.errcase -req.Cancel:return nil, errRequestCanceledConncase -req.Context().Done():return nil, req.Context().Err()case err : -cancelc:if err errRequestCanceled {err errRequestCanceledConn}return nil, err}
}// queueForDial queues w to wait for permission to begin dialing.
// Once w receives permission to dial, it will do so in a separate goroutine.
func (t *Transport) queueForDial(w *wantConn) {w.beforeDial()if t.MaxConnsPerHost 0 {go t.dialConnFor(w)return}t.connsPerHostMu.Lock()defer t.connsPerHostMu.Unlock()if n : t.connsPerHost[w.key]; n t.MaxConnsPerHost {if t.connsPerHost nil {t.connsPerHost make(map[connectMethodKey]int)}t.connsPerHost[w.key] n 1go t.dialConnFor(w)return}if t.connsPerHostWait nil {t.connsPerHostWait make(map[connectMethodKey]wantConnQueue)}q : t.connsPerHostWait[w.key]q.cleanFront()q.pushBack(w)t.connsPerHostWait[w.key] q
}// Transport is an implementation of RoundTripper that supports HTTP,
// HTTPS, and HTTP proxies (for either HTTP or HTTPS with CONNECT).
//
// By default, Transport caches connections for future re-use.
// This may leave many open connections when accessing many hosts.
// This behavior can be managed using Transports CloseIdleConnections method
// and the MaxIdleConnsPerHost and DisableKeepAlives fields.
//
// Transports should be reused instead of created as needed.
// Transports are safe for concurrent use by multiple goroutines.
//
// A Transport is a low-level primitive for making HTTP and HTTPS requests.
// For high-level functionality, such as cookies and redirects, see Client.
//
// Transport uses HTTP/1.1 for HTTP URLs and either HTTP/1.1 or HTTP/2
// for HTTPS URLs, depending on whether the server supports HTTP/2,
// and how the Transport is configured. The DefaultTransport supports HTTP/2.
// To explicitly enable HTTP/2 on a transport, use golang.org/x/net/http2
// and call ConfigureTransport. See the package docs for more about HTTP/2.
//
// Responses with status codes in the 1xx range are either handled
// automatically (100 expect-continue) or ignored. The one
// exception is HTTP status code 101 (Switching Protocols), which is
// considered a terminal status and returned by RoundTrip. To see the
// ignored 1xx responses, use the httptrace trace packages
// ClientTrace.Got1xxResponse.
//
// Transport only retries a request upon encountering a network error
// if the request is idempotent and either has no body or has its
// Request.GetBody defined. HTTP requests are considered idempotent if
// they have HTTP methods GET, HEAD, OPTIONS, or TRACE; or if their
// Header map contains an Idempotency-Key or X-Idempotency-Key
// entry. If the idempotency key value is an zero-length slice, the
// request is treated as idempotent but the header is not sent on the
// wire.
type Transport struct {idleMu sync.MutexcloseIdle bool // user has requested to close all idle connsidleConn map[connectMethodKey][]*persistConn // most recently used at endidleConnWait map[connectMethodKey]wantConnQueue // waiting getConnsidleLRU connLRUreqMu sync.MutexreqCanceler map[*Request]func(error)altMu sync.Mutex // guards changing altProto onlyaltProto atomic.Value // of nil or map[string]RoundTripper, key is URI schemeconnsPerHostMu sync.MutexconnsPerHost map[connectMethodKey]intconnsPerHostWait map[connectMethodKey]wantConnQueue // waiting getConns...
}// A wantConnQueue is a queue of wantConns.
type wantConnQueue struct {// This is a queue, not a deque.// It is split into two stages - head[headPos:] and tail.// popFront is trivial (headPos) on the first stage, and// pushBack is trivial (append) on the second stage.// If the first stage is empty, popFront can swap the// first and second stages to remedy the situation.//// This two-stage split is analogous to the use of two lists// in Okasakis purely functional queue but without the// overhead of reversing the list when swapping stages.head []*wantConnheadPos inttail []*wantConn
}还记得最开始介绍Transport结构体时有如下参数
MaxConnsPerHost控制某个host的所有连接包括创建中正在使用的以及空闲的连接(including connections in the dialing, active, and idle states)。一旦超过限制dial会阻塞(Zero means no limit)connsPerHost表示每个host(connectMethodKey)的目前连接个数connsPerHostWait表示每个host(connectMethodKey)等待建立的连接请求
queueForDial的作用是判断wantConn对应的host目前连接数是否超过Transport.MaxConnsPerHost如果没有超过则执行dialConnFor操作为该请求创建连接同时递增Transport.connsPerHost[wantConn]否则将wantConn放入connsPerHostWait列表中
接下来看dialConnFor如下
// dialConnFor dials on behalf of w and delivers the result to w.
// dialConnFor has received permission to dial w.cm and is counted in t.connCount[w.cm.key()].
// If the dial is cancelled or unsuccessful, dialConnFor decrements t.connCount[w.cm.key()].
func (t *Transport) dialConnFor(w *wantConn) {defer w.afterDial()pc, err : t.dialConn(w.ctx, w.cm)delivered : w.tryDeliver(pc, err)if err nil (!delivered || pc.alt ! nil) {// pconn was not passed to w,// or it is HTTP/2 and can be shared.// Add to the idle connection pool.t.putOrCloseIdleConn(pc)}if err ! nil {t.decConnsPerHost(w.key)}
}// tryDeliver attempts to deliver pc, err to w and reports whether it succeeded.
func (w *wantConn) tryDeliver(pc *persistConn, err error) bool {w.mu.Lock()defer w.mu.Unlock()if w.pc ! nil || w.err ! nil {return false}w.pc pcw.err errif w.pc nil w.err nil {panic(net/http: internal error: misuse of tryDeliver)}close(w.ready)return true
}func (t *Transport) dialConn(ctx context.Context, cm connectMethod) (pconn *persistConn, err error) {pconn persistConn{t: t,cacheKey: cm.key(),reqch: make(chan requestAndChan, 1),writech: make(chan writeRequest, 1),closech: make(chan struct{}),writeErrCh: make(chan error, 1),writeLoopDone: make(chan struct{}),}trace : httptrace.ContextClientTrace(ctx)wrapErr : func(err error) error {if cm.proxyURL ! nil {// Return a typed error, per Issue 16997return net.OpError{Op: proxyconnect, Net: tcp, Err: err}}return err}if cm.scheme() https t.DialTLS ! nil {var err errorpconn.conn, err t.DialTLS(tcp, cm.addr())if err ! nil {return nil, wrapErr(err)}if pconn.conn nil {return nil, wrapErr(errors.New(net/http: Transport.DialTLS returned (nil, nil)))}if tc, ok : pconn.conn.(*tls.Conn); ok {// Handshake here, in case DialTLS didnt. TLSNextProto below// depends on it for knowing the connection state.if trace ! nil trace.TLSHandshakeStart ! nil {trace.TLSHandshakeStart()}if err : tc.Handshake(); err ! nil {go pconn.conn.Close()if trace ! nil trace.TLSHandshakeDone ! nil {trace.TLSHandshakeDone(tls.ConnectionState{}, err)}return nil, err}cs : tc.ConnectionState()if trace ! nil trace.TLSHandshakeDone ! nil {trace.TLSHandshakeDone(cs, nil)}pconn.tlsState cs}} else {conn, err : t.dial(ctx, tcp, cm.addr())if err ! nil {return nil, wrapErr(err)}pconn.conn connif cm.scheme() https {var firstTLSHost stringif firstTLSHost, _, err net.SplitHostPort(cm.addr()); err ! nil {return nil, wrapErr(err)}if err pconn.addTLS(firstTLSHost, trace); err ! nil {return nil, wrapErr(err)}}}// Proxy setup.switch {case cm.proxyURL nil:// Do nothing. Not using a proxy.case cm.proxyURL.Scheme socks5:conn : pconn.connd : socksNewDialer(tcp, conn.RemoteAddr().String())if u : cm.proxyURL.User; u ! nil {auth : socksUsernamePassword{Username: u.Username(),}auth.Password, _ u.Password()d.AuthMethods []socksAuthMethod{socksAuthMethodNotRequired,socksAuthMethodUsernamePassword,}d.Authenticate auth.Authenticate}if _, err : d.DialWithConn(ctx, conn, tcp, cm.targetAddr); err ! nil {conn.Close()return nil, err}case cm.targetScheme http:pconn.isProxy trueif pa : cm.proxyAuth(); pa ! {pconn.mutateHeaderFunc func(h Header) {h.Set(Proxy-Authorization, pa)}}case cm.targetScheme https:conn : pconn.connhdr : t.ProxyConnectHeaderif hdr nil {hdr make(Header)}connectReq : Request{Method: CONNECT,URL: url.URL{Opaque: cm.targetAddr},Host: cm.targetAddr,Header: hdr,}if pa : cm.proxyAuth(); pa ! {connectReq.Header.Set(Proxy-Authorization, pa)}connectReq.Write(conn)// Read response.// Okay to use and discard buffered reader here, because// TLS server will not speak until spoken to.br : bufio.NewReader(conn)resp, err : ReadResponse(br, connectReq)if err ! nil {conn.Close()return nil, err}if resp.StatusCode ! 200 {f : strings.SplitN(resp.Status, , 2)conn.Close()if len(f) 2 {return nil, errors.New(unknown status code)}return nil, errors.New(f[1])}}if cm.proxyURL ! nil cm.targetScheme https {if err : pconn.addTLS(cm.tlsHost(), trace); err ! nil {return nil, err}}if s : pconn.tlsState; s ! nil s.NegotiatedProtocolIsMutual s.NegotiatedProtocol ! {if next, ok : t.TLSNextProto[s.NegotiatedProtocol]; ok {return persistConn{t: t, cacheKey: pconn.cacheKey, alt: next(cm.targetAddr, pconn.conn.(*tls.Conn))}, nil}}pconn.br bufio.NewReaderSize(pconn, t.readBufferSize())pconn.bw bufio.NewWriterSize(persistConnWriter{pconn}, t.writeBufferSize())go pconn.readLoop()go pconn.writeLoop()return pconn, nil
}// Add TLS to a persistent connection, i.e. negotiate a TLS session. If pconn is already a TLS
// tunnel, this function establishes a nested TLS session inside the encrypted channel.
// The remote endpoints name may be overridden by TLSClientConfig.ServerName.
func (pconn *persistConn) addTLS(name string, trace *httptrace.ClientTrace) error {// Initiate TLS and check remote host name against certificate.cfg : cloneTLSConfig(pconn.t.TLSClientConfig)if cfg.ServerName {cfg.ServerName name}if pconn.cacheKey.onlyH1 {cfg.NextProtos nil}plainConn : pconn.conntlsConn : tls.Client(plainConn, cfg)errc : make(chan error, 2)var timer *time.Timer // for canceling TLS handshakeif d : pconn.t.TLSHandshakeTimeout; d ! 0 {timer time.AfterFunc(d, func() {errc - tlsHandshakeTimeoutError{}})}go func() {if trace ! nil trace.TLSHandshakeStart ! nil {trace.TLSHandshakeStart()}err : tlsConn.Handshake()if timer ! nil {timer.Stop()}errc - err}()if err : -errc; err ! nil {plainConn.Close()if trace ! nil trace.TLSHandshakeDone ! nil {trace.TLSHandshakeDone(tls.ConnectionState{}, err)}return err}cs : tlsConn.ConnectionState()if trace ! nil trace.TLSHandshakeDone ! nil {trace.TLSHandshakeDone(cs, nil)}pconn.tlsState cspconn.conn tlsConnreturn nil
}重点分析dialConn该函数为connectMethod获取persistConn步骤如下
1.判断协议是https还是http如果是https则执行TLS握手过程(i.e. negotiate a TLS session)否则直接执行Transport.dial操作创建TCP连接
var zeroDialer net.Dialerfunc (t *Transport) dial(ctx context.Context, network, addr string) (net.Conn, error) {if t.DialContext ! nil {return t.DialContext(ctx, network, addr)}if t.Dial ! nil {c, err : t.Dial(network, addr)if c nil err nil {err errors.New(net/http: Transport.Dial hook returned (nil, nil))}return c, err}return zeroDialer.DialContext(ctx, network, addr)
}这里就是调用前面分析过的net.Dialer.DialContext函数了(前后对接起来了)不再展开分析
2.执行代理设置
3.设置persistConn的读写buffer
pconn.br bufio.NewReaderSize(pconn, t.readBufferSize())
pconn.bw bufio.NewWriterSize(persistConnWriter{pconn}, t.writeBufferSize())go pconn.readLoop()
go pconn.writeLoop()在分析完getConn之后回头来分析http.Client.Do的第二个步骤resp, err rt.RoundTrip(req)如下
// roundTrip implements a RoundTripper over HTTP.
func (t *Transport) roundTrip(req *Request) (*Response, error) {t.nextProtoOnce.Do(t.onceSetNextProtoDefaults)ctx : req.Context()trace : httptrace.ContextClientTrace(ctx)if req.URL nil {req.closeBody()return nil, errors.New(http: nil Request.URL)}if req.Header nil {req.closeBody()return nil, errors.New(http: nil Request.Header)}scheme : req.URL.SchemeisHTTP : scheme http || scheme httpsif isHTTP {for k, vv : range req.Header {if !httpguts.ValidHeaderFieldName(k) {return nil, fmt.Errorf(net/http: invalid header field name %q, k)}for _, v : range vv {if !httpguts.ValidHeaderFieldValue(v) {return nil, fmt.Errorf(net/http: invalid header field value %q for key %v, v, k)}}}}if t.useRegisteredProtocol(req) {altProto, _ : t.altProto.Load().(map[string]RoundTripper)if altRT : altProto[scheme]; altRT ! nil {if resp, err : altRT.RoundTrip(req); err ! ErrSkipAltProtocol {return resp, err}}}if !isHTTP {req.closeBody()return nil, badStringError{unsupported protocol scheme, scheme}}if req.Method ! !validMethod(req.Method) {return nil, fmt.Errorf(net/http: invalid method %q, req.Method)}if req.URL.Host {req.closeBody()return nil, errors.New(http: no Host in request URL)}for {select {case -ctx.Done():req.closeBody()return nil, ctx.Err()default:}// treq gets modified by roundTrip, so we need to recreate for each retry.treq : transportRequest{Request: req, trace: trace}cm, err : t.connectMethodForRequest(treq)if err ! nil {req.closeBody()return nil, err}// Get the cached or newly-created connection to either the// host (for http or https), the http proxy, or the http proxy// pre-CONNECTed to https server. In any case, well be ready// to send it requests.pconn, err : t.getConn(treq, cm)if err ! nil {t.setReqCanceler(req, nil)req.closeBody()return nil, err}var resp *Responseif pconn.alt ! nil {// HTTP/2 path.t.setReqCanceler(req, nil) // not cancelable with CancelRequestresp, err pconn.alt.RoundTrip(req)} else {resp, err pconn.roundTrip(treq)}if err nil {return resp, nil}// Failed. Clean up and determine whether to retry._, isH2DialError : pconn.alt.(http2erringRoundTripper)if http2isNoCachedConnError(err) || isH2DialError {t.removeIdleConn(pconn)t.decConnsPerHost(pconn.cacheKey)}if !pconn.shouldRetryRequest(req, err) {// Issue 16465: return underlying net.Conn.Read error from peek,// as weve historically done.if e, ok : err.(transportReadFromServerError); ok {err e.err}return nil, err}testHookRoundTripRetried()// Rewind the body if were able to.if req.GetBody ! nil {newReq : *reqvar err errornewReq.Body, err req.GetBody()if err ! nil {return nil, err}req newReq}}
}在执行pconn, err : t.getConn(treq, cm)获取连接(新创建的 or 复用空闲连接)后执行resp, err pconn.roundTrip(treq)如下
func (pc *persistConn) roundTrip(req *transportRequest) (resp *Response, err error) {...// Write the request concurrently with waiting for a response,// in case the server decides to reply before reading our full// request body.startBytesWritten : pc.nwritewriteErrCh : make(chan error, 1)pc.writech - writeRequest{req, writeErrCh, continueCh}resc : make(chan responseAndError)pc.reqch - requestAndChan{req: req.Request,ch: resc,addedGzip: requestedGzip,continueCh: continueCh,callerGone: gone,}var respHeaderTimer -chan time.TimecancelChan : req.Request.CancelctxDoneChan : req.Context().Done()for {testHookWaitResLoop()select {case err : -writeErrCh:if debugRoundTrip {req.logf(writeErrCh resv: %T/%#v, err, err)}if err ! nil {pc.close(fmt.Errorf(write error: %v, err))return nil, pc.mapRoundTripError(req, startBytesWritten, err)}if d : pc.t.ResponseHeaderTimeout; d 0 {if debugRoundTrip {req.logf(starting timer for %v, d)}timer : time.NewTimer(d)defer timer.Stop() // prevent leaksrespHeaderTimer timer.C}case -pc.closech:if debugRoundTrip {req.logf(closech recv: %T %#v, pc.closed, pc.closed)}return nil, pc.mapRoundTripError(req, startBytesWritten, pc.closed)case -respHeaderTimer:if debugRoundTrip {req.logf(timeout waiting for response headers.)}pc.close(errTimeout)return nil, errTimeoutcase re : -resc:if (re.res nil) (re.err nil) {panic(fmt.Sprintf(internal error: exactly one of res or err should be set; nil%v, re.res nil))}if debugRoundTrip {req.logf(resc recv: %p, %T/%#v, re.res, re.err, re.err)}if re.err ! nil {return nil, pc.mapRoundTripError(req, startBytesWritten, re.err)}return re.res, nilcase -cancelChan:pc.t.CancelRequest(req.Request)cancelChan nilcase -ctxDoneChan:pc.t.cancelRequest(req.Request, req.Context().Err())cancelChan nilctxDoneChan nil}}
}这里需要结合前面的pc.readLoop如下
func (pc *persistConn) readLoop() {closeErr : errReadLoopExiting // default value, if not changed belowdefer func() {pc.close(closeErr)pc.t.removeIdleConn(pc)}()tryPutIdleConn : func(trace *httptrace.ClientTrace) bool {if err : pc.t.tryPutIdleConn(pc); err ! nil {closeErr errif trace ! nil trace.PutIdleConn ! nil err ! errKeepAlivesDisabled {trace.PutIdleConn(err)}return false}if trace ! nil trace.PutIdleConn ! nil {trace.PutIdleConn(nil)}return true}// eofc is used to block caller goroutines reading from Response.Body// at EOF until this goroutines has (potentially) added the connection// back to the idle pool.eofc : make(chan struct{})defer close(eofc) // unblock reader on errors// Read this once, before loop starts. (to avoid races in tests)testHookMu.Lock()testHookReadLoopBeforeNextRead : testHookReadLoopBeforeNextReadtestHookMu.Unlock()alive : truefor alive {pc.readLimit pc.maxHeaderResponseSize()_, err : pc.br.Peek(1)pc.mu.Lock()if pc.numExpectedResponses 0 {pc.readLoopPeekFailLocked(err)pc.mu.Unlock()return}pc.mu.Unlock()rc : -pc.reqchtrace : httptrace.ContextClientTrace(rc.req.Context())var resp *Responseif err nil {resp, err pc.readResponse(rc, trace)} else {err transportReadFromServerError{err}closeErr err}if err ! nil {if pc.readLimit 0 {err fmt.Errorf(net/http: server response headers exceeded %d bytes; aborted, pc.maxHeaderResponseSize())}select {case rc.ch - responseAndError{err: err}:case -rc.callerGone:return}return}pc.readLimit maxInt64 // effictively no limit for response bodiespc.mu.Lock()pc.numExpectedResponses--pc.mu.Unlock()bodyWritable : resp.bodyIsWritable()hasBody : rc.req.Method ! HEAD resp.ContentLength ! 0if resp.Close || rc.req.Close || resp.StatusCode 199 || bodyWritable {// Dont do keep-alive on error if either party requested a close// or we get an unexpected informational (1xx) response.// StatusCode 100 is already handled above.alive false}if !hasBody || bodyWritable {pc.t.setReqCanceler(rc.req, nil)// Put the idle conn back into the pool before we send the response// so if they process it quickly and make another request, theyll// get this same conn. But we use the unbuffered channel rc// to guarantee that persistConn.roundTrip got out of its select// potentially waiting for this persistConn to close.// but afteralive alive !pc.sawEOF pc.wroteRequest() tryPutIdleConn(trace)if bodyWritable {closeErr errCallerOwnsConn}select {case rc.ch - responseAndError{res: resp}:case -rc.callerGone:return}// Now that theyve read from the unbuffered channel, theyre safely// out of the select that also waits on this goroutine to die, so// were allowed to exit now if needed (if alive is false)testHookReadLoopBeforeNextRead()continue}waitForBodyRead : make(chan bool, 2)body : bodyEOFSignal{body: resp.Body,earlyCloseFn: func() error {waitForBodyRead - false-eofc // will be closed by deferred call at the end of the functionreturn nil},fn: func(err error) error {isEOF : err io.EOFwaitForBodyRead - isEOFif isEOF {-eofc // see comment above eofc declaration} else if err ! nil {if cerr : pc.canceled(); cerr ! nil {return cerr}}return err},}resp.Body bodyif rc.addedGzip strings.EqualFold(resp.Header.Get(Content-Encoding), gzip) {resp.Body gzipReader{body: body}resp.Header.Del(Content-Encoding)resp.Header.Del(Content-Length)resp.ContentLength -1resp.Uncompressed true}select {case rc.ch - responseAndError{res: resp}:case -rc.callerGone:return}// Before looping back to the top of this function and peeking on// the bufio.Reader, wait for the caller goroutine to finish// reading the response body. (or for cancellation or death)select {case bodyEOF : -waitForBodyRead:pc.t.setReqCanceler(rc.req, nil) // before pc might return to idle poolalive alive bodyEOF !pc.sawEOF pc.wroteRequest() tryPutIdleConn(trace)if bodyEOF {eofc - struct{}{}}case -rc.req.Cancel:alive falsepc.t.CancelRequest(rc.req)case -rc.req.Context().Done():alive falsepc.t.cancelRequest(rc.req, rc.req.Context().Err())case -pc.closech:alive false}testHookReadLoopBeforeNextRead()}
}pc.readLoop() get http response(接收Response) pc.writeLoop() send http request(发送Request)
注意如下代码段(接收Response)
rc : -pc.reqch
trace : httptrace.ContextClientTrace(rc.req.Context())var resp *Response
if err nil {resp, err pc.readResponse(rc, trace)
} else {err transportReadFromServerError{err}closeErr err
}...
// readResponse reads an HTTP response (or two, in the case of Expect:
// 100-continue) from the server. It returns the final non-100 one.
// trace is optional.
func (pc *persistConn) readResponse(rc requestAndChan, trace *httptrace.ClientTrace) (resp *Response, err error) {if trace ! nil trace.GotFirstResponseByte ! nil {if peek, err : pc.br.Peek(1); err nil len(peek) 1 {trace.GotFirstResponseByte()}}num1xx : 0 // number of informational 1xx headers receivedconst max1xxResponses 5 // arbitrary bound on number of informational responsescontinueCh : rc.continueChfor {resp, err ReadResponse(pc.br, rc.req)if err ! nil {return}resCode : resp.StatusCodeif continueCh ! nil {if resCode 100 {if trace ! nil trace.Got100Continue ! nil {trace.Got100Continue()}continueCh - struct{}{}continueCh nil} else if resCode 200 {close(continueCh)continueCh nil}}is1xx : 100 resCode resCode 199// treat 101 as a terminal status, see issue 26161is1xxNonTerminal : is1xx resCode ! StatusSwitchingProtocolsif is1xxNonTerminal {num1xxif num1xx max1xxResponses {return nil, errors.New(net/http: too many 1xx informational responses)}pc.readLimit pc.maxHeaderResponseSize() // reset the limitif trace ! nil trace.Got1xxResponse ! nil {if err : trace.Got1xxResponse(resCode, textproto.MIMEHeader(resp.Header)); err ! nil {return nil, err}}continue}break}if resp.isProtocolSwitch() {resp.Body newReadWriteCloserBody(pc.br, pc.conn)}resp.TLS pc.tlsStatereturn
}// ReadResponse reads and returns an HTTP response from r.
// The req parameter optionally specifies the Request that corresponds
// to this Response. If nil, a GET request is assumed.
// Clients must call resp.Body.Close when finished reading resp.Body.
// After that call, clients can inspect resp.Trailer to find key/value
// pairs included in the response trailer.
func ReadResponse(r *bufio.Reader, req *Request) (*Response, error) {tp : textproto.NewReader(r)resp : Response{Request: req,}// Parse the first line of the response.line, err : tp.ReadLine()if err ! nil {if err io.EOF {err io.ErrUnexpectedEOF}return nil, err}if i : strings.IndexByte(line, ); i -1 {return nil, badStringError{malformed HTTP response, line}} else {resp.Proto line[:i]resp.Status strings.TrimLeft(line[i1:], )}statusCode : resp.Statusif i : strings.IndexByte(resp.Status, ); i ! -1 {statusCode resp.Status[:i]}if len(statusCode) ! 3 {return nil, badStringError{malformed HTTP status code, statusCode}}resp.StatusCode, err strconv.Atoi(statusCode)if err ! nil || resp.StatusCode 0 {return nil, badStringError{malformed HTTP status code, statusCode}}var ok boolif resp.ProtoMajor, resp.ProtoMinor, ok ParseHTTPVersion(resp.Proto); !ok {return nil, badStringError{malformed HTTP version, resp.Proto}}// Parse the response headers.mimeHeader, err : tp.ReadMIMEHeader()if err ! nil {if err io.EOF {err io.ErrUnexpectedEOF}return nil, err}resp.Header Header(mimeHeader)fixPragmaCacheControl(resp.Header)err readTransfer(resp, r)if err ! nil {return nil, err}return resp, nil
}这里可以看到ReadResponse即为读取TCP连接解析HTTP协议并构建http.Response内容返回
在readLoop成功返回http.Response之后会将Response塞到pc.reqch.ch管道中(case rc.ch - responseAndError{res: resp})如下
func (pc *persistConn) readLoop() {...alive : truefor alive {...rc : -pc.reqchtrace : httptrace.ContextClientTrace(rc.req.Context())var resp *Responseif err nil {resp, err pc.readResponse(rc, trace)} else {err transportReadFromServerError{err}closeErr err}bodyWritable : resp.bodyIsWritable()hasBody : rc.req.Method ! HEAD resp.ContentLength ! 0...if resp.Close || rc.req.Close || resp.StatusCode 199 || bodyWritable {// Dont do keep-alive on error if either party requested a close// or we get an unexpected informational (1xx) response.// StatusCode 100 is already handled above.alive false}...resp.Body bodyif rc.addedGzip strings.EqualFold(resp.Header.Get(Content-Encoding), gzip) {resp.Body gzipReader{body: body}resp.Header.Del(Content-Encoding)resp.Header.Del(Content-Length)resp.ContentLength -1resp.Uncompressed true}select {case rc.ch - responseAndError{res: resp}:case -rc.callerGone:return}...}
}而pc.roundTrip会读取该channel(case re : -resc)并返回http.Response如下
func (pc *persistConn) roundTrip(req *transportRequest) (resp *Response, err error) {...resc : make(chan responseAndError)pc.reqch - requestAndChan{req: req.Request,ch: resc,addedGzip: requestedGzip,continueCh: continueCh,callerGone: gone,}var respHeaderTimer -chan time.TimecancelChan : req.Request.CancelctxDoneChan : req.Context().Done()for {testHookWaitResLoop()select {case re : -resc:if (re.res nil) (re.err nil) {panic(fmt.Sprintf(internal error: exactly one of res or err should be set; nil%v, re.res nil))}if debugRoundTrip {req.logf(resc recv: %p, %T/%#v, re.res, re.err, re.err)}if re.err ! nil {return nil, pc.mapRoundTripError(req, startBytesWritten, re.err)}return re.res, nil}}
}golang http transport结构体关键字段
http.Client- Timeouthttp client请求生命周期超时设置(包括连接建立发请求重定向以及Response body读取等所有阶段)- RoundTrip(http.Transport实现)- MaxIdleConns控制所有hosts的空闲连接最大数目(Zero means no limit.)- MaxIdleConnsPerHost控制某个host的空闲连接(keep-alives)最大数目(设置为0则默认DefaultMaxIdleConnsPerHost2)- MaxConnsPerHost控制某个host的所有连接包括创建中正在使用的以及空闲的连接(including connections in the dialing, active, and idle states)。一旦超过限制dial会阻塞(Zero means no limit)- IdleConnTimeout空闲连接(keep-alives)超时时间- h2transportHTTP/2协议对应的transport- ForceAttemptHTTP2当Dial, DialTLS, or DialContext func or TLSClientConfig提供时默认情况下会禁止HTTP/2协议。当使用自定义的这些配置时需要设置ForceAttemptHTTP2字段开启HTTP2- DisableKeepAlives禁止HTTP keep-alives一个连接只用于一次请求(注意区分TCP keep-alives。HTTP keep-alives用于连接复用TCP keep-alives用于连接保活) - idleConn(map[connectMethodKey][]*persistConn)空闲连接池- idleConnWait(map[connectMethodKey]wantConnQueue)等待建立的连接池- connsPerHost(map[connectMethodKey]int)表示每个host(connectMethodKey)的目前连接个数- connsPerHostWait(map[connectMethodKey]wantConnQueue)表示每个host(connectMethodKey)等待建立的连接请求- (net.Dialer)DialContext指定底层TCP连接的创建函数- Timeout连接建立的超时时间操作系统的超时时间一般为3 minutes- Deadline与Timeout作用类似只不过限制了确定的超时时刻- LocalAddr本地地址TCP四元组的原始IP地址- DualStack(Deprecated)enabled RFC 6555 Fast Fallback Feature- FallbackDelayIPv6连接建立的等待时间如果超时则会切换到IPv4(A negative value disables Fast Fallback support.)- KeepAlive设置了活跃连接的TCP keep-alive探针间隔需要协议层以及操作系统支持(If zero, keep-alive probes are sent with a default value(currently 15 seconds))调用关键流程
step1 - http.NewRequest(method, url string, body io.Reader) 创建请求step2 - http.Client.Do(req *Request) 发送请求接收应答
整个http.Client.Do逻辑分为两道第一道执行send发送请求接收Response关闭Req.Body第二层对请求执行重定向等操作(若需要redirect)并关闭Resp.Bodyhttp.Client.Do(req) send(ireq *Request, rt RoundTripper, deadline time.Time)- setRequestCancel(req, rt, deadline) 设置请求超时时间- http.Client.RoundTrip(req) http.Client.RoundTrip(req) - http.Transport.t.getConn(treq, cm) 获取连接(新创建的 or 复用空闲连接) - http.Transport.queueForIdleConn(w *wantConn) 获取空闲连接- http.Transport.dialConnFor(w *wantConn) - http.Transport.dialConn(ctx context.Context, cm connectMethod) 创建新连接- http.Transport.dial(ctx context.Context, network, addr string) - http.Transport.DialContext(net.Dialer.DialContext)- http.persistConn.readLoop() read http.Response(读取响应内容并构建http.Response)- http.persistConn.writeLoop() write http.Request(发送请求) - http.persistConn.roundTrip(treq) 发送请求读取Response并返回step3 - http.Response.Body.Close() 关闭应答Body
Refs
duyanghao kubernetes-reading-notes
