diff --git a/diagrams/ads.svg b/diagrams/ads.svg deleted file mode 100644 index f2302abd..00000000 --- a/diagrams/ads.svg +++ /dev/null @@ -1,9 +0,0 @@ -participant Envoy as E [color="black"] -participant Management Server as M [color="black"] - -E->M: (V=X,R={},N=A,T=CDS) [color="green"] -M->E: (V=Y,R={foo:...},N=B,T=CDS) [color="gray"] -E->M: (V=,R={foo},N=,T=EDS) [color="green"] -M->E: (V=M,R={foo:...},N=D,T=EDS) [color="gray"] -E->M: (V=M,R={foo},N=D,T=EDS) [color="green"] -E->M: (V=Y,R={},N=B,T=CDS) [color="green"]Created with Raphaël 2.2.0EnvoyEnvoyManagement ServerManagement Server(V=X,R={},N=A,T=CDS)(V=Y,R={foo:...},N=B,T=CDS)(V=,R={foo},N=,T=EDS)(V=M,R={foo:...},N=D,T=EDS)(V=M,R={foo},N=D,T=EDS)(V=Y,R={},N=B,T=CDS) \ No newline at end of file diff --git a/diagrams/cds-eds-resources.svg b/diagrams/cds-eds-resources.svg deleted file mode 100644 index 7fbe958c..00000000 --- a/diagrams/cds-eds-resources.svg +++ /dev/null @@ -1,8 +0,0 @@ -participant Envoy as E [color="black"] -participant Management Server 0 as M0 [color="black"] -participant Management Server 1 as M1 [color="black"] - -E->M1: (V=..,R={},N=..,T=CDS) [color="green"] -E->M0: (V=X,R={foo},N=A,T=EDS) [color="green"] -M1->E: (V=M,R={foo:...,bar:...},N=D,T=CDS) [color="gray"] -E->M0: (V=X,R={foo,bar},N=A,T=EDS [color="green"]Created with Raphaël 2.2.0EnvoyEnvoyManagement Server 0Management Server 0Management Server 1Management Server 1(V=..,R={},N=..,T=CDS)(V=X,R={foo},N=A,T=EDS)(V=M,R={foo:...,bar:...},N=D,T=CDS)(V=X,R={foo,bar},N=A,T=EDS \ No newline at end of file diff --git a/diagrams/eds-distinct-stream.svg b/diagrams/eds-distinct-stream.svg deleted file mode 100644 index d00f0169..00000000 --- a/diagrams/eds-distinct-stream.svg +++ /dev/null @@ -1,10 +0,0 @@ -participant Envoy as E [color="black"] -participant Management Server 0 as M0 [color="black"] -participant Management Server 1 as M1 [color="black"] - -E->M0: (V=X,R={foo},N=A,T=EDS) [color="green"] -E->M1: (V=M,R={bar},N=D,T=EDS) [color="green"] -M0->E: (V=Y,R={foo:...,},N=B,T=EDS) [color="gray"] -E->M0: (V=Y,R={foo},N=B,T=EDS) [color="green"] -M1->E: (V=N,R={bar:...,},N=E,T=EDS) [color="gray"] -E->M1: (V=N,R={bar},N=E,T=EDS) [color="green"]Created with Raphaël 2.2.0EnvoyEnvoyManagement Server 0Management Server 0Management Server 1Management Server 1(V=X,R={foo},N=A,T=EDS)(V=M,R={bar},N=D,T=EDS)(V=Y,R={foo:...,},N=B,T=EDS)(V=Y,R={foo},N=B,T=EDS)(V=N,R={bar:...,},N=E,T=EDS)(V=N,R={bar},N=E,T=EDS) \ No newline at end of file diff --git a/diagrams/eds-same-stream.svg b/diagrams/eds-same-stream.svg deleted file mode 100644 index 1720ed03..00000000 --- a/diagrams/eds-same-stream.svg +++ /dev/null @@ -1,6 +0,0 @@ -participant Envoy as E [color="black"] -participant Management Server as M [color="black"] - -E->M: (V=X,R={foo,bar},N=A,T=EDS) [color="green"] -M->E: (V=Y,R={foo:...,bar:...},N=B,T=EDS) [color="gray"] -E->M: (V=Y,R={foo,bar},N=B,T=EDS) [color="green"]Created with Raphaël 2.2.0EnvoyEnvoyManagement ServerManagement Server(V=X,R={foo,bar},N=A,T=EDS)(V=Y,R={foo:...,bar:...},N=B,T=EDS)(V=Y,R={foo,bar},N=B,T=EDS) \ No newline at end of file diff --git a/diagrams/envoy-perf-script.svg b/diagrams/envoy-perf-script.svg deleted file mode 100644 index 74759e14..00000000 --- a/diagrams/envoy-perf-script.svg +++ /dev/null @@ -1,4795 +0,0 @@ - - - - - - - - - - - - - -Flame Graph - -Reset Zoom -Search - - -__ip_local_out (1 samples, 0.40%) - - - -inet_ehashfn (1 samples, 0.40%) - - - -sock_sendmsg (2 samples, 0.81%) - - - -skb_clone (1 samples, 0.40%) - - - -__netif_receive_skb (4 samples, 1.62%) - - - -ip_local_out (8 samples, 3.24%) -ip_.. - - -do_iter_readv_writev (1 samples, 0.40%) - - - -do_softirq_own_stack (1 samples, 0.40%) - - - -ipv4_mtu (1 samples, 0.40%) - - - -inet_recvmsg (3 samples, 1.21%) - - - -do_writev (19 samples, 7.69%) -do_writev - - -sk_reset_timer (1 samples, 0.40%) - - - -ip_finish_output (7 samples, 2.83%) -ip.. - - -__sk_dst_check (1 samples, 0.40%) - - - -ip_output (7 samples, 2.83%) -ip.. - - -tcmalloc::ThreadCache::ReleaseToCentralCache (1 samples, 0.40%) - - - -tcp_rcv_established (3 samples, 1.21%) - - - -sys_writev (8 samples, 3.24%) -sys.. - - -__libc_readv (5 samples, 2.02%) -_.. - - -tcp_transmit_skb (3 samples, 1.21%) - - - -release_sock (1 samples, 0.40%) - - - -__netif_receive_skb_core (1 samples, 0.40%) - - - -entry_SYSCALL_64_fastpath (2 samples, 0.81%) - - - -ip_queue_xmit (6 samples, 2.43%) -ip.. - - -tcp_send_delayed_ack (1 samples, 0.40%) - - - -tcp_push (7 samples, 2.83%) -tc.. - - -dev_queue_xmit_nit (1 samples, 0.40%) - - - -entry_SYSCALL_64_fastpath (8 samples, 3.24%) -ent.. - - -do_readv_writev (3 samples, 1.21%) - - - -ip_rcv (2 samples, 0.81%) - - - -__netif_receive_skb_core (3 samples, 1.21%) - - - -__softirqentry_text_start (2 samples, 0.81%) - - - -tcp_rcv_established (1 samples, 0.40%) - - - -sock_def_readable (1 samples, 0.40%) - - - -do_readv_writev (7 samples, 2.83%) -do.. - - -tcp_recvmsg (2 samples, 0.81%) - - - -tcp_push (12 samples, 4.86%) -tcp_push - - -ip_rcv (1 samples, 0.40%) - - - -vfs_writev (3 samples, 1.21%) - - - -ip_local_deliver (4 samples, 1.62%) - - - -__local_bh_enable_ip (4 samples, 1.62%) - - - -Envoy::Buffer::WatermarkBuffer::write (65 samples, 26.32%) -Envoy::Buffer::WatermarkBuffer::write - - -tcp_v4_do_rcv (1 samples, 0.40%) - - - -ip_rcv_finish (3 samples, 1.21%) - - - -__tcp_push_pending_frames (3 samples, 1.21%) - - - -sys_writev (13 samples, 5.26%) -sys_wr.. - - -ip_output (2 samples, 0.81%) - - - -__libc_writev (13 samples, 5.26%) -__libc.. - - -tcp_rcv_established (1 samples, 0.40%) - - - -sock_sendmsg (12 samples, 4.86%) -sock_s.. - - -Envoy::Network::ConnectionImpl::doWriteToSocket (98 samples, 39.68%) -Envoy::Network::ConnectionImpl::doWriteToSocket - - -copy_user_enhanced_fast_string (5 samples, 2.02%) -c.. - - -dev_gro_receive (1 samples, 0.40%) - - - -__wake_up_sync_key (3 samples, 1.21%) - - - -dev_queue_xmit (1 samples, 0.40%) - - - -entry_SYSCALL_64_fastpath (2 samples, 0.81%) - - - -tcp_rcv_established (2 samples, 0.81%) - - - -do_iter_readv_writev (1 samples, 0.40%) - - - -do_readv_writev (2 samples, 0.81%) - - - -do_readv_writev (12 samples, 4.86%) -do_rea.. - - -sock_sendmsg (2 samples, 0.81%) - - - -tcp_v4_rcv (4 samples, 1.62%) - - - -net_rx_action (1 samples, 0.40%) - - - -[libc-2.17.so] (1 samples, 0.40%) - - - -ip_queue_xmit (1 samples, 0.40%) - - - -ip_local_deliver_finish (3 samples, 1.21%) - - - -ip_output (6 samples, 2.43%) -ip.. - - -__netif_receive_skb (3 samples, 1.21%) - - - -do_readv_writev (5 samples, 2.02%) -d.. - - -sys_readv (5 samples, 2.02%) -s.. - - -ip_queue_xmit (3 samples, 1.21%) - - - -evbuffer_expand_fast_ (3 samples, 1.21%) - - - -__softirqentry_text_start (1 samples, 0.40%) - - - -netif_receive_skb_internal (1 samples, 0.40%) - - - -sock_read_iter (2 samples, 0.81%) - - - -do_readv_writev (15 samples, 6.07%) -do_readv.. - - -copy_user_enhanced_fast_string (1 samples, 0.40%) - - - -tcp_sendmsg (13 samples, 5.26%) -tcp_se.. - - -ip_finish_output2 (5 samples, 2.02%) -i.. - - -__netif_receive_skb (2 samples, 0.81%) - - - -ip_finish_output (4 samples, 1.62%) - - - -__libc_writev (8 samples, 3.24%) -__l.. - - -[unknown] (3 samples, 1.21%) - - - -__libc_readv (2 samples, 0.81%) - - - -do_readv (2 samples, 0.81%) - - - -__netif_receive_skb (4 samples, 1.62%) - - - -_raw_spin_lock_bh (1 samples, 0.40%) - - - -sock_def_readable (1 samples, 0.40%) - - - -ip_output (8 samples, 3.24%) -ip_.. - - -ipv4_dst_check (1 samples, 0.40%) - - - -tcp_sendmsg (11 samples, 4.45%) -tcp_s.. - - -tcp_recvmsg (1 samples, 0.40%) - - - -do_softirq (4 samples, 1.62%) - - - -[unknown] (19 samples, 7.69%) -[unknown] - - -skb_entail (1 samples, 0.40%) - - - -process_backlog (3 samples, 1.21%) - - - -__lock_text_start (1 samples, 0.40%) - - - -do_iter_readv_writev (2 samples, 0.81%) - - - -ip_finish_output (2 samples, 0.81%) - - - -__dev_queue_xmit (3 samples, 1.21%) - - - -ip_rcv_finish (2 samples, 0.81%) - - - -ip_output (1 samples, 0.40%) - - - -ip_finish_output2 (6 samples, 2.43%) -ip.. - - -copy_from_iter (1 samples, 0.40%) - - - -inet_sendmsg (2 samples, 0.81%) - - - -entry_SYSCALL_64_fastpath (19 samples, 7.69%) -entry_SYSC.. - - -entry_SYSCALL_64_fastpath (13 samples, 5.26%) -entry_.. - - -tcp_send_ack (1 samples, 0.40%) - - - -inet_recvmsg (1 samples, 0.40%) - - - -__GI___ioctl (1 samples, 0.40%) - - - -ip_finish_output2 (4 samples, 1.62%) - - - -do_writev (4 samples, 1.62%) - - - -inet_sendmsg (1 samples, 0.40%) - - - -ip_local_out (10 samples, 4.05%) -ip_l.. - - -sys_writev (1 samples, 0.40%) - - - -__tcp_ack_snd_check (1 samples, 0.40%) - - - -tcp_recvmsg (1 samples, 0.40%) - - - -sock_sendmsg (1 samples, 0.40%) - - - -copy_user_enhanced_fast_string (2 samples, 0.81%) - - - -do_readv_writev (19 samples, 7.69%) -do_readv_w.. - - -sys_writev (33 samples, 13.36%) -sys_writev - - -tcp_push_one (2 samples, 0.81%) - - - -tcp_transmit_skb (1 samples, 0.40%) - - - -fsnotify (1 samples, 0.40%) - - - -netif_rx_internal (1 samples, 0.40%) - - - -dev_hard_start_xmit (1 samples, 0.40%) - - - -ip_queue_xmit (1 samples, 0.40%) - - - -process_backlog (2 samples, 0.81%) - - - -do_softirq_own_stack (6 samples, 2.43%) -do.. - - -ip_local_deliver_finish (1 samples, 0.40%) - - - -do_writev (15 samples, 6.07%) -do_writev - - -__dev_queue_xmit (2 samples, 0.81%) - - - -copy_user_enhanced_fast_string (11 samples, 4.45%) -copy_.. - - -process_backlog (2 samples, 0.81%) - - - -Envoy::Network::ClientConnectionImpl::~ClientConnectionImpl (1 samples, 0.40%) - - - -ip_rcv_finish (3 samples, 1.21%) - - - -loopback_xmit (3 samples, 1.21%) - - - -do_writev (17 samples, 6.88%) -do_writev - - -do_readv_writev (17 samples, 6.88%) -do_readv_.. - - -__local_bh_enable_ip (1 samples, 0.40%) - - - -ip_local_deliver (3 samples, 1.21%) - - - -do_readv (2 samples, 0.81%) - - - -entry_SYSCALL_64_fastpath (2 samples, 0.81%) - - - -tcp_transmit_skb (6 samples, 2.43%) -tc.. - - -__netif_receive_skb_core (2 samples, 0.81%) - - - -tcp_sendmsg (17 samples, 6.88%) -tcp_sendmsg - - -sk_page_frag_refill (1 samples, 0.40%) - - - -__tcp_push_pending_frames (3 samples, 1.21%) - - - -tcp_transmit_skb (1 samples, 0.40%) - - - -sock_read_iter (3 samples, 1.21%) - - - -ip_rcv_finish (3 samples, 1.21%) - - - -ip_rcv (3 samples, 1.21%) - - - -__tcp_ack_snd_check (1 samples, 0.40%) - - - -__dev_queue_xmit (1 samples, 0.40%) - - - -sys_writev (15 samples, 6.07%) -sys_writev - - -queued_spin_lock_slowpath (1 samples, 0.40%) - - - -ip_local_deliver (1 samples, 0.40%) - - - -tcp_push (7 samples, 2.83%) -tc.. - - -do_iter_readv_writev (4 samples, 1.62%) - - - -sock_recvmsg (4 samples, 1.62%) - - - -xen_clocksource_read (1 samples, 0.40%) - - - -process_backlog (1 samples, 0.40%) - - - -do_readv (2 samples, 0.81%) - - - -__libc_writev (7 samples, 2.83%) -__.. - - -sys_readv (1 samples, 0.40%) - - - -__dev_queue_xmit (1 samples, 0.40%) - - - -ip_finish_output2 (1 samples, 0.40%) - - - -copy_user_enhanced_fast_string (2 samples, 0.81%) - - - -__skb_clone (1 samples, 0.40%) - - - -tcp_rcv_established (1 samples, 0.40%) - - - -ip_finish_output (1 samples, 0.40%) - - - -ip_rcv_finish (2 samples, 0.81%) - - - -pvclock_clocksource_read (1 samples, 0.40%) - - - -skb_release_all (1 samples, 0.40%) - - - -entry_SYSCALL_64_fastpath (1 samples, 0.40%) - - - -vfs_readv (3 samples, 1.21%) - - - -ip_finish_output (1 samples, 0.40%) - - - -do_softirq_own_stack (1 samples, 0.40%) - - - -__lock_text_start (1 samples, 0.40%) - - - -copy_from_iter (1 samples, 0.40%) - - - -vfs_writev (17 samples, 6.88%) -vfs_writev - - -evbuffer_get_length (1 samples, 0.40%) - - - -__softirqentry_text_start (1 samples, 0.40%) - - - -inet_sendmsg (6 samples, 2.43%) -in.. - - -evbuffer_drain (3 samples, 1.21%) - - - -sock_recvmsg (3 samples, 1.21%) - - - -do_softirq_own_stack (3 samples, 1.21%) - - - -netif_rx (1 samples, 0.40%) - - - -ip_finish_output (1 samples, 0.40%) - - - -__tcp_push_pending_frames (7 samples, 2.83%) -__.. - - -__tcp_v4_send_check (1 samples, 0.40%) - - - -__softirqentry_text_start (1 samples, 0.40%) - - - -sock_sendmsg (6 samples, 2.43%) -so.. - - -[unknown] (19 samples, 7.69%) -[unknown] - - -sk_stream_alloc_skb (1 samples, 0.40%) - - - -netif_rx (1 samples, 0.40%) - - - -tcp_v4_do_rcv (5 samples, 2.02%) -t.. - - -tcp_transmit_skb (1 samples, 0.40%) - - - -__tcp_push_pending_frames (8 samples, 3.24%) -__t.. - - -ip_finish_output2 (2 samples, 0.81%) - - - -ip_output (2 samples, 0.81%) - - - -skb_page_frag_refill (2 samples, 0.81%) - - - -ip_local_out (6 samples, 2.43%) -ip.. - - -Envoy::Network::ConnectionImpl::onFileEvent (164 samples, 66.40%) -Envoy::Network::ConnectionImpl::onFileEvent - - -tcp_write_xmit (1 samples, 0.40%) - - - -process_backlog (3 samples, 1.21%) - - - -dev_hard_start_xmit (1 samples, 0.40%) - - - -entry_SYSCALL_64_fastpath (3 samples, 1.21%) - - - -tcp_rcv_established (2 samples, 0.81%) - - - -net_rx_action (3 samples, 1.21%) - - - -tcp_transmit_skb (10 samples, 4.05%) -tcp_.. - - -tcp_rcv_established (1 samples, 0.40%) - - - -__lock_text_start (1 samples, 0.40%) - - - -process_backlog (1 samples, 0.40%) - - - -tcp_v4_rcv (2 samples, 0.81%) - - - -tcp_push (5 samples, 2.02%) -t.. - - -do_softirq_own_stack (1 samples, 0.40%) - - - -tcp_push (5 samples, 2.02%) -t.. - - -entry_SYSCALL_64_fastpath (1 samples, 0.40%) - - - -ip_local_out (1 samples, 0.40%) - - - -sock_def_readable (2 samples, 0.81%) - - - -do_softirq_own_stack (2 samples, 0.81%) - - - -__lock_text_start (1 samples, 0.40%) - - - -__lock_text_start (4 samples, 1.62%) - - - -tcp_write_xmit (10 samples, 4.05%) -tcp_.. - - -__libc_writev (2 samples, 0.81%) - - - -ip_finish_output2 (1 samples, 0.40%) - - - -do_iter_readv_writev (1 samples, 0.40%) - - - -tcp_stream_memory_free (1 samples, 0.40%) - - - -__lock_text_start (3 samples, 1.21%) - - - -do_softirq_own_stack (2 samples, 0.81%) - - - -ip_finish_output (1 samples, 0.40%) - - - -__tcp_push_pending_frames (1 samples, 0.40%) - - - -do_iter_readv_writev (3 samples, 1.21%) - - - -tcp_transmit_skb (1 samples, 0.40%) - - - -skb_entail (1 samples, 0.40%) - - - -tcp_send_delayed_ack (1 samples, 0.40%) - - - -ip_finish_output (5 samples, 2.02%) -i.. - - -tcp_v4_rcv (4 samples, 1.62%) - - - -sock_write_iter (13 samples, 5.26%) -sock_w.. - - -sock_write_iter (3 samples, 1.21%) - - - -ip_local_deliver (4 samples, 1.62%) - - - -ip_local_deliver (1 samples, 0.40%) - - - -process_backlog (3 samples, 1.21%) - - - -tcp_send_ack (1 samples, 0.40%) - - - -ip_local_deliver_finish (3 samples, 1.21%) - - - -ip_queue_xmit (1 samples, 0.40%) - - - -enqueue_to_backlog (1 samples, 0.40%) - - - -dev_hard_start_xmit (2 samples, 0.81%) - - - -do_softirq (2 samples, 0.81%) - - - -do_iter_readv_writev (17 samples, 6.88%) -do_iter_r.. - - -ip_rcv (6 samples, 2.43%) -ip.. - - -xen_hvm_callback_vector (1 samples, 0.40%) - - - -sock_def_readable (1 samples, 0.40%) - - - -tcp_v4_do_rcv (3 samples, 1.21%) - - - -__bpf_prog_run (1 samples, 0.40%) - - - -__alloc_pages_nodemask (1 samples, 0.40%) - - - -__dev_queue_xmit (4 samples, 1.62%) - - - -do_softirq (1 samples, 0.40%) - - - -do_iter_readv_writev (1 samples, 0.40%) - - - -net_rx_action (1 samples, 0.40%) - - - -inet_sendmsg (2 samples, 0.81%) - - - -sys_readv (1 samples, 0.40%) - - - -sk_filter_trim_cap (1 samples, 0.40%) - - - -sock_write_iter (3 samples, 1.21%) - - - -__lock_text_start (1 samples, 0.40%) - - - -ip_output (3 samples, 1.21%) - - - -tcp_data_queue (1 samples, 0.40%) - - - -__lock_text_start (1 samples, 0.40%) - - - -ip_output (1 samples, 0.40%) - - - -ip_local_deliver (3 samples, 1.21%) - - - -dev_queue_xmit (1 samples, 0.40%) - - - -ep_poll (1 samples, 0.40%) - - - -ip_rcv (3 samples, 1.21%) - - - -dev_hard_start_xmit (2 samples, 0.81%) - - - -vfs_writev (3 samples, 1.21%) - - - -skb_copy_datagram_iter (1 samples, 0.40%) - - - -__lock_text_start (2 samples, 0.81%) - - - -__libc_writev (6 samples, 2.43%) -__.. - - -ip_output (1 samples, 0.40%) - - - -ip_rcv_finish (5 samples, 2.02%) -i.. - - -sched_clock_cpu (1 samples, 0.40%) - - - -__GI___ioctl (1 samples, 0.40%) - - - -napi_gro_receive (1 samples, 0.40%) - - - -dev_queue_xmit (1 samples, 0.40%) - - - -tcp_write_xmit (1 samples, 0.40%) - - - -sys_writev (3 samples, 1.21%) - - - -ip_local_deliver (2 samples, 0.81%) - - - -__dev_queue_xmit (2 samples, 0.81%) - - - -tcp_write_xmit (7 samples, 2.83%) -tc.. - - -__local_bh_enable_ip (6 samples, 2.43%) -__.. - - -do_softirq_own_stack (2 samples, 0.81%) - - - -event_base_loop (189 samples, 76.52%) -event_base_loop - - -__local_bh_enable_ip (2 samples, 0.81%) - - - -dev_hard_start_xmit (1 samples, 0.40%) - - - -xen_evtchn_do_upcall (1 samples, 0.40%) - - - -dev_queue_xmit (1 samples, 0.40%) - - - -__netif_receive_skb (2 samples, 0.81%) - - - -sk_stream_alloc_skb (1 samples, 0.40%) - - - -Envoy::Network::ConnectionImpl::onWriteReady (141 samples, 57.09%) -Envoy::Network::ConnectionImpl::onWriteReady - - -do_readv_writev (2 samples, 0.81%) - - - -ip_queue_xmit (8 samples, 3.24%) -ip_.. - - -ip_finish_output (1 samples, 0.40%) - - - -entry_SYSCALL_64_fastpath (1 samples, 0.40%) - - - -do_writev (19 samples, 7.69%) -do_writev - - -ip_finish_output (8 samples, 3.24%) -ip_.. - - -sock_def_readable (4 samples, 1.62%) - - - -tcp_sendmsg (1 samples, 0.40%) - - - -tcp_transmit_skb (1 samples, 0.40%) - - - -Envoy::Network::FilterManagerImpl::onContinueReading (4 samples, 1.62%) - - - -process_backlog (4 samples, 1.62%) - - - -tcp_v4_rcv (1 samples, 0.40%) - - - -do_softirq (1 samples, 0.40%) - - - -sock_def_readable (1 samples, 0.40%) - - - -do_readv_writev (1 samples, 0.40%) - - - -ip_local_out (3 samples, 1.21%) - - - -epoll_dispatch (1 samples, 0.40%) - - - -__xfrm_policy_check2.constprop.43 (1 samples, 0.40%) - - - -vfs_writev (24 samples, 9.72%) -vfs_writev - - -__tcp_ack_snd_check (1 samples, 0.40%) - - - -ip_queue_xmit (1 samples, 0.40%) - - - -ip_queue_xmit (8 samples, 3.24%) -ip_.. - - -__tcp_ack_snd_check (1 samples, 0.40%) - - - -tcp_write_xmit (11 samples, 4.45%) -tcp_w.. - - -ip_finish_output2 (1 samples, 0.40%) - - - -tcp_v4_rcv (3 samples, 1.21%) - - - -ip_queue_xmit (4 samples, 1.62%) - - - -__libc_writev (1 samples, 0.40%) - - - -process_backlog (2 samples, 0.81%) - - - -tcp_v4_do_rcv (1 samples, 0.40%) - - - -process_backlog (5 samples, 2.02%) -p.. - - -__libc_readv (2 samples, 0.81%) - - - -ip_local_deliver_finish (1 samples, 0.40%) - - - -do_readv_writev (19 samples, 7.69%) -do_readv_w.. - - -[unknown] (1 samples, 0.40%) - - - -tcp_transmit_skb (1 samples, 0.40%) - - - -tcp_rcv_established (3 samples, 1.21%) - - - -__netif_receive_skb (1 samples, 0.40%) - - - -sock_read_iter (1 samples, 0.40%) - - - -tcp_rcv_established (2 samples, 0.81%) - - - -entry_SYSCALL_64_fastpath (2 samples, 0.81%) - - - -tcp_sendmsg (4 samples, 1.62%) - - - -Envoy::Network::ConnectionImplUtility::updateBufferStats (1 samples, 0.40%) - - - -loopback_xmit (1 samples, 0.40%) - - - -entry_SYSCALL_64_fastpath (13 samples, 5.26%) -entry_.. - - -tcp_transmit_skb (1 samples, 0.40%) - - - -do_writev (3 samples, 1.21%) - - - -[unknown] (36 samples, 14.57%) -[unknown] - - -process_backlog (5 samples, 2.02%) -p.. - - -tcp_push_one (1 samples, 0.40%) - - - -__skb_clone (1 samples, 0.40%) - - - -sock_write_iter (2 samples, 0.81%) - - - -xen_clocksource_get_cycles (1 samples, 0.40%) - - - -__netif_receive_skb (1 samples, 0.40%) - - - -tcp_transmit_skb (10 samples, 4.05%) -tcp_.. - - -do_readv_writev (8 samples, 3.24%) -do_.. - - -tcp_sendmsg (6 samples, 2.43%) -tc.. - - -_ZZN5Envoy6Thread6ThreadC4ESt8functionIFvvEEENUlPvE_4_FUNES5_ (211 samples, 85.43%) -_ZZN5Envoy6Thread6ThreadC4ESt8functionIFvvEEENUlPvE_4_FUNES5_ - - -__netif_receive_skb (6 samples, 2.43%) -__.. - - -skb_page_frag_refill (1 samples, 0.40%) - - - -ip_rcv_finish (3 samples, 1.21%) - - - -tcp_transmit_skb (7 samples, 2.83%) -tc.. - - -sock_recvmsg (2 samples, 0.81%) - - - -skb_clone (1 samples, 0.40%) - - - -__softirqentry_text_start (4 samples, 1.62%) - - - -entry_SYSCALL_64_fastpath (6 samples, 2.43%) -en.. - - -tcp_v4_do_rcv (2 samples, 0.81%) - - - -tcp_v4_do_rcv (3 samples, 1.21%) - - - -vfs_writev (2 samples, 0.81%) - - - -__lock_text_start (1 samples, 0.40%) - - - -ep_scan_ready_list.isra.10 (1 samples, 0.40%) - - - -net_rx_action (2 samples, 0.81%) - - - -tcp_sendmsg (1 samples, 0.40%) - - - -ip_local_deliver (2 samples, 0.81%) - - - -tcp_v4_do_rcv (1 samples, 0.40%) - - - -ip_local_deliver (1 samples, 0.40%) - - - -irq_exit (1 samples, 0.40%) - - - -security_socket_sendmsg (1 samples, 0.40%) - - - -entry_SYSCALL_64_fastpath (19 samples, 7.69%) -entry_SYSC.. - - -sys_writev (19 samples, 7.69%) -sys_writev - - -__wake_up_sync_key (1 samples, 0.40%) - - - -inet_recvmsg (2 samples, 0.81%) - - - -do_iter_readv_writev (32 samples, 12.96%) -do_iter_readv_writev - - -sk_reset_timer (1 samples, 0.40%) - - - -swiotlb_dma_mapping_error (1 samples, 0.40%) - - - -sys_writev (13 samples, 5.26%) -sys_wr.. - - -do_readv_writev (4 samples, 1.62%) - - - -copy_user_enhanced_fast_string (2 samples, 0.81%) - - - -ip_finish_output2 (4 samples, 1.62%) - - - -dev_queue_xmit (1 samples, 0.40%) - - - -ip_rcv (2 samples, 0.81%) - - - -do_writev (1 samples, 0.40%) - - - -__libc_readv (1 samples, 0.40%) - - - -tcp_recvmsg (4 samples, 1.62%) - - - -__alloc_skb (1 samples, 0.40%) - - - -__netif_receive_skb (3 samples, 1.21%) - - - -tcp_sendmsg (12 samples, 4.86%) -tcp_se.. - - -__netif_receive_skb_core (2 samples, 0.81%) - - - -entry_SYSCALL_64_fastpath (2 samples, 0.81%) - - - -__netif_receive_skb_core (3 samples, 1.21%) - - - -tcp_init_tso_segs (1 samples, 0.40%) - - - -do_readv_writev (2 samples, 0.81%) - - - -event_process_active_single_queue.isra.29 (167 samples, 67.61%) -event_process_active_single_queue.isra.29 - - -do_readv (1 samples, 0.40%) - - - -rw_verify_area (1 samples, 0.40%) - - - -ip_queue_xmit (4 samples, 1.62%) - - - -tcp_write_xmit (3 samples, 1.21%) - - - -tcp_init_tso_segs (1 samples, 0.40%) - - - -ip_local_out (2 samples, 0.81%) - - - -__kmalloc_reserve.isra.37 (1 samples, 0.40%) - - - -__kfree_skb (1 samples, 0.40%) - - - -sock_recvmsg (2 samples, 0.81%) - - - -loopback_xmit (1 samples, 0.40%) - - - -ip_rcv_finish (3 samples, 1.21%) - - - -tcp_tso_segs (1 samples, 0.40%) - - - -copy_from_iter (1 samples, 0.40%) - - - -__softirqentry_text_start (5 samples, 2.02%) -_.. - - -rw_verify_area (1 samples, 0.40%) - - - -sock_write_iter (7 samples, 2.83%) -so.. - - -[unknown] (3 samples, 1.21%) - - - -sock_write_iter (6 samples, 2.43%) -so.. - - -ip_local_out (1 samples, 0.40%) - - - -__release_sock (1 samples, 0.40%) - - - -event_active (18 samples, 7.29%) -event_active - - -sock_def_readable (1 samples, 0.40%) - - - -inet_sendmsg (8 samples, 3.24%) -ine.. - - -net_rx_action (5 samples, 2.02%) -n.. - - -tcp_write_xmit (7 samples, 2.83%) -tc.. - - -tcp_push (15 samples, 6.07%) -tcp_push - - -loopback_xmit (2 samples, 0.81%) - - - -lock_sock_nested (1 samples, 0.40%) - - - -do_softirq (4 samples, 1.62%) - - - -sock_read_iter (4 samples, 1.62%) - - - -tcp_sendmsg (8 samples, 3.24%) -tcp.. - - -do_softirq (3 samples, 1.21%) - - - -ip_local_out (2 samples, 0.81%) - - - -napi_gro_complete (1 samples, 0.40%) - - - -evbuffer_chain_new (3 samples, 1.21%) - - - -tcp_transmit_skb (10 samples, 4.05%) -tcp_.. - - -vfs_readv (2 samples, 0.81%) - - - -sched_clock_local (1 samples, 0.40%) - - - -tcp_v4_rcv (1 samples, 0.40%) - - - -vfs_writev (12 samples, 4.86%) -vfs_wr.. - - -sock_sendmsg (4 samples, 1.62%) - - - -sys_writev (4 samples, 1.62%) - - - -do_readv_writev (1 samples, 0.40%) - - - -irq_exit (1 samples, 0.40%) - - - -tcp_transmit_skb (2 samples, 0.81%) - - - -dev_queue_xmit (3 samples, 1.21%) - - - -Envoy::Event::FileEventImpl::assignEvents (164 samples, 66.40%) -Envoy::Event::FileEventImpl::assignEvents - - -tcp_sendmsg (15 samples, 6.07%) -tcp_send.. - - -ip_finish_output2 (1 samples, 0.40%) - - - -ip_output (4 samples, 1.62%) - - - -do_readv_writev (13 samples, 5.26%) -do_rea.. - - -do_softirq_own_stack (1 samples, 0.40%) - - - -__netif_receive_skb_core (1 samples, 0.40%) - - - -copy_user_enhanced_fast_string (2 samples, 0.81%) - - - -ip_queue_xmit (6 samples, 2.43%) -ip.. - - -tcp_send_delayed_ack (1 samples, 0.40%) - - - -copy_user_enhanced_fast_string (1 samples, 0.40%) - - - -dev_queue_xmit (1 samples, 0.40%) - - - -lock_timer_base (1 samples, 0.40%) - - - -sys_readv (2 samples, 0.81%) - - - -__local_bh_enable_ip (4 samples, 1.62%) - - - -ktime_get_with_offset (1 samples, 0.40%) - - - -tcp_transmit_skb (1 samples, 0.40%) - - - -__softirqentry_text_start (4 samples, 1.62%) - - - -skb_clone (1 samples, 0.40%) - - - -tcp_sendmsg (24 samples, 9.72%) -tcp_sendmsg - - -entry_SYSCALL_64_fastpath (3 samples, 1.21%) - - - -__wake_up_sync_key (1 samples, 0.40%) - - - -tcp_push (15 samples, 6.07%) -tcp_push - - -dev_hard_start_xmit (2 samples, 0.81%) - - - -copy_user_enhanced_fast_string (1 samples, 0.40%) - - - -Envoy::Filter::TcpProxy::onData (1 samples, 0.40%) - - - -ip_finish_output (10 samples, 4.05%) -ip_f.. - - -tcp_wfree (1 samples, 0.40%) - - - -__local_bh_enable_ip (4 samples, 1.62%) - - - -__put_page (1 samples, 0.40%) - - - -ip_local_out (7 samples, 2.83%) -ip.. - - -ip_local_deliver_finish (1 samples, 0.40%) - - - -ip_output (1 samples, 0.40%) - - - -ip_finish_output2 (6 samples, 2.43%) -ip.. - - -process_backlog (1 samples, 0.40%) - - - -do_readv (5 samples, 2.02%) -d.. - - -do_writev (24 samples, 9.72%) -do_writev - - -__netif_receive_skb_core (3 samples, 1.21%) - - - -ip_finish_output2 (7 samples, 2.83%) -ip.. - - -tcp_rcv_established (1 samples, 0.40%) - - - -tcp_sendmsg (27 samples, 10.93%) -tcp_sendmsg - - -ip_local_deliver (5 samples, 2.02%) -i.. - - -inet_sendmsg (15 samples, 6.07%) -inet_sen.. - - -copy_user_enhanced_fast_string (2 samples, 0.81%) - - - -ip_rcv (2 samples, 0.81%) - - - -ixgbevf_clean_rx_irq (1 samples, 0.40%) - - - -tcp_recvmsg (3 samples, 1.21%) - - - -tcp_v4_do_rcv (3 samples, 1.21%) - - - -dev_queue_xmit (4 samples, 1.62%) - - - -tcp_rcv_established (1 samples, 0.40%) - - - -[unknown] (3 samples, 1.21%) - - - -__netif_receive_skb_core (5 samples, 2.02%) -_.. - - -tcp_write_xmit (3 samples, 1.21%) - - - -__softirqentry_text_start (4 samples, 1.62%) - - - -ip_finish_output (8 samples, 3.24%) -ip_.. - - -__tcp_push_pending_frames (15 samples, 6.07%) -__tcp_pu.. - - -sock_def_readable (3 samples, 1.21%) - - - -tcp_nagle_check (1 samples, 0.40%) - - - -net_rx_action (4 samples, 1.62%) - - - -tcp_rcv_established (4 samples, 1.62%) - - - -ip_finish_output (2 samples, 0.81%) - - - -vfs_readv (1 samples, 0.40%) - - - -tcp_v4_do_rcv (3 samples, 1.21%) - - - -tcp_transmit_skb (1 samples, 0.40%) - - - -loopback_xmit (2 samples, 0.81%) - - - -netif_rx (1 samples, 0.40%) - - - -skb_put (1 samples, 0.40%) - - - -sched_clock_cpu (1 samples, 0.40%) - - - -tcp_v4_rcv (1 samples, 0.40%) - - - -__softirqentry_text_start (1 samples, 0.40%) - - - -net_rx_action (3 samples, 1.21%) - - - -tcp_transmit_skb (14 samples, 5.67%) -tcp_tra.. - - -net_rx_action (4 samples, 1.62%) - - - -tcp_rcv_established (1 samples, 0.40%) - - - -__lock_text_start (2 samples, 0.81%) - - - -xen_evtchn_do_upcall (1 samples, 0.40%) - - - -do_iter_readv_writev (24 samples, 9.72%) -do_iter_readv_.. - - -tcp_v4_do_rcv (4 samples, 1.62%) - - - -__tcp_push_pending_frames (11 samples, 4.45%) -__tcp.. - - -ip_finish_output2 (1 samples, 0.40%) - - - -entry_SYSCALL_64_fastpath (7 samples, 2.83%) -en.. - - -ip_finish_output2 (10 samples, 4.05%) -ip_f.. - - -__local_bh_enable_ip (2 samples, 0.81%) - - - -do_softirq (4 samples, 1.62%) - - - -vfs_readv (2 samples, 0.81%) - - - -ip_rcv_finish (1 samples, 0.40%) - - - -sys_writev (17 samples, 6.88%) -sys_writev - - -__netif_receive_skb_core (1 samples, 0.40%) - - - -tcp_sendmsg (3 samples, 1.21%) - - - -__wake_up_sync_key (1 samples, 0.40%) - - - -__free_pages_ok (1 samples, 0.40%) - - - -inet_recvmsg (1 samples, 0.40%) - - - -ip_local_deliver (2 samples, 0.81%) - - - -vfs_writev (13 samples, 5.26%) -vfs_wr.. - - -Envoy::Network::ConnectionImpl::write (1 samples, 0.40%) - - - -ip_rcv (4 samples, 1.62%) - - - -spdlog::logger::log<unsigned long> (1 samples, 0.40%) - - - -vfs_writev (7 samples, 2.83%) -vf.. - - -vfs_writev (6 samples, 2.43%) -vf.. - - -gettime (1 samples, 0.40%) - - - -tcp_v4_rcv (2 samples, 0.81%) - - - -do_softirq (1 samples, 0.40%) - - - -__softirqentry_text_start (2 samples, 0.81%) - - - -__tcp_push_pending_frames (1 samples, 0.40%) - - - -copy_user_enhanced_fast_string (1 samples, 0.40%) - - - -sock_def_readable (4 samples, 1.62%) - - - -ip_local_deliver_finish (3 samples, 1.21%) - - - -sk_free (1 samples, 0.40%) - - - -ip_queue_xmit (11 samples, 4.45%) -ip_qu.. - - -tcp_v4_rcv (3 samples, 1.21%) - - - -do_writev (12 samples, 4.86%) -do_wri.. - - -tcp_rcv_established (1 samples, 0.40%) - - - -sched_clock (1 samples, 0.40%) - - - -ip_finish_output2 (2 samples, 0.81%) - - - -__release_sock (1 samples, 0.40%) - - - -__wake_up_sync_key (1 samples, 0.40%) - - - -__libc_readv (2 samples, 0.81%) - - - -__netif_receive_skb (1 samples, 0.40%) - - - -alloc_pages_current (2 samples, 0.81%) - - - -clock_gettime (1 samples, 0.40%) - - - -tcp_v4_do_rcv (2 samples, 0.81%) - - - -tcp_tso_segs (1 samples, 0.40%) - - - -sock_sendmsg (15 samples, 6.07%) -sock_sen.. - - -tcp_rcv_established (1 samples, 0.40%) - - - -__softirqentry_text_start (1 samples, 0.40%) - - - -ip_local_out (8 samples, 3.24%) -ip_.. - - -start_thread (211 samples, 85.43%) -start_thread - - -__softirqentry_text_start (1 samples, 0.40%) - - - -__softirqentry_text_start (3 samples, 1.21%) - - - -vfs_readv (2 samples, 0.81%) - - - -[unknown] (3 samples, 1.21%) - - - -sock_sendmsg (8 samples, 3.24%) -soc.. - - -do_softirq (1 samples, 0.40%) - - - -ip_queue_xmit (10 samples, 4.05%) -ip_q.. - - -Envoy::Network::ConnectionImpl::write (21 samples, 8.50%) -Envoy::Netwo.. - - -__local_bh_enable_ip (5 samples, 2.02%) -_.. - - -do_iter_readv_writev (19 samples, 7.69%) -do_iter_re.. - - -vfs_writev (1 samples, 0.40%) - - - -ixgbevf_clean_rx_irq (1 samples, 0.40%) - - - -pvclock_clocksource_read (1 samples, 0.40%) - - - -vfs_writev (19 samples, 7.69%) -vfs_writev - - -do_readv (2 samples, 0.81%) - - - -__alloc_skb (1 samples, 0.40%) - - - -do_writev (2 samples, 0.81%) - - - -__netif_receive_skb_core (1 samples, 0.40%) - - - -do_iter_readv_writev (12 samples, 4.86%) -do_ite.. - - -sched_clock_cpu (1 samples, 0.40%) - - - -__copy_skb_header (1 samples, 0.40%) - - - -do_iter_readv_writev (8 samples, 3.24%) -do_.. - - -[unknown] (2 samples, 0.81%) - - - -__softirqentry_text_start (3 samples, 1.21%) - - - -ip_local_deliver_finish (4 samples, 1.62%) - - - -ip_output (4 samples, 1.62%) - - - -skb_release_data (1 samples, 0.40%) - - - -process_backlog (6 samples, 2.43%) -pr.. - - -vfs_writev (8 samples, 3.24%) -vfs.. - - -__skb_clone (1 samples, 0.40%) - - - -do_softirq_own_stack (5 samples, 2.02%) -d.. - - -ip_finish_output2 (1 samples, 0.40%) - - - -sock_write_iter (12 samples, 4.86%) -sock_w.. - - -[unknown] (17 samples, 6.88%) -[unknown] - - -_raw_spin_lock_bh (1 samples, 0.40%) - - - -net_rx_action (3 samples, 1.21%) - - - -entry_SYSCALL_64_fastpath (17 samples, 6.88%) -entry_SYS.. - - -net_rx_action (3 samples, 1.21%) - - - -sock_write_iter (24 samples, 9.72%) -sock_write_iter - - -ip_queue_xmit (1 samples, 0.40%) - - - -do_softirq_own_stack (1 samples, 0.40%) - - - -sock_def_readable (1 samples, 0.40%) - - - -do_iter_readv_writev (2 samples, 0.81%) - - - -mod_timer (1 samples, 0.40%) - - - -tcp_push (8 samples, 3.24%) -tcp.. - - -sk_free (1 samples, 0.40%) - - - -inet_sendmsg (13 samples, 5.26%) -inet_s.. - - -__libc_disable_asynccancel (1 samples, 0.40%) - - - -ip_local_out (1 samples, 0.40%) - - - -spdlog::logger::log<unsigned long, unsigned long> (1 samples, 0.40%) - - - -sock_write_iter (1 samples, 0.40%) - - - -__libc_writev (3 samples, 1.21%) - - - -netif_rx_internal (1 samples, 0.40%) - - - -entry_SYSCALL_64_fastpath (1 samples, 0.40%) - - - -tcp_rcv_established (4 samples, 1.62%) - - - -ip_output (2 samples, 0.81%) - - - -__pv_queued_spin_lock_slowpath (1 samples, 0.40%) - - - -dev_hard_start_xmit (1 samples, 0.40%) - - - -sched_clock (1 samples, 0.40%) - - - -ip_finish_output2 (1 samples, 0.40%) - - - -ip_finish_output (6 samples, 2.43%) -ip.. - - -__release_sock (1 samples, 0.40%) - - - -do_writev (6 samples, 2.43%) -do.. - - -ip_local_deliver_finish (3 samples, 1.21%) - - - -netif_rx_internal (1 samples, 0.40%) - - - -ip_queue_xmit (2 samples, 0.81%) - - - -__kfree_skb (1 samples, 0.40%) - - - -ip_local_deliver (2 samples, 0.81%) - - - -__wake_up_sync_key (1 samples, 0.40%) - - - -tcp_write_xmit (10 samples, 4.05%) -tcp_.. - - -__dev_queue_xmit (1 samples, 0.40%) - - - -vfs_readv (5 samples, 2.02%) -v.. - - -__local_bh_enable_ip (1 samples, 0.40%) - - - -sock_def_readable (2 samples, 0.81%) - - - -sys_readv (2 samples, 0.81%) - - - -__softirqentry_text_start (4 samples, 1.62%) - - - -ip_queue_xmit (2 samples, 0.81%) - - - -__tcp_push_pending_frames (2 samples, 0.81%) - - - -inet_sendmsg (12 samples, 4.86%) -inet_s.. - - -tcp_rcv_established (1 samples, 0.40%) - - - -tcp_rcv_established (2 samples, 0.81%) - - - -net_rx_action (6 samples, 2.43%) -ne.. - - -ip_finish_output2 (6 samples, 2.43%) -ip.. - - -bictcp_cwnd_event (1 samples, 0.40%) - - - -evbuffer_expand_fast_ (12 samples, 4.86%) -evbuff.. - - -tcp_v4_do_rcv (1 samples, 0.40%) - - - -__local_bh_enable_ip (1 samples, 0.40%) - - - -inet_sendmsg (12 samples, 4.86%) -inet_s.. - - -ip_local_out (1 samples, 0.40%) - - - -sock_sendmsg (32 samples, 12.96%) -sock_sendmsg - - -eth_type_trans (1 samples, 0.40%) - - - -[unknown] (1 samples, 0.40%) - - - -__libc_writev (19 samples, 7.69%) -__libc_wri.. - - -__libc_writev (24 samples, 9.72%) -__libc_writev - - -copy_user_enhanced_fast_string (5 samples, 2.02%) -c.. - - -ip_local_deliver_finish (2 samples, 0.81%) - - - -tcp_transmit_skb (6 samples, 2.43%) -tc.. - - -tcp_transmit_skb (1 samples, 0.40%) - - - -do_readv (2 samples, 0.81%) - - - -do_writev (13 samples, 5.26%) -do_wri.. - - -do_readv (1 samples, 0.40%) - - - -ip_finish_output2 (3 samples, 1.21%) - - - -skb_clone (1 samples, 0.40%) - - - -Envoy::Network::FilterManagerImpl::onWrite (1 samples, 0.40%) - - - -ip_local_deliver (1 samples, 0.40%) - - - -skb_copy_datagram_iter (2 samples, 0.81%) - - - -__lock_text_start (1 samples, 0.40%) - - - -sys_writev (24 samples, 9.72%) -sys_writev - - -tcp_write_xmit (6 samples, 2.43%) -tc.. - - -packet_rcv (1 samples, 0.40%) - - - -__netif_receive_skb_core (1 samples, 0.40%) - - - -free_compound_page (1 samples, 0.40%) - - - -rw_copy_check_uvector (1 samples, 0.40%) - - - -tcp_v4_rcv (3 samples, 1.21%) - - - -sock_def_readable (1 samples, 0.40%) - - - -net_rx_action (2 samples, 0.81%) - - - -ip_rcv_finish (1 samples, 0.40%) - - - -sys_readv (2 samples, 0.81%) - - - -__release_sock (1 samples, 0.40%) - - - -ip_local_out (1 samples, 0.40%) - - - -copy_user_enhanced_fast_string (4 samples, 1.62%) - - - -ip_finish_output (6 samples, 2.43%) -ip.. - - -arch_local_irq_save (1 samples, 0.40%) - - - -_raw_spin_lock (1 samples, 0.40%) - - - -__sk_dst_check (1 samples, 0.40%) - - - -do_softirq (1 samples, 0.40%) - - - -ip_local_out (2 samples, 0.81%) - - - -tcp_v4_do_rcv (1 samples, 0.40%) - - - -do_softirq_own_stack (4 samples, 1.62%) - - - -xen_evtchn_do_upcall (1 samples, 0.40%) - - - -tcp_push (1 samples, 0.40%) - - - -tcp_sendmsg (17 samples, 6.88%) -tcp_sendmsg - - -do_iter_readv_writev (6 samples, 2.43%) -do.. - - -tcp_transmit_skb (3 samples, 1.21%) - - - -dev_queue_xmit (1 samples, 0.40%) - - - -event_queue_remove_active (1 samples, 0.40%) - - - -__dev_queue_xmit (1 samples, 0.40%) - - - -__local_bh_enable_ip (1 samples, 0.40%) - - - -__softirqentry_text_start (2 samples, 0.81%) - - - -__alloc_skb (1 samples, 0.40%) - - - -do_iter_readv_writev (2 samples, 0.81%) - - - -skb_release_all (1 samples, 0.40%) - - - -__local_bh_enable_ip (3 samples, 1.21%) - - - -tcp_write_xmit (1 samples, 0.40%) - - - -__libc_writev (15 samples, 6.07%) -__libc_w.. - - -tcp_v4_rcv (1 samples, 0.40%) - - - -tcp_wfree (1 samples, 0.40%) - - - -vfs_writev (13 samples, 5.26%) -vfs_wr.. - - -tcp_transmit_skb (7 samples, 2.83%) -tc.. - - -sys_readv (2 samples, 0.81%) - - - -__netif_receive_skb (1 samples, 0.40%) - - - -__skb_clone (1 samples, 0.40%) - - - -evbuffer_read (3 samples, 1.21%) - - - -netif_rx_internal (1 samples, 0.40%) - - - -do_readv (3 samples, 1.21%) - - - -sock_write_iter (32 samples, 12.96%) -sock_write_iter - - -tcp_push (7 samples, 2.83%) -tc.. - - -inet_sendmsg (19 samples, 7.69%) -inet_sendmsg - - -ip_local_deliver (3 samples, 1.21%) - - - -entry_SYSCALL_64_fastpath (4 samples, 1.62%) - - - -__softirqentry_text_start (1 samples, 0.40%) - - - -tcp_transmit_skb (1 samples, 0.40%) - - - -sys_writev (2 samples, 0.81%) - - - -tcp_transmit_skb (10 samples, 4.05%) -tcp_.. - - -do_readv_writev (2 samples, 0.81%) - - - -ip_output (1 samples, 0.40%) - - - -[unknown] (3 samples, 1.21%) - - - -release_sock (1 samples, 0.40%) - - - -__lock_text_start (1 samples, 0.40%) - - - -__netif_receive_skb (2 samples, 0.81%) - - - -ip_local_deliver_finish (3 samples, 1.21%) - - - -ip_queue_xmit (1 samples, 0.40%) - - - -sys_readv (3 samples, 1.21%) - - - -__ip_local_out (1 samples, 0.40%) - - - -skb_entail (3 samples, 1.21%) - - - -do_readv_writev (13 samples, 5.26%) -do_rea.. - - -tcp_write_xmit (5 samples, 2.02%) -t.. - - -skb_clone (1 samples, 0.40%) - - - -inet_sendmsg (17 samples, 6.88%) -inet_send.. - - -__libc_readv (2 samples, 0.81%) - - - -tcp_v4_rcv (2 samples, 0.81%) - - - -sys_writev (7 samples, 2.83%) -sy.. - - -do_readv_writev (33 samples, 13.36%) -do_readv_writev - - -pthread_mutex_unlock (1 samples, 0.40%) - - - -tcp_write_xmit (1 samples, 0.40%) - - - -tcp_v4_do_rcv (1 samples, 0.40%) - - - -mod_timer (1 samples, 0.40%) - - - -ip_local_deliver_finish (1 samples, 0.40%) - - - -ip_rcv_finish (1 samples, 0.40%) - - - -__put_compound_page (1 samples, 0.40%) - - - -tcp_v4_do_rcv (2 samples, 0.81%) - - - -ip_finish_output2 (1 samples, 0.40%) - - - -inet_sendmsg (32 samples, 12.96%) -inet_sendmsg - - -__local_bh_enable_ip (6 samples, 2.43%) -__.. - - -__netif_receive_skb (5 samples, 2.02%) -_.. - - -ip_queue_xmit (7 samples, 2.83%) -ip.. - - -sock_write_iter (19 samples, 7.69%) -sock_write.. - - -do_readv_writev (24 samples, 9.72%) -do_readv_writev - - -inet_recvmsg (2 samples, 0.81%) - - - -loopback_xmit (1 samples, 0.40%) - - - -sock_write_iter (19 samples, 7.69%) -sock_write.. - - -ktime_get_with_offset (1 samples, 0.40%) - - - -evbuffer_write_atmost (1 samples, 0.40%) - - - -__tcp_ack_snd_check (1 samples, 0.40%) - - - -__libc_readv (2 samples, 0.81%) - - - -tcp_rcv_established (1 samples, 0.40%) - - - -do_softirq (3 samples, 1.21%) - - - -do_iter_readv_writev (1 samples, 0.40%) - - - -__netif_receive_skb_core (2 samples, 0.81%) - - - -__local_bh_enable_ip (1 samples, 0.40%) - - - -tcp_write_xmit (2 samples, 0.81%) - - - -tcp_v4_do_rcv (2 samples, 0.81%) - - - -ip_local_out (5 samples, 2.02%) -i.. - - -sock_write_iter (15 samples, 6.07%) -sock_wri.. - - -tcp_v4_rcv (3 samples, 1.21%) - - - -ip_finish_output2 (8 samples, 3.24%) -ip_.. - - -ip_finish_output (3 samples, 1.21%) - - - -ip_local_deliver_finish (2 samples, 0.81%) - - - -__libc_readv (2 samples, 0.81%) - - - -do_softirq_own_stack (6 samples, 2.43%) -do.. - - -do_softirq_own_stack (4 samples, 1.62%) - - - -ip_finish_output2 (1 samples, 0.40%) - - - -get_page_from_freelist (1 samples, 0.40%) - - - -__tcp_push_pending_frames (10 samples, 4.05%) -__tc.. - - -ip_finish_output (1 samples, 0.40%) - - - -tcp_v4_send_check (1 samples, 0.40%) - - - -tcp_v4_rcv (5 samples, 2.02%) -t.. - - -sys_writev (12 samples, 4.86%) -sys_wr.. - - -net_rx_action (1 samples, 0.40%) - - - -sock_sendmsg (3 samples, 1.21%) - - - -epoll_dispatch (1 samples, 0.40%) - - - -do_readv_writev (1 samples, 0.40%) - - - -sock_sendmsg (12 samples, 4.86%) -sock_s.. - - -copy_user_enhanced_fast_string (3 samples, 1.21%) - - - -__local_bh_enable_ip (4 samples, 1.62%) - - - -__sk_flush_backlog (1 samples, 0.40%) - - - -inet_recvmsg (1 samples, 0.40%) - - - -Envoy::Filter::TcpProxy::onData (3 samples, 1.21%) - - - -tcp_write_xmit (4 samples, 1.62%) - - - -__kfree_skb (1 samples, 0.40%) - - - -sock_read_iter (2 samples, 0.81%) - - - -validate_xmit_skb (1 samples, 0.40%) - - - -dev_hard_start_xmit (3 samples, 1.21%) - - - -ip_finish_output (4 samples, 1.62%) - - - -sk_page_frag_refill (1 samples, 0.40%) - - - -vfs_writev (15 samples, 6.07%) -vfs_writev - - -xen_clocksource_get_cycles (1 samples, 0.40%) - - - -ip_rcv_finish (4 samples, 1.62%) - - - -ip_local_out (8 samples, 3.24%) -ip_.. - - -dev_hard_start_xmit (1 samples, 0.40%) - - - -sock_write_iter (1 samples, 0.40%) - - - -__netif_receive_skb (3 samples, 1.21%) - - - -sock_recvmsg (1 samples, 0.40%) - - - -__libc_writev (2 samples, 0.81%) - - - -tcp_write_xmit (1 samples, 0.40%) - - - -ip_local_deliver (3 samples, 1.21%) - - - -entry_SYSCALL_64_fastpath (24 samples, 9.72%) -entry_SYSCALL_.. - - -entry_SYSCALL_64_fastpath (33 samples, 13.36%) -entry_SYSCALL_64_fas.. - - -ip_rcv_finish (2 samples, 0.81%) - - - -do_softirq (6 samples, 2.43%) -do.. - - -sys_readv (2 samples, 0.81%) - - - -sch_direct_xmit (1 samples, 0.40%) - - - -evbuffer_write_atmost (36 samples, 14.57%) -evbuffer_write_atmost - - -do_iter_readv_writev (4 samples, 1.62%) - - - -vfs_writev (33 samples, 13.36%) -vfs_writev - - -ip_queue_xmit (9 samples, 3.64%) -ip_q.. - - -skb_copy_datagram_iter (1 samples, 0.40%) - - - -tcp_push_one (1 samples, 0.40%) - - - -dev_hard_start_xmit (2 samples, 0.81%) - - - -tcp_v4_do_rcv (1 samples, 0.40%) - - - -tcp_nagle_check (1 samples, 0.40%) - - - -__lock_text_start (1 samples, 0.40%) - - - -ip_rcv (1 samples, 0.40%) - - - -do_writev (33 samples, 13.36%) -do_writev - - -vfs_writev (4 samples, 1.62%) - - - -ip_local_deliver_finish (5 samples, 2.02%) -i.. - - -__netif_receive_skb (1 samples, 0.40%) - - - -inet_sendmsg (18 samples, 7.29%) -inet_sendmsg - - -__wake_up_sync_key (1 samples, 0.40%) - - - -tcp_recvmsg (1 samples, 0.40%) - - - -tcp_queue_rcv (1 samples, 0.40%) - - - -tcp_rcv_established (1 samples, 0.40%) - - - -do_readv_writev (1 samples, 0.40%) - - - -ip_local_out (1 samples, 0.40%) - - - -ip_rcv (4 samples, 1.62%) - - - -inet_sendmsg (3 samples, 1.21%) - - - -tcp_push (1 samples, 0.40%) - - - -sock_read_iter (1 samples, 0.40%) - - - -release_sock (1 samples, 0.40%) - - - -tcp_write_xmit (7 samples, 2.83%) -tc.. - - -tcp_schedule_loss_probe (1 samples, 0.40%) - - - -sock_sendmsg (13 samples, 5.26%) -sock_s.. - - -copy_user_enhanced_fast_string (1 samples, 0.40%) - - - -inet_sendmsg (4 samples, 1.62%) - - - -do_iter_readv_writev (15 samples, 6.07%) -do_iter_.. - - -ip_rcv_finish (2 samples, 0.81%) - - - -tcp_ack (1 samples, 0.40%) - - - -sys_writev (3 samples, 1.21%) - - - -ip_local_deliver_finish (4 samples, 1.62%) - - - -event_changelist_remove_all_ (1 samples, 0.40%) - - - -ixgbevf_poll (1 samples, 0.40%) - - - -tcp_wfree (1 samples, 0.40%) - - - -dev_queue_xmit (4 samples, 1.62%) - - - -skb_release_data (1 samples, 0.40%) - - - -do_iter_readv_writev (3 samples, 1.21%) - - - -ip_output (6 samples, 2.43%) -ip.. - - -tcp_push_one (1 samples, 0.40%) - - - -sched_clock_local (1 samples, 0.40%) - - - -ip_output (1 samples, 0.40%) - - - -do_softirq (4 samples, 1.62%) - - - -sk_stream_alloc_skb (1 samples, 0.40%) - - - -Envoy::Network::ConnectionImpl::getReadBuffer (1 samples, 0.40%) - - - -do_iter_readv_writev (7 samples, 2.83%) -do.. - - -do_iter_readv_writev (19 samples, 7.69%) -do_iter_re.. - - -tcp_recvmsg (2 samples, 0.81%) - - - -sk_stream_alloc_skb (1 samples, 0.40%) - - - -envoy (247 samples, 100.00%) -envoy - - -__libc_writev (4 samples, 1.62%) - - - -sock_write_iter (8 samples, 3.24%) -soc.. - - -sock_recvmsg (1 samples, 0.40%) - - - -__libc_writev (12 samples, 4.86%) -__libc.. - - -__tcp_push_pending_frames (5 samples, 2.02%) -_.. - - -ksize (1 samples, 0.40%) - - - -__wake_up_sync_key (2 samples, 0.81%) - - - -__netif_receive_skb (4 samples, 1.62%) - - - -tcp_event_data_recv (1 samples, 0.40%) - - - -irq_exit (1 samples, 0.40%) - - - -copy_user_enhanced_fast_string (1 samples, 0.40%) - - - -enqueue_to_backlog (1 samples, 0.40%) - - - -__wake_up_sync_key (1 samples, 0.40%) - - - -Envoy::Network::ConnectionImpl::onReadReady (16 samples, 6.48%) -Envoy::N.. - - -copy_user_enhanced_fast_string (4 samples, 1.62%) - - - -__dev_queue_xmit (1 samples, 0.40%) - - - -tcp_write_xmit (14 samples, 5.67%) -tcp_wri.. - - -netif_rx (1 samples, 0.40%) - - - -do_iter_readv_writev (12 samples, 4.86%) -do_ite.. - - -sock_recvmsg (1 samples, 0.40%) - - - -vfs_readv (1 samples, 0.40%) - - - -sock_def_readable (1 samples, 0.40%) - - - -tcp_push (3 samples, 1.21%) - - - -entry_SYSCALL_64_fastpath (3 samples, 1.21%) - - - -dev_queue_xmit (2 samples, 0.81%) - - - -tcp_v4_rcv (3 samples, 1.21%) - - - -tcp_push (2 samples, 0.81%) - - - -sock_sendmsg (24 samples, 9.72%) -sock_sendmsg - - -tcp_rcv_established (1 samples, 0.40%) - - - -__netif_receive_skb_core (4 samples, 1.62%) - - - -evbuffer_chain_new (12 samples, 4.86%) -evbuff.. - - -vfs_writev (19 samples, 7.69%) -vfs_writev - - -ip_output (8 samples, 3.24%) -ip_.. - - -do_softirq (5 samples, 2.02%) -d.. - - -__local_bh_enable_ip (3 samples, 1.21%) - - - -do_iter_readv_writev (13 samples, 5.26%) -do_ite.. - - -ixgbevf_poll (1 samples, 0.40%) - - - -import_iovec (1 samples, 0.40%) - - - -do_softirq_own_stack (4 samples, 1.62%) - - - -tcp_v4_do_rcv (1 samples, 0.40%) - - - -ip_local_out (1 samples, 0.40%) - - - -__lock_text_start (2 samples, 0.81%) - - - -ip_finish_output (6 samples, 2.43%) -ip.. - - -__lock_text_start (4 samples, 1.62%) - - - -all (247 samples, 100%) - - - -loopback_xmit (2 samples, 0.81%) - - - -tcp_v4_rcv (1 samples, 0.40%) - - - -Envoy::Network::ConnectionImplUtility::updateBufferStats (1 samples, 0.40%) - - - -sch_direct_xmit (1 samples, 0.40%) - - - -xen_hvm_callback_vector (1 samples, 0.40%) - - - -__libc_writev (3 samples, 1.21%) - - - -netif_rx_internal (1 samples, 0.40%) - - - -__netif_receive_skb_core (4 samples, 1.62%) - - - -vfs_readv (2 samples, 0.81%) - - - -Envoy::Network::ConnectionImpl::~ConnectionImpl (2 samples, 0.81%) - - - -ip_rcv (2 samples, 0.81%) - - - -tcp_write_xmit (2 samples, 0.81%) - - - -[unknown] (2 samples, 0.81%) - - - -ip_rcv (3 samples, 1.21%) - - - -__tcp_push_pending_frames (12 samples, 4.86%) -__tcp_.. - - -ip_local_deliver_finish (2 samples, 0.81%) - - - -tcp_write_xmit (15 samples, 6.07%) -tcp_writ.. - - -do_writev (13 samples, 5.26%) -do_wri.. - - -do_iter_readv_writev (3 samples, 1.21%) - - - -do_softirq_own_stack (4 samples, 1.62%) - - - -tcp_write_xmit (1 samples, 0.40%) - - - -evbuffer_drain (3 samples, 1.21%) - - - -process_backlog (1 samples, 0.40%) - - - -entry_SYSCALL_64_fastpath (15 samples, 6.07%) -entry_SY.. - - -__softirqentry_text_start (6 samples, 2.43%) -__.. - - -__wake_up_sync_key (4 samples, 1.62%) - - - -ip_output (8 samples, 3.24%) -ip_.. - - -__wake_up_sync_key (2 samples, 0.81%) - - - -do_writev (7 samples, 2.83%) -do.. - - -copy_user_enhanced_fast_string (4 samples, 1.62%) - - - -sys_writev (2 samples, 0.81%) - - - -ip_local_deliver (3 samples, 1.21%) - - - -ip_rcv (3 samples, 1.21%) - - - -net_rx_action (1 samples, 0.40%) - - - -__tcp_push_pending_frames (5 samples, 2.02%) -_.. - - -sock_sendmsg (17 samples, 6.88%) -sock_send.. - - -xen_clocksource_read (1 samples, 0.40%) - - - -[unknown] (2 samples, 0.81%) - - - -__softirqentry_text_start (6 samples, 2.43%) -__.. - - -pvclock_clocksource_read (1 samples, 0.40%) - - - -__local_bh_enable_ip (2 samples, 0.81%) - - - -__libc_writev (33 samples, 13.36%) -__libc_writev - - -dev_queue_xmit (3 samples, 1.21%) - - - -__skb_clone (1 samples, 0.40%) - - - -__netif_receive_skb_core (4 samples, 1.62%) - - - -skb_entail (1 samples, 0.40%) - - - -sock_sendmsg (7 samples, 2.83%) -so.. - - -ktime_get_with_offset (1 samples, 0.40%) - - - -ip_rcv (1 samples, 0.40%) - - - -sk_page_frag_refill (2 samples, 0.81%) - - - -tcp_send_ack (1 samples, 0.40%) - - - -ip_local_out (6 samples, 2.43%) -ip.. - - -__tcp_push_pending_frames (7 samples, 2.83%) -__.. - - -entry_SYSCALL_64_fastpath (2 samples, 0.81%) - - - -inet_recvmsg (2 samples, 0.81%) - - - -entry_SYSCALL_64_fastpath (5 samples, 2.02%) -e.. - - -process_backlog (4 samples, 1.62%) - - - -net_rx_action (5 samples, 2.02%) -n.. - - -__inet_lookup_established (1 samples, 0.40%) - - - -__libc_writev (13 samples, 5.26%) -__libc.. - - -Envoy::Network::ConnectionImpl::doReadFromSocket (7 samples, 2.83%) -En.. - - -sock_write_iter (17 samples, 6.88%) -sock_writ.. - - -do_writev (8 samples, 3.24%) -do_.. - - -inet_sendmsg (24 samples, 9.72%) -inet_sendmsg - - -do_readv_writev (3 samples, 1.21%) - - - -netif_rx (2 samples, 0.81%) - - - -__netif_receive_skb_core (6 samples, 2.43%) -__.. - - -__dev_queue_xmit (3 samples, 1.21%) - - - -entry_SYSCALL_64_fastpath (12 samples, 4.86%) -entry_.. - - -do_softirq (2 samples, 0.81%) - - - -sys_epoll_wait (1 samples, 0.40%) - - - -__wake_up_sync_key (2 samples, 0.81%) - - - -ip_output (10 samples, 4.05%) -ip_o.. - - -sock_def_readable (2 samples, 0.81%) - - - -__libc_writev (17 samples, 6.88%) -__libc_wr.. - - -sock_write_iter (12 samples, 4.86%) -sock_w.. - - -do_iter_readv_writev (2 samples, 0.81%) - - - -ip_finish_output2 (1 samples, 0.40%) - - - -do_softirq_own_stack (3 samples, 1.21%) - - - -import_iovec (1 samples, 0.40%) - - - -do_readv_writev (6 samples, 2.43%) -do.. - - -do_writev (3 samples, 1.21%) - - - -do_readv_writev (2 samples, 0.81%) - - - -__netif_receive_skb_core (2 samples, 0.81%) - - - -net_rx_action (1 samples, 0.40%) - - - -__softirqentry_text_start (1 samples, 0.40%) - - - -tc_deletearray_nothrow (2 samples, 0.81%) - - - -sock_recvmsg (2 samples, 0.81%) - - - -do_readv_writev (3 samples, 1.21%) - - - -__netif_receive_skb (2 samples, 0.81%) - - - -ip_output (5 samples, 2.02%) -i.. - - -ip_rcv_finish (4 samples, 1.62%) - - - -pthread_mutex_unlock (1 samples, 0.40%) - - - -process_backlog (2 samples, 0.81%) - - - -tcp_rate_check_app_limited (1 samples, 0.40%) - - - -tcp_push_one (1 samples, 0.40%) - - - -ip_rcv (4 samples, 1.62%) - - - -inet_sendmsg (7 samples, 2.83%) -in.. - - -tcp_v4_do_rcv (1 samples, 0.40%) - - - -tcp_v4_do_rcv (1 samples, 0.40%) - - - -skb_release_data (1 samples, 0.40%) - - - -entry_SYSCALL_64_fastpath (1 samples, 0.40%) - - - -tcp_sendmsg (18 samples, 7.29%) -tcp_sendmsg - - -__alloc_skb (1 samples, 0.40%) - - - -__libc_writev (19 samples, 7.69%) -__libc_wri.. - - -ip_rcv_finish (1 samples, 0.40%) - - - -do_softirq (6 samples, 2.43%) -do.. - - -[unknown] (3 samples, 1.21%) - - - -dev_queue_xmit (2 samples, 0.81%) - - - -ip_local_deliver_finish (2 samples, 0.81%) - - - -[unknown] (3 samples, 1.21%) - - - -skb_copy_datagram_iter (1 samples, 0.40%) - - - -xen_hvm_callback_vector (1 samples, 0.40%) - - - -__dev_queue_xmit (1 samples, 0.40%) - - - -sock_sendmsg (18 samples, 7.29%) -sock_sendmsg - - -__dev_queue_xmit (3 samples, 1.21%) - - - -ip_queue_xmit (3 samples, 1.21%) - - - -tcp_recvmsg (2 samples, 0.81%) - - - -ip_finish_output (1 samples, 0.40%) - - - -__wake_up_sync_key (4 samples, 1.62%) - - - -__copy_skb_header (1 samples, 0.40%) - - - -ip_finish_output2 (8 samples, 3.24%) -ip_.. - - -tcmalloc::CentralFreeList::InsertRange (1 samples, 0.40%) - - - -Envoy::Server::WorkerImpl::threadRoutine (211 samples, 85.43%) -Envoy::Server::WorkerImpl::threadRoutine - - -__tcp_push_pending_frames (15 samples, 6.07%) -__tcp_pu.. - - -vfs_readv (2 samples, 0.81%) - - - -loopback_xmit (2 samples, 0.81%) - - - -__wake_up_sync_key (1 samples, 0.40%) - - - -copy_user_enhanced_fast_string (1 samples, 0.40%) - - - -sock_write_iter (4 samples, 1.62%) - - - -ip_rcv (1 samples, 0.40%) - - - -ip_output (1 samples, 0.40%) - - - -__wake_up_sync_key (1 samples, 0.40%) - - - -do_softirq (2 samples, 0.81%) - - - -ip_finish_output (1 samples, 0.40%) - - - -ip_local_out (4 samples, 1.62%) - - - -__fsnotify_parent (1 samples, 0.40%) - - - -[unknown] (3 samples, 1.21%) - - - -tcp_push (11 samples, 4.45%) -tcp_p.. - - -tcp_push (10 samples, 4.05%) -tcp_.. - - -sock_sendmsg (1 samples, 0.40%) - - - -inet_sendmsg (1 samples, 0.40%) - - - -__kmalloc_node_track_caller (1 samples, 0.40%) - - - -do_iter_readv_writev (1 samples, 0.40%) - - - -vfs_writev (2 samples, 0.81%) - - - -entry_SYSCALL_64_fastpath (2 samples, 0.81%) - - - -net_rx_action (2 samples, 0.81%) - - - -entry_SYSCALL_64_fastpath (2 samples, 0.81%) - - - -tc_malloc (1 samples, 0.40%) - - - -ip_queue_xmit (1 samples, 0.40%) - - - -tcp_transmit_skb (3 samples, 1.21%) - - - -skb_release_all (1 samples, 0.40%) - - - -sock_read_iter (2 samples, 0.81%) - - - -do_writev (2 samples, 0.81%) - - - -tcp_push (3 samples, 1.21%) - - - -tcp_sendmsg (6 samples, 2.43%) -tc.. - - -sock_read_iter (1 samples, 0.40%) - - - -kmem_cache_alloc_node (1 samples, 0.40%) - - - -sys_writev (6 samples, 2.43%) -sy.. - - -sock_def_readable (1 samples, 0.40%) - - - -sock_sendmsg (19 samples, 7.69%) -sock_sendmsg - - -inet_recvmsg (4 samples, 1.62%) - - - -tcp_sendmsg (2 samples, 0.81%) - - - -sk_stream_alloc_skb (1 samples, 0.40%) - - - -ip_local_out (4 samples, 1.62%) - - - -__libc_readv (3 samples, 1.21%) - - - -sys_writev (19 samples, 7.69%) -sys_writev - - -net_rx_action (1 samples, 0.40%) - - - -skb_copy_datagram_iter (3 samples, 1.21%) - - - -skb_clone (1 samples, 0.40%) - - - -eth_type_trans (1 samples, 0.40%) - - - -ip_queue_xmit (1 samples, 0.40%) - - - -__lock_text_start (1 samples, 0.40%) - - - -__tcp_push_pending_frames (7 samples, 2.83%) -__.. - - -sock_def_readable (1 samples, 0.40%) - - - -tcp_transmit_skb (5 samples, 2.02%) -t.. - - diff --git a/diagrams/error-detail-nack.svg b/diagrams/error-detail-nack.svg deleted file mode 100644 index 840d0277..00000000 --- a/diagrams/error-detail-nack.svg +++ /dev/null @@ -1,14 +0,0 @@ -participant Envoy as C -participant Management Server as S - -C->S: {resource_names={A}} -S->C: {resources={A}, version=1, nonce=1} -C->S: {resource_names={A}, version=1, nonce=1 (this is an ACK)} -Note over C: adding subscription\nto B -C->S: {resource_names={A, B}, version=1, nonce=1} -Note over S: sends B, but system version\ndoes not change because\nserver has not gotten a\nnew config -S->C: {resources={B}, version=1, nonce=2} -Note over C: intending to NACK B -C->S: {resource_names={A, B}, version=1, nonce=2, error_detail="B is invalid"} -Note over S: needs to look at error_detail\nto detect NACK instead of\n using version and nonce -Created with Raphaël 2.2.0EnvoyEnvoyManagement ServerManagement Server{resource_names={A}}{resources={A}, version=1, nonce=1}{resource_names={A}, version=1, nonce=1 (this is an ACK)}adding subscriptionto B{resource_names={A, B}, version=1, nonce=1}sends B, but system versiondoes not change becauseserver has not gotten anew config{resources={B}, version=1, nonce=2}intending to NACK B{resource_names={A, B}, version=1, nonce=2, error_detail="B is invalid"}needs to look at error_detailto detect NACK instead of using version and nonce diff --git a/diagrams/incremental-reconnect.svg b/diagrams/incremental-reconnect.svg deleted file mode 100644 index ef847234..00000000 --- a/diagrams/incremental-reconnect.svg +++ /dev/null @@ -1 +0,0 @@ -Created with Raphaël 2.2.0EnvoyEnvoyManagement ServerManagement Server{T=CDS}{R={(foo, v0), (bar, v0),nonce=n0, T=CDS}{response_nonce=n0, T=CDS} (ACK)Session is interrupted here. Reconnect.{initial_resource_versions={(foo, v0), (bar, v0)}, T=CDS} \ No newline at end of file diff --git a/diagrams/incremental.svg b/diagrams/incremental.svg deleted file mode 100644 index e0e93b8a..00000000 --- a/diagrams/incremental.svg +++ /dev/null @@ -1 +0,0 @@ -Created with Raphaël 2.2.0EnvoyEnvoyManagement ServerManagement Server{T=CDS}{R={(foo, v0)},nonce=n0, T=CDS}{response_nonce=n0, T=CDS} (ACK)spontaneous update of server{R={(bar, v0)},nonce=n1, T=CDS}{response_nonce=n1, error_detail="could not apply", T=CDS} (NACK)spontaneous resource list update{resource_list_subscribe=wc, T=CDS}{R={(wc, v0)},nonce=n2, T=CDS}{response_nonce=n2, T=CDS} (ACK) \ No newline at end of file diff --git a/diagrams/later-ack.svg b/diagrams/later-ack.svg deleted file mode 100644 index 7584c54f..00000000 --- a/diagrams/later-ack.svg +++ /dev/null @@ -1,8 +0,0 @@ -participant Envoy as E [color="black"] -participant Management Server as M [color="black"] - -E->M: (V=,R={foo},N=,T=EDS) [color="green"] -M->E: (V=X,R={foo:...},N=A,T=EDS) [color="gray"] -E->M: (V=,R={foo},N=A,T=EDS) [color="red"] -M->E: (V=Y,R={foo:...},N=B,T=EDS) [color="gray"] -E->M: (V=Y,R={foo},N=B,T=EDS) [color="green"]Created with Raphaël 2.2.0EnvoyEnvoyManagement ServerManagement Server(V=,R={foo},N=,T=EDS)(V=X,R={foo:...},N=A,T=EDS)(V=,R={foo},N=A,T=EDS)(V=Y,R={foo:...},N=B,T=EDS)(V=Y,R={foo},N=B,T=EDS) \ No newline at end of file diff --git a/diagrams/simple-ack.svg b/diagrams/simple-ack.svg deleted file mode 100644 index 148c2617..00000000 --- a/diagrams/simple-ack.svg +++ /dev/null @@ -1,6 +0,0 @@ -participant Envoy as E [color="black"] -participant Management Server as M [color="black"] - -E->M: (V=,R={foo},N=,T=EDS) [color="green"] -M->E: (V=X,R={foo:...},N=A,T=EDS) [color="gray"] -E->M: (V=X,R={foo},N=A,T=EDS) [color="green"]Created with Raphaël 2.2.0EnvoyEnvoyManagement ServerManagement Server(V=,R={foo},N=,T=EDS)(V=X,R={foo:...},N=A,T=EDS)(V=X,R={foo},N=A,T=EDS) \ No newline at end of file diff --git a/diagrams/simple-nack.svg b/diagrams/simple-nack.svg deleted file mode 100644 index f2d41383..00000000 --- a/diagrams/simple-nack.svg +++ /dev/null @@ -1,6 +0,0 @@ -participant Envoy as E [color="black"] -participant Management Server as M [color="black"] - -E->M: (V=,R={foo},N=,T=EDS) [color="green"] -M->E: (V=X,R={foo:...},N=A,T=EDS) [color="gray"] -E->M: (V=,R={foo},N=A,T=EDS) [color="red"]Created with Raphaël 2.2.0EnvoyEnvoyManagement ServerManagement Server(V=,R={foo},N=,T=EDS)(V=X,R={foo:...},N=A,T=EDS)(V=,R={foo},N=A,T=EDS) \ No newline at end of file diff --git a/diagrams/stale-requests.svg b/diagrams/stale-requests.svg deleted file mode 100644 index 0b440c97..00000000 --- a/diagrams/stale-requests.svg +++ /dev/null @@ -1,11 +0,0 @@ -participant Envoy as E [color="black"] -participant Management Server as M [color="black"] - -E->M: (V=X,R={foo},N=A,T=EDS) [color="green"] -Note right of M: Stale -E->M: (V=X,R={foo,bar},N=A,T=EDS) [color="green"] -M->E: (V=Y,R={foo:...,bar:...},N=B,T=EDS) [color="gray"] -E->M: (V=X,R={foo,baz},N=A,T=EDS) [color="green"] -Note right of M: Stale -E->M: (V=Y,R={foo,baz},N=B,T=EDS) [color="green"] -M->E: (V=Z,R={foo:...,baz:...},N=C,T=EDS) [color="gray"]Created with Raphaël 2.2.0EnvoyEnvoyManagement ServerManagement Server(V=X,R={foo},N=A,T=EDS)Stale(V=X,R={foo,bar},N=A,T=EDS)(V=Y,R={foo:...,bar:...},N=B,T=EDS)(V=X,R={foo,baz},N=A,T=EDS)Stale(V=Y,R={foo,baz},N=B,T=EDS)(V=Z,R={foo:...,baz:...},N=C,T=EDS) \ No newline at end of file diff --git a/diagrams/update-race.svg b/diagrams/update-race.svg deleted file mode 100644 index 1428bc7a..00000000 --- a/diagrams/update-race.svg +++ /dev/null @@ -1,11 +0,0 @@ -participant Envoy as E [color="black"] -participant Management Server 0 as M0 [color="black"] -participant Management Server 1 as M1 [color="black"] - -E->M1: (V=..,R={},N=..,T=CDS) [color="green"] -E->M0: (V=X,R={foo},N=A,T=EDS) [color="green"] -M1->E: (V=M,R={foo:...,bar:...},N=D,T=CDS) [color="gray"] -E->M0: (V=X,R={foo,bar},N=A,T=EDS [color="green"] -Note right of M0: Management server 0 replies\nbefore processing resource update at X -M0->E: (V=Y,R={foo:...,},N=B,T=EDS) [color="gray"] -E->M0: (V=Y,R={foo,bar},N=B,T=EDS [color="green"]Created with Raphaël 2.2.0EnvoyEnvoyManagement Server 0Management Server 0Management Server 1Management Server 1(V=..,R={},N=..,T=CDS)(V=X,R={foo},N=A,T=EDS)(V=M,R={foo:...,bar:...},N=D,T=CDS)(V=X,R={foo,bar},N=A,T=EDSManagement server 0 repliesbefore processing resource update at X(V=Y,R={foo:...,},N=B,T=EDS)(V=Y,R={foo,bar},N=B,T=EDS \ No newline at end of file diff --git a/xds_protocol.rst b/xds_protocol.rst deleted file mode 100644 index 71c5f646..00000000 --- a/xds_protocol.rst +++ /dev/null @@ -1,857 +0,0 @@ -.. _xds_protocol: - -xDS REST and gRPC protocol -========================== - -Envoy discovers its various dynamic resources via the filesystem or by -querying one or more management servers. Collectively, these discovery -services and their corresponding APIs are referred to as *xDS*. -Resources are requested via *subscriptions*, by specifying a filesystem -path to watch, initiating gRPC streams, or polling a REST-JSON URL. The -latter two methods involve sending requests with a :ref:`DiscoveryRequest ` -proto payload. Resources are delivered in a -:ref:`DiscoveryResponse ` -proto payload in all methods. We discuss each type of subscription -below. - -Resource Types --------------- - -Every configuration resource in the xDS API has a type associated with it. Resource types follow a -:repo:`versioning scheme `. Resource types are versioned independent of the -transports described below. - -The following v3 xDS resource types are supported: - -- :ref:`envoy.config.listener.v3.Listener ` -- :ref:`envoy.config.route.v3.RouteConfiguration ` -- :ref:`envoy.config.route.v3.ScopedRouteConfiguration ` -- :ref:`envoy.config.route.v3.VirtualHost ` -- :ref:`envoy.config.cluster.v3.Cluster ` -- :ref:`envoy.config.endpoint.v3.ClusterLoadAssignment ` -- :ref:`envoy.extensions.transport_sockets.tls.v3.Secret ` -- :ref:`envoy.service.runtime.v3.Runtime ` - -The concept of `type URLs `_ -appears below, and takes the form ``type.googleapis.com/`` -- e.g., -``type.googleapis.com/envoy.config.cluster.v3.Cluster`` for a ``Cluster`` resource. In various requests from -Envoy and responses by the management server, the resource type URL is stated. - - -Filesystem subscriptions ------------------------- - -The simplest approach to delivering dynamic configuration is to place it -at a well known path specified in the :ref:`ConfigSource `. -Envoy will use ``inotify`` (``kqueue`` on macOS) to monitor the file for -changes and parse the -:ref:`DiscoveryResponse ` proto in the file on update. -Binary protobufs, JSON, YAML and proto text are supported formats for -the -:ref:`DiscoveryResponse `. - -There is no mechanism available for filesystem subscriptions to ACK/NACK -updates beyond stats counters and logs. The last valid configuration for -an xDS API will continue to apply if an configuration update rejection -occurs. - -.. _xds_protocol_streaming_grpc_subscriptions: - -Streaming gRPC subscriptions ----------------------------- - -API flow -~~~~~~~~ - -For typical HTTP routing scenarios, the core resource types for the client's configuration are -``Listener``, ``RouteConfiguration``, ``Cluster``, and ``ClusterLoadAssignment``. Each ``Listener`` resource -may point to a ``RouteConfiguration`` resource, which may point to one or more ``Cluster`` resources, -and each ``Cluster`` resource may point to a ``ClusterLoadAssignment`` resource. - -Envoy fetches all ``Listener`` and ``Cluster`` resources at startup. It then fetches whatever -``RouteConfiguration`` and ``ClusterLoadAssignment`` resources that are required by the ``Listener`` and -``Cluster`` resources. In effect, every ``Listener`` or ``Cluster`` resource is a root to part of Envoy's -configuration tree. - -A non-proxy client such as gRPC might start by fetching only the specific ``Listener`` resources -that it is interested in. It then fetches the ``RouteConfiguration`` resources required by those -``Listener`` resources, followed by whichever ``Cluster`` resources are required by those -``RouteConfiguration`` resources, followed by the ``ClusterLoadAssignment`` resources required -by the ``Cluster`` resources. In effect, the original ``Listener`` resources are the roots to -the client's configuration tree. - -Variants of the xDS Transport Protocol -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Four Variants -^^^^^^^^^^^^^ - -There are four variants of the xDS transport protocol used via streaming gRPC, which cover all -combinations of two dimensions. - -The first dimension is State of the World (SotW) vs. incremental. The SotW approach was the -original mechanism used by xDS, in which the client must specify all resource names it is -interested in with each request (except when making a wildcard request in LDS/CDS), and the server -must return all resources the client has subscribed to in each request (in LDS/CDS). This means -that if the client is already subscribing to 99 resources and wants to add an additional one, it -must send a request with all 100 resource names, rather than just the one new one. And the server -must then respond by sending all 100 resources, even if the 99 that were already subscribed to have -not changed (in LDS/CDS). This mechanism can be a scalability limitation, which is why the -incremental protocol variant was introduced. The incremental approach allows both the client and -server to indicate only deltas relative to their previous state -- i.e., the client can say that -it wants to add or remove its subscription to a particular resource name without resending those -that have not changed, and the server can send updates only for those resources that have changed. -The incremental protocol also provides a mechanism for lazy loading of resources. For details on -the incremental protocol, see :ref:`Incremental xDS ` below. - -The second dimension is using a separate gRPC stream for each resource type vs. aggregating all -resource types onto a single gRPC stream. The former approach was the original mechanism used by -xDS, and it offers an eventual consistency model. The latter approach was added for environments -in which explicit control of sequencing is required. For details, see :ref:`Eventual consistency -considerations ` below. - -So, the four variants of the xDS transport protocol are: - -1. State of the World (Basic xDS): SotW, separate gRPC stream for each resource type -2. Incremental xDS: incremental, separate gRPC stream for each resource type -3. Aggregated Discovery Service (ADS): SotW, aggregate stream for all resource types -4. Incremental ADS: incremental, aggregate stream for all resource types - -RPC Services and Methods for Each Variant -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -For the non-aggregated protocol variants, there is a separate RPC service for each resource type. -Each of these RPC services can provide a method for each of the SotW and Incremental protocol -variants. Here are the RPC services and methods for each resource type: - -- Listener: Listener Discovery Service (LDS) - - SotW: ListenerDiscoveryService.StreamListeners - - Incremental: ListenerDiscoveryService.DeltaListeners -- RouteConfiguration: Route Discovery Service (RDS) - - SotW: RouteDiscoveryService.StreamRoutes - - Incremental: RouteDiscoveryService.DeltaRoutes -- ScopedRouteConfiguration: Scoped Route Discovery Service (SRDS) - - SotW: ScopedRouteDiscoveryService.StreamScopedRoutes - - Incremental: ScopedRouteDiscoveryService.DeltaScopedRoutes -- VirtualHost: Virtual Host Discovery Service (VHDS) - - SotW: N/A - - Incremental: VirtualHostDiscoveryService.DeltaVirtualHosts -- Cluster: Cluster Discovery Service (CDS) - - SotW: ClusterDiscoveryService.StreamClusters - - Incremental: ClusterDiscoveryService.DeltaClusters -- ClusterLoadAssignment: Endpoint Discovery Service (EDS) - - SotW: EndpointDiscoveryService.StreamEndpoints - - Incremental: EndpointDiscoveryService.DeltaEndpoints -- Secret: Secret Discovery Service (SDS) - - SotW: SecretDiscoveryService.StreamSecrets - - Incremental: SecretDiscoveryService.DeltaSecrets -- Runtime: Runtime Discovery Service (RTDS) - - SotW: RuntimeDiscoveryService.StreamRuntime - - Incremental: RuntimeDiscoveryService.DeltaRuntime - -In the aggregated protocol variants, all resource types are multiplexed on a single gRPC stream, -where each resource type is treated as a separate logical stream within the aggregated stream. -In effect, it simply combines all of the above separate APIs into a single stream by treating -requests and responses for each resource type as a separate sub-stream on the single aggregated -stream. The RPC service and methods for the aggregated protocol variants are: - -- SotW: AggregatedDiscoveryService.StreamAggregatedResources -- Incremental: AggregatedDiscoveryService.DeltaAggregatedResources - -For all of the SotW methods, the request type is :ref:`DiscoveryRequest -` and the response type is :ref:`DiscoveryResponse -`. - -For all of the incremental methods, the request type is :ref:`DeltaDiscoveryRequest -` and the response type is :ref:`DeltaDiscoveryResponse -`. - -Configuring Which Variant to Use -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -In the xDS API, the :ref:`ConfigSource ` message indicates how to -obtain resources of a particular type. If the :ref:`ConfigSource ` -contains a gRPC :ref:`ApiConfigSource `, it points to an -upstream cluster for the management server; this will initiate an independent bidirectional gRPC -stream for each xDS resource type, potentially to distinct management servers. If the -:ref:`ConfigSource ` contains a :ref:`AggregatedConfigSource -`, it tells the client to use :ref:`ADS -`. - -Currently, the client is expected to be given some local configuration that tells it how to obtain -the :ref:`Listener ` and :ref:`Cluster ` resources. -:ref:`Listener ` resources may include a -:ref:`ConfigSource ` that indicates how the -:ref:`RouteConfiguration ` resources are obtained, and -:ref:`Cluster ` resources may include a -:ref:`ConfigSource ` that indicates how the -:ref:`ClusterLoadAssignment ` resources are obtained. - -Client Configuration -"""""""""""""""""""" - -In Envoy, the bootstrap file contains two :ref:`ConfigSource ` -messages, one indicating how :ref:`Listener ` resources are obtained and -another indicating how :ref:`Cluster ` resources are obtained. It also -contains a separate :ref:`ApiConfigSource ` message indicating -how to contact the ADS server, which will be used whenever a :ref:`ConfigSource -` message (either in the bootstrap file or in a :ref:`Listener -` or :ref:`Cluster ` resource obtained from a -management server) contains an :ref:`AggregatedConfigSource -` message. - -In a gRPC client that uses xDS, only ADS is supported, and the bootstrap file contains the name of -the ADS server, which will be used for all resources. The :ref:`ConfigSource -` messages in the :ref:`Listener ` and -:ref:`Cluster ` resources must contain :ref:`AggregatedConfigSource -` messages. - -The xDS transport Protocol -~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Transport API version -^^^^^^^^^^^^^^^^^^^^^ - -In addition the resource type version described above, the xDS wire protocol has a -transport version associated with it. This provides type versioning for messages such as -:ref:`DiscoveryRequest ` and :ref:`DiscoveryResponse -`. It is also encoded in the gRPC method name, so a server -can determine which version a client is speaking based on which method it calls. - -Basic Protocol Overview -^^^^^^^^^^^^^^^^^^^^^^^ - -Each xDS stream begins with a :ref:`DiscoveryRequest ` from the -client, which specifies the list of resources to subscribe to, the type URL corresponding to the -subscribed resources, the node identifier, and an optional resource type instance version -indicating the most recent version of the resource type that the client has already seen (see -:ref:`ACK/NACK and resource type instance version ` for details). - -The server will then send a :ref:`DiscoveryResponse ` containing -any resources that the client has subscribed to that have changed since the last resource type -instance version that the client indicated it has seen. The server may send additional responses -at any time when the subscribed resources change. - -Whenever the client receives a new response, it will send another request indicating whether or -not the resources in the response were valid (see -:ref:`ACK/NACK and resource type instance version ` for details). - -All server responses will contain a :ref:`nonce`, and -all subsequent requests from the client must set the -:ref:`response_nonce ` field to the most recent -nonce received from the server on that stream. This allows servers to determine which response a -given request is associated with, which avoids various race conditions in the SotW protocol -variants. Note that the nonce is valid only in the context of an individual xDS stream; it does -not survive stream restarts. - -Only the first request on a stream is guaranteed to carry the node identifier. -The subsequent discovery requests on the same stream may carry an empty node -identifier. This holds true regardless of the acceptance of the discovery -responses on the same stream. The node identifier should always be identical if -present more than once on the stream. It is sufficient to only check the first -message for the node identifier as a result. - -.. _xds_ack_nack: - -ACK/NACK and resource type instance version -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Every xDS resource type has a version string that indicates the version for that resource type. -Whenever one resource of that type changes, the version is changed. - -In a response sent by the xDS server, the -:ref:`version_info` field indicates the current -version for that resource type. The client then sends another request to the server with the -:ref:`version_info` field indicating the most -recent valid version seen by the client. This provides a way for the server to determine when -it sends a version that the client considers invalid. - -(In the :ref:`incremental protocol variants `, the resource type instance -version is sent by the server in the -:ref:`system_version_info` field. -However, this information is not actually used by the client to communicate which resources are -valid, because the incremental API variants have a separate mechanism for that.) - -The resource type instance version is separate for each resource type. When using the aggregated -protocol variants, each resource type has its own version even though all resource types are being -sent on the same stream. - -The resource type instance version is also separate for each xDS server (where an xDS server is -identified by a unique :ref:`ConfigSource `). When obtaining -resources of a given type from multiple xDS servers, each xDS server will have a different notion -of version. - -Note that the version for a resource type is not a property of an individual xDS stream but rather -a property of the resources themselves. If the stream becomes broken and the client creates a new -stream, the client's initial request on the new stream should indicate the most recent version -seen by the client on the previous stream. Servers may decide to optimize by not resending -resources that the client had already seen on the previous stream, but only if they know that the -client is not subscribing to a new resource that it was not previously subscribed to. For example, -it is generally safe for servers to do this optimization for wildcard LDS and CDS requests, and it -is safe to do in environments where the clients will always subscribe to exactly the same set of -resources. - -An example EDS request might be: - -.. code:: yaml - - version_info: - node: { id: envoy } - resource_names: - - foo - - bar - type_url: type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment - response_nonce: - -The management server may reply either immediately or when the requested -resources are available with a :ref:`DiscoveryResponse `, e.g.: - -.. code:: yaml - - version_info: X - resources: - - foo ClusterLoadAssignment proto encoding - - bar ClusterLoadAssignment proto encoding - type_url: type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment - nonce: A - -After processing the :ref:`DiscoveryResponse `, Envoy will send a new -request on the stream, specifying the last version successfully applied -and the nonce provided by the management server. The version provides Envoy and the -management server a shared notion of the currently applied configuration, -as well as a mechanism to ACK/NACK configuration updates. - -ACK -^^^ - -If the update was successfully applied, the -:ref:`version_info ` will be **X**, as indicated -in the sequence diagram: - -.. figure:: diagrams/simple-ack.svg - :alt: Version update after ACK - -NACK -^^^^ - -If Envoy had instead rejected configuration -update **X**, it would reply with :ref:`error_detail ` -populated and its previous version, which in this case was the empty -initial version. The :ref:`error_detail ` has -more details around the exact error message populated in the message field: - -.. figure:: diagrams/simple-nack.svg - :alt: No version update after NACK - -In the sequence diagrams, the following format is used to abbreviate messages: - -- *DiscoveryRequest*: (V=version_info,R=resource_names,N=response_nonce,T=type_url) -- *DiscoveryResponse*: (V=version_info,R=resources,N=nonce,T=type_url) - -After a NACK, an API update may succeed at a new version **Y**: - - -.. figure:: diagrams/later-ack.svg - :alt: ACK after NACK - -The preferred mechanism for a server to detect a NACK is to look for the presence of the -:ref:`error_detail ` field in the request sent by -the client. Some older servers may instead detect a NACK by looking at both the version and the -nonce in the request: if the version in the request is not equal to the one sent by the server with -that nonce, then the client has rejected the most recent version. However, this approach does not -work for APIs other than LDS and CDS for clients that may dynamically change the set of resources -that they are subscribing to, unless the server has somehow arranged to increment the resource -type instance version every time any one client subscribes to a new resource. Specifically, -consider the following example: - -.. figure:: diagrams/error-detail-nack.svg - :alt: detecting NACK from error_detail instead of version and nonce - -ACK and NACK semantics summary -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -- The xDS client should ACK or NACK every :ref:`DiscoveryResponse ` - received from the management server. The :ref:`response_nonce - ` field tells the server which of its responses - the ACK or NACK is associated with. -- ACK signifies successful configuration update and contains the - :ref:`version_info ` from the - :ref:`DiscoveryResponse `. -- NACK signifies unsuccessful configuration and is indicated by the presence of the - :ref:`error_detail ` field. The :ref:`version_info - ` indicates the most recent version that the - client is using, although that may not be an older version in the case where the client has - subscribed to a new resource from an existing version and that new resource is invalid (see - example above). - -.. _xds_protocol_resource_update: - -When to send an update -^^^^^^^^^^^^^^^^^^^^^^ - -The management server should only send updates to the Envoy client when -the resources in the :ref:`DiscoveryResponse ` have changed. Envoy replies -to any :ref:`DiscoveryResponse ` with a :ref:`DiscoveryRequest ` containing the -ACK/NACK immediately after it has been either accepted or rejected. If -the management server provides the same set of resources rather than -waiting for a change to occur, it will cause needless work on both the client and the management -server, which could have a severe performance impact. - -Within a stream, new :ref:`DiscoveryRequests ` supersede any prior -:ref:`DiscoveryRequests ` having the same resource type. This means that -the management server only needs to respond to the latest -:ref:`DiscoveryRequest ` on each stream for any given resource type. - -.. _xds_protocol_resource_hints: - -How the client specifies what resources to return -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -xDS requests allow the client to specify a set of resource names as a hint to the server about -which resources the client is interested in. In the SotW protocol variants, this is done via the -:ref:`resource_names ` specified in the -:ref:`DiscoveryRequest `; in the incremental protocol variants, -this is done via the :ref:`resource_names_subscribe -` and -:ref:`resource_names_unsubscribe -` fields in the -:ref:`DeltaDiscoveryRequest `. - -Normally (see below for exceptions), requests must specify the set of resource names that the -client is interested in. The management server must supply the requested resources if they exist. -The client will silently ignore any supplied resources that were not explicitly requested. When -the client sends a new request that changes the set of resources being requested, the server must -resend any newly requested resources, even if it previously sent those resources without having -been asked for them and the resources have not changed since that time. If the list of resource -names becomes empty, that means that the client is no longer interested in any resources of the -specified type. - -For :ref:`Listener ` and :ref:`Cluster ` resource -types, there is also a "wildcard" mode, which is triggered when the initial request on the stream -for that resource type contains no resource names. In this case, the server should use -site-specific business logic to determine the full set of resources that the client is interested -in, typically based on the client's :ref:`node ` identification. Note -that once a stream has entered wildcard mode for a given resource type, there is no way to change -the stream out of wildcard mode; resource names specified in any subsequent request on the stream -will be ignored. - -Client Behavior -""""""""""""""" - -Envoy will always use wildcard mode for :ref:`Listener ` and -:ref:`Cluster ` resources. However, other xDS clients (such as gRPC clients -that use xDS) may specify explicit resource names for these resource types, for example if they -only have a singleton listener and already know its name from some out-of-band configuration. - -Grouping Resources into Responses -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -In the incremental protocol variants, the server sends each resource in its own response. This -means that if the server has previously sent 100 resources and only one of them has changed, it -may send a response containing only the changed resource; it does not need to resend the 99 -resources that have not changed, and the client must not delete the unchanged resources. - -In the SotW protocol variants, all resource types except for :ref:`Listener -` and :ref:`Cluster ` are grouped into responses -in the same way as in the incremental protocol variants. However, -:ref:`Listener ` and :ref:`Cluster ` resource types -are handled differently: the server must include the complete state of the world, meaning that all -resources of the relevant type that are needed by the client must be included, even if they did -not change since the last response. This means that if the server has previously sent 100 -resources and only one of them has changed, it must resend all 100 of them, even the 99 that were -not modified. - -Note that all of the protocol variants operate on units of whole named resources. There is -no mechanism for providing incremental updates of repeated fields within a named resource. -Most notably, there is currently no mechanism for incrementally updating individual -endpoints within an EDS response. - -Duplicate Resource Names -^^^^^^^^^^^^^^^^^^^^^^^^ - -It is an error for a server to send a single response that contains the same resource name -twice. Clients should NACK responses that contain multiple instances of the same resource name. - -Deleting Resources -^^^^^^^^^^^^^^^^^^ - -In the incremental proocol variants, the server signals the client that a resource should be -deleted via the :ref:`removed_resources ` -field of the response. This tells the client to remove the resource from its local cache. - -In the SotW protocol variants, the criteria for deleting resources is more complex. For -:ref:`Listener ` and :ref:`Cluster ` resource types, -if a previously seen resource is not present in a new response, that indicates that the resource -has been removed, and the client must delete it; a response containing no resources means to delete -all resources of that type. However, for other resource types, the API provides no mechanism for -the server to tell the client that resources have been deleted; instead, deletions are indicated -implicitly by parent resources being changed to no longer refer to a child resource. For example, -when the client receives an LDS update removing a :ref:`Listener ` -that was previously pointing to :ref:`RouteConfiguration ` A, -if no other :ref:`Listener ` is pointing to :ref:`RouteConfiguration -` A, then the client may delete A. For those resource types, -an empty :ref:`DiscoveryResponse ` is effectively a no-op -from the client's perspective. - -.. _xds_protocol_resource_not_existed: - -Knowing When a Requested Resource Does Not Exist -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The SotW protocol variants do not provide any explicit mechanism to determine when a requested -resource does not exist. - -Responses for :ref:`Listener ` and :ref:`Cluster ` -resource types must include all resources requested by the client. However, it may not be possible -for the client to know that a resource does not exist based solely on its absence in a response, -because the delivery of the updates is eventually consistent: if the client initially sends a -request for resource A, then sends a request for resources A and B, and then sees a response -containing only resource A, the client cannot conclude that resource B does not exist, because -the response may have been sent on the basis of the first request, before the server saw the -second request. - -For other resource types, because each resource can be sent in its own response, there is no way -to know from the next response whether the newly requested resource exists, because the next -response could be an unrelated update for another resource that had already been subscribed to -previously. - -As a result, clients are expected to use a timeout (recommended duration is 15 seconds) after -sending a request for a new resource, after which they will consider the requested resource to -not exist if they have not received the resource. In Envoy, this is done for -:ref:`RouteConfiguration ` and :ref:`ClusterLoadAssignment -` resources during :ref:`resource warming -`. - -Note that this timeout is not strictly necessary when using wildcard mode for :ref:`Listener -` and :ref:`Cluster ` resource types, because -in that case every response will contain all existing resources that are relevant to the -client, so the client can know that a resource does not exist by its absence in the next -response it sees. However, using a timeout is still recommended in this case, since it protects -against the case where the management server fails to send a response in a timely manner. - -Note that even if a requested resource does not exist at the moment when the client requests it, -that resource could be created at any time. Management servers must remember the set of resources -being requested by the client, and if one of those resources springs into existence later, the -server must send an update to the client informing it of the new resource. Clients that initially -see a resource that does not exist must be prepared for the resource to be created at any time. - -Unsubscribing From Resources -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -In the incremental protocol variants, resources can be unsubscribed to via the -:ref:`resource_names_unsubscribe -` field. - -In the SotW protocol variants, each request must contain the full list of resource names being -subscribed to in the :ref:`resource_names ` field, -so unsubscribing to a set of resources is done by sending a new request containing all resource -names that are still being subscribed to but not containing the resource names being unsubscribed -to. For example, if the client had previously been subscribed to resources A and B but wishes to -unsubscribe from B, it must send a new request containing only resource A. - -Note that for :ref:`Listener ` and :ref:`Cluster ` -resource types where the stream is in "wildcard" mode (see :ref:`How the client specifies what -resources to return ` for details), the set of resources being -subscribed to is determined by the server instead of the client, so there is no mechanism -for the client to unsubscribe from resources. - -Requesting Multiple Resources on a Single Stream -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -For EDS/RDS, Envoy may either generate a distinct stream for each -resource of a given type (e.g. if each :ref:`ConfigSource ` has its own -distinct upstream cluster for a management server), or may combine -together multiple resource requests for a given resource type when they -are destined for the same management server. While this is left to -implementation specifics, management servers should be capable of -handling one or more :ref:`resource_names ` for a given resource type in -each request. Both sequence diagrams below are valid for fetching two -EDS resources ``{foo, bar}``: - -|Multiple EDS requests on the same stream| |Multiple EDS requests on -distinct streams| - -Resource updates -^^^^^^^^^^^^^^^^ - -As discussed above, Envoy may update the list of :ref:`resource_names ` it -presents to the management server in each :ref:`DiscoveryRequest ` that -ACK/NACKs a specific :ref:`DiscoveryResponse `. In addition, Envoy may later -issue additional :ref:`DiscoveryRequests ` at a given :ref:`version_info ` to -update the management server with new resource hints. For example, if -Envoy is at EDS version **X** and knows only about cluster ``foo``, but -then receives a CDS update and learns about ``bar`` in addition, it may -issue an additional :ref:`DiscoveryRequest ` for **X** with ``{foo,bar}`` as -``resource_names``. - -.. figure:: diagrams/cds-eds-resources.svg - :alt: CDS response leads to EDS resource hint update - -There is a race condition that may arise here; if after a resource hint -update is issued by Envoy at **X**, but before the management server -processes the update it replies with a new version **Y**, the resource -hint update may be interpreted as a rejection of **Y** by presenting an -**X** :ref:`version_info `. To avoid this, the management server provides a -``nonce`` that Envoy uses to indicate the specific :ref:`DiscoveryResponse ` -each :ref:`DiscoveryRequest ` corresponds to: - -.. figure:: diagrams/update-race.svg - :alt: EDS update race motivates nonces - -The management server should not send a :ref:`DiscoveryResponse ` for any -:ref:`DiscoveryRequest ` that has a stale nonce. A nonce becomes stale -following a newer nonce being presented to Envoy in a -:ref:`DiscoveryResponse `. A management server does not need to send an -update until it determines a new version is available. Earlier requests -at a version then also become stale. It may process multiple -:ref:`DiscoveryRequests ` at a version until a new version is ready. - -.. figure:: diagrams/stale-requests.svg - :alt: Requests become stale - -An implication of the above resource update sequencing is that Envoy -does not expect a :ref:`DiscoveryResponse ` for every :ref:`DiscoveryRequests ` -it issues. - -.. _xds_protocol_resource_warming: - -Resource warming -~~~~~~~~~~~~~~~~ - -:ref:`Clusters ` and -:ref:`Listeners ` -go through warming before they can serve requests. This process -happens both during :ref:`Envoy initialization ` -and when the ``Cluster`` or ``Listener`` is updated. Warming of -``Cluster`` is completed only when a ``ClusterLoadAssignment`` response -is supplied by management server. Similarly, warming of ``Listener`` is -completed only when a ``RouteConfiguration`` is supplied by management -server if the listener refers to an RDS configuration. Management server -is expected to provide the EDS/RDS updates during warming. If management -server does not provide EDS/RDS responses, Envoy will not initialize -itself during the initialization phase and the updates sent via CDS/LDS -will not take effect until EDS/RDS responses are supplied. - -.. _xds_protocol_eventual_consistency_considerations: - -Eventual consistency considerations -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Since Envoy's xDS APIs are eventually consistent, traffic may drop -briefly during updates. For example, if only cluster **X** is known via -CDS/EDS, a ``RouteConfiguration`` references cluster **X** and is then -adjusted to cluster **Y** just before the CDS/EDS update providing -**Y**, traffic will be blackholed until **Y** is known about by the -Envoy instance. - -For some applications, a temporary drop of traffic is acceptable, -retries at the client or by other Envoy sidecars will hide this drop. -For other scenarios where drop can't be tolerated, traffic drop could -have been avoided by providing a CDS/EDS update with both **X** and -**Y**, then the RDS update repointing from **X** to **Y** and then a -CDS/EDS update dropping **X**. - -In general, to avoid traffic drop, sequencing of updates should follow a -make before break model, wherein: - -- CDS updates (if any) must always be pushed first. -- EDS updates (if any) must arrive after CDS updates for the respective clusters. -- LDS updates must arrive after corresponding CDS/EDS updates. -- RDS updates related to the newly added listeners must arrive after CDS/EDS/LDS updates. -- VHDS updates (if any) related to the newly added RouteConfigurations must arrive after RDS updates. -- Stale CDS clusters and related EDS endpoints (ones no longer being referenced) can then be removed. - -xDS updates can be pushed independently if no new -clusters/routes/listeners are added or if it's acceptable to temporarily -drop traffic during updates. Note that in case of LDS updates, the -listeners will be warmed before they receive traffic, i.e. the dependent -routes are fetched through RDS if configured. Clusters are warmed when -adding/removing/updating clusters. On the other hand, routes are not -warmed, i.e., the management plane must ensure that clusters referenced -by a route are in place, before pushing the updates for a route. - -.. _xds_protocol_TTL: - -TTL -~~~ - -In the event that the management server becomes unreachable, the last known configuration received -by Envoy will persist until the connection is reestablished. For some services, this may not be -desirable. For example, in the case of a fault injection service, a management server crash at the -wrong time may leave Envoy in an undesirable state. The TTL setting allows Envoy to remove a set of -resources after a specified period of time if contact with the management server is lost. This can -be used, for example, to terminate a fault injection test when the management server can no longer -be reached. - -For clients that support the *xds.config.supports-resource-ttl* client feature, A TTL field may -be specified on each :ref:`Resource `. Each resource will have its own TTL -expiry time, at which point the resource will be expired. Each xDS type may have different ways of -handling such an expiry. - -To update the TTL associated with a *Resource*, the management server resends the resource with a -new TTL. To remove the TTL, the management server resends the resource with the TTL field unset. - -To allow for lightweight TTL updates ("heartbeats"), a response can be sent that provides a -:ref:`Resource ` with the :ref:`resource ` -unset and version matching the most recently sent version can be used to update the TTL. These -resources will not be treated as resource updates, but only as TTL updates. - -SotW TTL -^^^^^^^^ - -In order to use TTL with SotW xDS, the relevant resources must be wrapped in a -:ref:`Resource `. This allows setting the same TTL field that is used for -Delta xDS with SotW, without changing the SotW API. Heartbeats are supported for SotW as well: -any resource within the response that look like a heartbeat resource will only be used to update the TTL. - -This feature is gated by the *xds.config.supports-resource-in-sotw* client feature. - -.. _xds_protocol_ads: - -Aggregated Discovery Service -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -It's challenging to provide the above guarantees on sequencing to avoid -traffic drop when management servers are distributed. ADS allow a single -management server, via a single gRPC stream, to deliver all API updates. -This provides the ability to carefully sequence updates to avoid traffic -drop. With ADS, a single stream is used with multiple independent -:ref:`DiscoveryRequest `/:ref:`DiscoveryResponse ` sequences multiplexed via the -type URL. For any given type URL, the above sequencing of -:ref:`DiscoveryRequest ` and :ref:`DiscoveryResponse ` messages applies. An -example update sequence might look like: - -.. figure:: diagrams/ads.svg - :alt: EDS/CDS multiplexed on an ADS stream - -A single ADS stream is available per Envoy instance. - -An example minimal ``bootstrap.yaml`` fragment for ADS configuration is: - -.. literalinclude:: ../_include/ads.yaml - -.. _xds_protocol_delta: - -Incremental xDS -~~~~~~~~~~~~~~~ - -Incremental xDS is a separate xDS endpoint that: - -- Allows the protocol to communicate on the wire in terms of - resource/resource name deltas ("Delta xDS"). This supports the goal - of scalability of xDS resources. Rather than deliver all 100k - clusters when a single cluster is modified, the management server - only needs to deliver the single cluster that changed. -- Allows the Envoy to on-demand / lazily request additional resources. - For example, requesting a cluster only when a request for that - cluster arrives. - -An Incremental xDS session is always in the context of a gRPC -bidirectional stream. This allows the xDS server to keep track of the -state of xDS clients connected to it. There is no REST version of -Incremental xDS yet. - -In the delta xDS wire protocol, the nonce field is required and used to -pair a :ref:`DeltaDiscoveryResponse ` -to a :ref:`DeltaDiscoveryRequest ` -ACK or NACK. Optionally, a response message level :ref:`system_version_info ` -is present for debugging purposes only. - -:ref:`DeltaDiscoveryRequest ` can be sent in the following situations: - -- Initial message in a xDS bidirectional gRPC stream. -- As an ACK or NACK response to a previous :ref:`DeltaDiscoveryResponse `. In this case the :ref:`response_nonce ` is set to the nonce value in the Response. ACK or NACK is determined by the absence or presence of :ref:`error_detail `. -- Spontaneous :ref:`DeltaDiscoveryRequests ` from the client. This can be done to dynamically add or remove elements from the tracked :ref:`resource_names ` set. In this case :ref:`response_nonce ` must be omitted. - -In this first example the client connects and receives a first update -that it ACKs. The second update fails and the client NACKs the update. -Later the xDS client spontaneously requests the "wc" resource. - -.. figure:: diagrams/incremental.svg - :alt: Incremental session example - -On reconnect the Incremental xDS client may tell the server of its known -resources to avoid resending them over the network by sending them in -:ref:`initial_resource_versions `. -Because no state is assumed to be preserved from the previous stream, the reconnecting -client must provide the server with all resource names it is interested in. Note that for wildcard -requests (CDS/LDS/SRDS), the request must have no resources in both -:ref:`resource_names_subscribe ` and -:ref:`resource_names_unsubscribe `. - -.. figure:: diagrams/incremental-reconnect.svg - :alt: Incremental reconnect example - -Resource names -^^^^^^^^^^^^^^ - -Resources are identified by a resource name or an alias. Aliases of a -resource, if present, can be identified by the alias field in the -resource of a :ref:`DeltaDiscoveryResponse `. The resource name will be -returned in the name field in the resource of a -:ref:`DeltaDiscoveryResponse `. - -.. _xds_protocol_delta_subscribe: - -Subscribing to Resources -^^^^^^^^^^^^^^^^^^^^^^^^ - -The client can send either an alias or the name of a resource in the -:ref:`resource_names_subscribe ` field of a :ref:`DeltaDiscoveryRequest ` in -order to subscribe to a resource. Both the names and aliases of -resources should be checked in order to determine whether the entity in -question has been subscribed to. - -A :ref:`resource_names_subscribe ` field may contain resource names that the -server believes the client is already subscribed to, and furthermore has -the most recent versions of. However, the server *must* still provide -those resources in the response; due to implementation details hidden -from the server, the client may have "forgotten" those resources despite -apparently remaining subscribed. - -.. _xds_protocol_unsubscribe: - -Unsubscribing from Resources -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -When a client loses interest in some resources, it will indicate that -with the :ref:`resource_names_unsubscribe ` field of a -:ref:`DeltaDiscoveryRequest `. As with :ref:`resource_names_subscribe `, these -may be resource names or aliases. - -A :ref:`resource_names_unsubscribe ` field may contain superfluous resource -names, which the server thought the client was already not subscribed -to. The server must cleanly process such a request; it can simply ignore -these phantom unsubscriptions. - -Knowing When a Requested Resource Does Not Exist -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -When a resource subscribed to by a client does not exist, the server will send a :ref:`Resource -` whose :ref:`name ` field matches the -name that the client subscribed to and whose :ref:`resource ` -field is unset. This allows the client to quickly determine when a resource does not exist without -waiting for a timeout, as would be done in the SotW protocol variants. However, clients are still -encouraged to use a timeout to protect against the case where the management server fails to send -a response in a timely manner. - -REST-JSON polling subscriptions -------------------------------- - -Synchronous (long) polling via REST endpoints is also available for the -xDS singleton APIs. The above sequencing of messages is similar, except -no persistent stream is maintained to the management server. It is -expected that there is only a single outstanding request at any point in -time, and as a result the response nonce is optional in REST-JSON. The -`JSON canonical transform of -proto3 `__ -is used to encode :ref:`DiscoveryRequest ` and :ref:`DiscoveryResponse ` -messages. ADS is not available for REST-JSON polling. - -When the poll period is set to a small value, with the intention of long -polling, then there is also a requirement to avoid sending a -:ref:`DiscoveryResponse ` unless a change to the underlying resources has -occurred via a :ref:`resource update `. - -.. |Multiple EDS requests on the same stream| image:: diagrams/eds-same-stream.svg -.. |Multiple EDS requests on distinct streams| image:: diagrams/eds-distinct-stream.svg