This is Network Expect, version 0.22.


Version 0.22 (July 31, 2014)
-----------------------------

* libwireshark 1.12.x support. As usual, API changes in libwireshark
required minor changes in our code. And as before, the new libwireshark
1.12.x support removes support for previous versions of libwireshark,
i.e. there are no #defines and other tricks to keep support for multiple
versions of libwireshark.

* Fix inability to create bottom of stack MPLS tag. Reported by Yousuke
Takahashi.


Version 0.21 (July 17, 2013)
-----------------------------

* libwireshark 1.10.x support. Minor API changes in libwireshark
required minor changes in our code. libwireshark 1.8 apparently did
not require changes compared to libwireshark 1.6, so if a version of
Network Expect that works with libwireshark 1.8 is needed, the Network
Expect 0.20 can be used. As before, the new libwireshark 1.10.x support
removes support for previous versions of libwireshark, i.e. there are
no #defines and other tricks to keep support for multiple versions of
libwireshark.


Version 0.20 (March 22, 2012)
-----------------------------

* libwireshark 1.6.x support. Unfortunately this means that Network
Expect cannot build against previous libwireshark versions since there
has been no work to support multiple libwireshark versions via #ifdef's.
Wireshark 1.6.x is now required. Use Network Expect 0.19 if Wireshark
1.4.x is needed for some reason.


Version 0.19 (June 09, 2011)
----------------------------

* libwireshark 1.4.x support. Unfortunately this breaks libwireshark
1.2.x support since 1.4.x and 1.2.x are not ABI or API compatible and
there has been no work to support multiple libwireshark versions via
#ifdef's. Wireshark 1.4.x is now required. Use Network Expect 0.18 if
Wireshark 1.2.x is needed for some reason.

* New command "peek" that allows to read arbitrary elements of a BArray
object. Syntax:

peek <barray variable name> [/FMT] [offset]

/FMT: similar to format used in gdb 'x' command and our own 'x'
    and 'barray examine' commands.

offset: offset into BArray object to start grabbing data.

Usage example:

netexpect> spawn_network -i eth0 icmp and host turbo
netexpect> expect_network 1
netexpect> barray dump _(packet)
00000000  0000 0c07 ac01 001f e21a 7380 0800 4500 ..........s...E.
00000010  0054 0000 4000 4001 8ceb 4066 58bf ac12 .T..@.@...@fX...
00000020  6886 0800 9601 0e66 0001 72a2 5e4d 91a4 h......f..r.^M..
00000030  0600 0809 0a0b 0c0d 0e0f 1011 1213 1415 ................
00000040  1617 1819 1a1b 1c1d 1e1f 2021 2223 2425 .......... !"#$%
00000050  2627 2829 2a2b 2c2d 2e2f 3031 3233 3435 &'()*+,-./012345
00000060  3637                                    67
netexpect> peek _(packet) /1xb
0
netexpect> peek _(packet) /1xb 1
0
netexpect> peek _(packet) /1xb 2
12
netexpect> peek _(packet) /1xb 3
7
netexpect> peek _(packet) /1xh 2
3079
netexpect> peek _(packet) /1xh 4
44033
netexpect> calc 0xac01
        44033
netexpect> peek _(packet) /4xh 16
84 0 16384 16385
netexpect> set mylist [peek _(packet) /4xh 0]
0 3079 44033 31
netexpect> set element2 [lindex $mylist 2]
44033

* Random bug fixes here and there; there really aren't many new features
in this release but I am putting it out to have a very recent release
that supports libwireshark 1.4.x since now that libwireshark 1.6.x is
out I want to start working on porting netexpect to it.


Version 0.18 (June 30, 2010)
----------------------------

* libwireshark 1.2.x support. Unfortunately this breaks libwireshark
1.0.x support since 1.2.x and 1.0.x are not ABI or API compatible and
there has been no work to support multiple libwireshark versions via
#ifdef's. Wireshark 1.2.x is now required. Use Network Expect 0.17 if
Wireshark 1.0.x is needed for some reason.

* Improvements to the data() PDU: previously, a data() PDU, i.e. raw
data, used the simple syntax "data(data = <a payload specification>)".
For example:

ip(dst = cisco.com)/icmp-echo()/data(data = '1234567890')

If one wanted to include random data it was necessary to specify so
in the payload specification. For example, to have 20 bytes of random
data one would do:

data(data = 'random:20')

While the payload specification is reasonably flexible, a few improvements
to the data() PDU were easy to implement:

1. Many "data" parameters can now be specified. For example:

data(data = '1234', data = '56', data = '7890')

would produce the raw data "1234567890".

2. There is now a "random" parameter that allow to specify... well,
random data. For example:

data(random(length = 20) )

would specify 20 bytes of random data.

The nice thing about random(length = nn) compared to "random:nn" (in a
payload specification) is that in the former case each time a PDU is
built, new random data is created. In the latter case, the random data
is created at PDU definition time so each time the PDU is built the data
is the same. In other words:

ip(dst = cisco.com)/icmp-echo(id = 1..4)/data(data = 'random:20') ->
4 packets, data never changes (random only once).

ip(dst = cisco.com)/icmp-echo(id = 1..4)/data(random(length = 20) ) ->
4 packets, data is always different (random always).

* Add support for Wireshark configuration profiles. This allows to
use different configuration profiles for Wireshark and netexpect. The
configuration profile to use is specified via nexp's -C option (just
like tshark's -C option).

* In the dumbhex module, we already handled properly numbers that start
with "0x" but we didn't handle properly numbers that start with "\x",
i.e. "\x12\x34\x05". Fixed this so we can import from stuff like:

pkt = (
    "\x18\x40\xf9\x12\x01\x00\xc0\xa8"
    "\x01\x23\x06\xb8\x4a\x00\x32\x00"
)

We continue to be dumb, though, which means that "\x1\x23" is converted
to 0x12, 0x30 instead of 0x01, 0x23.

* Allow to specify a raw data PDU without using "data(data = '...')".
For example, now we can do:

send_network ip(dst = cisco.com)/tcp(dst = 80, psh, ack)/'GET / HTTP/1.0\n\r'

instead of:

send_network ip(dst = cisco.com)/tcp(dst = 80, psh, ack)/data(data = 'GET / HTTP/1.0\n\r')

* Add SNTP PDU builder support. Example of an SNTP PDU definition:

sntp(version = 3, delay = -1.5, dispersion = 65535.4, orig = 35.4, tx = 2.4)

* Add Cisco Dynamic Trunking Protocol (DTP) PDU builder support. This
"should" allow for negotiating a trunk port with a switch, just like one
of the Yersinia attacks.

Here's an example of how to create a valid DTP PDU:

dot3(dst = 01:00:0c:cc:cc:cc)/llc(dsap=0xaa, ssap=0xaa, ctrl = 0x3)/snap(oui='\\x00\\x00\\x0c', code = 0x2004)/dtp(options(tlv(type='DTPDomain', value="mydomain"), tlv(type='DTPType', value="\\xa5"), tlv(type = 'DTPStatus', value="\\x10") ) )

* New "ws" Tcl command that allows to change behavior of some parts
of libwireshark. Right now the only thing that is possible is to set
Wireshark preferences. For example:

ws setprefs ip.defragment:FALSE

* Create Tcl BArray (byte array) objects for libwireshark fields of type
FT_PROTOCOL. This is important because it allows netexpect script to
get full and direct access to a dissected layer/protocol. If a PDU is
fragmented/segmented across multiple packets, and libwireshark is doing
re-assembly then this change means that we get access to re-assembled
data in a very sinple manner.

Re-assembling fragmented/segmented data (IP, TCP, etc.) is not an
easy task given that fragments can arrive out of order, can be
re-transmitted, etc, but leveraging libwireshark for this makes this an
easy job.

Look in the examples directory for a sample script that looks in a PCAP
file for PNG, JPEG, or just generic binaries that have been transferred
over HTTP, and saves anything it finds to separate files on disk.

* New "packet length" command that returns the length of a Tcl "packet"
object.

* Change naming of Tcl variables created by the dissection process:
before we were handling name collisions by keeping track of the position
of the layer the dissection variable belonged to. For example, ip.src
in IP transported by Ethernet (frame.protocols = eth:ip) would get the
variable name ip.src, but IP inside an ICMP message (frame.protocols =
eth:ip:icmp:ip) would get ip.src3. This was not intuitive, incorrect
(since libwireshark may insert non-data protocols like "expert" in some
places), and really hard to handle from Tcl scripts. Now we don't keep
track of layer positions and just name a variable by pre-pending the
names of the protocols all the way to the root of the dissection tree.
For example, for ip.src in eth:ip:icmp:ip, the name of ip.src in the
embedded IP packet would be "icmp.ip.src".

* Create Tcl dissection variables in a specific Tcl namespace.
The actual namespace to use is specified via the global variable
"dissection_ns". This helps prevent pollution of the global Tcl
namespace.

* Do not create a "data" variable with no data if a "data" protocol has
not been dissected. This aligns things better with how libwireshark
dissects packets.

* Do not offer a global Tcl variable that can control whether Tcl
variables created during packet dissection are overwritten if
libwireshark variables with the same name already exist. Instead, we now
never overwrite a variable, and if a variable with the same name exists,
we convert the variable to a Tcl list so we have all different values of
the same variable.

* When making libwireshark dissection results available to the Tcl world
in the form of Tcl variables, make the Tcl variables hold initially a
binary representation instead of a string representation. This should
make dissection faster and make it easier to handle dissection results.

* Add a "-namespace" option to the Tcl "packet dissect" command that
allows to specify a Tcl namespace that will hold dissection variables.

* Add "count" suboption to the "ipaddr" and "num" Tcl commands to
get the number of values that a numspec Tcl object type can get. For
example:

set ip [ipaddr new 192.168.1.0/24]
set n [ipaddr count ip]

will set "n" to 256.

* Workaround OS X's (and possibly other BSDs') seemingly broken select()
system call when usind on BPF file descriptors so the "expect_network"
and "send_expect" commands work on OS X as they work on Linux.

* Add a -cancel switch to the expect_network_background command to
cancel all running expect_network_background commands.

* Other random bug fixes.


Version 0.17 (October 25, 2009)
-------------------------------

* Addition of a "generic" PDU builder to libpbuild so it is (hopefully)
easier to create new PDUs -- now a new protocol can be added to
libpbuild by writing little or, in some cases, no C code. The new
protocol is described by a C structure that contains all the needed
information so the generic builder can put things in place.

* New framework command "send_tcl" that takes a PDU definition and
produces the equivalent binary data as a string. This new command
behaves similar to the "send_network" command but instead of sending
traffic to a network speaker sends traffic to a Tcl object and sets the
result of the command to that object. The Tcl object is actually a list
so if the PDU definition results in several packets then each packet
will be a separate element of the list.

* New framework command "packet new" that takes a PDU definition and
builds it, putting the resulting packet in a Tcl Packet object. Example:

netexpect> set pkt [packet new ip(dst = 192.168.1.1)/icmp-echo()/data(data = '1234')]
 10.10.11.40 -> 192.168.1.1  ICMP Echo (ping) request
netexpect> packet stats pkt
Protocol      % Packets      Packets        Bytes  End Packets    End Bytes
===========================================================================
frame           100.00%            1           32            0            0
  raw           100.00%            1           32            0            0
    ip          100.00%            1           32            0            0
      icmp      100.00%            1           32            1           32
1 packet, 32 bytes

* It is now possible to import hexadecimal dumps into Byte Arrays or
directly into Packet objects by using the "hex-import" sub-command of
the "barray" and "packet" commands. For example:

netexpect> set hexdump {
>0x0000:  4500 0054 0000 4000 4001 5f5f 0a74 bc3d
>0x0010:  ac12 6886 0800 c565 d748 0001 284f d64a
>0x0020:  6cb3 0500 0809 0a0b 0c0d 0e0f 1011 1213
>0x0030:  1415 1617 1819 1a1b 1c1d 1e1f 2021 2223
>0x0040:  2425 2627 2829 2a2b 2c2d 2e2f
>}
netexpect> set pkt [packet hex-import -ignore-prefixes 1 $hexdump]
10.116.188.61 -> 172.18.104.134 ICMP Echo (ping) request
netexpect> packet show pkt
[...]

This comes handy when one needs to analyze packets stored in memory
buffers and we can obtain an hexadecimal dump of those buffers, for
example.

* Add pbuild PDU support for Ethernet 802.3 frames.

* Add pbuild PDU support for 802.2 Logical Link Control.

* Add pbuild PDU support for 802.2 SNAP.

* Add pbuild PDU support for Cisco VLAN Trunking Protocol (VTP).

* This should be the last release that support libwireshark 1.0.x since
1.2.x is not API- or ABI-compatible with 1.0.x.

* Use the same configure.ac code used by the Tcl extension tclreadline
to find the Tcl library and the include file.


Version 0.16 (September 12, 2009)
---------------------------------

* New CLI-based program "tgn" that allows to easily generate traffic
from a shell prompt. A couple of examples:

shell# tgn -o eth0 "ip(dst = 192.168.1.1)/icmp-echo(id = 'random')"

shell# tgn "ip(src = 192.168.0.1, dst = <192.168.0.10, 192.168.0.11>," \
    "ttl = <1, 2>)/tcp(src = 'random', dst = 22..25, window = 16384," \
    "syn, seq = 'random', ack-seq = 0)"

shell# tgn -w /tmp/cap -c 5 "ether()/ip(dst = 1.2.3.4++)/" \
    "icmp-echo(seq = 0++)" && wireshark /tmp/cap

The PDU definition uses the same syntax as PDU definitions in the
"send_network" command.

See tgn(1) for details.

* Work on the netexpect-libwireshark has been merged into the trunk.
Dropping "libwireshark" from version numbers because of this.


Version 0.15libwireshark (September 3, 2009)
--------------------------------------------

It's been 15 months since the last, official, Network Expect release.
Life has been busy. However, Network Expect development hasn't been
completely dead, as you'll see from the following release notes:

- Several improvements to Network Expect's PCAP re-write and replay
capabilities. The Network Expect wiki has a page dedicated to PCAP
re-write replay scenarios:

http://netexpect.org/wiki/RewriteAndReplay.

- New CLI option "-fixups" for the "send_network" command that "fixes
up" a packet before sending it. "fixing a packet up" means recalculating
layer 3 and 4 protocol checksums (IP, UDP, TCP, ICMP, IGMP). Useful in
PCAP re-writing and replaying scenarios.

- New Tcl command "islocal" that receives as a parameter a network
address (a.b.c.d/nn or a.b.c.d/nn.nn.nn.nn) and an IP address, and
returns 1 if the IP address is local to the network address or 0 if it
is remote.

For an example of how this can be used see:

http://netexpect.org/wiki/moin.cgi/RewriteAndReplay#head-a62fd39814b7ea2518944febda2abd0a7dbfbfbe

- Add a new Tcl object to represent numeric specifications (numspecs)
offered by the libnumbers module. Also add two new Tcl commands, "num"
and "ipaddr" to manipulated numeric specifications. Here's an example of
their use:

netexpect> set number [num new 1++]
1++
netexpect> num next number
1
netexpect> num next number
2
netexpect> num next number
3
netexpect> num next number
4
netexpect> set addr [ipaddr new 192.168.1.1/24]
192.168.1.1/24
netexpect> ipaddr next addr
192.168.1.0
netexpect> ipaddr next addr
192.168.1.1
netexpect> ipaddr next addr
192.168.1.2
netexpect> ipaddr next addr
192.168.1.3
netexpect> 

- Add to libnumbers the ability to have numeric specifications that
increment IP addresses, like in:

netexpect> set base_ip [ipaddr new 192.168.1.1++]
192.168.1.1++
netexpect> for {set i 0} {$i < 10} {incr i} {puts [ipaddr next base_ip]}
192.168.1.1
192.168.1.2
192.168.1.3
192.168.1.4
192.168.1.5
192.168.1.6
192.168.1.7
192.168.1.8
192.168.1.9
192.168.1.10
netexpect> 

- New "packet stats" Tcl command that provides the same functionality
as Wireshark's Protocol Hierarchy Statistics dialog but for our Tcl
"packet" objects. Here's an example of its use with a TCP traceroute:

----------------------------------------------------------------------
netexpect> spawn_network -i eth0
netexpect> send_expect -tries 2 -delay 0.010 ip(id = random, dst = cisco.com, ttl=1..30)/tcp(src = random, dst = 80, s
yn)
Begin stimulus injection (pass 1/2)... done (sent 30 packets)
...........!.!.!.!..!!.!...!.!.!...!.!.!..!.!.!.!.!.!.!..!..!..!..!..!.!.!.!.!.!
Sent 30 packets; received 30 answers; 0 unanswered
netexpect> packet stats _(received)
Protocol      % Packets      Packets        Bytes  End Packets    End Bytes
===========================================================================
frame           100.00%           30         1990            0            0
  eth           100.00%           30         1990            0            0
    ip          100.00%           30         1990            0            0
      icmp       63.33%           19         1330           19         1330
      tcp        36.67%           11          660           11          660
netexpect> packet stats _(sent)
Protocol      % Packets      Packets        Bytes  End Packets    End Bytes
===========================================================================
frame           100.00%           30         1200            0            0
  raw           100.00%           30         1200            0            0
    ip          100.00%           30         1200            0            0
      tcp       100.00%           30         1200           30         1200
----------------------------------------------------------------------

- A common technique used when generating network traffic is spoofing.
Spoofing is sometimes necessary because one doesn't want the operating
system where the traffic generation tool is running to generate traffic
(like a TCP RST) that may disrupt the test.

One of the challenges related to spoofing traffic on Ethernet networks
is that return traffic may not reach the system where the traffic
generation tool is being run because there is nothing that responds to
ARP requests for the MAC address of the (ghost) system with the spoofed
(and unallocated) IP address.

Because spoofing is a very common task, Network Expect 0.15libwireshark
adds a new command to the Network Expect framework that allows to easily
create "ARP responders" that will respond to ARP requests for spoofed IP
addresses.

The new command is called "ghost" (because I call these "ARP responders"
"ghosts") and here's a quick introduction of how it's used:

----------------------------------------------------------------------
netexpect> show ghosts
Ghosts:
  Number of ghosts: 0 (0 ghost control blocks)
netexpect> ghost with ip 192.168.1.0/24 and mac 00:00:5e:cc:90:9f on interface eth0
netexpect> ghost with ip 10.10.10.10 and mac random on interface eth0
netexpect> show ghosts
Ghosts:
  Number of ghosts: 2 (1 ghost control block)
  Ghost #0:
    Name: ghost0
    Interface: eth0
    MAC address: 00:00:5e:cc:90:9f
    IP address: 192.168.1.0/24
  Ghost #1:
    Name: ghost1
    Interface: eth0
    MAC address: 00:00:5e:ef:93:b7
    IP address: 10.10.10.10
netexpect> ghost kill ghost0
netexpect> ghost kill ghost1
netexpect> show ghosts
Ghosts:
  Number of ghosts: 0 (0 ghost control blocks)
netexpect> 
----------------------------------------------------------------------

A ghost will respond to both ARP requests and ICMP echo requests to its
IP address (the later because it's a nice troubleshooting tool to make
sure ghosts are reachable.)

This concept, by the way, is not a new one - other tools have it
(Unicornscan and its fantip and arpd from honeyd, for example) but
netexpect was missing it, and I think it's definitely a must. Otherwise
you have to go outside the framework to get it.

- New framework command "tgn" (for Traffic Generator) that allows to
start traffic generators that generate traffic in the background while
script processing continues in the foreground. Each traffic generator is
implemented with a separate thread.

Usage of the "tgn" command is as follows:

tgn create -rate 100 ip(dst = 10.0.0.1)/tcp(syn, src = random, dst = 80)
tgn info
tgn start
tgn stop
tgn reset
tgn destroy

There is no limit on the number of concurrent traffic generators. TGNs
are created with "tgn new" and destroyed with "tgn destroy". Each
generator can be independently started or stopped via "tgn start" and
"tgn stop" respectively. "tgn info" provides information about the
existing TGNs.

- It is now possible to create a valid TCP or UDP PDU without a valid IP
packet that carries the TCP or UDP PDU. To be able to do this, TCP or
UDP PDU definitions now accept "srcip" and "dstip" parameters to specify
source and destination IP adresses that will be used for TCP or UDP
checksum calculation.

- Added a sample Naptha implementation using netexpect's ghost and tgn
commands. Documented this at http://netexpect.org/wiki/Naptha.

- Big re-write of the libpdus subsystem in an attempt to make it
easier to develop new PDUs. During the re-write, libpdus was renamed
to libpbuild (packet build library; think the opposite of libpcap, the
packet capture library).

libpbuild is the netexpect subsystem responsible for PDU creation.
netexpect commands like "send_network" use libpbuild to create PDUs
based on PDU definitions supplied by the user, e.g.
"send_network ip(dst = cisco.com)/icmp-echo(id='random')".

There's a lot of room for improvement. In particular, it'd be nice for
libpbuild to also free the PDU developer from having to write PDU build
code for the most simple PDU parts (like Scapy does). The improvements
will have to wait, though, and in the mean time, what we have now should
be better than what we had.

Note: this 0.15libwireshark builds against Wireshark 1.0.x. I have not
tried yet to build against Wireshark 1.2.x but it's in the to-do list.


Version 0.14libwireshark (June 5, 2008)
---------------------------------------

- When dissecting certain protocols libwireshark may re-use field
names. For example, the dissector for the bootp protocol will use
bootp.option.type, bootp.option.length, and bootp.option.value for each
of the bootp options in a bootp message. With the current approach of
one Tcl variable per field name we only have access to the value of
the last field since previous values get overwritten. This release now
detects reuse of field names and creates Tcl lists for such cases so
scripts can have access to everything.

As an example of the new behavior, the following script fragment would
print out the length of bootp option 50 in a bootp message:

expect_network {<condition that gets us a bootp message>} {
    # "expect_network" just dissected a bootp message

    # Get index of length corresponding to option 50
    set index [lsearch ${bootp.option.type} 50]

    if {$index != -1} {
        # Get the length
        set length [lindex ${bootp.option.length} $index]

        puts "Length of option 50 is $length"
    }
}

- Add a new global Tcl variable ("overwrite") that turns on or off the new
behavior described above. It defaults to 0, which means do not overwrite
fields with same names.

- Add the capability to use libwireshark display filters. Just as PCAP
capture filters, libwireshark display filters are tied to network
listeners. libwireshark display filters can only be used with the
"expect_network" command, i.e. other commands that read packets from
listeners, like "send_expect" don't use libwireshark display filters.

libwireshark display filters are defined via the "-R" CLI switch (just
like tshark's "-R" switch) of the "spawn_network" command.

The following two commands are equivalent, but one uses a PCAP filter
and the other a libwireshark display filters:

spawn_network -i eth0 -R "icmp && ip.src == 1.2.3.4"
spawn_network -i eth0 icmp and src host 1.2.3.4

Version 0.13libwireshark (May 29, 2008)
---------------------------------------

- Added example of a simple TCP traceroute that uses send_network and
expect_network commands, instead of the more powerful send_expect
command.

- Broken macros in byteorder.h were causing endianess problems on all
architectures but i386. Refreshed byteorder.h with a fresh copy from the
Samba source code. This is the most important fix in this release.

- Explicitely include stddef.h so compilation of
src/pdus/builders/pdu-*.c doesn't fail on some systems.

- Replace the "-peername" switch of the "spawn_network" Tcl command with
a "-4tuple" switch that returns a Tcl list of four members that consists
of local IP address, local port number, remote IP address, and remote
port number. See the 0trace.nexp example to see how to use this.

- Support the Linux "cooked" mode link layer encapsulation in the
"send_expect" command. Useful when using the universal TUN/TAP driver.

- Minor improvements to the 0trace.nexp example.

Version 0.12libwireshark (May 15, 2008)
---------------------------------------

- Proof of concept version developed on a separate SVN branch that uses
libwireshark from the Wireshark project for all packet dissection tasks.
This gives Network Expect access to hundreds of protocols supported by
Wireshark.

The negative side of this new reliance on libwireshark is that the build
process has become a little more complicated. I have only tested the
build process on my Debian unstable development environment so I am sure
building on other platforms and even Linux distribution is bound to hit
some snags.

Version 0.11
------------

- Initial public release.
