Wednesday, December 2, 2009

IP Tunneling and Amazon VPC

Glenn Brunette has successfully been able to access the Amazon Virtual Private Cloud using OpenSolaris as a customer gateway.  He describes the general concept in his blog along with pointers to a tool he's developed to automate the configuration.  This OpenSolaris customer gateway uses core technologies such as IP tunneling, IPsec, and BGP to provide redundant secure links to the cloud (see the functional diagram that Glenn provides for a depiction of this).

The IP tunnel configuration using OpenSolaris allowed Glenn and his team (including Dan McDonald and others) to troubleshoot the operation of BGP over these tunnels using the observability provided by the Clearview project (integrated in OpenSolaris build 125), which would not have been possible before.

Thanks for doing this fantastic work Glenn!

Thursday, November 12, 2009

Fluendo DVD player initial reaction

I bought the Fluendo DVD player for OpenSolaris last night, and there seem to be some very rough edges.  For one, the /usr/bin/fluendo-dvd script doesn't work out of the box and spews shell syntax errors.  It assumes that "/bin/sh" is actually bash, which isn't the case on OpenSolaris.  Changing the first line of the script to "#!/bin/bash" fixes the problem and the binary launches.

I've installed the player on two systems, both running development build 126 of OpenSolaris.  One is my Sun Ultra 40 desktop, and the other is my Toshiba Portege r500 laptop.  One common gripe in general is that the player has no control buttons at all (e.g. play, stop, pause, etc.).  To control the player, one needs to go through the "DVD Player" menu, which is very odd.
There is also no evidence of the capability to navigate forward or backwards through a movie at higher or lower rates.

OpenSolaris itself is not contributing to a positive user experience, as after watching a movie for any more than ten minutes results in the audio stream being corrupted by what sounds like static clicks and hisses.  Stopping and restarting the player causes the audio issue to go away, but it comes back after a short time.  This exact same issue occurs for other gstreamer applications, so this is not fluendo-dvd player problem.  There is likely a bug in the audio framework.

Aside from these common issues, the player is unable to play movies on my Toshiba Portege laptop.  The first time I attempted to play a movie (after having applied the above fix to the launcher script), the application crashed with a segmentation fault.  I have the core dump if anyone from Fluendo wishes to debug the issue (the segfault occurs in fluendo_css_descrambler_descramble()).  From that point on, any attempt to play a movie results in the following popup.
I'm not sure what to make of that.  Perhaps there are some file permission issues on this system, but the error is cryptic enough that there's no hope of diagnosing what the problem is.

On the positive side, on the one system I'm able to get it working, the video quality is excellent.

The laptop is the main platform from which I'd like to use this, so the current situation is disappointing.  I'm hoping that these are simple bugs that can be expediently fixed, and that the mail I sent to the support channel at Fluendo last night will be answered (I'd expect so since the 20 Euros one pays for this includes 1 year of support).

OpenSolaris DVD player from Fluendo

Fluendo released their DVD player for OpenSolaris today!

http://www.fluendo.com/shop/product/fluendo-dvd-player/

Thursday, October 8, 2009

IPv6 in Shared-Stack Zones

I was recently at an OpenSolaris user-group meeting where a question was asked regarding how IPv6 could be used from a shared-stack zone.  For the benefit of anyone who has a similar question, here is an example of a working configuration:


bash-3.2# zoneadm list -iv
  ID NAME             STATUS     PATH                           BRAND    IP    
   0 global           running    /                              native   shared
   - test             installed  /export/home/test              native   excl  
   - test2            installed  /export/home/test2             native   shared

The exclusive-stack zone "test" has all of its own networking configured within it, so IPv6 inherently just works there.  The question, however, was about shared-stack, and so I setup the "test2" zone to demonstrate this.


bash-3.2# zonecfg -z test2
zonecfg:test2> add net
zonecfg:test2:net> set physical=e1000g0
zonecfg:test2:net> set address=fe80::1234/10
zonecfg:test2:net> end
zonecfg:test2> add net
zonecfg:test2:net> set physical=e1000g0
zonecfg:test2:net> set address=2002:a08:39f0:1::1234/64
zonecfg:test2:net> end
zonecfg:test2> verify
zonecfg:test2> commit
zonecfg:test2> exit
bash-3.2# zonecfg -z test2 info
zonename: test2
zonepath: /export/home/test2
brand: native
...
net:
address: 10.8.57.111/24
physical: e1000g0
defrouter not specified
net:
address: fe80::1234/10
physical: e1000g0
defrouter not specified
net:
address: 2002:a08:39f0:1::1234/64
physical: e1000g0
defrouter not specified

Here I configured a link-local address fe80::1234/10, and a global address 2002:a08:39f0:1::1234/64.  Each interface within each zone requires a link-local address for use with neighbor-discovery, and the global address is the address used for actual IPv6 communication by applications and services. The global address' prefix is one that is configured on the link to which the interface is connected.  In the zone, we end up with:

bash-3.2# zlogin test2 ifconfig -a6
lo0:1: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1
        inet6 ::1/128
e1000g0:2: flags=2000841<UP,RUNNING,MULTICAST,IPv6> mtu 1500 index 2
        inet6 fe80::1234/10
e1000g0:3: flags=2000841<UP,RUNNING,MULTICAST,IPv6> mtu 1500 index 2
        inet6 2002:a08:39f0:1::1234/64

The global zone has IPv6 connectivity using this same prefix as well as a default IPv6 route: [2]

bash-3.2# netstat -f inet6 -rn

Routing Table: IPv6
Destination/Mask            Gateway                   Flags Ref   Use    If
--------------------------- --------------------------- ----- --- ------- -----
2002:a08:39f0:1::/64        2002:a08:39f0:1:214:4fff:fe1e:1e72 U       1       0 e1000g0:1
fe80::/10                   fe80::214:4fff:fe1e:1e72    U       1       0 e1000g0
default                     fe80::1                     UG      1       0 e1000g0

From the non-global zone, we have IPv6 connectivity:

bash-3.2# zlogin test2 ping -sn 2002:8194:aeaa:1:214:4fff:fe70:5530
PING 2002:8194:aeaa:1:214:4fff:fe70:5530 (2002:8194:aeaa:1:214:4fff:fe70:5530): 56 data bytes
64 bytes from 2002:8194:aeaa:1:214:4fff:fe70:5530: icmp_seq=0. time=4.654 ms
64 bytes from 2002:8194:aeaa:1:214:4fff:fe70:5530: icmp_seq=1. time=2.632 ms
64 bytes from 2002:8194:aeaa:1:214:4fff:fe70:5530: icmp_seq=2. time=2.501 ms
64 bytes from 2002:8194:aeaa:1:214:4fff:fe70:5530: icmp_seq=3. time=2.571 ms
^C
----2002:8194:aeaa:1:214:4fff:fe70:5530 PING Statistics----
4 packets transmitted, 4 packets received, 0% packet loss
round-trip (ms)  min/avg/max/stddev = 2.501/3.090/4.654/1.044

The zone can then be configured to use DNS or local hosts to resolve names to IPv6 addresses in order to utilize IPv6 more effectively.

Friday, September 25, 2009

Clearview IP Tunneling in OpenSolaris

I integrated Clearview IP Tunneling (the final component of the Clearview project) into the ON consolidation this week.  It will be included in OpenSolaris build 125 which will make its way to the dev repository in due time.  Thanks to all who participated including the Clearview project team (past and present), and members of various OpenSolaris communities who contributed by doing design and code reviews.  This brings a close to a project that Meem and I conceived years ago while doodling network interface requirements on his whiteboard.  We've now delivered every component that we initially identified as the solutions to meet our requirements.  That's something to be proud of.

With this integration, IP tunnel links can be created using dladm, be given meaningful names using link vanity naming, observed using traditional network observability tools such as snoop and wireshark, assigned to exclusive stack non-global zones, and created from within non-global zones.

This integration also enables the use of dladm in general from within exclusive stack non-global zones.  Aside from the IP tunnel subcommands which are supported from such zones, all of the show-* subcommands now work in such zones, allowing administrators to view datalink configuration pertinent to the zone.  This is a first step towards gradually expanding the set of datalink features available in zones.

Enjoy, and feel free to communicate with us regarding this project at clearview-discuss@opensolaris.org.

Monday, November 17, 2008

Observe Loopback and Inter-Zone IP Packets With OpenSolaris

I'm happy to announce that the IP Observability Devices component of the Clearview project has integrated into OpenSolaris build 103 (also see Phil Kirk's announcement to the ON community).  This adds the following new capabilities to OpenSolaris:

  • Network observability at the IP layer for traditional DLPI-based tools such as snoop
  • Observability of loopback IP packets
  • Observability of inter-zone IP packets
  • Tools such as snoop can be run from within a non-global zone to observe packets associated with that zone
  • Snoop filtering based on zone id

The snoop command has grown a new "-I <interface-name>" option to access this feature.  Its semantics are to snoop the IP interface named <interface-name> at the IP layer.  When observing a particular IP interface with this facility, packets that have a source or destination IP address assigned to that interface can be observed, as well as packets that are forwarded to or from that IP interface, and broadcast and multicast packets received by that interface.  Additional internal filtering is performed to ensure that an observer from a non-global zone can only see packets that belong to that zone, with the exception of the global zone, from which packets to or from any zone that shares its stack can be observed.  Any IP interface visible through "ifconfig -a" can be observed using this feature.

We are also working towards integrating support for these IP Observability Devices into Wireshark and tcpdump in the near future.


Here are some examples using snoop:

Example 1: Observing the Loopback Interface


bash-3.2# snoop -I lo0
Using device ipnet/lo0 (promiscuous mode)
localhost -> localhost    ICMP Echo request (ID: 37110 Sequence number: 0)
localhost -> localhost    ICMP Echo reply (ID: 37110 Sequence number: 0)

The lo0 interface has the 127.0.0.1 address assigned to it, and so any communication using the address 127.0.0.1 is seen above (in this case, I was simply doing "ping 127.0.0.1").  Snoop's verbose output mode displays a new "ipnet" header that precedes all IP packets observed:

bash-3.2# snoop -v -I lo0
Using device ipnet/lo0 (promiscuous mode)
IPNET:  ----- IPNET Header -----
IPNET: 
IPNET:  Packet 1 arrived at 10:40:33.68506
IPNET:  Packet size = 108 bytes
IPNET:  dli_version = 1
IPNET:  dli_type = 4
IPNET:  dli_srczone = 0
IPNET:  dli_dstzone = 0
IPNET: 
...

Note above that the source and destination zone ids are displayed.  In this case, I was running "ping 127.0.0.1" in the global zone, and so both the source and destination zone ids are "0".


Example 2: Running Snoop From a Non-Global Zone


bash-3.2# zoneadm list -v
ID NAME             STATUS     PATH                           BRAND    IP
0 global           running    /                              native   shared
4 test             running    /zones/test                    native   shared
bash-3.2# zlogin test
[Connected to zone 'test' pts/2]
...
bash-3.2# ifconfig -a
lo0:1: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
bge0:1: flags=201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500 index 2
        inet 10.8.57.34 netmask ffffff00 broadcast 10.8.57.255
lo0:1: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1
        inet6 ::1/128
bge0:2: flags=202000841<UP,RUNNING,MULTICAST,IPv6,CoS> mtu 1500 index 2
        inet6 2002:a08:39f0:1::f/64
bash-3.2# snoop -I bge0
Using device ipnet/bge0 (promiscuous mode)
whitestar1-2.East.Sun.COM -> mf-ubur-01.East.Sun.COM DNS C 253.57.8.10.in-addr.arpa. Internet PTR ?
mf-ubur-01.East.Sun.COM -> whitestar1-2.East.Sun.COM DNS R 2.0.0.224.in-addr.arpa. Internet PTR ALL-ROUTERS.MCAST.NET.
whitestar1-6.East.Sun.COM -> whitestar1-2.East.Sun.COM TCP D=22 S=62117 Syn Seq=195630514 Len=0 Win=49152 Options=<mss
whitestar1-2.East.Sun.COM -> whitestar1-6.East.Sun.COM TCP D=62117 S=22 Syn Ack=195630515 Seq=195794440 Len=0 Win=49152
whitestar1-6.East.Sun.COM -> whitestar1-2.East.Sun.COM TCP D=22 S=62117 Ack=195794441 Seq=195630515 Len=0 Win=49152
whitestar1-2.East.Sun.COM -> whitestar1-6.East.Sun.COM TCP D=62117 S=22 Push Ack=195630515 Seq=195794441 Len=20 Win=491

Although not evident from the snoop output above, whitestar1-2 is 10.8.57.34 (the bge0:1 IP address in this non-global zone), and whitestar1-6 is actually an IP address in another zone on the same system.  By snooping the bge0 interface, the user sees all packets associated with the bge0 IP addresses in the zone; even those that are locally delivered to other zones.  Using snoop's verbose output mode allows us to see which zones these packets are flowing between:

bash-3.2# snoop -v -I bge0 whitestar1-6
Using device ipnet/bge0 (promiscuous mode)
IPNET:  ----- IPNET Header -----
IPNET: 
IPNET:  Packet 1 arrived at 10:44:10.86739
IPNET:  Packet size = 76 bytes
IPNET:  dli_version = 1
IPNET:  dli_type = 4
IPNET:  dli_srczone = 0
IPNET:  dli_dstzone = 4
IPNET: 
...

We can see above that the packet was from the global zone to the test zone.

Example 3: Filtering by Zone ID


Filtering by zone id can be useful on a system that has multiple zones.  In this example, an administrator in the global zone observes packets being sent to or from IP addresses in the "test" zone.

bash-3.2# zoneadm list -v
ID NAME             STATUS     PATH                           BRAND    IP
0 global           running    /                              native   shared
4 test             running    /zones/test                    native   shared
bash-3.2# snoop -I bge0 zone 4
Using device ipnet/bge0 (promiscuous mode)
whitestar1-6.East.Sun.COM -> whitestar1-2.East.Sun.COM TCP D=22 S=61658 Syn Seq=374055417 Len=0 Win=49152 Options=<mss
whitestar1-2.East.Sun.COM -> whitestar1-6.East.Sun.COM TCP D=61658 S=22 Syn Ack=374055418 Seq=374124525 Len=0 Win=49152
whitestar1-6.East.Sun.COM -> whitestar1-2.East.Sun.COM TCP D=22 S=61658 Ack=374124526 Seq=374055418 Len=0 Win=49152

This can be particularly useful with the loopback interface, as the 127.0.0.1 address is shared among all shared-stack zones, and it can be difficult to associate a loopback packet to an application in a zone.

Note that there is a pending RFE to also be able to enter a zone name as well as a zone id as the argument to the snoop "zone" filtering primitive.  For now, the zone id is the only allowable argument.

Thursday, June 5, 2008

Clearview Vanity Naming BigAdmin Article

I expanded upon one of my previous blog entries on network datalink vanity naming in OpenSolaris into a more thorough article with more examples.  The result is the following BigAdmin article:

http://www.sun.com/bigadmin/sundocs/articles/vnamingsol.jsp

Enjoy.