January 23, 1996
 
Table of Contents

1.0: About Miscellaneous Networking Programs
2.0: Debugging Miscellaneous Networking Programs
  2.1: Debugging etherfind and snoop
  2.2: Debugging inetd
  2.3: Debugging portmapper and rpcbind
3.0: Common How Tos
  3.1: How To Use etherfind?
  3.2: How To Use snoop?
  3.3: How To Add a New Service to inetd?
  3.4: How To Cause inetd to Log All requests?
  3.5: How To Add New RPC Services?
4.0: Some Frequently Asked Questions
  4.1: etherfind/snoop Questions
  4.2: inetd Questions
  4.3: portmapper/rpcbind Questions
5.0: Patches
  5.1: etherfind/snoop Patches
  5.2: inetd Patches
  5.3: portmapper/rpcbind Patches
6.0: Known Bugs & RFEs
  6.1: Bugs
7.0: References
  7.1: Important Man Pages
  7.2 Sunsolve Documents
  7.3 Sun Educational Services
  7.4: Solaris Documentation
  7.5: Third Party Documentation
  7.6: RFCs
8.0: Supportability
9.0: Additional Support

1.0: About Miscellaneous Networking Programs
 
This Tip Sheet documents a wide variety of information concerning a
number of the more miscellaneous Sun networking programs. It includes
information on: etherfind, snoop, inetd, portmapper and rpcbind. It is
intended as an introduction to these services, and as a guide to the
most common problems involving them. Other Tip Sheets exist for the
larger Network Packages, including: Automounter, DNS, the Ethernet
Interface, FTP, NIS, NIS+, NFS, PPP, Remote Login programs, Routing
and TCP/IP.

ETHERFIND (SunOS) and SNOOP (Solaris) are programs which root can use
to examine the packets on the network. They are primarily diagnostic
tools, used to debug network problems. Solaris' snoop has much
improved functionality over SunOS' etherfind.

INETD, or the internet daemon, controls many standard internet
programs, including: rlogin, telnet, ftp, tftp, talk and finger. It
automatically starts the programs up, when requests come in, and those
programs are terminated when the remote connection ends.

PORTMAPPER (SunOS) and RPCBIND (Solaris) control all of the RPC
programs, including: NIS, NIS+ and NFS. They register RPC daemons, and
forward RPC requests on to those daemons. Portmapper and rpcbind are
essentially the same program, though rpcbind implements a newer
version of the standard.

2.0 Debugging Misc Network Programs 

Included below are some short notes on debugging each of the programs
detailed in this document.

2.1: Debugging etherfind and snoop

etherfind and snoop are both themselves simple debugging programs, and
there really is not a lot that should go wrong with them. You should
however be aware that etherfind and snoop will not always show you all
the information concerning the machine that etherfind/snoop is running
on.

2.2: Debugging inetd

Section 3.4 describes how you can turn on logging for inetd. This
should provide a vast amount of additional information. inetd also has
a -d flag which can provide even more debugging information.

When debugging inetd problems, you should also carefully examine the
/etc/inetd.conf file. Some inetd.conf parsers are very picky, and if a
particular entry is giving you trouble, you should make sure there are
no control characters in the entry, and that the inetd.conf fields are
seperated by tabs.

/etc/services is also relevant to inetd, as it indexes service names
and port numbers.

2.3: Debugging portmapper and rpcbind
-------------------------------------

The 'rpcinfo' program can be used to investigate problems with
portmapper and rpcbind. It can be used in a number of different ways.

To list the RPC services available from a machine:

  %% rpcinfo -p macine-name

To determine whether a specific TCP service is running:

  %% rpcinfo -t machine-name program-name

To determine whether a specific UDP service is running:

  %% rpcinfo -u machine-name program-name

The list of all of the program-names is in /etc/rpc. This file should
also be investigated for errors.

In general, if rpcinfo is totally failing (even with -p) you should
investigate portmapper/rpcbind on the machine, to be sure that it is
running.

If an individual program is not responding (with -t or -u) that
usually means that either the program is failing, or rpcbind is not
correctly registering that program. The first is more usually the
case, and so based on this information, most investigations should
continue with looking at the individual RPC daemon.

3.0 Common How Tos

3.1: How To Use etherfind?

etherfind is typically used to examine all of the packets going into
or out of a machine, or in between two machines. It is very important
to note that you should usually run etherfind on an uninvolved
machine. In addition, your etherfind must be on the same subnet as the
machines you are examing. For example, if you want to investigate
traffic between machine A and machine B, you should run etherfind on
machine C, which is on the same subnet as either machine A or B.

If you must run etherfind on one of the involved machines, you will
still get some information, but you may not be seeing everything
relevent to the machine.

To see all the packets involving a certain machine:

  # etherfind host machine-name

To see all the packets between two machines:

  # etherfind between machine1 machine2

To see all broadcast packets on a network:

  # etherfind broadcast

Standardly, etherfind will give you the following information about
every packet: length, protocol, source machine, source port,
destination machine, destination port:

  Using interface le0
                                            icmp type
   lnth proto         source     destination   src port   dst port
     60  tcp             psi         crimson       1023      login
     60  tcp             psi         crimson       1023      login
     60  tcp          ellery           prism       7777      34333

A few things can be done to extract more information out of etherfind.

To get more information about RPC packets (including routing, NIS and
NFS):

  # etherfind -r [additional args]

ie:

  # etherfind -r between host1 host2

To get slightly more verbose output:

  # etherfind -v [additional args]

To get the hex values of the packets:

  # etherfind  -x [additional args]

In general, etherfind can give you a limited amount of information
regarding what packets a machine is receiving, and from who. The
etherfind man page gives additional information on other etherfind
options.

etherfind is only available under SunOS.

3.2: How To Use snoop?

Like etherfind (see Section 3.1, above), you may wish to run snoop to
find out information on network packets. The same rules apply: you
need to run snoop on an unrelated machine on the same subnet.

To see all the packets involving a certain machine:

  # snoop machine-name

To see all the packets between two machines:

  # snoop machine1 and machine2

To see all the packets on a network:

  # snoop

Here's what a standard snoop output looks like:

  Using device /dev/le (promiscuous mode)
         jfn35 -> smiley-116   NFS C CREATE FH=5882 .cshrc
       marlowe -> jfn35        TCP D=33237 S=48864     Ack=3354199170
Seq=2006870529 Len=0 Win=8632
    smiley-116 -> jfn35        NFS R CREATE OK FH=6AD4
         jfn35 -> smiley-116   NFS C WRITE FH=6AD4 at 0 for 10
    smiley-116 -> jfn35        NFS R WRITE OK
      sun-soft -> psi          RLOGIN C port=1017 
           psi -> sun-soft     RLOGIN R port=1017 /dev/le (promiscuous

You'll note that each line tells us the sending and the receiving
machine, some information about what program is running, and some of
the arguments that are being passed. For example, on the first line,
we see the file that is being created via NFS ".cshrc" and on the last
line, we see the output being passed on by rlogin "/dev/le
(promiscuous". For most packets, you'll see a message sent out (marked
with a "C"), and then a reply to that message (marked with an "R").

A few things can be done to extract more information out of snoop.

To produce slightly verbose output:

  # snoop -V

To produce very verbose output:

  # snoop -v

The verbose output on snoop is very much so, and it's not very easy to
look at in real time. For that reason, you can save snoop output to a
file:

  # snoop -o filename

You will just see an incrementing number as packets are saved. Hit ^C
when you are down collecting information.

You can then read the information back in:

  # snoop -i filename

You can look at the entire snoop output in verbose mode:

  # snoop -i filename -v

Alternatively, you can view individual packets in verbose mode:

  # snoop -i filename -p12 -v

If you run 'snoop -i filename' and realize that packet 12 is the
interesting one, you would then run the above command to see exactly
what was in the twelfth packet.

snoop can give you extremely good information on exactly what each
packet on the network contains. The snoop man page gives additional
information on other snoop options.

snoop is only available under Solaris.

3.3: How To Add a New Service to inetd?

In order to add a new service to the inetd, you must first add a new
entry to /etc/services, which correlates the service to a port number.
Afterwards, add a new entry to inetd.conf, and restart the inetd:

  # kill -HUP <pid_inetd>

When a call is made to the port number in question, inetd should now
automatically start up your new program.

3.4: How To Cause inetd to Log All Requests?

This is done by adding the -t switch to inetd. This should be done in
the /etc/rc (SunOS) or /etc/rc2.d/S72inetsvc (Solaris). This will log
all service requests to syslog, provided syslog.conf has been setup to
capture daemon.notice type messages (by default daemon.notice message
are logged to /var/adm/messages).

3.5: How To Add a New RPC Service?

When you are adding a new RPC service, you should be sure to update
the file /etc/rpc. Nothing else need be done, as the RPC program will
register itself with portmapper/rpcbind when it starts up.

4.0 Some Frequently Asked Questions

4.1: etherfind/snoop Questions

Q: Why does etherfind fail with the following error:

  "nit open:  no such device"

A: Your kernel configuration file must contain the following lines:

  pseudo-device   snit            # streams NIT
  pseudo-device   pf              # packet filter
  pseudo-device   nbuf            # NIT buffering module
  pseudo-device   clone           # clone device

If any of these are missing, you should add them, and rebuild your
kernel. In addition, make sure that /dev/nit exists. It should be
major number 37, minor number 40.

Q1: Why do I only see half the packets with etherfind/snoop?
Q2: Why do I not see my own machines packets with etherfind/snoop?

A: When you use etherfind or snoop, you will not always see all the
packets going out of your machine, due to limitations in the design.
For this reason, you should start etherfind or snoop on a machine
unrelated to the machines that you want to investigate. In addition,
it must be on the same subnet as one of the machines you wish to
examine. For example, if you wish to look at all the packets between
machines A and B, you must start etherfind/snoop up on machine C,
which is on the same subnet as either A or B. If you start
etherfind/snoop on the involved machines, you will only see some of
the traffic.

(The one exception to the above occurs when you want to verify that a
packet is actually getting to a machine. In that case, it is safe to
run etherfind/snoop on the machine. You will always see all packets
coming into the machine, but may not see all outgoing packets.)

4.2: inetd Questions

Q: What does it mean when the following error constantly prints:

  "<service> server failing (looping), service terminated"

A: Literally, this means that the <service> in question is getting
more than 40 requests in a 60 second period, resulting in requests for
that <service> being denied.

Often, this may be the result of some underlying problem. You may have
some program trying to access the <service> that is in an infinite
loop. You should examine your programs, and make sure that this is not
the case, before you take any other steps.

This can also be the result of faster machines making legitimate
requests for the service at a rapid rate. In this case, you must
reconfigure your inetd to accept requests more rapidly. This can be
done by killing and restarting inetd with the -r option. To allow your
inetd to accept 80 connections every 60 seconds, you would type the
following on SunOS:

  # inetd -r 80 60

And the following on Solaris:

  # inetd -s -r 80 60

These modifications should also be made to /etc/rc (SunOS) or
/etc/rc2.d/S72inetsvc (Solaris) to make the changes permanent.

Note: The -r functionality is not available by default on SunOS, or on
Solaris earlier than 5.4. For those older operating systems you must
install the inetd patch noted in Section 5.0 before you will be able
to use the -r flag.

Q: Why do I constantly get the following two errors:

  "mountd[X]: couldn't register TCP MOUNTPROG"
  "inetd[X]: mountd/rpc/udp server failing (looping), service terminated"

A: This problem occurs most often on SunOS machines. It typically
means that you are starting rpc.mountd from the rc.local, but also
have a line in your inetd.conf:

  mountd/1       dgram   rpc/udp wait root /usr/etc/rpc.mountd   rpc.mountd

You can resolve this problem by commenting out the mountd line in the
/etc/inetd.conf file, and then killing and restarting your inetd:

  # kill -HUP inetd-pid

Q: What does the following error mean, on inetd startup:

  "couldn't find ISTATE environment variable. Try -s flag."

A: This occurs only on Solaris machines. By default, inetd will try
and access SAF, and exit if SAF is not available. The default is to
have SAF not running, and so inetd should usually be started up with
the -s variable:

  # Run inetd in "standalone" mode (-s flag) so that it doesn't have
  # to submit to the will of SAF.  Why did we ever let them change inetd?
  #
  /usr/sbin/inetd -s

4.3: portmapper/rpcbind Questions

Q: What version of the portmapper standard does Sun use?

A: SunOS machines use v.2 of portmapper. Solaris machines use both v.2
and v.3 of portmapper. Due to bugs on some old non-Sun systems that
use v.2 of portmapper, the Solaris v.3 does not always work well on
networks with them. Steps have been taken to correct this in the
portmapper/rpcbind patches listed in Section 5.3 below. In addition,
Solaris 2.5 or higher should automatically deal well with misbehaving
v.2 portmappers. Bug-ID 1169003 explains which bugs precisely were
fixed in these patches.

5.0: Patches 

The following is the list of all of the etherfind/snoop, inetd and
portmapper/rpcbind related patches for 4.1.3, 4.1.3_u1, 4.1.4, 5.3 and
5.4. If you are having problems with those programs, installing the
patches is a good place to start, especially if you recognize the
general symptoms noted below.

In order for a machine to be stable, all of the recommended patches
should be installed as well. The list of recommended patches for your
operating system is available from sunsolve.sun.com.

5.1: etherfind/snoop Patches

101954-06 SunOS 4.1.3_U1: le jumbo patch  
102430-02 SunOS 4.1.4: le patch that fixes sun4m ethernet hang problems  

  Fix a wide variety of problems with the le interface, including one
  which could cause le1 to hang if etherfind was started up. Should be
  installed if you are using etherfind on a machine with more than one
  le interface.

101751-05 TRI/S 3.0.1: Token Ring patch  

  Fixes a wide variety of problems with the tr interface, including one
  which would cause snoop to work poorly. Should be installed on any
  TRI/S 3.0.1 machine, but is a necessity if you are trying to run snoop
  on tr.

101766-10 SunLink FDDI/S 3.0: Jumbo Patch  

  Fixes a wide variety of problems with the FDDI interface, including
  one which could cause snoop to crash the machine. Should be
  installed on any FDDI/S 3.0 machine, but is a necessity if you are
  trying to run snoop over FDDI.

5.2: inetd Patches

100178-09 SunOS 4.1.1 4.1.2 4.1.3: inetd "broken server detection" breaks  
101618-01 SunOS 4.1.3_U1: inetd "broken server detection" breaks on fast  
102416-01 SunOS 4.1.4: inetd patch  

  Fix a number of minor problems with inetd. Also add the -r option to
  inetd, so that faster machines can rapidly access inetd.

101786-02 SunOS 5.3: inetd fixes  

  Fixes a bug which could cause inetd cpu usage to go to 100%%. Also
  adds the -r option to inetd, so that faster machines can rapidly
  access inetd.

102922-03 SunOS 5.4: inetd fixes  

  Fixes a bug which could cause inetd cpu usage to go to 100%.

5.3: portmapper/rpcbind Patches

100482-07 SunOS 4.1 4.1.1 4.1.2 4.1.3: ypserv and ypxfrd Jumbo Patch  

  Corrects certain portmapper security problems related to NIS.

102034-01 SunOS 5.3: portmapper security hole  
102070-01 SunOS 5.4: Bugfix for rpcbind/portmapper  
102071-01 SunOS 5.4_x86: Bugfix for rpcbind/portmapper  

  Each of these correct a security problem in rpcbind.

101318-75 SunOS 5.3: Jumbo patch for kernel (includes libc, lockd)  
101973-15 SunOS 5.4: fixes for libnsl and ypbind  

  Each of these patches includes fixes for libnsl, which affects the
  running of rpcbind. Should be installed on any machine providing
  RPC services. Also include fixes for incompatibilities between
  v.2 and v.3 of rpcbind.

6.0: Known Bugs And RFEs 

This section documents known opens bugs & rfes with the Sun
miscellaneous networking programs.
6.1: Bugs

1184919 print daemon doesn't always start up

  This bug documents a problem with the SunOS inetd when NIS is
  running. It can cause rlogind and other inetd-spawned programs to
  fail with the message "unknown service". This Bug-ID has been closed
  as will not fix. If you are experiencing it, you may work-around it
  by putting a 'sleep 30' statement after the start of 'ypbind' in
  your rc.local file.

7.0 References

7.1: Important Man Pages

etherfind               (etherfind)
inetd                   (inetd)
ientd.conf              (inetd)
nit                     (etherfind)
portmap                 (portmapper)
rpcbind                 (rpcbind)
rpcinfo                 (portmapper/rpcbind)
snoop                   (snoop)

7.2: Sunsolve Documents

The following SRDBs cover some topics not necessarily covered in this
document.

7.2.1: Sun SRDBs

4403    Breakdown of output from etherfind
4725    How to decode Ethernet header from etherfind hex output
4769    How to use rpcinfo program to troubleshoot RPC daemons?

7.3: Sun Educational Services

[pending]

7.4: Solaris Documentation

_System and Network Administration_, Part #800-3805-10 

  The Network Information Service chapter provides some minimal
  information on network programs in general.

7.5: Third Party Documentation

_TCP/IP Illustrated, Volume 1_, by W. Richard Stevens, published by
Addison-Wesley

  An excellent in-depth overview of TCP/IP networking. Includes info
  on how RPC works, and some notes on snoop and etherfind.

_TCP/IP Network Administration_, by Craig Hunt, published by O'Reilly
& Associates

  Contains an overview of networking, including some minimal notes on
  inetd.

8.0: Supportability 

SunService is not responsible for the initial configuration of your
network. In addition, SunService can not diagnose your performance
problems, or suggest performance tuning guidelines.
 
We can help resolve problems where network programs are not behaving
correctly, but in such cases the contact must be a administrator with
a good understanding of the network.

9.0: Additional Support
 
For initial configuration, or performance tuning guidelines, please
contact your local SunService office for possible consulting
offerings. Sun's Customer Relations organization can put you in touch
with your local SunIntegration or Sales office. You can reach Customer
Relations at 800-821-4643.

SunWeb Home SunWeb Search SunSolve Home Simple Search

Sun Proprietary/Confidential: Internal Use Only
Feedback to SunSolve Team