                  Zebra hacking notes
                  for: GNU Zebra 0.93b
                  by: Pilot
                  rev: 1.6
                  URL: http://pilot.org.ua/zebra/

Conventions:
"zebra" is GNU Zebra software AKA Zebra routing suite AKA the
whole pack of binaries.
"zebrad" is the main zebra daemon AKA platform abstraction level
"protod" is any of protocol daemons: ospfd, bgpd etc.
All file paths are relative to zebra source topdir e.g. zebra-0.93b/
--> means direct function call
[-->] means addition of thread onto master
RT is routing table
RR is route record
Note: I use tab width 2

Source tree layout:
lib/ contains static lib, which protod links and uses.


There are 7 binaries in full zebra installation. Locations of main():
zebra:  zebra/main.c
ospfd:  ospfd/ospf_main.c
bgpd:   bgpd/bgp_main.c
ospf6d: ospf6d/ospf6_main.c
ripd:   ripd/rip_main.c
ripngd: ripngd/ripng_main.c
vtysh:  vtysh/vtysh_main.c


Upon startup some functions are called, cmd_init() (lib/command.c) is among them.
cmd_init() initializes protod's CLI with the same set of basic commands:
quit, help, enable, show run etc. All daemons run cmd_init(1)

Every protod has its own instance of variable cmdvec (lib/command.c).
cmdvec is a vector (lib/vector.h) of structures of type cmd_node (lib/command.h).
There are 5 nodes initially installed by cmd_init() (command.c):
auth_node
view_node
auth_enable_node
enable_node
config_node
Each cmdvec's node defines current atomic state of the current copy of protod's
CLI and corresponding command prompt. e.g. when CLI is in state view_node, you
see something like "router>" and when in enable_node, "router#".

When cmd_init() installs that initial nodes, each one is assigned a handler for
dumping its current portion of config file.

Then each node is filled with its set of commands. There is an argument (int terminal)
to cmd_init, which defines whether "interactive" commands should be put into
nodes or not. I currently see no clear reason to do that, but vtysh/vtysh.c
calls cmd_init(0).
Finally cmd_init() calls srand(time(NULL)) for unknown reason.

So cmdvec contains nodes, nodes contain lists of commands. cmd_init() is done.

Then debug_init() (protod/xxxx.c) is called. Each protod defines a separate
debug_init(). Except ospf6d. Except vtysh.
debug_init() locations:
zebra:  zebra/debug.c
bgpd:   bgpd/bgp_debug.c
ospfd:  ospfd/ospf_dump.c
ripd:   ripd/rip_debug.c
ripngd: ripngd/ripng_debug.c


Next call: vty_init(). (lib/vty.c)
vty_init() adds new node vty_node into cmdvec and fills it with commands relevant
to vty line. It also extends nodes view_node, enable_node, config_node by additional
commands.

Next call: run memory_init(). (lib/memory.c) This call extends view_node and enable_node
with various "show memory xxx" commands. Except ripngd.

Next call: access_list_init() (lib/filter.c). This function adds commands
for managing different kinds of access lists. IPv6 case is handled too.

Next call: prefix_list_init() (lib/plist.c). The same as above for prefix lists.

Now we have a standard set of commands. Time to learn more protod specific commands
and to do more initialization.

protod_init() (same file as main()) comes to the scene. Very specific code for
each protod inside.
NOTE: thread creation demands on this process, see below.

Each routing daemon is zebrad client via UNIX/TCP socket.
Data relevane to communication between protod and zebrad are stored in zclient structure.
Files: lib/zclient.[h|c]
zclient has its own event classification:
enum event {ZCLIENT_SCHEDULE, ZCLIENT_READ, ZCLIENT_CONNECT};

zclient_event() places events into protod master queue
schedule zclient_connect() as EVENT
connect - zclient_connect() as TIMER
read - zclient_read() as READ
Stack: zclient_init() [--->] zclient_connect() ---> zclient_start() [--->] zclient_read()
zclient_read() registers itself each time before return
Where is the rest of engine ? It is created during XXX_vty_init() and friends.
Each CLI command has its handling function, which in turn spawns new thread (again, not
real thread, but "execution request"). When protod starts and sees 'router XXX' clause,
protod process gets new thread ! After parsing subsequent commands, e.g. network 10.0.0.0/8,
more threads get created. Finally you get engine filled with hord of hooks.

Function: protod_zebra_init(). (protod/XXXX_zebra.c) protod connects to zebrad.
{
May be called XXX_zclient_init().
Inside above zclient_init() (lib/zclient.c) is called. This function performs misc setup
for protod routing structures. One inmportant thing: event is sheduled via zclient_event()
(lib/zclient.c). This means that new thread(s) will be added to master thread structure,
which will initiate connection to zebrad.
After that record zclient gets filled with hooks for routing events: interface/route up/down,
add/remove, and so on. Hooks for access list updates are assigned separately via
access_list_[add|delete]_hook (lib/filter.c). Except ospf6d, except vtysh.
XXX_zebra_init() is done.
}

It's time to educate protod's CLI with its cpecific commands. XXX_vty_init(), XXX_vty_show_init()
and XXX_vty_if_init() do it.

Common section again:

sort_node(). Elements (commands) gets sorted inside their nodes.

struct master contains 1 thread at that moment: sheduled event to connect to zebrad.

Now config can be read and understood: vty_read_config() (lib/vty.c) does something
like the following:
{
look around()
check ()
look around()
fopen()
vty_read_file()
fclose()
host_config_set()

vty_read_file() (lib/vty.c) reads the FD line-by-line.
First a vty is created. Then it is switched into terminal mode and config_node and its
config_node is filled with contents of the file: config_from_file() (lib/command.c).
config_from_file() calls cmd_execute_command_strict() (lib/command.c) for each line of the file,
because current node is config_node (in keychain_key_node cmd_execute_command_strict() behaves
differently).

host_config_set() (lib/command.c) remembers full path for the file from which current
configuration was read. Config will be saved to the same file from which it was read. ?

}

Important note on vty_read_config():
/*
void
vty_read_config (char *config_file,
         char *config_current_dir,
         char *config_default_dir)
You should remember what happens before this function is called (ospfd example):
vty_read_config (config_file, config_current, config_default); // ospf_main.c:main()

#define OSPF_DEFAULT_CONFIG   "ospfd.conf" // ospfd.h
char config_current[] = OSPF_DEFAULT_CONFIG; // ospf_main.c
char config_default[] = SYSCONFDIR OSPF_DEFAULT_CONFIG; // ospf_main.c
char *config_file = NULL; // ospf_main.c:main()

SYSCONFDIR is defined by configure script
config_file is initialized from -f option and NULL otherwise

step A. if (config_file != NULL), fullpath is determined (user could specify conf file from
current dir and that dir must be remembered to write to the same file later)
NB: vty_use_backup_config() tries to recover .conf from .conf.sav
step B. otherwise config_current_dir is tried the same way. If there is neither .conf
nor .conf.sav or config_current_dir == NULL, pass through.
step C. config_default_dir is processed the same way

One important thing is that all zebra daemons pick up their configuration from either .conf
or .conf.sav, if any exists in the current dir. This may lead to unexpected results.
At least for me it costed 2 hours of router downtime, because I restarted zebra and had
an old zebra.conf in current dir (not /etc/zebra).

My first attempt to resolve this was to redefine config_current:
char config_current[] = SYSCONFDIR OSPF_DEFAULT_CONFIG; // ospf_main.c
But that's the wrong way:

lib/vty.c:
2189: if (config_current_dir)
2190: {
2191:   confp = fopen (config_current_dir, "r");
...
2201: if (confp == NULL)
2202: {
... // not reached when we have ./protod.conf
2247: }
2248: else
2249: {
2250:   /* Rleative path configuration file. */
2251:   cwd = getcwd (NULL, MAXPATHLEN);
2252:   fullpath = XMALLOC (MTYPE_TMP, 
2253:       strlen (cwd) + strlen (config_current_dir) + 2);
2254:   sprintf (fullpath, "%s/%s", cwd, config_current_dir);
2255: }  
2256: }  
2257: vty_read_file (confp);
2258:
2259: fclose (confp);
2260:
2261: host_config_set (fullpath);

Have you got it ? look at lines 2191, 2254, 2257 and 2261 !
vty_read_config() reads .conf from config_current_dir, but remembers it for future use
as CWD + config_current dir. Here is the trap: you will not be able to "write file" from vty.
I did not see the relevance between config_current in ospf_main.c and the code above.

In this case the minimal patch to avoid unexpected troubles is to assign NULL to
config_current_dir. vty_read_config will never try to pick up configs from current dir:

diff -urN zebra-0.93b.orig/lib/vty.c zebra-0.93b/lib/vty.c
--- zebra-0.93b.orig/lib/vty.c	2002-08-18 17:49:50 +0300
+++ zebra-0.93b/lib/vty.c	2003-03-11 23:46:26 +0200
@@ -2154,6 +2154,7 @@
   FILE *confp = NULL;
   char *fullpath;
 
+  config_current_dir = NULL;
   /* If -f flag specified. */
   if (config_file != NULL)
     {

*/

After daemonizing, creating PID file and so on protod creates socket for its vty and launches
available threads from struct master in infinite loop.


*** Zebra threads ***
This is the main engine of protod. There is no main infinite loop with lots of checks or
subsequent function calls. Here is the main loop:
while (thread_fetch (master, &thread))
  thread_call (&thread);
			

Files: lib/thread.[h|c]
thread.h:
struct thread is not really a thread, but io request or an atomic operation or
any asyncronous block of code, which can be define das function call.
Threads are executed in FIFO order.
struct thread_list --- container for thread structs, linked list
struct thread_master --- complex structure. Each protod keeps own instance of variable "master"
of this type. This structure is intended to 

*Thread types (thread.h):
#define THREAD_READ           0
#define THREAD_WRITE          1
#define THREAD_TIMER          2
#define THREAD_EVENT          3
#define THREAD_READY          4
#define THREAD_UNUSED         5

*utilities
thread_fetch() is designed to fetch ready threads from "master" and to return them
{
	If there is any EVENT thread, it is deleted from the queue and returned.
	If nothing was found, continue:
	If event queue is empty, TIMERs are checked. Timers are also removed from timers queue before
	returning them. (expired timers only)
	If nothing was found, continue:
	READY threads are checked.
	If nothing was found, continue:
	READ/WRITE/EXPECT requests are checked: fdsets are copied and thread_timer_wait() called.
	thread_timer_wait() calculates timeout for select() performed at the fdsets so, that pending
	timers will not get timed out during select().
	If no IO event occurs after select(), nothing happens. Otherwise READ request are run, then
	WRITE requests.
	Finally there is one extra check for READY threads. So in fully loaded master READY request
	queue reduces twice faster.
	If nothing of above was found, thread_fetch() loops until something happens.
}
thread_trim_head(): picks up the head of the thread list, deletes it from the list and returns it.
thread_run() marks thread as unused.

TODO:
redef THREAD_YIELD_TIME_SLOT to (100 * 1000L)
rename thread_get() to thread_alloc_and_init()

************* ripd **************************

RIP is the most simple, stupid and implementable IGP one can invent.
Its main purpose (to my opinion) is to be coded into xDSL modem
firmware or to be run at old 386 router booting from a floppy.
If you plan to build a /20 AS with full mesh and a hord of a routers,
it is not the best choice. Any way, due to its simplicity ripd happens
to appear much simpler than e.g. bgpd and I will start from it.

Resources: RFC 2453 or any other good explanation of RIP.

RO == read-only
RW == read-write
R == read, possible write
WO == write-only

Look into ripd/rip_main.c. Skip common init. I see keychain_init()
(lib/keychain.c).
It installs new CLI node and fills it with a list of commands relevant
to RIP timed keys authentication.

Next call: rip_init() (ripd/rip.c)
[ few calls ]
install_element (CONFIG_NODE, &router_rip_cmd);
{
router_rip_cmd is DEFUNed. It calls rip_create() (ripd/rip.c),
 {
which allocates and initializes structure representing RIP router in the
whole. One line:
rip->obuf = stream_new (1500);
makes me to think about possible MTUs in the Net. It will work Ok for
ATM, most PPP and Ethernet interfaces. But what if MTU is 8k ? I consider
extra 16kbytes of the ripd process will not slow the whole router down.
Then 2 threads get scheduled as events: RIP_READ from zebrad
(this schedules rip_read(), which reschedules the thread each time)
and RIP_UPDATE_EVENT to fd 1 (stdout) (re-registers itself too)
 }
Further rip_version/metric/default do not schedule new threads behind
the scene. But rip_timers_cmd schedules RIP_UPDATE_EVENT to force new
timer values into work.
rip_route_cmd and rip_distance* do not schedule new threads but only change
global RIP router parameters.
The rest seems to be standard (debug, access/prefix/distribute_list hook init).
}

rip_if_init()
{
// init_structures
// add hooks to interface state changes
// create INTERFACE_NODE and fill it with commands
}

rip_peer_init() does struct init

Now we are done with the init. ripd begins to exec ripd.conf and meets
"router rip". Above RIP_READ and RIP_UPDATE_EVENT get scheduled.

rip_read() (ripd/ripd.c) receives UDP datagrams among other things.
{
BTW this function is commented very well. A lot of nested "if"s check
recvd packet's fields. I consider it to be a different function and/or
to optimize the conditionals. Later.
After all the tests we've got to the actual packet processing:
rip_response_process (packet, len, &from, ifp); (ripd/ripd.c)
 {
Even more checks. Thoughts for a separate packet validator get stronger.
if_valid_neighbor() is defined in rip_interface.c, but declared in ripd.h (WTF?)
It performs something like Linux's /proc/sys/net/ipv4/conf/XXX/rp_filter
function. If the packet does not belong to the subnet from which it was
recvd (PTP case is handled too), it will not be possible to return response
to that address and anyway it is fake/error. Discard it.
Finally after discarding and complaining about invalid packets/field we
get some amount of RRs, which can be learned:
rip_rte_process (rte, from, ifp);
 }
rip_request_process (packet, len, &from, ifp);
 {
Here we process our own RIP RT and decide what to answer. Then
rip_send_packet ((caddr_t) packet, size, from, ifp);
 }
}
there is new version: rip_read_new(), but it's orphane.


Finally here is the source map for ripd:

rip_debug.[c|h]
debug handling (CLI and logging)

rip_interface.c
address classification
interface flag/address management
zebrad messages handling
DEFUNs for CLI commands
neighbour list management
different utilities
hooks

rip_main.c
int main() etc. etc. etc.

rip_offset.c
offset lists mangement (used from rip_rte_process() to mangle with RIP hops count)
DEFUNs for relevant CLI commands

rip_peer.c
peer management functions. used from rip_[response|request]_process to identify
neighbours.

rip_routemap.c
route maps support

rip_snmp.c
SNMP support

rip_zebra.c
zebrad interface

ripd.[c|h]
high-level functions performing RIP-term operations


I look into route processing code deeper.

rip_rte_process() (ripd/ripd.c)
let's strip code and leave comments only:
/* Make prefix structure. */
/* Make sure mask is applied. */
/* Apply input filters. */
/* Once the entry has been validated, update the metric by
adding the cost of the network on wich the message
arrived. If the result is greater than infinity, use infinity
(RFC2453 Sec. 3.9.2) */
/* Zebra ripd can handle offset-list in. */
/* If offset-list does not modify the metric use interface's
metric. */
/* Set nexthop pointer. */
/* Check nexthop address. */
/* Get index for the prefix. */
/* Check to see whether there is already RIP route on the table. */
if (rinfo)
{
/* Redistributed route check. */
/* Local static route. */
}
if (!rinfo)
{
/* Now, check to see whether there is already an explicit route
   for the destination prefix.  If there is no such route, add
   this route to the routing table, unless the metric is
   infinity (there is no point in adding a route which
   unusable). */
// and a lot of...
																					
}
else
{
// lots too
}

Too much code for one function. Let's try to look at the data flow:
The central structure is extern struct rip *rip; (ripd/ripd.h).
Native comments are /* */, my comments are //
/* RIP structure. */
struct rip
{
  /* RIP socket. */
  int sock;
	// RO by rip_send_packet()
	// RO by rip_if_down()
	// RO by rip_interface_wakeup
  // RW by rip_create()
	// RW by rip_clean()

  /* Default version of rip instance. */
  u_char version;
	// RO by rip_read()
	// RO by rip_update_process()
	// RO by show_ip_protocols_rip_cmd()
	// RO by config_write_rip()
	// RO by rip_request_interface()
	// RO by rip_request_neighbor()
	// RO by rip2IfConfSend()
	// RW by rip_create()
	// RW by rip_version_cmd()
	// RW by no_rip_version_cmd()

  /* Output buffer of RIP. */
  struct stream *obuf;
	// R  by rip_output_process()
	// RW by rip_create()

  /* RIP routing information base. */
  struct route_table *table;
	// RO by show_ip_rip_cmd()
	// R  by rip_rte_process()
	// R  by rip_response_process()
	// R  by rip_redistribute_add()
	// R  by rip_redistribute_delete()
	// R  by rip_request_process()
	// R  by rip_output_process()
	// RW by rip_clear_changed_flag()
	// RW by rip_redistribute_withdraw
	// RW by rip_create()
	// RW by rip_update_default_metric()
	// RW by rip_clean()
	// RW by rip_if_down()

  /* RIP only static routing information. */
  struct route_table *route;
  
  /* RIP neighbor. */
  struct route_table *neighbor;
	// RO by rip_update_process()
	// RO by rip_request_neighbor_all()
	// RO by rip_neighbor_lookup()
	// RO by config_write_rip_network()
	// RW by rip_neighbor_lookup()
	// RW by rip_neighbor_delete()
	// RW by rip_create()
	// RW by rip_clean()
  
  /* RIP threads. */
  struct thread *t_read;
	// RW by rip_read()
	// RW by rip_event()
	// RW by rip_clean()

  /* Update and garbage timer. */
  struct thread *t_update;
	// RO by show_ip_protocols_rip_cmd()
	// RW by rip_update()
	// RW by rip_event()
	// RW by rip_clean()

  /* Triggered update hack. */
  int trigger;
  struct thread *t_triggered_update;
  struct thread *t_triggered_interval;

  /* RIP timer values. */
  unsigned long update_time;
	// WO by rip_create()
	// WO by rip_timers_cmd()
	// WO by no_rip_timers_cmd()
	// RO by rip_event
	// RO by show_ip_protocols_rip_cmd()
	// RO by config_write_rip()

  unsigned long timeout_time;
	// WO by rip_create()
	// WO by rip_timers_cmd()
	// WO by no_rip_timers_cmd()
	// RO by show_ip_protocols_rip_cmd()
	// RO by config_write_rip()
	// RO by rip_timeout_update()

  unsigned long garbage_time;
	// WO by rip_create()
	// WO by rip_timers_cmd()
	// WO by no_rip_timers_cmd()
	// RO by show_ip_protocols_rip_cmd()
	// RO by config_write_rip()
	// RO by rip_timeout()
	// RO by rip_rte_process()
	// RO by rip_redistribute_delete()
	// RO by rip_redistribute_withdraw()

  /* RIP default metric. */
  int default_metric;
	// RO by rip_output_process()
	// RW by rip_create()
	// RO by rip_update_default_metric()
	// RW by rip_default_metric_cmd()
	// RW by no_rip_default_metric_cmd()
	// RO by show_ip_protocols_rip_cmd()
	// RO by config_write_rip()

  /* RIP default-information originate. */
  u_char default_information;
  char *default_information_route_map;

  /* RIP default distance. */
  u_char distance;
  struct route_table *distance_table;

  /* For redistribute route map. */
  struct
  {
    char *name;
    struct route_map *map;
    int metric_config;
    u_int32_t metric;
  } route_map[ZEBRA_ROUTE_MAX];
};

Here I see, that structure rip is accessed almost ripd-wide. I've got not much sense.
Another attempt to make things more clear:

Data flow:
1. init.
// common part
extern struct thread_master *master;
extern struct host host;
vector cmdvec;
struct zclient *zclient;
// ripd custom
rip_offset_list_master
extern struct rip *rip;
peer_list

2. startup

After vty_read_config() many structures get changed and new threads scheduled.

3. main loop

Call request queue (master) entries are fetched and executed. How often does the loop
repeats ? Inside thread_fetch() there is a call to process READ threads via calculating
amount of time before next timer. Then select() is called with that timeout. Any protod
always has READ thread to zebrad, so we don't get mad rolling loop. (at least i hope so)
Initially ripd has 2 requests: 1 to read from zebrad and 1 to send updates.
Sending updates is done via scheduling rip_update() with current rip->update_time

After vty_read_config() request to accept UDPs get added. If an UDP gets recvd (rip_read()),
the whole bunch of checks get performed at the packet itself and at the contents of RTEs
inside it. If any action appears to be performed, rip_request_process() or rip_response_process()
is called. Peer stats id updated. If it happens to make decision at incoming RTEs,
rip_rte_process() gets called. Much more checks. Authentication. Access/prefix/distribution lists.
It may even lead to routing table modification. All repeating threads reschedule themselves.

A rough model is complete.


******************** ospfd ******************

Resources: RFC2328 (OSPF Version 2)

ospfd's main file (ospfmain.c) looks like any other protod:

// std CLI commands
cmd_init (1);
debug_init ();
vty_init ();
memory_init ();
access_list_init ();
prefix_list_init ();

// ospfd specific CLI commands
{
  /* OSPFd inits. */
  ospf_init ();
  ospf_if_init ();
  ospf_zebra_init ();

  /* OSPF vty inits. */
  ospf_vty_init ();
  ospf_vty_show_init ();

  ospf_route_map_init ();
#ifdef HAVE_SNMP
  ospf_snmp_init ();
#endif /* HAVE_SNMP */
#ifdef HAVE_OPAQUE_LSA
  ospf_opaque_init ();
#endif /* HAVE_OPAQUE_LSA */
}
// standard calls follow


Let alone standard thread_master and friends.
ospfd important original structures:

ospfd/ospf_nsm.[c|h]
struct {
  int (*func) ();
  int next_state;
} NSM
This structure reflects neighbour state mashine behaiour defined in RFC.
Each neighbour's state can be one of a fixed set. There is also a fixed
set of possible events which can change current state. For each current state
and any possible event corresponding reaction is defined via next neighbour
state and new scheduled function. Many such combination are senseless and
are ignored or cause neighbour state reset.

ospfd/ospfd.h
{
struct ospf --- OSPF router instance representation
struct ospf_area --- OSPF area data
}
ospfd/ospf_route.h
{
struct route_external
struct ospf_route
}
ospfd/ospf_neighbor.h
{
struct ospf_neighbor
}
ospfd/ospf_lsdb.h
{
struct ospf_lsdb --- main OSPF LSDB table (LSA record set ?)
}

Data get into ospfd through ospf_recv_packet() (ospf_packet.c), this function
only reads packet from wire.
I've forgot to look up what struct stream is, but it's simply a data chunk
representation. (lib/stream.h). ospf_recv_packet() is called only from ospf_read()
(ospf_packet.c). ospf_read() appears to be a self-respawning "thread" and is
initially scheduled from ospf_new() (ospfd.c).
If the packet is self-originated, it is destroyed. More sanity checks follow.
Then if the packet is lucky enough to meet all criteria, line 2271 gets exec'ed:
  switch (ospfh->type)
    {
    case OSPF_MSG_HELLO:
      ospf_hello (iph, ospfh, ibuf, oi, length);
      break;
    case OSPF_MSG_DB_DESC:
      ospf_db_desc (iph, ospfh, ibuf, oi, length);
      break;
    case OSPF_MSG_LS_REQ:
      ospf_ls_req (iph, ospfh, ibuf, oi, length);
      break;
    case OSPF_MSG_LS_UPD:
      ospf_ls_upd (iph, ospfh, ibuf, oi, length);
      break;
    case OSPF_MSG_LS_ACK:
      ospf_ls_ack (iph, ospfh, ibuf, oi, length);
      break;
    default:
      zlog (NULL, LOG_WARNING,
	    "interface %s: OSPF packet header type %d is illegal",
	    IF_NAME (oi), ospfh->type);
      break;
    }
Each function performs further analysis of the packet.

Let's trace the case when HELLO packet arrives into freshly configured router
assuming that all packet data is correct, interface is active and
network is configured etc.:
[packet] -> ospf_recv_packet() -> ospf_hello()
Surprise ! ospfd rejects non-zero ToS bit.
708: rn = route_node_get (oi->nbrs, &key);
lib/table.c:route_node_get() looks up existence of current neighbor in current
interface neighbor list. The list is RT with /32 RRs for each known neighbor
(oi->nbrs). In case of a fresh-born router ospf_nbr_new() is called.
Adding neighbor is done via macros OSPF_NSM_EVENT_EXECUTE which is actually
an immediate call to ospf_nsm_event with indirect arguments (nbr,
NSM_HelloReceived).
ospfd/ospf_nsm.c:ospf_nsm_event() is the place where contents of struct NSM
gets its move.
The function determines next nbr state first via calling event handler
(nsm_hello_received() this case) and if the call suggests no next state,
default next state from NSM is assumed.
code of nsm_hello_received() is self-expained well. return value is 0,
so ospf_nsm_event() assigns NSM_Init to nbr->state and calls nsm_change_state()
and nsm_timer_set().
nsm_change_stae() fixes misc stats and schedules events.
{
At the end of function code (line 779) there is a special handling of NSM_Init:
HELLO timer gets reset at the interface:
782: OSPF_ISM_TIMER_ON (oi->t_hello, ospf_hello_timer, 1);
After macro expanding this means:
if (!(oi->t_hello)) (oi->t_hello) = thread_add_timer (master, (ospf_hello_timer), oi, (1))
So we have a timer scheduled.
Let's exit the function and continue.
}
nsm_timer_set() adjusts set of currently assigned timers for current nbr state.

Then different event may be executed depending on network type and interface
priority to perform DR election. I did not look into the rest well.

Now we have a scheduled timer. Wait. Wait. Wait. Oops ! fetch_thread() decides
that it's time to execute ospf_hello_timer().
ospfd/ospf_ism.c:ospf_hello_timer() (line 273)
The timer gets cancelled and then assigned again but with v_hello delay.
ospf_hello_send() is called.
ospfd/ospf_packet.c:2830:ospf_hello_send() allocates, constructs and puts onto
interface outgoing queue an OSPF packet. Then OSPF_ISM_WRITE_ON() is called,
which is a macros (of course), defined in ospf_ism.h
Some time later the packet will be emitted from the interface by scheduled WRITE
thread.

***************************** zebra sockets
This time I decide to break functionality of zebra sockets. ;-)
First thing is that sockets live in /tmp and this is hard-coded.
I guess that it should break LFS. To fix it in a fast and ugly way
I could change the #define's, but I currently have no LFS to
reference. The fix should be trivial.

The second is that they have -rwx------ root:root permissions and
ownership. This causes users, which wish to run vtysh, to be root.
I consider it unreasonable, because need to administer routing
must not imply root privileges.
So we have a call to vty_serv_socket in all protod main()s.

lib/vty.c:1961
vty_serv_sock (const char *hostname, unsigned short port, char *path)
hostname is NULL by default (if you don't specify --vty_addr option),
port is default port of protod (260x),
path is #define'd macros ("/tmp/.xxx").

If to take into accounf the case with Linux, IPv6 & vtysh,
after C preprocessing C compiler will have:
{
	if (port)
	{
		vty_serv_sock_addrinfo (hostname, port);
	}
	vty_serv_un (path);
}
First call starts listening for incoming vty connections, 2nd
is what I need now.
lib/vty.c:1816
vty_serv_un (char *path)
The function is trivial too: a classical example on UNIX domain
sockets. Three things to look at:
1. char *path could be const char * path :)
2. More code is needed to handle group name, chown(), chmod(),
GID resolving, command-line option handling etc. See diff below.
3. vty_event (VTYSH_SERV, sock, NULL);

Note: --retain was ignored in ripngd.
Note: zebra/zebra binary lacks dependency on lib/libzebra.a
Note: vtysh/vtysh.c:vtysh_connect() tries to guess permission
to open socket comparing socket owner and EUID. Groups are not
taken into account. It seems to me that the simplest method is
just to call connect() and look if errno == EACCESS or EPERM.
The same behaviour for checking if the file is socket or not.
Why not to use ENOTSOCK ? The same for ECONNREFUSED.
Note: nice name: vtysh_completion_entry_fucntion

Here is the diff:

diff -urN zebra-0.93b.orig/bgpd/bgp_main.c zebra-0.93b/bgpd/bgp_main.c
--- zebra-0.93b.orig/bgpd/bgp_main.c	2002-06-22 00:14:39 +0300
+++ zebra-0.93b/bgpd/bgp_main.c	2003-04-03 00:19:19 +0300
@@ -45,6 +45,7 @@
   { "vty_port",    required_argument, NULL, 'P'},
   { "retain",      no_argument,       NULL, 'r'},
   { "no_kernel",   no_argument,       NULL, 'n'},
+  { "group",       required_argument, NULL, 'g'},
   { "version",     no_argument,       NULL, 'v'},
   { "help",        no_argument,       NULL, 'h'},
   { 0 }
@@ -57,6 +58,9 @@
 /* Route retain mode flag. */
 int retain_mode = 0;
 
+/* UNIX socket group owner */
+char *socket_group = NULL;
+
 /* Master of threads. */
 struct thread_master *master;
 
@@ -89,6 +93,7 @@
 -P, --vty_port     Set vty's port number\n\
 -r, --retain       When program terminates, retain added route by bgpd.\n\
 -n, --no_kernel    Do not install route to kernel.\n\
+-g, --group=XXX    Change UNIX socket group to XXX and permit rw for group\n\
 -v, --version      Print program version\n\
 -h, --help         Display this help and exit\n\
 \n\
@@ -113,7 +118,7 @@
   vty_read_config (config_file, config_current, config_default);
 
   /* Create VTY's socket */
-  vty_serv_sock (vty_addr, vty_port ? vty_port : BGP_VTY_PORT, BGP_VTYSH_PATH);
+  vty_serv_sock (vty_addr, vty_port ? vty_port : BGP_VTY_PORT, BGP_VTYSH_PATH, socket_group);
 
   /* Try to return to normal operation. */
 }
@@ -197,7 +202,7 @@
   /* Command line argument treatment. */
   while (1) 
     {
-      opt = getopt_long (argc, argv, "df:hp:A:P:rnv", longopts, 0);
+      opt = getopt_long (argc, argv, "df:hp:A:P:rng:v", longopts, 0);
     
       if (opt == EOF)
 	break;
@@ -227,6 +232,9 @@
 	case 'r':
 	  retain_mode = 1;
 	  break;
+	case 'g':
+	  socket_group = optarg;
+	  break;
 	case 'n':
 	  bgp_option_set (BGP_OPT_NO_FIB);
 	  break;
@@ -270,7 +278,7 @@
   pid_output (pid_file);
 
   /* Make bgp vty socket. */
-  vty_serv_sock (vty_addr, vty_port, BGP_VTYSH_PATH);
+  vty_serv_sock (vty_addr, vty_port, BGP_VTYSH_PATH, socket_group);
 
   /* Print banner. */
   zlog_info ("BGPd %s starting: vty@%d, bgp@%d", ZEBRA_VERSION,
diff -urN zebra-0.93b.orig/lib/vty.c zebra-0.93b/lib/vty.c
--- zebra-0.93b.orig/lib/vty.c	2002-08-18 17:49:50 +0300
+++ zebra-0.93b/lib/vty.c	2003-04-03 22:54:08 +0300
@@ -21,6 +21,7 @@
  */
 
 #include <zebra.h>
+#include <grp.h>
 
 #include "linklist.h"
 #include "buffer.h"
@@ -1813,7 +1814,7 @@
 
 /* VTY shell UNIX domain socket. */
 void
-vty_serv_un (char *path)
+vty_serv_un (const char *path, const char *group_name)
 {
   int ret;
   int sock, len;
@@ -1851,7 +1852,35 @@
       close (sock);	/* Avoid sd leak. */
       return;
     }
-
+	/* Adjust ownership/permissions to allow non-root vty access.
+	 * If group_name == NULL, original behaviour is kept.
+	 */
+	if (group_name)
+	{
+		struct group * group_info = getgrnam (group_name);
+		if (!group_info)
+			perror ("getgrnam() failed");
+		else
+		{
+			gid_t group_id = group_info->gr_gid;
+			/* make it -rw-rw---- */
+			ret = chmod (path, S_IRUSR | S_IWUSR | S_IRGRP | S_IWGRP);
+			if (ret)
+			{
+				perror ("fchmod");
+				close (sock);
+				return;
+			}
+			/* make it <what it was before>:group_name */
+			ret = chown (path, -1, group_id);
+			if (ret)
+			{
+				perror ("fchown");
+				close (sock);
+				return;
+			}
+		}
+	} /* if (group_name) */
   ret = listen (sock, 5);
   if (ret < 0)
     {
@@ -1958,7 +1987,7 @@
 
 /* Determine address family to bind. */
 void
-vty_serv_sock (const char *hostname, unsigned short port, char *path)
+vty_serv_sock (const char *hostname, unsigned short port, const char *path, const char *group)
 {
   /* If port is set to 0, do not listen on TCP/IP at all! */
   if (port)
@@ -1977,7 +2006,7 @@
     }
 
 #ifdef VTYSH
-  vty_serv_un (path);
+  vty_serv_un (path, group);
 #endif /* VTYSH */
 }
 
diff -urN zebra-0.93b.orig/lib/vty.h zebra-0.93b/lib/vty.h
--- zebra-0.93b.orig/lib/vty.h	2002-08-18 17:34:00 +0300
+++ zebra-0.93b/lib/vty.h	2003-04-02 23:03:06 +0300
@@ -191,7 +191,7 @@
 int vty_out (struct vty *, const char *, ...) PRINTF_ATTRIBUTE(2, 3);
 void vty_read_config (char *, char *, char *);
 void vty_time_print (struct vty *, int);
-void vty_serv_sock (const char *, unsigned short, char *);
+void vty_serv_sock (const char *, unsigned short, const char *, const char *);
 void vty_close (struct vty *);
 char *vty_get_cwd (void);
 void vty_log (const char *, const char *, va_list);
diff -urN zebra-0.93b.orig/ospf6d/ospf6_main.c zebra-0.93b/ospf6d/ospf6_main.c
--- zebra-0.93b.orig/ospf6d/ospf6_main.c	2002-06-22 01:34:21 +0300
+++ zebra-0.93b/ospf6d/ospf6_main.c	2003-04-03 00:19:20 +0300
@@ -43,6 +43,9 @@
 /* Default port values. */
 #define OSPF6_VTY_PORT             2606
 
+/* UNIX socket group owner */
+char *socket_group = NULL;
+
 /* ospf6d options, we use GNU getopt library. */
 struct option longopts[] = 
 {
@@ -51,6 +54,7 @@
   { "pid_file",    required_argument, NULL, 'i'},
   { "vty_addr",    required_argument, NULL, 'A'},
   { "vty_port",    required_argument, NULL, 'P'},
+  { "group",       required_argument, NULL, 'g'},
   { "version",     no_argument,       NULL, 'v'},
   { "help",        no_argument,       NULL, 'h'},
   { 0 }
@@ -93,6 +97,7 @@
 -i, --pid_file     Set process identifier file name\n\
 -A, --vty_addr     Set vty's bind address\n\
 -P, --vty_port     Set vty's port number\n\
+-g, --group=XXX    Change UNIX socket group to XXX and permit rw for group\n\
 -v, --version      Print program version\n\
 -h, --help         Display this help and exit\n\
 \n\
@@ -231,7 +236,7 @@
   /* Command line argument treatment. */
   while (1) 
     {
-      opt = getopt_long (argc, argv, "df:hp:A:P:v", longopts, 0);
+      opt = getopt_long (argc, argv, "df:hp:A:P:g:v", longopts, 0);
     
       if (opt == EOF)
         break;
@@ -255,6 +260,9 @@
         case 'P':
           vty_port = atoi (optarg);
           break;
+		case 'g':
+		  socket_group = optarg;
+		  break;
         case 'v':
           print_version (progname);
           exit (0);
@@ -306,7 +314,7 @@
 
   /* Make ospf vty socket. */
   vty_serv_sock (vty_addr,
-		 vty_port ? vty_port : OSPF6_VTY_PORT, OSPF6_VTYSH_PATH);
+		 vty_port ? vty_port : OSPF6_VTY_PORT, OSPF6_VTYSH_PATH, socket_group);
 
   /* Print start message */
   zlog_notice ("OSPF6d (Zebra-%s ospf6d-%s) starts",
diff -urN zebra-0.93b.orig/ospfd/ospf_main.c zebra-0.93b/ospfd/ospf_main.c
--- zebra-0.93b.orig/ospfd/ospf_main.c	2002-07-04 08:06:41 +0300
+++ zebra-0.93b/ospfd/ospf_main.c	2003-04-03 00:19:20 +0300
@@ -61,12 +61,16 @@
   { "help",        no_argument,       NULL, 'h'},
   { "vty_addr",    required_argument, NULL, 'A'},
   { "vty_port",    required_argument, NULL, 'P'},
+  { "group",       required_argument, NULL, 'g'},
   { "version",     no_argument,       NULL, 'v'},
   { 0 }
 };
 
 /* OSPFd program name */
 
+/* UNIX socket group owner */
+char *socket_group = NULL;
+
 /* Master of threads. */
 struct thread_master *master;
 
@@ -88,6 +92,7 @@
 -i, --pid_file     Set process identifier file name\n\
 -A, --vty_addr     Set vty's bind address\n\
 -P, --vty_port     Set vty's port number\n\
+-g, --group=XXX    Change UNIX socket group to XXX and permit rw for group\n\
 -v, --version      Print program version\n\
 -h, --help         Display this help and exit\n\
 \n\
@@ -197,7 +202,7 @@
     {
       int opt;
 
-      opt = getopt_long (argc, argv, "dlf:hA:P:v", longopts, 0);
+      opt = getopt_long (argc, argv, "dlf:hA:P:g:v", longopts, 0);
     
       if (opt == EOF)
 	break;
@@ -221,6 +226,9 @@
 	case 'P':
 	  vty_port = atoi (optarg);
 	  break;
+	case 'g':
+	  socket_group = optarg;
+	  break;
 	case 'v':
 	  print_version (progname);
 	  exit (0);
@@ -278,7 +286,7 @@
 
   /* Create VTY socket */
   vty_serv_sock (vty_addr,
-		 vty_port ? vty_port : OSPF_VTY_PORT, OSPF_VTYSH_PATH);
+		 vty_port ? vty_port : OSPF_VTY_PORT, OSPF_VTYSH_PATH, socket_group);
 
   /* Print banner. */
   zlog (NULL, LOG_INFO, "OSPFd (%s) starts", ZEBRA_VERSION);
diff -urN zebra-0.93b.orig/ripd/rip_main.c zebra-0.93b/ripd/rip_main.c
--- zebra-0.93b.orig/ripd/rip_main.c	2002-06-22 00:14:39 +0300
+++ zebra-0.93b/ripd/rip_main.c	2003-04-03 00:19:20 +0300
@@ -43,6 +43,7 @@
   { "vty_addr",    required_argument, NULL, 'A'},
   { "vty_port",    required_argument, NULL, 'P'},
   { "retain",      no_argument,       NULL, 'r'},
+  { "group",       required_argument, NULL, 'g'},
   { "version",     no_argument,       NULL, 'v'},
   { 0 }
 };
@@ -57,6 +58,9 @@
 /* Route retain mode flag. */
 int retain_mode = 0;
 
+/* UNIX socket group owner */
+char *socket_group = NULL;
+
 /* RIP VTY bind address. */
 char *vty_addr = NULL;
 
@@ -85,6 +89,7 @@
 -A, --vty_addr     Set vty's bind address\n\
 -P, --vty_port     Set vty's port number\n\
 -r, --retain       When program terminates, retain added route by ripd.\n\
+-g, --group=XXX    Change UNIX socket group to XXX and permit rw for group\n\
 -v, --version      Print program version\n\
 -h, --help         Display this help and exit\n\
 \n\
@@ -130,7 +135,7 @@
   vty_read_config (config_file, config_current, config_default);
 
   /* Create VTY's socket */
-  vty_serv_sock (vty_addr, vty_port, RIP_VTYSH_PATH);
+  vty_serv_sock (vty_addr, vty_port, RIP_VTYSH_PATH, socket_group);
 
   /* Try to return to normal operation. */
 }
@@ -189,7 +194,7 @@
     {
       int opt;
 
-      opt = getopt_long (argc, argv, "df:hA:P:rv", longopts, 0);
+      opt = getopt_long (argc, argv, "df:hA:P:rg:v", longopts, 0);
     
       if (opt == EOF)
 	break;
@@ -216,6 +221,9 @@
 	case 'r':
 	  retain_mode = 1;
 	  break;
+	case 'g':
+	  socket_group = optarg;
+	  break;
 	case 'v':
 	  print_version (progname);
 	  exit (0);
@@ -259,7 +267,7 @@
   pid_output (pid_file);
 
   /* Create VTY's socket */
-  vty_serv_sock (vty_addr, vty_port, RIP_VTYSH_PATH);
+  vty_serv_sock (vty_addr, vty_port, RIP_VTYSH_PATH, socket_group);
 
   /* Execute each thread. */
   while (thread_fetch (master, &thread))
diff -urN zebra-0.93b.orig/ripngd/ripng_main.c zebra-0.93b/ripngd/ripng_main.c
--- zebra-0.93b.orig/ripngd/ripng_main.c	2002-06-22 00:14:39 +0300
+++ zebra-0.93b/ripngd/ripng_main.c	2003-04-03 00:19:20 +0300
@@ -49,6 +49,7 @@
   { "vty_addr",    required_argument, NULL, 'A'},
   { "vty_port",    required_argument, NULL, 'P'},
   { "retain",      no_argument,       NULL, 'r'},
+  { "group",       required_argument, NULL, 'g'},
   { "version",     no_argument,       NULL, 'v'},
   { 0 }
 };
@@ -58,6 +59,9 @@
 /* Route retain mode flag. */
 int retain_mode = 0;
 
+/* UNIX socket group owner */
+char *socket_group = NULL;
+
 /* Master of threads. */
 struct thread_master *master;
 
@@ -81,6 +85,7 @@
 -A, --vty_addr     Set vty's bind address\n\
 -P, --vty_port     Set vty's port number\n\
 -r, --retain       When program terminates, retain added route by ripngd.\n\
+-g, --group=XXX    Change UNIX socket group to XXX and permit rw for group\n\
 -v, --version      Print program version\n\
 -h, --help         Display this help and exit\n\
 \n\
@@ -174,7 +179,7 @@
     {
       int opt;
 
-      opt = getopt_long (argc, argv, "dlf:hA:P:v", longopts, 0);
+      opt = getopt_long (argc, argv, "dlf:hA:P:rg:v", longopts, 0);
     
       if (opt == EOF)
 	break;
@@ -204,6 +209,9 @@
 	case 'r':
 	  retain_mode = 1;
 	  break;
+	case 'g':
+	  socket_group = optarg;
+	  break;
 	case 'v':
 	  print_version (progname);
 	  exit (0);
@@ -238,7 +246,7 @@
 
   /* Create VTY socket */
   vty_serv_sock (vty_addr,
-		 vty_port ? vty_port : RIPNG_VTY_PORT, RIPNG_VTYSH_PATH);
+		 vty_port ? vty_port : RIPNG_VTY_PORT, RIPNG_VTYSH_PATH, socket_group);
 
   /* Process id file create. */
   pid_output (pid_file);
diff -urN zebra-0.93b.orig/vtysh/vtysh.c zebra-0.93b/vtysh/vtysh.c
--- zebra-0.93b.orig/vtysh/vtysh.c	2002-05-08 15:47:39 +0300
+++ zebra-0.93b/vtysh/vtysh.c	2003-04-03 01:27:43 +0300
@@ -1515,8 +1515,7 @@
 	  exit (1);
 	}
       
-      if (euid != s_stat.st_uid 
-	  || !(s_stat.st_mode & S_IWUSR)
+      if (!(s_stat.st_mode & S_IWUSR)
 	  || !(s_stat.st_mode & S_IRUSR))
 	{
 	  fprintf (stderr, "vtysh_connect(%s): No permission to access socket\n",
diff -urN zebra-0.93b.orig/zebra/main.c zebra-0.93b/zebra/main.c
--- zebra-0.93b.orig/zebra/main.c	2002-06-22 00:14:39 +0300
+++ zebra-0.93b/zebra/main.c	2003-04-03 00:43:30 +0300
@@ -46,6 +46,9 @@
 /* Route retain mode flag. */
 int retain_mode = 0;
 
+/* UNIX socket group owner */
+char *socket_group = NULL;
+
 /* Don't delete kernel route. */
 int keep_kernel_mode = 0;
 
@@ -62,6 +65,7 @@
   { "vty_addr",    required_argument, NULL, 'A'},
   { "vty_port",    required_argument, NULL, 'P'},
   { "retain",      no_argument,       NULL, 'r'},
+  { "group",       required_argument, NULL, 'g'},
   { "version",     no_argument,       NULL, 'v'},
   { 0 }
 };
@@ -93,6 +97,7 @@
 -A, --vty_addr     Set vty's bind address\n\
 -P, --vty_port     Set vty's port number\n\
 -r, --retain       When program terminates, retain added route by zebra.\n\
+-g, --group=XXX    Change UNIX socket group to XXX and permit rw for group\n\
 -v, --version      Print program version\n\
 -h, --help         Display this help and exit\n\
 \n\
@@ -195,7 +200,7 @@
     {
       int opt;
   
-      opt = getopt_long (argc, argv, "bdklf:hA:P:rv", longopts, 0);
+      opt = getopt_long (argc, argv, "bdklf:hA:P:rg:v", longopts, 0);
 
       if (opt == EOF)
 	break;
@@ -230,6 +235,9 @@
 	case 'r':
 	  retain_mode = 1;
 	  break;
+	case 'g':
+	  socket_group = optarg;
+	  break;
 	case 'v':
 	  print_version (progname);
 	  exit (0);
@@ -304,7 +312,7 @@
 
   /* Make vty server socket. */
   vty_serv_sock (vty_addr,
-		 vty_port ? vty_port : ZEBRA_VTY_PORT, ZEBRA_VTYSH_PATH);
+		 vty_port ? vty_port : ZEBRA_VTY_PORT, ZEBRA_VTYSH_PATH, socket_group);
 
   while (thread_fetch (master, &thread))
     thread_call (&thread);

**************************** vtysh again
One useful thing to do for looking-glasses is to disable subshell invocation:

diff -urN zebra-0.93b.orig/vtysh/vtysh.c zebra-0.93b/vtysh/vtysh.c
--- zebra-0.93b.orig/vtysh/vtysh.c	2002-05-08 15:47:39 +0300
+++ zebra-0.93b/vtysh/vtysh.c	2003-04-03 23:52:57 +0300
@@ -1777,15 +1777,19 @@
 
   install_element (VIEW_NODE, &vtysh_ping_cmd);
   install_element (VIEW_NODE, &vtysh_traceroute_cmd);
+/*
   install_element (VIEW_NODE, &vtysh_telnet_cmd);
   install_element (VIEW_NODE, &vtysh_telnet_port_cmd);
+*/
   install_element (ENABLE_NODE, &vtysh_ping_cmd);
   install_element (ENABLE_NODE, &vtysh_traceroute_cmd);
+/*
   install_element (ENABLE_NODE, &vtysh_telnet_cmd);
   install_element (ENABLE_NODE, &vtysh_telnet_port_cmd);
   install_element (ENABLE_NODE, &vtysh_start_shell_cmd);
   install_element (ENABLE_NODE, &vtysh_start_bash_cmd);
   install_element (ENABLE_NODE, &vtysh_start_zsh_cmd);
+*/
 
   install_element (RMAP_NODE, &set_metric_cmd);
   install_element (RMAP_NODE, &set_ip_nexthop_cmd);

****************************** forwarding
I've met this when was ivestigating why zebra builds for Solaris in
ALTLinux BTE. The thing was in configure script, but between other
things i looked into lib/ipforward_proc.c. I guess that this could
unify the things:


diff -urN zebra-0.93b-orig/zebra/ipforward_proc.c zebra-0.93b/zebra/ipforward_proc.c
--- zebra-0.93b-orig/zebra/ipforward_proc.c	2000-01-11 09:11:49 +0200
+++ zebra-0.93b/zebra/ipforward_proc.c	2003-02-25 00:31:13 +0200
@@ -22,8 +22,6 @@
 
 #include <zebra.h>
 
-char proc_net_snmp[] = "/proc/net/snmp";
-
 static void
 dropline (FILE *fp)
 {
@@ -40,22 +38,20 @@
   int ipforwarding = 0;
   char *pnt;
   char buf[10];
+  int sscanfresult = 0;
 
-  fp = fopen (proc_net_snmp, "r");
+  fp = fopen (proc_ipv4_forwarding, "r");
 
   if (fp == NULL)
     return -1;
 
-  /* We don't care about the first line. */
-  dropline (fp);
-  
-  /* Get ip_statistics.IpForwarding : 
+  /* Get kernel flag
      1 => ip forwarding enabled 
-     2 => ip forwarding off. */
-  pnt = fgets (buf, 6, fp);
-  sscanf (buf, "Ip: %d", &ipforwarding);
+     0 => ip forwarding off. */
+  pnt = fgets (buf, 2, fp);
+  sscanfresult = sscanf (buf, "%d", &ipforwarding);
 
-  if (ipforwarding == 1)
+  if (ipforwarding == 1 && sscanfresult == 1)
     return 1;
 
   return 0;
