Article 32014 of comp.os.vms: Path: utkcs2!emory!sol.ctr.columbia.edu!ucselx!bionet!raven.alaska.edu!acad3.alaska.edu!sxrmh1 From: sxrmh1@acad3.alaska.edu Newsgroups: comp.os.vms Subject: RE: RWASTed processes Message-ID: <1991Jun26.131514.1@acad3.alaska.edu> Date: 26 Jun 91 17:15:14 GMT Sender: news@raven.alaska.edu (USENET News System) Organization: University of Alaska Fairbanks Lines: 737 Nntp-Posting-Host: acad3.alaska.edu Note this post is about 700 lines long... I do think you need to be more specific. There are many many reasons why a process could go into the RWASTed state (Resource Wait - Asynchronous System Trap). If it is one of the very many quotas (PQLs) that's been exhausted, a little kernel mode code to "temporarily" up a quota would work (and has for me). If it is caused by a device like a tape drive, I haven't figured that one out yet, so I schedule a reboot for later. If it is caused by a munged mailbox and the message router, again, I just schedule a reboot for later. For the others, that I can't think of right now, I can't recall what I've done, or what code I've set up for them. .. then I get to work with the System Dump Analyzer and try to find out why, before the reboot occurs. I don't advocate the kernel mode code method - that doesn't fix the real problem, just allows a bypass. I guarantee DEC will say "reboot" for any and all RWASTed processes to get the process slots back. Read the articles from the DSN system attached below... ------------------------------------------------------------------------------ .. RWAST - Is used when a process is waiting for something to happen, when that event has occurred, an AST will be delivered to the process. The most common occurrences happen waiting for a quota to be returned which has been used up by I/O requests, or for a channel to become no longer busy. Please see separate article for more details on this resource wait state. .. ------------------------------------------------------------------------------ Reasons Why A Process Might Go Into RWAST State Last Technical Review: 6-SEP-1988 Size: 159 lines COPYRIGHT (c) 1988, 1989, 1990 by Digital Equipment Corporation. ALL RIGHTS RESERVED. No distribution except as provided under contract. COMPONENT: SCHEDULER OP/SYS: VMS, Version 5.n LAST TECHNICAL REVIEW: 24-JUN-1988 SOURCE: Customer Support Center/Colorado Springs USA SUBJECT: The following is a list of reasons why a process might go into the state RWAST. It lists the PC, when known, and the facility and module that contains that PC. DISCUSSION: To find the necessary information for a process in RWAST, invoke the SDA utility with the command ANALYZE/SYSTEM. Most of the following information assumes that the first two commands issued in SDA are READ SYS$SYSTEM:SYSDEF and READ/EXECUTIVE. At the SDA prompt, do a SHOW SUMMARY and find the process in question. Then use the process index number from the second column of the SHOW SUMMARY display in a SET PROCESS/INDEX=nnn command. Now use the SDA command EXAMINE @PC. If this command returns an error message about being unable to access location C0000068, this means that for some reason an AST could not be delivered to that process to get the necessary information. The SDA command EXAMINE R4 (or any other register) will return the contents of that register. The notes that follow will provide what type of data structure is in each register. If the notes mention an offset in a data structure, there are two ways to examine that offset within SDA. EXAMINE @R6+CCB$W_IOC will examine the field CCB$W_IOC when R6 points to the CCB data structure. Since this is a word field, use only the rightmost word returned by the EXAMINE command. The other method is to FORMAT @R6 and look for the CCB$W_IOC field in the output. PC Facility Module Affected by SET PROCESS /NORESOURCE EXE$DASSGN+0084 SYS SYSDASSGN.LIS N R5 - Channel number R6 - CCB address Reason: CCB$W_IOC is non-zero. Channel is busy. The process will wait until that channel becomes no longer busy. EXE$SNGLEQUOTA+005E SYS EXSUBROUT.LIS Y R1 - amount of required quota R2 - address of quota, in PCB, PHD, or JIB for process R3 - size of field to check in bits Reason: Waiting for a quota at address R2 which is R3 bits in size. anywhere SYS PAGEFAULT.LIS N Reason: DIOCNT is less than or equal to 0, a direct I/O is charged to the process for a pagefault from disk, but is unavailable. This puts a limit of 32767 on DIOLM in AUTHORIZE for any process. EXE$CANCEL+0189 SYS SYSCANCEL.LIS Y R5 - UCB address R6 - CCB address R7 - channel number Reason: BIOCNT is zero and a buffered I/O is required for that cancel operation. unknown SYS SYSCREDEL.LIS N R2 - VA being deleted R3 - SVAPTE for virtual address R4 - PCB for process Note: Address will be in MMG$DELPAG Reason: DIOLM > DIOCNT (some DIO outstanding) when deleting a page mapped by pfn. Process will wait until all direct I/O has completed. unknown SYS SYSCRMPSC.LIS N R4 - PCB address R7 - IRP address R10 - SEC address (global section descriptor?) Reason: Waiting for I/O to complete for the IRP as specified in R7. This is done for a process when IPL is above ASTDEL. This is intended for use only during system initialization. unknown SYS SYSDELPRC.LIS N R4 - PCB address Reason: PCB$B_DPC is nonzero when process is being deleted. This means that the XQP is active and the process will wait until all XQP activity has completed. EXE$DELPRC+00FB SYS SYSDELPRC.LIS N R4 - PCB address Reason: PCB$W_PRCCNT is nonzero when process is being deleted Process owns subprocess. The process will wait until all subprocesses have been deleted before completing the deletion of this process. EXE$SUSPND+0144 SYS SYSPCNTRL.LIS N R4 - PCB address Reason: PCB$B_DPC is nonzero when attempting to force a hard suspend on a process. A process can not have a hard (KERNEL) suspend until all XQP activity has completed. unknown SYS SYSQIOREQ.LIS N R4 - PCB address R5 - UCB address R7 - CCB address Note: The PC should be in either EXE$QIO or SYS$QIO Reason: QIO system service call with a channel which has access or deaccess pending, must wait for access or deaccess to complete before retrying I/O. CLUSTRLOA+7C88 SYSLOA CSPCALL.LIS N R4 - PCB address R2 - CSD address Reason: Waiting for block transfer to CSP (Cluster Server Process) on another node of the cluster to complete. CSD is appended to end of ACB examine @r2-ACB$K_CSPLNG+ACB$B_STS has ACB$V_STS_WAIT set. Verify that the CLUSTER_SERVER process on the other node exists and is running properly. LTDRIVER+16EC LAT LTDRIVER.LIS N R4 - PCB address Reason: Waiting for TQE to execute, 1/2 second wait Used during LAT shutdown when waiting for all CSB's for this link to run down. All UCBs relating to each CSB are sent a data set hangup which should cause all process related to the UCB to be deleted. This enables the reference count for that UCB to go to 0. This allows the UCB to be deleted. Once all UCBs relating to a CSB are deleted, then the CSB is deleted. When the CSBs are deleted, then LTDRIVER will allow the process requesting the LAT shutdown to continue executing. unknown DRIVER DSDRIVER.LIS Y R2 - UCB address R3 - IRP address R4 - PCB address R6 - CCB address Note: PC is within DSDRIVER Reason: A IO$_CRESHAD fucntion was called on a unit which has UCB$V_MSCP_INITING bit set. The process is put in RWAST until the associated UCB has finished initialization. ------------------------------------------------------------------------------ Procedure To Determine Why A Process Is In An RWAST State Last Technical Review: 23-OCT-1987 Size: 533 lines COPYRIGHT (c) 1988, 1989, 1990 by Digital Equipment Corporation. ALL RIGHTS RESERVED. No distribution except as provided under contract. COMPONENT: Scheduler OP/SYS: VMS LAST TECHNICAL REVIEW: 23-OCT-1987 SOURCE: Customer Support Center/Colorado Springs USA VERSION INFORMATION: Symptom Identified On: VMS, Version 4.n SYMPTOM: The $ SHOW SYSTEM command display indicates that a process is in RWAST state and the process seems locked. How can you determine why the process is in this state? ANALYSIS: The RWAST is a general purpose 'Resource Wait' state. It indicates the wait is expected to be satisfied by the delivery and/or enqueueing of an AST to the process. Reasons why a process goes into RWAST include: waiting for an I/O to complete on a channel exhausted an AUTHORIZE or SYSGEN quota waiting for a file system or lock request to complete waiting for a subprocess to terminate Other database articles discuss print symbionts in RWAST and RWAST states on systems running VMS Workstation Software (VWS). Processes in the RWAST state can NOT be deleted until the condition they are waiting for is met. If you can not identify what the process is waiting for, you will have to reboot the system to eliminate the process. If the process in RWAST state is a user-written program, it is possible to receive an error status rather than VMS putting the process into resource wait for some system service calls. Usually, the status indicates either a quota problem or insufficient pooled memory. This is accomplished by using the SYS$SETRWM system service call as described in the "VAX/VMS Version 4.4 System Service Reference Manual", (April 1986), p. SYS-376. PROCEDURE: To find out why processes are in an RWAST state, use the System Dump Analyzer (SDA): 1. Invoke SDA: $ ANALYZE/SYSTEM VAX/VMS System analyzer SDA> SHOW SUMMARY ! find the RWAST process and its INDEX Current process summary ----------------------- Extended Indx Process name Username State Pri PCB PHD Wkset -- PID -- ---- --------------- ----------- ----- --- -------- -------- ---- 20200080 0000 NULL COM 0 800024A8 80002328 0 20200081 0001 SWAPPER HIB 16 80002748 800025C8 0 20201005 0005 KILEY KILEY LEF 4 80363C50 82CEEE00 211 20200086 0006 ERRFMT SYSTEM HIB 7 8030CA80 80A2FA00 88 20200087 0007 CACHE_SERVER SYSTEM HIB 16 80317F70 80C3AE00 62 2020104F 004F SMITH SMITH RWAST 6 8036CE90 82DF4800 200 ---- ----- 2. Set your default to the RWAST process using its INDEX value: SDA> SET PROCESS/INDEX=4F !Selects the process in RWAST state, !in this case, the SMITH process NOTE: If you have tried to delete the process, SDA may not permit you to set your process to the RWAST process. In this case, you would receive the following error: %SDA-E-NOTINPHYS, xxxxxx: not in physical memory If you receive this error, you may have to format the PCB and/or JIB to figure out the problem. The address for the PHD and PCB can be found from the SHOW SUMMARY display. The address for the JIB will be at offset PCB$L_JIB in the formatted PCB. Keep this in mind if SDA will not allow you 'normal' access to the data structures that follow. If you can get no access to the process data structures, for example the process header is outswapped, you may have to reboot the system and wait for the problem to occur again. If it happens again, you may be able to catch the data structures in memory and analyze the Resource Wait state more thoroughly. 3. Find the process Program Counter (PC): SDA> EXAMINE @PC and see if it evaluates to one of the following symbols: A. EXE$DASSGN+6D, in which case you go to step 4 B. EXE$MULTIQUOTA+032, in which case you go to step 5 C. EXE$DCLEXH+0A5, in which case you go to step 6 D. EXE$DCLEXH+141, in which case you go to step 7 Other RWAST states are possible but very rare. If the PC does not evaluate to one of the above symbols, you will have to reboot the system to eliminate the hung process. Take a crash dump of the system to determine why the process was in RWAST. Occasionally, the RWAST process will clear itself if the process waits long enough and the AST somehow gets satisfied. 4. If the PC is at EXE$DASSGN+6D, the process is waiting for an I/O request to complete. Many times, the device it is waiting for I/O on will be shown as 'Busy' in the SHOW PROCESS/CHANNEL display: SDA> SHOW PROCESS/CHANNEL !Look for a status of "Busy" !and this will determine the !device that is waiting for the I/O Process index: 004F Name: SMITH Extended PID: 22E0124D ------------------------------------------------------------------ Process active channels ----------------------- Channel Window Status Device/file accessed ------- ------ ------ -------------------- 0010 00000000 LFILNG$DUA14: 00C0 00000000 Busy LPA0: <-- This device is 00D0 00000000 MBA1: blocking process In the above example, you have only 1 BUSY channel, so this must be the channel causing the process to hang in RWAST. If you have multiple BUSY channels, you can identify which one is causing the RWAST state with the following commands: SDA> READ SYS$SYSTEM:SYSDEF.STB !read in system symbols SDA> EXAMINE @R6+CCB$W_IOC !number of outstanding requests 8041BBBA: 00180001 "...." !in lower word (1) ---- SDA> DEFINE UCB=@(@R6+CCB$L_UCB) !define UCB address SDA> EXAMINE UCB+UCB$W_UNIT !low order word is unit number UCB+054: 00000000 "...." !here it is unit # 0 ---- SDA> EXAMINE @(UCB+UCB$L_DDB)+DDB$T_NAME;8 ! device name 8041BBBA: 2041504C "LPA." ! device is LPA0: --- NOTE: If the device is a printer connected to a terminal port and the symbiont is waiting for an X/ON to be delivered, occasionally turning the printer OFF and back ON again will cause an X/ON to be sent back to the VAX. This allows the I/O to complete, permitting the print symbiont to continue. If the device is a printer connected to a printer port (LPA0, LCA0...), the VAX thinks the printer is offline. This may indicate a hardware problem with the printer or controller, if it is really online. Again, turning the printer OFF and back ON again may help. 5. If the value returned is EXE$MULTIQUOTA+032, the RWAST state indicates the process has run out of a quota. A SHOW SYSTEM display will often show the process continuing to accumulate CPU time. A quick check to help determine which quota the process has exhausted is the following: SDA> SHOW PROCESS !for the SMITH process Process index: 004F Name: SMITH Extended PID: 2020104F ------------------------------------------------------------ Process status: 02040001 RES,PHDRES PCB address 8036CE90 JIB address 8064F3C0 PHD address 82DF4800 Swapfile disk address 01002821 Master internal PID 0020004F Subprocess count 0 Internal PID 0020004F Creator internal PID 00000000 Extended PID 2020104F Creator extended PID 00000000 State RWAST Termination mailbox 0000 Base priority 4 AST's active NONE UIC [00022,000016] AST's remaining 16 Mutex count 0 Buffered I/O count/limit 0/18* Waiting EF cluster 0 Direct I/O count/limit 18/18* Starting wait time 1B001B1B BUFIO byte count/limit 30478/31936* Event flag wait mask 00000001 # open files allowed left 75 Local EF cluster 0 E4000000 Timer entries allowed left 10 Local EF cluster 1 00000000 Active page table count 0 Global cluster 2 pointer 00000000 Process WS page count 140 Global cluster 3 pointer 00000000 Global WS page count 60 Look in the lower right hand portion of the display, denoted by asterisks (*), to see if any quotas are down to zero. In the example above, you can see that Buffered I/O count/limit is zero. The number before the slash (/) is the amount of this quota left. The number after the slash (/) is the total amount allowed. These fields relate to the following Authorization (UAF) records and SYSGEN PQL parameters if the process is a detached process: Authorization SYSGEN PQLs ------------- ----------- AST's remaining - ASTLM PQL_DASTLM Buffered I/O count/limit - BIOLM PQL_DBIOLM Direct I/O count/limit - DIOLM PQL_DDIOLM BUFIO byte count/limit - BYTLM PQL_DBYTLM # open files allowed left - FILLM PQL_DFILLM Timer Entries allowed left - TQELM PQL_DTQELM Once it has been determined which quota needs to be increased for processes or subprocesses, increase that value within the User Authorization File, the SYSGEN PQL parameter (if it is a detached process), or on the SYS$CREPRC system service (if a process is creating the detached process). The new value is then used when a new process is created or when you log out and log back in again. At this point, reboot the system to eliminate the RWAST process waiting for quota. Hopefully, you have increased the parameter to a value high enough that you will not see the new process go into the RWAST state again. If the RWAST process is waiting for a quota and the quota does not appear to be any of these, you can format and display the Job Information Block (JIB), Process Control Block (PCB), and the Process Header (PHD) to locate the quota problem. R2 contains the address of the insufficient quota. To determine the insufficient quota, do the following: SDA> READ SYS$SYSTEM:SYSDEF ! read system definitions SDA> EXAMINE R2 R2: 8036CECA "JNG." ! obtain value contained in R2 Next, locate the addresses of the PCB (Process Control Block) and the JIB (Job Information Block) from the top of the SHOW PROCESS display. The value found in R2 will be pointing somewhere in one of these two data structures. Identify which data structure would contain the value in R2 and format that data structure. In this case, R2 would be in the PCB so the PCB needs to be formatted: SDA> FORMAT 8036CE90 ! Formatting PCB of process 8036CE90 PCB$L_SQFL 80002180 8036CE94 PCB$L_SQBL 80002180 . . . . . . . . . . . . . . . . 8036CEC6 PCB$W_PPGCNT 008C 8036CEC8 PCB$W_ASTCNT 0010 -----> "8036CECA" PCB$W_BIOCNT 0000 <-- This address matches R2. 8036CECC PCB$W_BIOLM 0012 The value for PCB$W_BIOCNT is zero, indicating that the quota is depleted. BIOLM needs to increased beyond 18 or the 8036CECE PCB$W_DIOCNT 0012 application program . . . . . . . . . . . . . . . . modified to not have 8036CF0C 00000000 so many outstanding +------<8036CF10 PCB$L_JIB 8064F3C0 buffered I/O requests | . . . . . . . . . . . . . . . . outstanding at once. | | If the address is not found in the PCB, format the JIB. | The JIB address can be found from either the SHOW PROCESS | display or the PCB$L_JIB value above: | ------> SDA> FORMAT 8064F3C0 ! Formatting JIB of process 6. If the value comes back with EXE$DCLEXH+0A5, the process may be waiting for the file system to complete a request or for a lock request. EXE$DCLEXH+0A5 is returned if you have tried to delete the process, whose former state was probably LEF. A quick check to see if the RWAST process is waiting for an XQP file request to complete is to format the PCB and look for a non-zero value in PCB$B_DPC. If the process is being forced to wait under these circumstances, the SDA 'SHOW PROCESS' command displays the PROCESS status as "DELPEN" or "SUSPEN". In the following example, the process is in the delete pending state with a status of "DELPEN". v | | +-----+ SDA> SHOW PROCESS | | Process index: 00DF Name: Mike Mc. | Extended PID: 000007DF -------------------------------------V----------------------- Process status: 02040023 RES,DELPEN,RESPEN,PHDRES PCB address 80339230 JIB address 804FE7E0 PHD address 8092D000 Swapfile disk address 010065A1 Master internal PID 000700DF Subprocess count 0 Internal PID 000700DF Creator internal PID 00000000 Extended PID 000007DF Creator extended PID 00000000 State RWAST Termination mailbox 0000 Current priority 6 AST's enabled KESU Base priority 4 AST's active NONE UIC [00310,000014] AST's remaining 78 Mutex count 0 Buffered I/O count/limit 49/50 Waiting EF cluster 0 Direct I/O count/limit 17/18 Starting wait time 1B011B1A BUFIO byte count/limit 35606/36000 Event flag wait mask 00000001 # open files allowed left 60 Local EF cluster 0 A300002E Timer entries allowed left 50 Local EF cluster 1 FBC00000 Active page table count 0 Global cluster 2 pointer 00000000 Process WS page count 213 Global cluster 3 pointer 00000000 Global WS page count 102 SDA> FORMAT 80339230 ! format the PCB 80339230 PCB$L_SQFL 80002180 80339234 PCB$L_SQBL 8032D5C0 . . . . . . . . . . . . . 8033925A PCB$B_DPC 01 <--NON-ZERO, waiting for XQP (file . . . . . . . . . . . system) activity to complete o If the value is zero (00), the RWAST is not waiting for the XQP and you can check for outstanding lock requests using the command SDA> SHOW PROCESS/LOCK. You will often find a lock in either "Waiting for" or "Converting to" state. (Two other database articles describe how to trace lock requests on both clustered and nonclustered systems.) The process holding the lock this process is waiting for is often in a RWxxx state itself, and solving that other process's problem may clear up this process's RWAST state. If you are at this address, EXE$DCLEXH+0A5, and the process is not getting CPU time or waiting for XQP or lock operations to complete, reboot the system to eliminate the process. Take a crash dump for later examination if the problem occurs often. o If the value is non-zero, the process is waiting for an XQP file system request. Use the following information to analyze further: WARNING: Do not use the SDA command "SHOW PROCESS/CHANNELS" on a V5 system to troubleshoot an RWAST process; the process issuing the command may also hang. SDA> SHOW PROCESS/CHANNELS Process index: 00DF Name: Mike Mc. Extended PID: 000007DF ------------------------------------------------------------- Process active channels ----------------------- Channel Window Status Device/file accessed ------- ------ ------ -------------------- 0010 00000000 DUA2: 0040 00000000 VTA52: 0050 00000000 VTA52: 0070 00000000 Busy DUA2: <--- Device waiting for XQP The most common occurrence of a RWAST process waiting for XQP is that SYSGEN parameter PAGEDYN has been used up. If this is the case, increase the value of PAGEDYN; you should do this with AUTOGEN. PAGEDYN should normally be at least 1/3 (33%) free, and up to 40% free on a busy system. To determine how much PAGEDYN has been used, issue the following command: SDA> SHOW POOL/SUMMARY/PAGE Paged dynamic storage pool -------------------------- Summary of paged pool contents 108 UNKNOWN = 357984 (20%) 2 LOG = 83728 (4%) . . . . . . . . . . 1 CI = 96 (0%) 1 CLU = 2384 (0%) Total space used = 1719808 out of 1988608 total bytes, 268800 bytes left Total space utilization = 86% <--- indicates that only 14% is left To determine if the RWAST process is waiting for PAGEDYN, you can do the following: Create the following macro to set up a symbol definition to format data structures to get necessary symbols defined for the next step: $ MACRO/OBJ=F11BDEF.STB SYS$INPUT+SYS$LIBRARY:LIB/LIB $F11BCDEF GLOBAL $F11BDEF GLOBAL .END SDA> READ F11BDEF !read in data structures created by macro SDA> SHOW DEVICE DUA2 !DUA2 is the device shown busy from the !SDA command SHOW PROCESS/CHANNELS Look for the AQB address in the following information: I/O data structures ------------------- DUA2 RA80 UCB address: 80484B90 Device status: 00021810 online,valid,unload,lcl_valid Characteristics: 1C4D4108 dir,rct,fod,shr,avl,mnt,elg,idv,odv,rnd 00000221 clu,mscp,nnm Owner UIC [000001,000001] Operation count 274887 ORB address 80484C96 PID 00000000 Error count 0 DDB address 80BA3D20 Alloc. lock ID 000400AF Reference count 1 DDT address 80550C78 Alloc. class 2 Online count 1 VCB address 8048AD70 . . . . . . . . . . . . . . . . . . . . . . . Press RETURN for more. I/O data structures ------------------- --- Volume Control Block (VCB) 8048AD70 --- Volume: TUBORGPAGE Lock name: TUBORGPAGE Status: A0 extfid,system Status2: 05 writethru,mountver ------------------------------------------------------------+ *** Here it is | v Mount count 1 Rel. volume 0 AQB address 80BA4AA0 Transactions 2 Max. files 29651 RVT address 80484B90 Free blocks 34020 Rsvd. files 9 FCB queue 808A9D10 Window size 7 Cluster size 3 Cache blk. 80768460 Vol. lock ID 000200B1 Def. extend sz. 5 Block. lock ID 001A00A7 Record size 0 SDA> FORMAT 80BA4AA0 80BA4AA0 AQB$L_ACPQFL 80BA4AA0 . . . . . . . . . . . . . . 80BA4AB6 AQB$B_CLASS 00 80BA4AB7 00 80BA4AB8 AQB$L_BUFCACHE 80274380 <----- ! Look for this value . . . . . . . . . . . . . . SDA> FORMAT 80274380 ! formatting the AQB$L_BUFCACHE 80274380 F11BC$L_BUFBASE 80297400 . . . . . . . . . . . . . . 802743C8 F11BC$Q_POOL_WAITQ 802743C8 <----- !Look for this value. 802743CC 802743C8 ! The forward link and the ! backward link point to the 802743D0 802743D0 ! forward link address which 802743D4 802743D0 ! indicates the queue is empty. 802743D8 802743D8 ! In this example, it is not 802743DC 802743D8 ! waiting for PAGEDYN...If it ! was waiting for PAGEDYN, 802743E0 802743E0 ! different address values 802743E4 802743E0 ! for the forward and backward ! link would be displayed and 802743E8 F11BC$L_POOLAVAIL 0000002F ! they would not have the same 802743EC 0000044A ! value, indicating that the . ! queue is not empty, waiting . ! for PAGEDYN. SDA> EXIT 7. If the value comes back with EXE$DCLEXH+141, the process is waiting for a subprocess to terminate before it can terminate. Many times the subprocess it is waiting for is also in a RWxxx state. To check for this occurrence, format the PCB as described above and check the PCB$W_PRCCNT field. This field contains the number of subprocesses this process is waiting for. The following DCL command procedure can be used to check all subprocesses on the system to find which process has the RWAST parent process. The subprocess Username and Process Name are displayed followed by its parent process PID. $ a = """ $ pid = "" $ context = "" $LOOP: $ pid = F$PID(context) $ IF pid .EQS. "" THEN GOTO DONE $ pid = a+pid+a $ owner = F$GETJPI('pid,"OWNER") $ if owner .eqs. "" then goto loop $ username = F$GETJPI('pid,"USERNAME") $ prcnam = F$GETJPI('pid,"PRCNAM") $ imagename = F$GETJPI('pid,"IMAGNAME") $ imagename = F$PARSE(imagename,,,"NAME") $ text = F$FAO("!8AS !8AS !8AS ",username,prcnam,owner) $ write sys$output text $ GOTO LOOP $DONE: If no subprocess is found for the parent process, the parent process will wait forever and the system will have to be rebooted to eliminate this process. It could be that privileged code is altering the subprocess PCB$L_OWNER field so process termination does not know about the parent process. To find the subprocess in a crash dump, you need to locate the OWNER field whose PID matches that of the parent process in RWAST. This can be done by displaying every owner field in every PCB available on the system. 1) SDA> READ SYS$SYSTEM:SYSDEF 2) SDA> SHOW SUM !to get all the PCB addresses 3) then for each PCB address you can enter: SDA> EXAM +PCB$L_OWNER When you find the process whose owner/parent is the PID of the process in RWAST, you can start analyzing why the subprocess is not terminating. Hope this helps... Randy sxrmh1@acad3.alaska.edu Article 32146 of comp.os.vms: Path: utkcs2!memstvx1!ukma!usenet.ins.cwru.edu!magnus.acs.ohio-state.edu!zaphod.mps.ohio-state.edu!swrinde!cs.utexas.edu!uunet!mcsun!hp4nl!nkivxh!cal!gege Newsgroups: comp.os.vms Subject: Re: Deletion of RWAST processes... Message-ID: <36@cal.enst.fr> From: gege@cal.enst.fr (Why not ?) Date: 28 Jun 91 10:24:22 GMT Sender: daemon@ucbvax.BERKELEY.EDU References: <2961434@MVB.SAIC.COM> <1991Jun27.082051.403@beckman.com> Organization: Centre de calcul, E.N.S.T. Paris Lines: 113 In article <1991Jun27.082051.403@beckman.com>, dsroberts@beckman.com writes: > > In message, <0094aaeb.64d08240.19859>, > asks: > >>I would be grateful for advice on how to delete processes >>that are hung in a RWAST state. >> Thanks in advance, >> A. Schach (SCHACH@LBL.BITNET) >> > > We had an RWAST process once that had a busy mailbox open. Using DCL, we > opened the mailbox and wrote to it then closed it and this freed up the RWAST. > I have not seen it happen again, but it was a fascinating exercise. > -- here is a hack to clear the resource wait mode bit in the PCB. it allows process waiting for resources to disappear. it won't help against *every* RWAST state, but common problems due to decwindows are solved by this program. ;++ ; This little hack forces the target process to set itself in NO RESOURCE WAIT MODE. ; ; Author : Alain PANNETIER TIBET - PARIS - (1) 40 41 70 34. ;-- .TITLE FORCE_NO_RESOURCE_WAIT_MODE .LIBRARY 'SYS$SHARE:LIB.MLB' .LINK 'SYS$SYSTEM:SYS.STB' /SELECTIVE_SEARCH .DISABLE GLOBAL ; take care of misspelling errors at assembly time $DSCDEF ; Descriptors structures and constants $IPLDEF ; Interrupt priority levels $PCBDEF ; Process Control Block $SSDEF ; System services condition values ;+ ; LOCAL STORAGE ;- .PSECT DATA, WRT, NOEXE, LONG ; Pid_str : .WORD 0 ; Empty Dynamic string descriptor .BYTE DSC$K_DTYPE_T ; .BYTE DSC$K_CLASS_D ; .LONG 0 ; Pid: .BLKL 1 ; External pid Prompt: .ASCID 'Target Process PID : ' ; Prompt .EXTERNAL EXE$epid_to_pcb ; Gets the PCB address from an extended pid .EXTERNAL LIB$get_foreign ; Get/prompt for argument .EXTERNAL OTS$cvt_tz_l ; Converts hexadecimal text to binary ;+ ; MAIN PROGRAM ;- .PSECT CODE,WRT,EXE,LONG .ENTRY FORCE_NO_RESOURCE_WAIT_MODE ,^m<> PUSHAQ Prompt ; PUSHAQ Pid_str ; CALLS # 2, G^lIB$get_foreign ; Get the pid if not specified on command line BLBC R0,1$ PUSHAL Pid ; Convert to binary ( assumes hexa input ) PUSHAQ Pid_str ; CALLS #2,G^OTS$cvt_tz_l ; BLBC R0,1$ $CMKRNL_S routin = Set_norswm ; Kick into kernel mode 1$: $EXIT_S R0 ; Let DCL signal returned status ;+ ; KERNEL MODE MAIN TREATMENT ;- .ENTRY SET_NORSWM,^M ; Save all GPRs ;+ ; This routine is executed in Kernel mode ; It addresses the status field of the target processe's PCB ;- LOCK lockname = SCHED, - ; This macro from LIB.MLB lockipl = #IPL$_SYNCH, - ; Lock sheduler database savipl = R2 ; Remember current IPL MOVL Pid, R0 ; input of EXE$epid_to_pcb in R0, output in R0 as well JSB G^EXE$epid_to_pcb ; Get corresponding PCB address TSTL R0 ; If the PCB address returned is null, assume process disapeared BNEQ 1$ ; Else continue. UNLOCK lockname = SCHED, - ; Unlock and newipl = R2 ; restore original IPL before returning MOVL #SS$_NONEXPR, R0 ; with appropriate status. RET 1$: BBSSI #PCB$V_SSRWAIT, PCB$L_STS(R0), 2$ ; Set the no resource wait mode bit in target PCB's status 2$: UNLOCK lockname = SCHED, - ; Unlock and newipl = R2 ; restore original IPL before returning MOVL #SS$_NORMAL, R0 ; with success status. RET ; .END FORCE_NO_RESOURCE_WAIT_MODE ================================================================================ ! Guillaume Gerard ! Bitnet GEGE@FRINT51 ! ! Systems responsible ! Email gerard@enst.fr ! ! French Telecom University ! X400 C=FR AD=ATLAS PD=TELECPARIS ! ! FAX: Ha! I won !! ! PSI 2080750412855::gerard ! ================================================================================ %SYSTEM-W-TMNYFNGRS, too many fingers on keyboard Article 32379 of comp.os.vms: Path: utkcs2!emory!swrinde!zaphod.mps.ohio-state.edu!usc!rpi!crdgw1!uunet!sunquest!alpha.sunquest.com!gavron From: gavron@alpha.sunquest.com (Ehud Gavron 602-885-7700x.2546) Newsgroups: comp.os.vms Subject: Re: RWAST processes Message-ID: <5JUL91105638@alpha.sunquest.com> Date: 5 Jul 91 17:56:38 GMT References: <0094aaeb.64d08240.19859@SSRL01.SLAC.STANFORD.EDU> <1991Jul1.104948.1@vmsa.technion.ac.il> <362@motto.UUCP> Sender: news@sunquest.UUCP Reply-To: gavron@alpha.sunquest.com Organization: Sunquest VMS Internals, Tucson AZ Lines: 93 News-Software: VAX/VMS VNEWS 1.4-a3 In article <362@motto.UUCP>, murray@motto.UUCP (Murray S. Kucherawy) writes... #shirley@SSRL01.SLAC.STANFORD.EDU writes: #>I would be grateful for advice on how to delete processes #>that are hung in a RWAST state. # #ben@VMSA.TECHNION.AC.IL.UUCP writes: #>The best answer is REBOOT. There are alternatives, but only the lucky #>survive... # #Not in all cases. Occasionally I see an RWAST-state process on one of our #VMS machines, and if I can track it back to a hung session on another machine #(a login over DECnet) and stop that other process, both die gracefully. Ok, I wasn't going to post, because this is a nontrivial issue, and the chances are that if you do The Wrong Thing, your system will crash. However, I have used these ideas successfully many a time... 1. If there are more than one process stuck in RWAST, make sure you're not going to get stuck there too. SHO DEV D and check for 0 blocks, SHOW MEM and check for litte free pages, little IRPs, SRPs, or LRPs, or no page file space (middle column under V5.x). 2. At that stage, or if there is only one such beastie, go into SDA. Do a show proc/chan on the process. Note which channel is busy. Disk File: Usually a result of full disk. You should have caught this with the SHOW DEV D. Delete unnecessary files. Mailbox: If you're really clever, you can follow from the channel to the IRP to the function and figure out if it's a read or write the mailbox is failing on. In either case, it's usually a synchronization problem - somebody wrote after someone had already tried to read. At least one of those will hang, unless the code was designed with that in mind. I usually do this: $ spawn/nowait/notify/inpu=nl: - copy sys$Login:login.com MBAnnn: $ spawn/nowait/notify/inpu=nl: - type MBAnnn: This has the effect of reading to and writing from the mailbox. If the second prints the results of the first, you'll find that you probably cleared your RWASTed process in the process. (Nifty, eh?) If it didn't, you just added two more RWASTed jobs, but the problem is internal to the mailbox system, and you really ought to reboot. Tape: Put a formatted tape online, push the online button and bring it offline and online a few times. This is the one that often fails, necessitating a reboot. The problem(s) are in the magtape driver and acp... 3. If there was no busy channel (highly unlikely) then check the process quotas (listed when you do SHOW/PROCESS in SDA) and see the BUFLIM/CNT, DIOLM/CNT, etc, all are nonzero. If they are, then you know what resource it ran out of; either run some kernel code to add to the LM part of the JIB/PCB or release the resource in another way. 4. If you're even more clever than that, and want to take the scientfici approach (ie the non AL/EG/JMS method) then what you want to do is catch the RWASTed process, examine its stack, and step back through the call frames, figuring out which resource it ran out of and give it to it. Note that this is also the only way to catch and release MUTEX waits... #It's those RWASTs that come from tape drive errors that are real buggers... Agreed! If you ever have an RWAST situation, and you can't afford to reboot, and you must solve the problem, there are other workaround... catch me offline and we can di$cu$$ $olution$. # #=============================== Murray S. Kucherawy ========================== #Motorola Canada, Ltd. Shared Systems Division, Toronto [on work term] #University of Waterloo, Ontario, Canada 2B Math/Computer Science #Internet: murray@motto.UUCP (work) mskucherawy@watmath.UWaterloo.ca (UW) #UUCP: uunet!utai!lsuc!motto!murray uunet!watmath!mskucherawy Ehud -- Ehud Gavron (EG76) gavron@vesta.sunquest.com Article 276 of vmsnet.internals: Path: utkcs2!memstvx1!ukma!hri.com!spool.mu.edu!mips!swrinde!sdd.hp.com!hamblin.math.byu.edu!arizona.edu!mvb.saic.com!macro32 Newsgroups: vmsnet.internals Subject: RE: Killing off an RWAST process Message-ID: <009519CC.105C6080.1343@WKUVX1.BITNET> From: goathunter@WKUVX1.BITNET (Hunter Goatley, WKU) Date: 14 Nov 91 11:32:56 GMT Organization: Macro32<==>Vmsnet.Internals Gateway X-Gateway-Source-Info: Mailing List Lines: 151 "George D. Greenwade" writes: > >The Subject: just about says it all. Is there a program somewhere which >can kill off a process hung in RWAST state? We have a few problems with >UCX and processes occasionally get hung up. With privileges, you can do a >STOP/ID= and the process is essentially gone (i.e., you can no longer do >privileged functions on the process id), but a SHOW USER shows the process >still there (as does a SHOW SYSTEM). > >The only way we know of to get rid of the process is to do a system >shutdown -- that can't be the elegant way you >MACRO/BLISS/vmsnet.internals-types would do it! Your insight on how to >more properly handle this would be truly appreciated. That's the way you get rid of them: rebooting. Theoretically, there are a few things you might be able to do to provide some of the resources they are waiting on, but most of the time you have to reboot. Ideally, you'd find out what the process is waiting on and maybe adjust some SYSGEN parameters. I've included below a message from Gerard K. Newman that was posted to comp.os.vms a few years ago. The specifics may be a little out of date, but I believe most of the text is accurate (I just skimmed it before including it here, so I haven't verified any of the info). Hunter ------ Hunter Goatley, VAX Systems Programmer, Western Kentucky University goathunter@wkuvx1.bitnet, 502-745-5251 ------------------------------------------------------------------------------ From: Richard Steinberger Subject: Processes in a RWAST state Date: Sat 14 Mar 87 10:33:01-PST ... Is rebooting the only option to getting rid of a process in the RWAST state? If so, why does DEC allow VMS to create processes that it can't delete? Sigh. This is probably one of the things that causes the most frustration to people who do not have extensive backgrounds into VMS internals. To answer your question, yes, rebooting is sometimes the only way to get rid of a process in RWAST state. [allow me to apologize in advance for the length of this one...] Now that that's over with... a little more information on RWAST, and resource waits (what the RW stands for) in general. Resource wait is an involuntary wait state that VMS will place a process in while waiting for a resource to become available. This resource can usually be identified by the rest of the name (RWxxx) displayed by SHOW SYSTEM. Resource waits are a subset of the scheduler state SCH$C_MWAIT, which is shorthand for miscellaneous resource/MUTEX wait. A MUTEX is a mutual-exclusion semaphore, which is something VMS uses internally to protect various data structures, such as the I/O database, logical name tables, and a few other odds and ends which really aren't important in this discussion. You can tell the difference between a MUTEX wait and a resource wait by examining the event flag wait mask (PCB$L_EFWM in the PCB, JPI$_EFWM) if the process' scheduler state is SCH$C_MWAIT. If PCB$L_EFWM is negative, then it is the system-space address of the MUTEX which the process is attempting to gain access to. If it isn't, it's a small positive integer which is an indication of resource wait state the process is in. These are the known resource waits (as of VMS V4.x) and a little bit about what they mean: 1 RWAST Waiting for an AST (see below). 2 RWMBX Mailbox full. A process attempted to write more data into a mailbox than the buffer quota for that mailbox allows. 3 RWNPG Waiting for non-paged pool. 4 RWPFF Page file full. The page file to which this process is assigned is full. 5 RWPAG Waiting for paged pool. 6 RWBRK $BRKTHRU wait. 7 RWIMG Image activator interlock. 8 RWQUO A pooled quota has been exceeded. Use SDA to figure out which one. 9 RWLCK Lock ID database is full? (can anybody fill me in?). 10 RWSWP Swap file full. 11 RWMPE Waiting for the modified page write to empty the modified page list. 12 RWMPB Waiting for the modified page writer (to do something, but I'm not sure what. Can anybody fill me in?). 13 RWSCS Waiting for a systems communications services (cluster) event. 14 RWCLU Waiting for a cluster transition. RWAST is sort of the catch-all resource wait state. Routines inside VMS will place a process into RWAST hoping that the next kernel mode AST queued to the process will have called SCH$RAVAIL to report resource avaialability for the resource in question. By far the most popular reason to get placed in RWAST is a lack of a non-pooled quota, typically BYTLM, BIOLM, DIOLM or ASTLM (this is not an exhaustive list, but you get the idea). The BYTLM quota as it comes "out of the box" from DEC (4096 bytes) in the default account is so pitifully small that you can't even use DECnet effectively. For normal users I grant a BYTLM quota of 24000 bytes. You can tell if this is happening by using SDA on the running system to examine the PCB (the SDA SHOW PROCESS/INDEX=nn/PCB command) for the process that's stuck in RWAST. If some of the non-pooled quotas have zeroes for the "count" portion (really, the amount remaining) then there's your problem. Another popular reason for processes getting stuck in RWAST is a hangup in last-channel deassign. You can tell if this is happening if you look at the process in question with SDA and see EXE$DASSGN+6D or so floating near the top of that process' stack (use the SDA SHOW STACK/INDEX=nnn command). In this case R6 points to a data structure called a CCB (channel control block) which will generally have outstanding I/O that VMS is waiting to have complete before it completely deassigns the I/O channel. If a process is stuck for this reason you can sometimes unstick it by jostling whatever I/O device is involved (you can figure out which by deciphering the CCB, or using the SDA SHOW PROCESS/CHANNELS/INDEX=nnn command). This is not an exhaustive list of why processes get placed in RWAST, it's just two cases I see a fair amount. There are probably dozens of other scenarios. Why can't you kill a process in RWAST, you ask? Well, VMS put the process in that state to wait for a given resource. To delete that process before the resource comes available could leave the system in an inconsistent state, or cause system data structures to be corrupted. The actual mechanics behind it are such that the special kernel mode AST to delete the process remains queued until some process in VMS calls SCH$RAVAIL with the appropriate arguments to knock the process out of resource wait, and which time the process will be deleted. It is possible to break a process out of a quota related resource wait by writing a little program to go patch the PCB and quota in question and call SCH$RAVAIL, but it's definitely not for the novice (and probably there are real reasons why this shouldn't be done, either, but I've successfully done it). Could someone from DEC (or anyplace else, for that matter) please comment on the interpretations of RWxxx above if I'm completely off base on some of them? This information could really be helpful if more people had it. gkn -------------------------------------- Arpa: GKN@SDSC.ARPA Bitnet: GKN@SDSC Span: SDSC::GKN (5.600) USPS: Gerard K. Newman San Diego Supercomputer Center P.O. Box 85608 San Diego, CA 92138 AT&T: 619.534.5076 ------- Article 62688 of comp.os.vms: Newsgroups: comp.os.vms Path: cs.utk.edu!gatech!howland.reston.ans.net!paladin.american.edu!news.univie.ac.at!scsing.switch.ch!aragorn.unibe.ch!vision.rs.ch!WINTER From: winter@vision.rs.ch (Martin Winter) Subject: Re: how to kill a process in RWAST state? Message-ID: <1993Mar26.141812.17564@aragorn.unibe.ch> Sender: news@aragorn.unibe.ch Reply-To: winter@vision.rs.ch Organization: RadioSuisse Ltd, Berne, Switzerland References: <9303252259.AA00526@mx.nsi.nasa.gov> Date: Fri, 26 Mar 1993 14:18:12 GMT Lines: 61 In article <9303252259.AA00526@mx.nsi.nasa.gov>, YONG YUAN writes: >Hi, > We are running VAX/VMS V5.4-3 with VWS 4.3. What happened is: I opened >a VT200 window on the VR260 monitor, as my password has expired I was prompted >to enter a new password, at this time I try to stop this process by enter >+C, yes, it was STOPPED, but the window carrying this process was DEAD >there and can't be deleted or removed. I can see that process by SHOW USER and >SHOW SYSTEM, SHOW PROCESS/ID= tells me the process is >suspended, but SET PROCESS/RESUME (or STOP) tells me: >NONEXISTENT PROCESS. The state of that process is RWAST (from SHOW SYSTEM), >or Mutex & Misc Resource Wait (from MONITOR STATE). > Is there anyone can tell me what's going on and how to deal with it >(except reboot)? > Thanks in advance! do $ ANALYZE /SYSTEM which brings up the SDA SDA> SHOW SUMMARY now look for the process which is in RWAST SDA> SET PROCESS /INDEX=process_index (process_index, is the index of the process from the SHOW SUMMARY) SDA> SHOW PROCESS /CHANNEL this gives you the allocated channels for this process. You will notify one or more channels with the status "busy" If there is more than 1 busy channel, then just take the first and try it there - if it does not work then use the next one and so on. Now you have to remove the busy of this channels. first of all you have to do a $ STOP /ID=pid if you haven't done it at this time. It will give you an error, but it marks the process for deletion and will delete the process, as soon as the reason goes away. If the busy channel was MBAxxx: Try to copy something into and out of the mailbox. $ COPY SYS$LOGIN:LOGIN.COM MBAxxx: $ COPY MBAxxx: NLA0: NETxxx: RTAxxx: Try to find the correct network channel in NCP $ MCR NCP SHOW KNOWN LINKS and disconnect it (you may have to guess which one it is) with $ MCR NCP DISCONNECT LINK nnnnn There are a few other type of channels... but it's usually one of these. If you hit the right thing, then the process will go away. Good luck! Martin with SHOW PROC ---------------------------------------------------------------------------- Radio Suisse Ltd, Internet: winter@vision.rs.ch Laupenstrasse 18a postmaster@rs.ch 3008 Bern PSI-Mail: +22846431062::WINTER (public X.25) Switzerland X.400: C=CH; A=ARCOM; P=RS; O=DM; S=WINTER, G=MARTIN ---------------------------------------------------------------------------- Article 62750 of comp.os.vms: Path: cs.utk.edu!darwin.sura.net!zaphod.mps.ohio-state.edu!swrinde!elroy.jpl.nasa.gov!nntp-server.caltech.edu!SOL1.GPS.CALTECH.EDU!CARL From: carl@SOL1.GPS.CALTECH.EDU (Carl J Lydick) Newsgroups: comp.os.vms Subject: Re: how to kill a process in RWAST state? Date: 27 Mar 1993 06:25:13 GMT Organization: HST Wide Field/Planetary Camera Lines: 25 Distribution: world Message-ID: <1p0s09INN59k@gap.caltech.edu> References: <9303252259.AA00526@mx.nsi.nasa.gov>,<1993Mar26.141812.17564@aragorn.unibe.ch> Reply-To: carl@SOL1.GPS.CALTECH.EDU NNTP-Posting-Host: sol1.gps.caltech.edu In article <1993Mar26.141812.17564@aragorn.unibe.ch>, winter@vision.rs.ch (Martin Winter) writes: =If the busy channel was = MBAxxx: Try to copy something into and out of the mailbox. = $ COPY SYS$LOGIN:LOGIN.COM MBAxxx: = $ COPY MBAxxx: NLA0: You'd be well-advised to substitute $ SPAWN/NOWAIT COPY SYS$LOGIN:LOGIN>COM MBXxxx: for the first command. Otherwise, if the request on the mailbox was a write request, your process will hang and attempts to stop it will result in its going into the RWAST state as well. = NETxxx: = RTAxxx: Try to find the correct network channel in NCP = $ MCR NCP SHOW KNOWN LINKS = and disconnect it (you may have to guess which one it is) with = $ MCR NCP DISCONNECT LINK nnnnn -------------------------------------------------------------------------------- Carl J Lydick | INTERnet: CARL@SOL1.GPS.CALTECH.EDU | NSI/HEPnet: SOL1::CARL Disclaimer: Hey, I understand VAXen and VMS. That's what I get paid for. My understanding of astronomy is purely at the amateur level (or below). So unless what I'm saying is directly related to VAX/VMS, don't hold me or my organization responsible for it. If it IS related to VAX/VMS, you can try to hold me responsible for it, but my organization had nothing to do with it. Article 81713 of comp.os.vms: Path: cs.utk.edu!ornl!fnnews.fnal.gov!uwm.edu!cs.utexas.edu!convex!seas.smu.edu!mic!montagar!davidc From: davidc@avatar.montagar.com (David L. Cathey) Newsgroups: comp.os.vms Subject: Re: RWAST - Check value of VBSS_ENABLE Message-ID: <1994Jan29.015311.18683@avatar.montagar.com> Date: 29 Jan 94 01:53:11 CDT References: <2iapus$rhd@infosrv.edvz.univie.ac.at> Organization: Montagar Software Concepts, Plano TX Lines: 38 In article <2iapus$rhd@infosrv.edvz.univie.ac.at>, mader@aimep0.imep.univie.ac.at (WOIFAL) writes: > Hello. > > On our VAXStation 3100M76 running VMS 6.0 are some (one batch and a > subprocess of it) suspended processes, which can't be removed ( by me so far). > > The batch sometimes is RWAST then it's even COMputing! > The sub is LEFO. > Trying do stop it by stop/id= aren't answered by anything the try to show it > (show/proc/id=) produces the SYSTEM-F-SUSPENDED message. Your batch process is in RWAST because it is waiting for the sub- process to be deleted. Until the subprocess goes away, you will *never* get the batch job to delete. The "undeletable" process is LEFO is a little odd, but still a possibility since you are running V6.0. V6.0 provides "virtual balance set slots", but unless you set your parameters correctly, you *cannot* outswap processes that are waiting for direct I/O. If you have SYSGENned the system with reduced balance set slots, you can lock up the system (I've done this on mine as well). The sub-process probably can't get a balance set slot so the $DELPRC can take effect. Try killing off a process that is in LEF that you don't need. Also, verify that VBSS_ENABLE is set to 2 or 3. If it is set to 1, then this is almost certainly your problem. You may also want to increase BALSETCNT. For more information about RWASTed processes, you can get a copy of my session notes of the presentation I did for the U.S. DECUS Symposium. Send a message to FILESERV@MONTAGAR.COM with the following in the body of the message: SEND RWAST - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - David L. Cathey |INET: davidc@montagar.com Montagar Software Concepts |UUCP: ...!montagar!davidc P. O. Box 260772, Plano TX 75026-0772 |Fone: (214)-618-2117