Feasibility Study for Potential fP Software

[Keith' Weatherhead]

Fairlight wrote:

> Hi Everyone,
> 
> I'm trying to do some market research for a potential ***filePro-related***
> product.  It can't get much more filePro-related, so I'm not considering
> this off-topic.  I would appreciate a few moments of anyone's time who's
> willing to participate.
[most of the question deleted]
> I appreciate your time, thanks!
> 
> Bests,
> 
> Mark Luljak
> Fairlight Consulting

Mark,

This would be a great project, however I think you will find it 
mostly non-profitable, at least at the beginning and you will really 
have to decide that you are doing it for the "bragging rights" of 
saying you did it, before really looking for any "true" compensation.

That said, here is my take based on 10 years of mainframe experience 
in a distributed environment that touched five contintents on a 
private network of better than a quarter million users where we 
could control things that cannot be controled on the INet.

First, you would need to build the transaction logging system. 
To-date, as far as I know (thru 5.0.14) there is no @delete.  This 
would help greatly.

Second, there is not enough granularity in the @time return.  You 
will need hundredths if not thousands of a second to resolve 
transaction contention issues, for the order of application.  I had 
started doing what I am laying out below and ran into issues.  I had 
requested this time granularity, but I am not aware of it ever being 
implemented.

You will need to develop a few transaction codes that your system 
will use and what has priority if two have the exact same.  Simple 
codes such as:
N = New Record
C = Change
D = Delete

ClusterID      (Host Code)
ApplicationID  (Allowing more than one Application per Host)
The User Name  (Regardless of platform)
Station Address  (tty, or IP address)
Date  (yyyy/mm/dd)
Time  (hhmmss###)
Reserved Space for additional controls
FileName
Qualifier
Entire Record's Data w/ 20 byte header

You will have to impose, in your transaction system, the maximum 
record size to be how ever much these things (with the exception of 
the Data Record) less than the current record limits.  Technically, 
you will also want to preserve the current 20-byte control area as 
part of the Data Record, outside of your transaction control stuff 
as you will need your stuff to control the transaction logs and make 
it possible to cross platforms without loss of either your control 
information of FP's control information.

Now, from my experience, there are dsome things that just have to 
work and be usable before really embarking on clustinering.

Perfect a Transaction Logging and Recovery System ! ! !

You should be able to do the following.

1. Put out a test database.
2. Back up the test database.
3. Process a series of transactions, adds, updates, deletes, etc.
4. Backup the updated database.
5. Resore the backup from step 2.
6. Use the Transaction Log from Step #3 with your Recovery tool.
7. Backup this recovered database.
8. Choose method of puttng databases, from steps #4 and #7, 
side-by-side and doing a binary bit-level.

Now if this passes.  Add the the recovery tools to extract or remove 
updates from users or processes.  Why, if an errant process was 
done, with a transaction recovery tool you should be able to go to 
the most current backup and restore the database and reapply all 
transaction logs from the backup to the pouint of desired recovery, 
EXCEPT, for the errant transactions and essentially remove the 
effects of a bad process.

At this point you will have a product that is sellable and worth a 
lot to a large scale FP site, that cannot afford big outages and has 
to meet other data integrity issues that FP alone does not truely 
meet today.

****

Second Phase - II

Now in a multi-hosted environment you will need a way to sync times 
and time deltas in order to handle either recoveries  OR  data 
synchs.  This is where GMT offsets and a differential time field may 
need to be added to the tranasction logs.

You could issue a transaction sequence ID, would have to be 
alphanumberic to be great enough to not run out within too short a 
time window.  If you were going to try and meet an update synch of 
less than say 5 mins I think I would do the following.

Have the application program write the transaction log entries with 
two spare fields to allow another process to flag whether is has 
queued this entry to be sent to the other hosts (local or remote).

I would have a process that traverses the transaction log (keeping 
track of the point that it has currently processed thru looking for 
another entry to have been recorded.

Have it record in its own tracking file the transactionID and which 
hosts it has successfully transferred the transaction too.

Once it has been acknowledged by the other hosts (all that the 
trnasction had to be sync's with) it should mark the official 
transaction log that the transaction had been completed (that 2nd 
spare field from above)and move an archive copy of the transactionID 
to a history file.  This information would be needed in the event of 
a recovery that had to remove tranactions and would have to sync the 
other hosts in the process.

I would havve a process listening on each host for sync records from 
other hosts and posted sync records (from that current host) to the 
other hosts, you would then only need the database application tool 
to be running on the incoming side to read up transactions from an 
EXTERNAL transaction log and apply them to the local copy of the 
databases.

Due to a record being added on one host and then deleted from a 
different host, this is where the time differentials would be very 
tricky.  Also, record numbers would be totally worthless as Record 
50 in FILEA could be being created, at the exact same instant, by 
two different users, on different hosts, for different purposes.  As 
such every record MUST have a unique-indexed key, period.  Once 
could not afford to have the database updater (you should now see 
that the recovery tool, created in phase one is this process) doing 
a scans of the database to find a record to update or delete.

Finally you would need, in the transaction logs, to have the ability 
too do the following.
 From the (implementation or backup) point.  For a given file, given 
(record number or field key) produce a log of all pertinent tracking 
information, such as: Transaction ID, date, time and userID.  If you 
need specifc data, you would go back to the appropiate transaction 
logs, if still in retention (another issue entirely) and using the 
recovery tool, either recovery or dump (to file or printer) the 
contents you are trying to examine.

At this point you have abilities that rival major databases in the 
Oracle and DB2 rehlms, yet with a light weight character base I/F.

I would be happy to contine this discussion as it is a neat project.

If you could get FP-Tech to add an @delete ane higher time 
granularity your job would be much easier than it will as of today.

Best of Luck !

Keith
-- 
- - - - - - - - - - - - - - - - - - - - - - - - - - - -
Keith F. Weatherhead                   keithw at ddltd.com

Discus Data, LTD                  Voice: (815) 237-8467
3465 S Carbon Hill Rd               Fax: (815) 237-8641
Braceville, IL  60407       Nat'l_Pager: (815) 768-8098
- - - - - - - - - - - - - - - - - - - - - - - - - - - -