Feasibility Study for Potential fP Software

[Fairlight]

Hi Everyone,

I'm trying to do some market research for a potential ***filePro-related***
product.  It can't get much more filePro-related, so I'm not considering
this off-topic.  I would appreciate a few moments of anyone's time who's
willing to participate.

The project would amount to clustering filePro databases.  That's the core
and basic description of the intent.  The degree and intensity of the
clustering is not yet determined.  Currently, I know a few sites that are
looking at trying to keep a hot spare system around, and who could also use
a better way to sync multiple web servers than they're currently using.  It
could scale up to the best high-availability you could get for filePro,
depending on several factors.

By clustering, I'm denoting the following:

* Multiple filePro systems would have the ability to sync their data.

* Syncs might be in near-realtime, or the granularity might be a bit
  lower.

* Data sync would include add, update, and delete on a record-transactional
  basis.  These would not be whole-file copies, or even rsync-type diffs.
  The record updates would get handled on a transactional basis, as if
  they'd been entered through IUA (or imported by one method or another).

* There would be no master/slave relationship.  Due to the desire for
  failover and the attendant desire to have failback automatically handled,
  a master/slave relationship is wholly undesirable.  The fact that it
  would not require a master also means it -could- be built to accomodate
  more than two nodes.  (Read:  3+ machine clusters.)  I'm looking at the
  ability to do mutual propogation.

* The clustering would not rely on physical topography.  IOW, no LAN-only
  solution.  I'd be aiming for WAN viability over TCP/IP.  Primarily, this
  is so that people could have their in-house system and keep it synced
  with a co-located system, or several.

* Considering the nature of the beast, it would obviously have to be fault
  tolerant and handle when one of possibly several machines were down, and
  handle the updates gracefully.

* At any point, every machine would ideally have as much of the other
  systems' data as possible, and be mutually propogating.  This serves a
  round-robin or load balanced input scenario, be it multi-location or
  simply multi-system (web server cluster).

Okay, that's what I'm looking at potentially achieving.  Here come the
questions:

+ Is there an event type that will let you perform a task on record
  deletion?  That's a technical question that I need answered.  There are
  ways of working around it if there's not, but it would help if there was
  an @delete or something similar.  That's my only current technical
  question about fP itself.

+ How granular would you want the syncing to be?  What exactly would the
  threshhold need to be to make it worth your while.  Obviously "instant"
  is a bit of a misnomer given communication time is required.  But would
  you want this to be as near realtime as possible and perpetually running
  (possibly taking up a license seat at each node), or would a "run every
  'x' minutes" scenario work sufficiently?

+ Would the fact that any table restructures have to be propogated by other
  means (copy, manual restructure of all pertinent nodes, etc.) be a strike
  against such a product?  I feel it exceeds the scope of the product.  What
  I envision is taking a node entirely out of all other nodes' "push" lists,
  updating the node with the new layout, and re-activating the newly updated
  node on all push lists.  Is this reasonable?

+ How necessary would it be to encrypt the data at the application level?
  Does it need to be implemented to work over in-the-clear transport
  protocols, or could this rely on transport-level encryption (i.e., SSL/TLS, 
  SSH, etc.), or external application enryption like GPG?

+ Similarly, is it highly desirable to allow multiple transport methods
  (outdated rcp/rsh, ftp, etc.), or is it allowable to mandate the presence
  of certain software ([open]ssh, rsync, or possibly even apache, depending
  on the direction I think is best) as dependencies?

+ Should it support only 5.6, or should it support all the way back to 4.8?

+ Part and parcel of the last question, but also independantly--how much
  of the communications work should be handled internally to filePro even
  if it's a 5.6-only product?  Are you willing to "settle" for scp/ssh or
  http as a transport, or do you -really- mandate a filePro-only processing
  solution with fP Sockets?

+ Would single-record/table-based transactional updates be sufficient, or
  would it need to also handle "transactions" in the RDBMS sense of going
  as far towards all records being in place at the same time?  The latter
  is not actually possible in the ACID sense in filePro TTBOMK, but I'm
  wondering how close to that I'd need to focus and accomodate, if at all.
  Given the non-ACID nature of fP to begin with, is just handling the
  records themselves sufficient?

+ Understand that I envision the product as at -least- needing CALL
  statements in pertinent places, inserted into existing code.  This would
  be more of an API than one sweeping daemon that monitored all tables for
  changes.  Is that a problematic approach, or would you be willing to make
  fairly minor changes to your code in order to get this functionality?
  Is a monitoring daemon preferable?  (In my current trains of thought,
  doing it from an external monitor would result in far more performance
  overhead and worse performance in terms of time.  Mostly because you have
  to know there was a change, then you have to find what that change actually
  was.  If you can base this on events in an API-like fashion, that's a lot
  faster response time and a lot less overhead looking for what's already
  known by filePro at the time.)

+ How much would you be willing to pay for such a product?

+ How do you feel it should be licensed?  Per node cost?  "Entity" license?
  (Entity being a corporation or organisation, since "site" may not apply
  with off-site systems in the mix.)  I'm not a big fan of copy protection
  BS, so node-count based licensing would be on the honour system.

+ If the product existed -right now- and could be proven viable, would you
  purchase it within the next 3-6 months?  Do you figure you'd ever purchase
  or seriously look at purchasing such a package?  If not, why?

I think that wraps up my immediate questions.  I'm looking at the technical
viability of such a beast, currently, in various forms.  Some are more
readily attainable than others.  Before I do more technical research or
invest more time thinking about it, I want some frank and honest feedback
from the community--as much of it is reachable without having fP-Tech's
customer list.  :)

You -can- reply to the list, and I'll certainly read the thread.  However,
I'll also take private feedback via clusterfp at fairlite.com.  This address
will remain viable for two weeks before I shut it down.

I'd like to say that while I certainly value feedback that comes in
privately, I'd like to see a public discussion of this topic, really.  The
balance of views and counterpoints to certain technical arguments may prove
invaluable in deciding key factors--including whether to even proceed.
Additionally, negative feedback is just as valuable; if you think the
proposed product is a daft idea and you'd never go for it, please give me
that feedback as well, along with the reasons.  I'd like to know what I'm
dealing with in as great detail as possible.

I'll be blunt:  I've wasted weeks developing software that people said
they wanted and on which I took a nice, large loss.  I think this has the
potential to keep filePro a bit more in the game, make it more scalable a
solution, and benefit both the filePro product and community.  It solves
a number of different problem scenarios I've encountered.  But I'll also
be damned if I'm going to flush a few months of time and effort away on
something this intricate for very little return.  I'd rather not do it
at all, or wait until someone ponies up the cash for their own custom
development and just do it as a contract project, than go through the
"lossware" experience again.  Please, be honest and thoughtful in your
responses if you give them.

I appreciate your time, thanks!

Bests,

Mark Luljak
Fairlight Consulting