The Bivio OLTP Platform (BOP)
BOP is a transactional Model-View-Controller (MVC) application framework. With over 900 classes and 500 tests, BOP gives you all you need to create secure, scalable, and maintanable applications. A typical BOP application provides hundreds of end-user functions implemented in 10,000 lines of code or less.
Every day you probably use an Online Transaction Processing (OLTP) system. Telephone billing systems, banks, airline reservation systems, grocery stores, etc. are all examples of classical OLTP systems.
OLTP systems went out of fashion in the 1980s in favor of client-server computing and standalone PC applications. Out of this grew the Web. However, web developers today are making the same mistakes as classical OLTP programmers made in the 1970s.
The Bivio OLTP Platform (BOP) takes the lessons learned from OLTP systems and adds in the Model-View-Control (MVC) software pattern, an object-relational (OR) layer, and a declarative presentation layer. This makes BOP ideally suited to build large-scale, web-delivered applications. In fact that's how BOP got started; we had a complex service to build and BOP was designed to meet the requirements. This article explains why traditional web solutions were inappropriate and how BOP is designed.
Short course on OLTP
OLTP systems are characterized by many concurrent short-lived transactions. OLTP systems usually run on a 24x7 basis and must support thousands of concurrent users while maintaining quick response times. Here are some other characteristics of OLTP systems:
user authentication and authorization,
distributed transaction processing,
workload balancing,
transaction queueing and prioritization,
application parallelization,
ACID (atomic, consistent, isolated, durable) transactions, and data dependent routing.
The core of an OLTP system is the Transaction Monitor (TM). The TM is the container for application services. TMs are responsible for many tasks, which include naming, message routing, load balancing, configuration management, and security.
As you can see, Websites and OLTP systems have a lot in common.
The First Websites
Web servers started out as simple file servers. A site was a collection of static documents. The web server's job was serving these documents to browsers upon request. The demand for information increased including dynamic content.
To satisfy the need for dynamic content, programmers wrote software which created web pages offline from databases and other sources; just like the batch processing systems of the 1960s were used to create daily, weekly, and monthly printouts. The web servers would still serve these documents on demand, but the information was still static. The demand for true dynamic, online services grew out of these simple batch systems.
Database-backed Websites
Today, most web applications do more than serve files. They need to store information users give them in ways that can be easily retrieved. CGI was introduced to allow the web server to execute simple business logic. The solution was adequate for the first few small websites, but rapidly fell apart as sites grew. CGI programs are inefficient and loosely coupled. People cobbled together solutions to make CGI faster, but the lack of cohesion was the real problem. Web developers were trying to build complex integrated applications using loosely coupled collections of programs.
'The next evolutionary step was the introduction of the database-backed website a.k.a. the cart-backed horse.
A database-backed website is a software application, not just a collection of files. You can't just plug in a database and have the web server magically store and retrieve your data. You have to program the website. If all you know is HTML, you can't program anything. You have to learn a programming language. You have to tell the computer what to do in a language it can understand.
A zillion products allow you to create database-backed websites, e.g. ColdFusion, PHP, and ACS. These systems take great pains to avoid the word programming, because programming is "hard" and HTML coding is "easy". Web development is the process of coding server pages in an extension of HTML. The web developer programs database accesses and business logic in the HTML extension language.
The number one mistake made in the OLTP systems of the 1970s was mixing business logic and presentation logic. A classical example is embedding IBM 3270 terminal escape codes in the business logic of a COBOL program. With web-page-centric systems like ColdFusion et al, your programs are HTML templates with business logic sprinkled around or programs with embedded HTML. Application maintenance is a nightmare. If, say, you would like to generate XML from the same logic, you are out of luck. Web developers are forced to re-write the presentation logic and the business logic.
Database-backed websites are great for simple problems. However, when Bivio started building its investment club application, we knew that we would be programming lots of business logic and rendering the results in various formats including HTML, XML, PDF, and even email messages. We needed more support than any HTML template language could provide.
Application Servers
An application server is a grown up database-backed website. The big step was to recognize that a website is an application, not just a collection of programs and files. An application server has a mechanism for connecting business logic with a template language. The idea is that graphic designers don't need to know how to program. They just use the template language with their favorite WYSIWYG HTML editor. The resultant pages access the business logic written by programmers. Well, almost.
A server page is still a program. It's written in a template language, e.g. Sun's JSP or Microsoft's ASP. The server page must access the business logic written by the programmers. The business logic needs to make values available to the server page. Any way you slice it, there's inter-program communication going on. The problem is that by going to great lengths to separate graphic designers from programmers, the application server vendors have created a huge maintenance problem.
Here's one line from Sun's official J2EE tutorial on Java Server Pages (JSP):
<input type="text" name="username" size="25">
How does the graphic designer know that the display size of a username is 25? Every database field has limits. Is the maximum size of a username 25 characters? Well, not according to this example. HTML text fields can have a maxsize attribute, why wasn't it used? More importantly why is it necessary for the graphic designer to write this code at all?
Most page-centric application servers do not allow you to separate business and presentation logic. Some application servers, e.g. J2EE, pretend to create a separation but fail to give developers an architecture. You can write business logic in server pages, tag libraries, or "beans". But, for example, is the number of rows returned from a database query business logic or display logic? No guidance is given.
In application server systems of today, the presentation layer drives the business logic. Even leading-edge application servers which use declarative languages like XMLC, do not address the control logic issue. For maintainability, you need a control layer, which is what differentiates OLTP systems from application servers.
Bivio OLTP Platform (BOP)
In 1999 Bivio built and launched a web-based tax accounting service for investment partnerships (stock clubs, private equity funds, angel funds, etc.). The accounting service was developed concurrently with email and file services. The application servers available in 1999 were fragile and provided little or no support for complex applications. We had to develop an application infrastructure to support these first three applications.
Our first step was to recognize the parallels between our problem and the problems solved by OLTP systems. We also realized that the Model-View-Controller (MVC) software pattern applied quite nicely to our problem.
MVC differentiates BOP from application servers and classical OLTP systems. BOP gets its controller architecture from OLTP systems. Its Model-View paradigm comes from GUI toolkits such as Swing. Its declarative page language has roots in Tcl/Tk.
A BOP website is a list of tasks (units of work). Each task is identified internally by a code name, and externally by one or more URLs. The dispatcher uses the task table to determine which task is requested and whether the incoming user has permissions to execute the task.
Once a task has been authorized, it executes a list of models, actions, and views. A model is stateful business logic. Actions are stateless business logic. Views generate HTML, XML, PDF, or email messages. Views present the results of the business logic, i.e. the data contained in the Model state.
Models come in three flavors: Property, List, and Form. Property models define the database schema, including relationships, constraints, and types. Property model meta-data is used by our Object-Relational (OR) layer so that you probably don't need to know SQL. List models are views on Property models with builtin paging support. Form models encapsulate and abstract HTTP forms. All models share the same meta-data interface.
Views use model meta-data to create widget hierarchies during initialization. At rendering time model state and meta-data are used to navigate the hierarchies. A Password widget is selected by the presentation configuration logic, because a Model's field is of type Password. Invalid input data is available via the Form model along with the detected error code. This info is extracted automatically by the field's widget. BOP's widget library is so rich, you usually do not need to know HTML to construct pages.
OLTP systems have run-time support including configuration management, version checking, logging, real-time debugging, and transaction resource management. The BOP operational support components provide these necessities and more.
BOP Advantages
BOP's primary advantage is as an alternative model for web development. Application servers promote a page-centric model. BOP's task-centric model allows you to think in terms of operations, not output (results).
The control logic is very similar to an OLTP's system message dispatch table. The single point of entry has many advantages as it allows you to centralize important operations such as the decision whether to commit or rollback a transaction. Error redirects are handled centrally and therefore consistently.
What didn't come from OLTP systems is the clear separation of business and presentation logic. Application servers allow you to do this, but they don't encourage it.
With BOP, the presentation logic is declarative, just like HTML is supposed to be. You specify a field name and the page configuration language knows its type and, say, whether it should display a list (select) box or enumerate the choices as radio buttons. You declare general page styles and insert content (forms, tables, etc.) into those pages.
The OR layer is also declarative. You define the fields of a Property model. The OR engine generates all the SQL you need tocreate, read, update, and delete rows in the database. Type validators, converters, and meta-data are shared across all models. List models are often just a declaration of the Property model fields desired and the join equivalences.
The operational support intrinsic to OLTP systems but missing from most web servers is included in BOP. Decades of experience building distributed systems has left its mark on BOP's architecture.
Security
Security of OLTP systems has traditionally been a trust issue. Programmers were careful not to leak information about one user to another.
With BOP, security for tasks is role-based. The user is authenticated and has an authorized role for a security realm. Tasks cannot be executed without this security check. Every task operates within a realm.
Every model (table) has a security field identifying the realm the data belongs to. When you load a row from the table, BOP's OR layer ensures only data for the currently authorized realm is loaded.
Scalability
Scalability is an intrinsic quality of most OLTP systems. The transaction monitor distributes the request load across an arbitrary number of stateless servers. Statelessness is the key to scalability.
BOP servers are stateless. Users are authenticated and authorized for every request. The performance is quite acceptable. The request set up time is under 20 milliseconds with a modest hardware configuration.
The database is the scalability bottleneck in an OLTP system. If you can partition the database in a meaningful way, you have instant scalability. The security identifier can be used to partition a table.
Conclusion
The Bivio OLTP Platform was developed in 1999 to build a complex system: investment club accounting, reports, taxes, broker account aggregation, quote database, file sharing, message boards, and support interface. Although the system has grown since its first release the BOP's architecture has remained the same.
The groupware application today comprises 300 distinct user-level tasks. The implementation is less than 70K non-comment-lines of code including presentation text. Systems of similar complexity usually require an order of magnitude more code.
In summary, BOP is a proven and efficient infrastructure on which to build web-delivered applications.