Co-Engineering Applications and Adaptive Business Technologies in Practice: Enterprise Service Ontologies, Models, and Frameworks

Why is this needed?

Dynamic economic context demands service agility and requires overcoming challenges in the way we

Motivating Adaptive Complex Enterprise (ACE) Frameworks for

What is the Adaptive Complex Enterprise (ACE)?

Integrates business and engineering knowledge needed for practice and research through the Adaptive Complex Enterprise service ontology and framework. This integrated knowledge seeks to give students and professionals conceptual tools for improved practice and a basis for the formalization of experiential learning for further research advancement.

Dynamic Context: Driven by a rapidly changing global environment that includes communications, competition, business growth, and technology innovation, ‘services’ now account for a pre-dominance of the labor force in most economies. To thrive, the service enterprise must sense, respond, and adapt to external conditions and leverage new opportunities (Haeckel 1999). In addition to addressing externally-driven uncertainty, the service enterprise must also embrace change and variation in interactions within its own organizations and systems that are getting more and more complex. These have been identified as issues to be addressed in the emerging field of Services Science Management and Engineering .

“Economic conditions like the recent recession are likely to make real-time delivery of products and services all the more attractive. Real-time enterprise technology can enable companies to answer queries instantly, monitor business on a continuous basis for quicker responses to shifting market conditions, and offer new products and services as new data becomes available. Behind the drive towards real-time enterprise is new hardware that gathers more real-world data than ever before, and software technology that eases or eliminates the trade-off between integration and flexibility. The technology is poised to have a tremendous effect on internal corporate operations while at the same time increasing the economy's fluidity--and possibly its turbulence ” .

These trends severely challenge traditional disconnected business, technology, and architecture frameworks that are designed to manage within simpler and more static and environments. While IT (Information Technology) with SOA (service-oriented architecture concepts and related technologies) is now positioned, it is not yet practically implementable as a tool for the service economy. This requires the development of a new integrated discipline for managing complex systems. Such a discipline must enable new tools and methods that are responsive to the external environment, making adaptive behaviors possible. This book is a contribution towards that goal.

Motivating Adaptive Enterprise Frameworks: As companies seek to stay competitive and offer more services through product innovations and variations new processes and information needs arise. These needs are often not met despite the overall investment in enterprise applications (Gartner 2003) . The resulting systems – composed of both organizations and software - are often found to be rigid (Nates 2003) . To succeed we have to think of systems differently – as complex systems of made of a small number of interacting patterns .

Services innovation requires emergent architectures frameworks: Service-oriented organizations typically have the dual challenge of embracing complexity due to externally-driven variation while simultaneously managing the evolution of enabling complex IT systems. However, confusing build-versus-buy-versus-adapt options confront the businesses looking to overcome IT limitations. Of these, the buy option is usually pursued first. However, when off-the-shelf applications fall short, service-oriented organizations continue to invest in enterprise-level initiatives and custom integration to improve performance. These investments often fail to deliver the expected value and return. This is due to application challenges such as overlapping application functionality, existing lack of services, and the lack of a practice methodology for developing realistic estimates of costs and risks. It is no wonder that only a small fraction of enterprise projects are on time and on budget. As IT use gets more pervasive within the extended enterprise, these challenges make innovation and adaptation to externally-driven change even tougher. These challenges need to be jointly addressed by the Business and IT organizations. This requires enterprise architectures that can facilitate the emergence of new externally-driven adaptive behaviors.

Creational Versus Operational and Continual Improvement: Traditional Computer Science curriculum has focused primarily on the creational aspects of software and hardware systems. The challenges of enhancing the overall functionality and performance of a deployed system is left primarily unaddressed within the curriculum. With the increasing complexity of deployed systems, there is a need for continual improvement approaches that take a more systemic view and system engineering methods for creative tasks in the context of an operational system.

Holistic, interdisciplinary practice frameworks are required for effective practice. Thousands of relevant books and publications provide an explosion of information - strategies, initiatives, technologies, and practices – all promising to improve the performance of today’s Business-IT systems. Five Forces, Balanced Scorecard, Enterprise Architecture frameworks, Customer Relationship Management, Enterprise Resource Management, Supply Chain Integration, Product Data Management, eStrategy, Re-engineering, World Class, Six Sigma, Total Quality Management, Just In Time, Lean, Concurrent Engineering, Enterprise Integration, Workflow Management, Middleware, Web services, Service-Oriented Architectures, Security, and Requirements and Organization Engineering are just a few among the topics covered.

However, this material is very poorly integrated for effective practice (Denning 2003, Armour 2003). Today’s knowledge for engineering complex systems is at best descriptive, ad hoc, and pre-scientific . Underlying principles are not abstracted for high-bandwidth communication and effective application by interdisciplinary implementation teams. How can we work locally in technical teams, yet make decisions that further overall business goals? When do we know that the issues we address are adequate and complete? When do we know when we have a truly new idea? How do we asses project impact on organizations and existing systems? How do useful best-practice frameworks relate in a deployment? How can we reduce custom code and reduce maintenance costs? These are just some of the questions that need to be addressed (Computing Curricula 2005).

Re-invention is in Part Due to Failure in Industry and Academic Collaboration. Despite the common elements across projects and businesses, most major IT projects in enterprise integration are often traditionally treated as “one-offs”. Further, there is no shared ontology and basis for shared principles. The various disciplines involved understand the same enterprise integration concepts in a unique way. For example, the ubiquitous notion of a ‘process’ means different things to the distinct practicing disciplines of business, systems engineering, and computer science. To illustrate the point, a systems engineer will be concerned with the behavior of the process while the computer scientist is more concerned with the representation of process behavior.

Consequently, strategies for integration are not only re-invented on a project-by-project basis but left to the particular experiences of the team of practitioners . With some exceptions, much of the needed knowledge is still proprietary and deployed at high costs. To counter the proprietary trends, both end customers and technology and service vendors such as IBM and Hewlett Packard promote open practices and re-use at the more conceptual levels (Herzum 1999).

While the practitioner, analyst, and academic communities have developed useful frameworks over the last few decades they typically have two shortcomings. Either they focus on a very limited slice of the problem or the problem is too abstracted. In either case research in successful IT practice and deployment is scarce. (See Denning 2003 for a discussion). Within academia green field assumptions also make research less relevant. Thus, unrelated and abstract frameworks lead to a lot of overlapping materials, inefficient communication, and re-invention without the evolution of fundamental underlying concepts.

Deeper Principles are Critical to Increased Re-Use and Reduced Service Costs. Service organizations often focus on differentiation as a competitive strategy; on the other hand the business return on IT is tied to increasing reuse. Businesses defining their success based on uniqueness are indeed different when examined on the surface but often very similar when represented at deeper levels (Long & Denning 1995) . In contrast, requirements communicated at the surface level generates a remarkably diverse number of IT implementaions. If every IT need is treated as a unique service project, it becomes difficult for technology vendors and IT workers to provide IT supported solutions and systems that are cost effective.

What is the Adaptive Complex Enterprise (ACE)?

ACE is a conceptual view of the complex interactions between services provided by any collection of organizations and systems; and Co-engineering is the theory that allows us to analyze, reason about, and improve the ACE performance. The service-interaction-based ontology is also introduced as the basis of the ACE representation (Adaptive Complex Systems 2005). This dynamic structure allows us to create work products that surface performance issues that can be analyzed and Co-engineered.

The ACE framework herein is based on a service interaction ontology that provides a deeper approach for defining principles that can be taught, practiced, and researched. Leveraging the representational methods of computer science, the analysis techniques of systems engineering, and business decision-making, the ACE framework leads to a shared understanding and increased communication bandwidth for Co-engineering of the Business-IT attributes of complex systems. Requirements and solution cost-benefit tradeoffs can now be discussed at deeper levels. Thus richer dialog can now replace the feeling of helplessness and déjà vu felt by experienced project managers as they face a lack of understanding within each new project team. New tools and methods can be developed. The integrative framework is based both on recent technology advances and conceptual successes of Complexity Theory.

Co-engineering simultaneously engineers the Business tactics, IT, Operations, and Strategy (BioS) performance based on established business and systems engineering principles and technology. The ACE framework, including both the representation and the Co-engineering theory, builds upon transaction theory to integrate Porter’s Model, The Balanced Scorecard, Lean, The Open Group Architecture Framework, Organizational Engineering, IT Infrastructure Library, Component Business Modeling, Business Process Management, Service-Oriented Architecture, and Autonomic Concepts.

The prescriptive performance-centered Co-engineering method integrates these proven and existing best practices at a deep level using precise service-interaction ontology. The result of Co-engineering is a roadmap for achieving overall complex system goals and business aspects, like risk management, value creation, return, and payback. Within the organization, business requirements for IT are more clearly defined and deployed to enable business service solutions.

Building upon the core of proven frameworks, Co-engineering also allows us to analyze the ACE Work Products and align performance to the value add desired by the BioS stakeholders. At the same time, it provides a framework to leverage emerging opportunities for managing complex IT systems better with IT itself. Such an active ACE architecture advances adaptation by providing a conceptual structure for real-time monitoring of the BioS layers. The alignment across the BioS is achieved by defining global policies for adaptation and providing the context for locally improving autonomous resources.

Using case studies throughout, we show how a small number of ACE framework patterns can be dynamically applied to permit local and lightweight implementations by teams of practitioners. Thus, the ACE framework provides a body of integrated knowledge for high-bandwidth communication and for implementation by interdisciplinary project teams in a highly distributed fashion.

Approach

At a glance:

What is the ACE approach to Knowledge Integration?

This is a book on service-interaction-based creation of value from business, information flow, operations, and technology performance perspectives. It is not just a book on any single aspect - such as integration, or security or compliance - but on an interaction language for representing and meeting commitments between all the stakeholder and resource perspectives. The result is a conceptual ACE representation of an enterprise with performance traceability that allows the Co-engineering of BioS perspectives. Thus the ACE framework allows service-oriented organizations to leverage the speed and dynamic flexibility provided by IT to run the ever more adaptive businesses. (See Christensen 2004 for the need for a more predictive capability and the special issues - Service Orientation 2007 and Services Science 2006 for and excellent background on services).

This book is an evolutionary one based on consolidated experiences of hundreds of companies. It is a book for the successful introduction of SOA projects into the services organization. And it is a book that describes more than planning for SOA implementation; it provides patterns for strategic planning, execution, and improvement of business services using specific Work Products. Finally, it is a book on in-the-large and in-the-small ACE representation to empower teams to identify and engineer the Business-IT tradeoffs through good decision making, investments, and implementations.

Thus, this is an integrative, rigorous, yet practical book intended for the advanced student or the practitioner with some real-world experience. It also identifies future research and opportunities for industry–university collaboration that is critical to furthering the discipline of services science (Services Science Management and Engineering). To this end, the book is presented in a manner suitable for the practitioner within a services organization and it also has extensive end-notes and annotated references into a vast body of knowledge in three disciplines - Business, Systems Engineering, and Computer Science.

The ACE and Co-engineering framework of this book has evolved over the last two decades of business process solution experience of the authors applying workflow technology to adapt industrial-age enterprise systems to enable new services and business processes. The underlying change-management patterns themselves have been abstracted and validated though projects with over a hundred and fifty different companies and the experiences of even larger communities such as International Standards Organization (e.g. ISO 9000, ISO 20000), Supply Chain Council, Object Management Group, OGC, ITIL, W3C, OASIS, and others.

The main motivation of this book is to provide knowledge that more successfully Co-engineers - anticipates and better predicts - and thus enables an organization to become an Adaptive Complex Enterprise for services delivery.

ACE Knowledge Integration

The approach taken in the ACE Framework is to define a small number of patterns based on the universal and fundamental ontology of customer-provider service transactions. The ontology serves to unify a selection of patterns and methods discovered and often re-discovered across the disciplines. (See Alexander 1999 for an overview of patterns).

Componentization: The recent progress in IT technologies, especially in the areas of middleware, workflow execution, and Web services, makes it possible to compose business process enabling solutions by leveraging existing applications (Berners-Lee 2001). There is now also a wealth of experience that allows us to re-use specifications and write adaptors to existing enterprise applications treating them as components within larger systems and business components (Herzum 1999). Vendors such as Hewlett Packard, IBM, and Microsoft and organizations like the OASIS , Object Management Group , W3C , TOGAF , and National Institute of Standards and Technologies now represent the experiences of numerous companies in this area.

Complementing the Case Method with Predictive Approaches: Broadly speaking, the widely accepted ‘case method has been used to develop decision-making skills by focusing on the behavior of the business . Here within the business, the IT details are primarily treated as a black box (Christensen 1999). The Co-engineering method of this book complements the case method by relating the business behavior to its internal layers and dimensions - operations, execution, and IT. This treatment of the internal layers as a white-box enables teams to explore cause-and-effect relationships and better predict the exact areas of maximum system improvement and IT leverage. By reasoning about the internal workings of the Business-IT system, management can now better identify implementation strategies. In summary, for improved business management, the black-box case approach identifies the faults of omission. The white-box Co-engineering approach discovers faults of commission, pinpointing those parts of the system whose implementation is faulty and needs to be adjusted.

ACE can now be Represented at Ever-Deeper Levels Using a Single Pattern Framework: ACE is based on easy-to-learn patterns incorporating uncertainty – dynamically and recursively applied patterns called fractals (Mandelbrot & Freeman 1983) - to elaborate each black box layer with white-box details, repeatedly (CACM 2005). Fractals allow all these internal dimensions to be represented and related more precisely with external events. Thus, macro-level business services and objectives are related to micro-level operating performance targets. The resulting structure now provides cause-and-effect traceability leading to better adaptation.

There is a growing and diverse community that has been applying fractals, and the more general Systems Theory (Skyttner 2005) , Complexity Theory (Kiel 1994, Kelly 1998), or Chaos Theory , to characterize and understand the behaviors of complex systems and the manner in which they respond to unexpected demands and changes in their environment. Some examples of these works and initiatives are reported within the Santa Fe Institute , the Next Generation Manufacturing (Jordan & Michel 2000, Kiel 1994) , and the European Union’s Sixth Generation Framework . In general, the exact use of Complexity Theory varies within projects. It has been applied as a model for simulations aimed at understanding growth in biological systems, fluctuations in economic systems, and turbulence in fluids. Complexity Theory also has been used to increase understanding through analogy between socio-economic systems and biological systems for greater understanding of socio-economic phenomena .

Continual Improvement: Finally, we integrate the concepts underlying Complexity Theory with the pattern in Deming 1982 - plan, do, study, and act - to locally improve the service transactions performed within the ACE at both in-the-large, e.g. business, and in-the-small, e.g. IT, dimensions. The service transaction is itself achieved through the interactions and services of the IT infrastructure made of people, processes, and software applications.

We use examples and case studies to show how the methods and principles can be deployed. By using them along with current technologies we can implement patterns for more flexible cost-effective SOA solutions for the enterprise. We will show how the ACE Co-engineering methods can form the basis for experimentation and science.

Audiences and Learning Objectives

The book is intended for the practicing professional and the advanced student in any of the disciplines of Computer Science, Operations Management, Management Information Systems, Systems Engineering, or Public Policy. Aimed at the advanced or mature student, the book uses an integrative interdisciplinary approach to applying materials from several disciplines to understand and solve concrete business problems using IT and other technologies, as and when applicable (Figure 1).

For practitioners at the executive and senior technical management levels with a background in business, operations, or information technologies, the book provides the precise knowledge necessary to understand and isolate the deeper issues and challenges in applying IT to develop solutions to business problems.

For the advanced senior or graduate student with work experience, the book should be used as part of an integrative program enabling the student to make deep linkages and thus better decisions in the professional world. It is expected that the student will participate in a practicum – such as a two-term industry-sponsored and industry-relevant project - applying the methods of the book. Basic project steps and deployment work products are introduced at the end of each chapter to facilitate the learning process.

Solution Architect: We introduce the solution architect role as an analyst that has the skills of both the Business and IT systems analyst to function as an integrator. The business-type benefits from the ‘white-box’ approach followed here in many ways. The Co-engineering method connects the Business analyst and the IT systems analyst with Work Products and tools to negotiate the ACE solution tradeoffs without needing very detailed disciplinary knowledge.

Finally, the knowledge presented here benefits IT workers in technology companies as well as IT workers working in companies with internal IT departments. IT workers in technology companies can deliver products better targeted to the service economy. Similarly, IT workers within internal IT departments can better understand the larger context in which their services have to be delivered. For background, a basic knowledge of business strategy, cost accounting practices, project management, databases, programming, and process/data modeling is assumed. Knowledge of UML and sysML is a requirement and is used to represent enterprise architectures.

Learning Objectives

The student or architect will learn to represent and manage the improvement of complex systems with the following skills:

The skills are based on an enterprise framework that brings order and meaning to observations that may otherwise seem chaotic. It explains complexity in a simpler manner by establishing principles. This positions the student to conduct future research in more predictive management techniques and technologies. With this knowledge, the student will be able to play the role of a solution architect within the organization.

Scope and Learning Objects

The scope of topics and relationships is presented in Figure 2. The ACE Co-engineering framework begins by analyzing external influences on the organization. These in turn create Strategy requirements that guide the Business, Operations, and the enabling IT infrastructure of the organization viewed as ACE. The Co-engineering principles help identify evolutionary and revolutionary methods for improvement. The related learning objects are alos as illustrated Figure 2.

Finally, the chapters contain examples of the business-related IT problems solved by the prescriptive Co-engineering steps. They address the real world challenges using rigorous principles. We also include the rationale for why the prescriptive methods work. Typically this will be presented in the form of end notes and references that index into a large body of related knowledge in three disciplines. This background work is presented for academic pursuit and the main flow is not interrupted for the reader with primary interest in practice.

Thus this book is written as a framework and an organized introduction to a vast amount of related interdisciplinary knowledge and underlying concepts that are the basis for Co-engineering complex systems. Many ideas are also presented for additional research. An extensive glossary and templates are provided at the end. We also take advantage of links to wikipedia and other accessible sources of reasonably factual information. The papers referenced here are selected for historical value and accessibility for the researcher . Finally, each chapter has suggested readings that form a good background for future chapters. 0.2

Preparing for Projects

The instructor is encouraged to identify enterprise architecture projects and project sponsors. Preferred sponsorship would be from the senior management looking to advance the use of IT to improve efficiency or for service innovation.