Table of Contents

Chapter 1: Introduction__________________________________________________3

Chapter 2: Literature Review_____________________________________________ 6

Chapter 3: Methodology_________________________________________________16

3.1 Step One: Performance Metric Collection ___________________________16

3.2 Step Two: Identify Information Requirement_________________________19

3.2.1 Task One: Identification of Use Cases_______________________19

3.2.2 Task Two: Identification of Classes and Attributes_____________22

3.2.3 Class Diagram_ ________________________________________23

3.2.4 Information Requirement_________________________________24

3.3 Step Three: Implementation______________________________________ 25

3.3.1 Relational Database_____________________________________25

3.3.2 Introduction to Online Benchmarking System: The Web Interface 32

3.3.3 Performance Analysis Based on SMART____________________34

Chapter 4: Website Documentation_______________________________________47

4.1 Purpose of Documentation_______________________________________47

4.2 What you need to know_________________________________________47

4.3 ASP Code Structure____________________________________________48

4.4 A Walkthrough of Online SMART Application______________________52

4.5 ASP Databases________________________________________________ 57

4.5.1 ASP _________________________________________________58

4.5.2 ADO_________________________________________________59

4.5.3 The ADO object model__________________________________ 62

4.6 System Evaluation_____________________________________________ 63

Chapter 5: Conclusion__________________________________________________64

Future Research__________________________________________________65

Appendix A: Performance Metric List_____________________________________66

Appendix B: Questionnaire______________________________________________82

Appendix C: Sample survey/Metric Evaluation Package_______________________100

Appendix D: Feedback from Survey Evaluation______________________________110

Appendix E: Use Case List_______________________________________________124

Appendix F: Classes and Attributes________________________________________129

Appendix G: Snap shots of Online Benchmarking System______________________ 135

Reference____________________________________________________________142

CHAPTER 1: INTRODUCTION

In order to remain competitive, transportation service providers, such as trucking firms, must constantly evaluate their operations, management structures, information systems, and customer relations with respect to their competitors. One of the standard approaches used by industry to evaluate performance is known as benchmarking. Benchmarking is a process by which companies determine the best practices that lead to exceptional performance; however, benchmarking is a complicated and time-consuming process. First, the data necessary to support a benchmarking analysis can be difficult and time-consuming to obtain. For example, companies may be reluctant to share the data necessary to determine best practices because of proprietary or competitive concerns. Second, performance metrics required to perform the benchmarking can be difficult to determine and difficult to interpret their relevance to best practices. Third, finding the best practice in a subject area can be a lengthy task that requires expertise and analysis techniques. Fourth, communication with benchmarking partners must be effective and require full cooperation from all parties. As a result, a system is needed that can aid in the process of benchmarking. The system or tool should partially or completely solve the problems mentioned above. In particular, the system should allow the users to track the performance metrics in specific subject areas, such as customer service or product service performance. Since users are interested in identifying best practices and performance metrics, the system should facilitate the data collection process within these subject areas. In addition, the system must be designed in a user friendly way so that users can quickly apply and learn the system. In turn, communication of all parties involved is improved because the system allows a systematic approach to benchmarking.

There are many systematic ways of conducting a benchmarking study; however, an “online benchmarking” system is one of the innovative ways that one can use to benchmark a transportation company. This research examines the development of an on-line benchmarking system to facilitate the benchmarking of transportation providers. We have implemented the system in prototype form as an “on-line benchmarking system (OBS)”. In particular, the system allows transportation providers to recommend performance metrics, rate their importance, classify the metrics into service categories, and develop other specifications for a performance measurement system. In addition, the system allows for the on-line collection of performance metric values through an on-line survey process. The data is collected and stored in a database for later benchmarking analysis against other participating benchmarking parties in an anonymous fashion. Finally, the system facilitates best practice analysis through a multi-criteria process based on a balanced score card approach. The metrics are categorized according to the four categories in the balance scorecard approach to performance measurement.

This project addresses the need for timely, accurate and comprehensive information concerning the performance and capabilities of transportation and local delivery providers in the form of an on-line benchmarking database. In addition to an on-line benchmarking database, this project examines the development of innovative technologies and methodologies to allow the interactive analysis of transportation providers compared to best practices.

Benchmarking surveys are one of the most popular techniques used to collect performance data. For instance, a transportation provider may perform a survey to examine some specific operations, such as, customer relations, delivery operations, or e-commerce strategies of peer-group companies. In this project, we investigate the development of a benchmarking system with online capabilities to facilitate benchmarking survey and data collection. The system will not only contain information from the survey but will also contain information about the surveys, such as performance metrics, survey questions, subject areas, etc.

The primary goal of this project is to support innovation and dissemination of knowledge within the area of benchmarking analysis for transportation carriers. The following are contributions of the project.

§ Standardized performance metrics for transportation carriers within all areas of operation.

§ Innovative tools for the display and comparison of benchmark results to indicate the performance of Transportation Company.

In this report, we present the details of the collected performance metrics and the structure of the online benchmarking system in the context of trucking carriers. In addition, an online application based on Simple Multi-Attribute Rating Techniques (SMART) will be discussed in full. SMART is the method that we use to prioritize the performance of carriers. We begin with a review of literature relevant to the benchmarking of transportation service providers.

CHAPTER 2: LITERATURE REVIEW

According to Zivan (1992), then Xerox's vice president for logistics and distribution, a pioneer in developing benchmarking processes, defines benchmarking as the heart of the planning process for any company that holds customer satisfaction as its highest priority. Indeed, benchmarking is an integral part of any quality improvement process. Performance metrics are used to indicate the performance of an organization within a benchmarking analysis and within performance measurement systems.

According to Watson, et. al (1999), performance metrics can be defined as the analytical tools in the performance measurement process that take measurements, display results, and determine subsequent actions. A specific value of a metric indicates the performance of a specific area in an organization. There are many performance metrics that are important to a company’s operation. In general, an enterprise will have hundreds of potential performance metrics to be incorporated into their performance measurement systems. Because of the large number of potential performance metrics, it is often very useful to classify the metrics into subject areas. For example, Watson et al. (1999) proposed a comprehensive logistics performance framework and a best practice template in their work. A total of one hundred and twelve metrics were identified and categorized into four groups, which were “cycle time”, “quality”, “financial”, and “resource”. In addition, a list of eighty-two best practices was complied from two hundred and sixty six best practices that matched the appropriate metrics. In addition, their work “identifies benchmarking that links qualitative values (best practices) with quantitative measurement (performance metrics), across the value chain” (1999).

Bravo and Related Research

In this section, we discuss the research project known as BRAVO performed by Eindhoven University in the Netherlands and research related to the BRAVO study. The key findings related to this research are:

1) The importance of segmentation (or categorization) of trucking companies.

2) The identification of critical success factors or critical performance metrics (financial and operational metrics only) that directly affect the performance of companies.

3) The identification of a performance metrics list that are related to the financial and operational aspects of the company.

4) The identification of relationships between financial and operational performance metrics.

In the article “Performance measurement in the transportation and distribution sector” by Donselaar et. al. (1998), the authors address the importance of using performance indicators other than financial because of the growing trend of quality improvement in many industries. In addition, the authors are interested in identifying the relationship between financial and operational measures that had an obvious impact on the performance of a company. As a result, the authors conduct a research project known as BRAVO that concentrated on logistics performance from the transportation providers’ point of view. The BRAVO project studies marketing, information technology, inter-firm co-operation, strategy and operations of trucking companies.

In order to measure performance, three different scores are used to distinguish successful and less successful companies. The scores are named BRAVO-1, BRAVO-2, and BRAVO-3. “BRAVO-1 is a score in the long term and consists of ratios concerning the growth, productivity, solvency and profitability of the company. BRAVO-3 is a score to measure the operational performance at segment level. This score is defined as the turnover per segment (or category) in proportion to the relevant costs in the segment. BRAVO-2 score is derived from the BRAVO-3 scores. This is done by taking the weighted sum of the BRAVO-3 scores over the segments in which the company operates. Generally, the authors use a weighting method to calculate the BRAVO score. However, no detail on the weighting method is given.

The objective of the BRAVO project is to develop a list of performance metrics that was critical to the operations of transportation service providers. The project team conducted two separate surveys. One survey was conducted on shippers and another one was conducted on transportation providers. About 150 companies responded to the survey. The companies were categorized into successful and less successful group by looking at their strategy, marketing, information technology, cooperation, and operations aspects.

In the BRAVO project, the authors segmented or categorized all of the 150 companies into three different groups, which were “transportation”, “distribution” and “warehousing”. In addition, trucks were divided into general container trucks and specialized trucks. As a result, performance data (financial, operational) was collected according to the segmentation (categorization). Based on the segmentation of the trucking companies, the authors constructed two models, which were the “Transportation” model and “Distribution Model”. Both models attempted explicitly to identify the relationship between financial performance metrics and operational performance metrics. The models were constructed with performance metrics arranged hierarchically. In addition, mathematical relationships were primarily used as the indicators of the relationship that existed among the various metrics.

After the models had been completed, the authors constructed a method that was used to identify the “Critical Success Factors (CSF)”, or critical performance metrics (which are the performance metrics that have a crucial impact on the operations of companies). The approach used by the authors to identify the CSF and the results of the project were reported by Wouters et. al. (1999) The method was made up of five steps:

1) Perform a one-factor correlation analysis on all the measures to find out which operational measures are strongly associated with financial measures.

2) Compare the average value of the operational performance measures for the companies that are performing well.

3) Find the operational measures that have the highest impact

4) Discuss with experts to decided if the data need further segmentation (segmentation is used by the authors to indicate the categorization of metrics)

5) Repeat step 2 to 4 if necessary with new segmentation.

In the following, we discuss the CSF as identified by the Bravo authors using the approach above. We begin by first looking into the Transportation model, and then the Distribution model.

Transportation model

The model consisted of many operational factors that related to one another. The purpose of the model was to determine the most influential factors that were associated with operational performance. In this model, companies were further segmented into Short Distance group and Long Distance group because the average distance per trip had a significant effect on the BRAVO-3 score. The results in the short distance sub-segment showed that both “average wages per driver” and “average load in a truck” were the most important metrics that were highly correlated to the performance of companies. Improvement made to these metrics could result in better operational performance; therefore, the metrics were classified as “Critical Success Factor (CSF)”. On the other hand, the results in the long distance sub-segment showed that “turnover per trip”, “turnover per km per 1000kg”, and “turnover per truck” were highly correlated with the BRAVO-3 score (Note that “turnover” was used by the authors to indicate “revenue”); however, the three metrics should be used to diagnose the situation of a company only, because they do not suggest a solution the operational performance of a company due to their relation to financial performance. Therefore, the metrics were not classified as CSF. On the other hand, the “percentage of km (kilometer) driven empty” and “distance traveled per trip” were considered as CSFs. Therefore, improvement made to these metrics can directly improve the operational performance.

The Distribution Model

In the distribution model, the focus of operation lies on stopping instead of driving. Therefore, operational performance of companies is determined mostly by stopping operations. In this model, companies were further divided into small drop weight and large drop weight groups.

Results for the small drop weight companies showed that “variable cost per km” was directly proportional to the operational performance of the company. Even though turnover per km was highly correlated with operational performance, the metric was associated with financial performance as well. Therefore, this metric could not be considered as CSF. Other important operational metrics included “number of stops per trip” and “average distance between two stops”.

Results for the large drop weight companies suggested that “turnover per km” was highly correlated with BRAVO-3 score. Since “turnover per km” was associated with financial performance, it was not classified as CSF; however, “distance between two stops” was determined to be a CSF. In this case, the shorter the distance between two stops, the better the performance of the company. Finally, the size of a company (defined as number of trucks and annual revenue of the company) was determined to be a CSF.

All the CSF identified in the Bravo research enables individual companies to study the company’s current situation so that improvement plans could be carried out in the future. The limitation of the project was that it concentrated on operational performance and disregarded overhead costs, because overhead was not considered significant in this sector. In addition, company performance was compared with one CSF only. However, there might be more than one CSF that could account for the success or failure of a company.

The research served as an important guideline for our project so that we have better understanding of certain performance metrics of a transportation company. On the other hand, CSF and some other metrics that were presented in this study were included in our performance metric list. For example, revenue/truck, average load, average speed, number of truck per company and load factor. The different models and their evaluations were also important because the authors studied the problem from different point of views. In addition, the segmentation of transportation companies in the BRAVO project highlighted the importance of categorization when comparing performance. In the article “An innovative survey in the transportation and distribution sector”, Donselaar stressed on the importance of proper categorization of trucking companies when conducting the BRAVO project. In the authors own words: “Segmentation is crucial for benchmarking: comparing apples and pears makes little sense (1997).” In fact, the author argued that segmentation of transportation and Distribution Companies were important in the selection of CSFs and it was an important condition for the measurement of success. In our project, we integrated Doonselar’s segmentation concept by including an important “carrier categorization” questionnaire in our benchmarking system.

The BRAVO research indicates that the selection of key performance metrics is extremely important so that the performance of a company can be accurately and precisely measured. Since the selection of key performance metrics is important, we investigate more on this subject. Caplice et al (1994) have a structured approach to select crucial performance metrics. Caplice et. al.(1994) proposed eight criteria to consider when selecting performance metrics for logistics and business purposes, which were: 1) Validity, 2) Robustness, 3) Usefulness, 4) Integration, 5) Economy, 6) Compatibility, 7) Level of detail, and 8) Behavioral soundness. In addition, they suggested three forms of measurement that could be used to capture the performance of business activities, which were 1) Utilization, 2) Productivity, and 3) Effectiveness (1994).

Surveys can be an effective tool to identify areas that may contain key performance metrics. A study by Tang et al. (2001) applied the Analytical Hierarchy Process (AHP) to measure the performance of outbound logistic processes within Taiwan’s notebook computer industry. By conducting a survey, they identified seven key performance metric areas that closely related to logistic activities: 1) information system, 2) customer service, 3) order process, 4) distribution process, 5) assembly process, 6) inventory control, and 7) warehouse management.

The identification of relationships in performance metrics can be useful in constructing a performance measurement system or conducting a benchmarking project. Boyd and Cox (1997) used a technique known as the “negative branch” to evaluate the cause and effect relationships of performance metrics within an organization. According to them, the construction of a “Negative branch” is a four step process: “1) Write down the positive effects that are expected to result from the action, 2) Write down the negative effects that might result from the action, 3) Connect the proposed solution with your suspected positive and negative effects by cause-and-effect relationships, and 4) Read the negative branches from bottom up using if-then logic, scrutinizing every statement and logical connection along the way, and make necessary corrections (1997).” For example, if on-time delivery is a performance metric, then drivers may be forced to exceed the time allowed for consecutive driving hours in order to consistently meet on-time delivery. This process helps to identify trade-offs between the metrics.

In addition to the categorization of transportation company, we realize that proper categorization of performance metrics is crucial. Well categorized metrics reduce the chances of using the wrong metrics and make the proper undertanding/analyzing of certain metrics possible. Kaplan and Norton (2001) create a performance metric framework known as the balanced scorecard. The balanced scorecard is a “multifaceted tool for communications, alignment, improvement, and control that integrates critical non-financial performance measures into the basic management structure of the organization (1992)”. According to the authors, the balanced scorecard “retains measures of financial performance – the lagging outcome indicators- but supplements these with mewasures on the drivers, the lead indicators, of future fianancial performance (2000).” The framework was divided into four perspectives: customer, internal business, learning and growth, and financial (2000). The learning and growth perspective includes employee training and corporate cultural attitudes related to both individual and corporate self-improvement (2002). The internal business perspective refers to internal business processes. Performance metrics that are based on this perspective enable the managers to know the status of their business. Furthermore, the metrics tell whether the products and services conform to customer requirements (2002). On the other hand, the customer perspective is based on the importance of customer satisfaction in all businesses (2002). Finally, the financial perspective is based on timely and accurate financial data from the management. By using balanced scorecard, we can categorized virtually all metrics into these perspective. According to Brewer and Speh (2000), this framework “balances the inclination to overemphasize financial performance by incorporating metrics related to business process measures, innovation and learning measures, and customer satisfaction measures”. In addition, Brewer and Speh developed a method that applied the balance scorecard to measure the performance of supply chain.

In the project, we integrated the four perspectives of Kaplan and Norton’s balanced scorecard into the metrics categorization. In other words, the performance metrics that we have identified for transportation providers within the on-line benchmarking system are classified according to balanced scorecard framework. Indeed, the integration of these perspectives into our categorization facilitate the understanding and dissemination of the metrics.

CHAPTER 3: METHODOLOGY

In order to satisfy our objectives, the following procedures were executed step by step:

(1) Performance metric collection: We collected a total of seventy-nine performance metrics that were most related to the various operations of trucking service providers. All the metrics were categorized to the four balanced scorecard perspectives accordingly.

(2) Identify information requirements: We identified the “tasks” that we needed to perform so that the objectives of this project could be fulfilled. In order to identify the information requirements, we utilized the Unified Modeling Language (See section 3.2.1 for more detail) to analyze the situation.

(3) Implementation: After we had identified all the required tasks, we needed to implement the tasks. The implementation was mainly achieve by (i) creating a data-driven dynamic website, (ii) creating a relational database, and (iii) implementing an online decision tools based on the collected metrics known as “Simple Multi-Attribute Rating Techniques” or SMART.

3.1 Step One: Performance Metric Collection

A performance metric is an indicator of performance in a system. A metric may have a certain value and unit that measures and indicates the system performance. The importance of performance metrics in benchmarking is that the metrics highlight the important aspects of a system that can be improved. In other words, performance measurement evaluates performance relative to a defined goal. A benchmarking team may identify the performance metrics of a process so that a complete performance measurement system can be designed. By improving the values of certain metrics, the performance of a system can be improved.

In order to facilitate online benchmarking, we need a comprehensive list of performance metrics in all areas of interest for transportation providers. In order to develop a comprehensive list of potential metrics, we examined literature related to trucking service providers and discussed metrics with experts within the industry. After identifying potential metrics, we categorized the metrics into the four categories associated with the balance scorecard. The grouping of metrics was subjectively based on their functionalities and properties. For instance, the “unloading time” metric was categorized into the “operational measures” group because “unloading time” was directly related to the time needed to unload goods in an operation.

By using the balanced scorecard groupings, we attempted to cover all the major operations within a company. In other words, the metrics collection is not meant to be a complete list of metrics, but to serve as a resourceful reference for transportation service providers (mainly for trucking service providers). The four groups of performance metrics covered the most important performance areas that can be improved in a company. For instance, if a manager wishes to improve customer relations, the manager can look into the “customer” group for useful metric(s). Similarly, the manager can look into other groups as needed.

As indicated in Table 1, the metrics are classified according to the categories in the balanced scorecard. The metric table is not rigid. In fact, one can add more metrics into the table so that the collection can be more comprehensive.

Internal Business Process	Financial Measures	Learning and growth	Customer
Loading size	Direct cost per truck	Absentee rate	Claims settling rate
Average length of haul	Maintenance cost /mile	Average # of workdays	% Of active customer
Average speed	Driver’s gross pay	Driver’s weekly work hours	Delivery accuracy
Operating hour/Truck/day	Operating Revenue/wages	Average nights a driver’s home/week	% Correct order
%Distance driven empty	Revenue/driver	Training received	% Order returned
Trailer loading rate	Revenue/(Driving time)	Recognition	Delivery date compliance
Trailer unloading rate	Revenue/Trip	Accident rate	Re-purchase rate
Load Factor	Revenue/Truck	Availability rate	% Order on time
Miles/Truck/week	Operating Revenue/Distance	Education	Service termination rate
Average loaded miles	Revenue/Hour	Turnover rate	Claims rate
Average empty miles	Maintenance Cost	# Grievances/year	Complete order rate
# Stops per route	Logistics operating expenses	Recruitment rate	Damage free rate
% Routes repeated	% Return on asset	Truck operator’s experience	On-time delivery
Average MPG	% Return on investment	# Truck operators	On-time order entry
Trailer, tractor ratio	Revenue growth rate	Employment opportunities/year	On-time loading
% Driven not empty	% Market share	% Offers accepted	Correct destination rate
Time driven between 2 stops	Rental Expenses	Promotion rate	Data Entry Accuracy
Tracking accuracy	Insurance expenses
Inventory record accuracy	Annual inventory usage
Pick rate	Inventory asset value
Fill rate/Quantity	Operating revenue/year
Material handling rate
Inventory Turnover
% Complete sales
Loads/trailer/year
Average # of dispatches/load

Table 1: Performance Metrics

For each metric, we developed a detailed metric description that included a name, a textual description of the metric, a formula for computing the metric if applicable, the measurement units for the metric, the purpose of the metric, and a place for comments or discussion about the metric. The following is an example of such a description:

Name: Loading Capacity

Definition: Maximum load per trailer in tons

Formula: N/A

Units: Tons

Purpose: Used to identify the loading capacity of trailers.

Discussion: This metrics is needed to derive other metrics, such as “Average load of truck”.

Please refer to “Appendix A” for complete list of metric descriptions.

3.2 Step Two: Identify Information requirement

Before we can identify the information requirements, we need to understand the technical requirements of the project. Information requirements are the requirements of the project that one has to identify and satisfy so that the objectives of the project can be met. In order to identify the project requirements, we performed the following two tasks:

Examine system-user interaction through use cases.
Identify all major classes and attributes.

The following are the detailed descriptions of the tasks that we have performed.

3.2.1 Task one: Identification of Use Cases

Unified Modeling Language (UML)

Before we get into the discussion of use cases, we need to introduce the UML, which is the parent of use case. We utilized UML to develop the representation of information requirements.

As defined by the Rational software company, the definition of UML is:

The Unified Modeling Language (UML) is a language for specifying, visualizing, constructing, and documenting the artifacts of software systems, as well as for business modeling and other non-software systems. The UML represents a collection of best engineering practices that have proven successful in the modeling of large and complex systems. (Source: www.rational.com)

In addition, UML provides the application modeling language for,

Business process modeling with ‘use cases’.
Class and object modeling.
Component modeling.
Distribution and deployment modeling.

(Source: www.rational.com)

Since UML is a powerful tool to model a problem, we utilize it to model our information requirements. In other words, UML helped us to identify all the requirements in this project so that we knew exactly what we wanted to build. By using UML, we identified all major classes and attributes in the project. In addition, we identified use cases of the system as well. In the following section, we will discuss use cases in detail.

As we have seen, use case is one of the components of UML. A UML use case can be used to show all the functionalities in a given system. A use case consists of four major elements:

(i) Actor: An actor is anything that interacts with a system. It can be a human actor or non-human actor.

(ii) Definition: The description of the user-system interaction

(iii) Scenario: If different scenarios exist in a use case. Then, list all possible scenarios.

(iv) Use Case: Name of the use case has to be defined

In this project, we brainstormed a list of possible uses of the online benchmarking system. Then, the list was further refined. The refinement was done in the way that only specific uses of the system that satisfied the objective requirements was allowed. The following is an example of a use case that describes the interaction of the user with the system. Please refer to Appendix E for the complete listing of use cases. Indeed, the use cases shown in Appendix E encompass most if not all of the possible uses and scenarios that might occur during system/user interactions.

Use Case: Take Survey

Definition: Surveys are available online so that users who are interested in participating in taking a survey can fill out an electronic survey form easily. The survey respondent has to go to a designated web page that contains the survey.

Actor: Transportation service provider (TSP) – Transportation service provider is the company that provides transportation service.

Scenarios:

Actor (TSP)	System
Click “Survey” link.
	Display all surveys that are available in the system.
Select a survey by clicking on it.
	Display the survey questions.
Fill out the questionnaire. Click “Submit” button after the user completed the questionnaire.
	Perform error checking - If error is detected, display error message and ask user to correct the error. - Else, display message to thank the survey taker.

In the following, we discuss the important concept of classes and attributes that contribute to our database design.

3.2.2 Task Two: Identification of Classes and Attributes

Major classes and their corresponding attributes were identified. Classes describe the objects that we have to store in the system so that the system contains all the functionality stated in the use cases. Attributes are all the properties that are associated with classes. In the following, an example of the registered user class and its attribute are given. Please refer to Appendix F for all major classes and their attributes.

Class: Registered User

Description: A registered user is a person who registered online in the benchmarking website so that he/she can use some features in the website.

Attribute: First Name

Description: A name that describes the first name of the registered user.

Domain: A text string

Attribute: Last Name

Description: A name that describes the last name of the registered user.

Domain: A text string

Attribute: Company Name

Description: A name that describes the name of the company that the registered user currently works.

Domain: A text string

Attribute: Phone number

Description: A phone number is a text string of number that can be used to make a phone call. This is the phone number of the company that a registered user currently works.

Domain: An text string in the form ###-###-####

Attribute: Email address

Description: A user name and domain name that can be used to send or receive email.

Domain: A text string in the form UserName@DomainName

Attribute: User name

Description: A user name that is selected by the registered user as the login information and validation.

Domain: A text string or number or combination of both.

Attribute: password

Description: A string or number or combination of both that is selected by the registered user as the login validation and information.

Domain: A text string or number or combination of both

3.2.3 Class Diagram

After we had identified all the classes and attributes, a class diagram that indicates the relationship between the classes was drawn. Since the benchmarking website is a data driven website, we need a database to support it. Note that the class diagram served as an important guide to the fundamental design/structure of our database. The class diagram is shown below:

Figure 1: Class Diagram

Relationship Interpretation

The relationship between two distinct classes can be interpreted this way: The relationship between “Registered User” and “Scorecard” indicates that, a registered user can create many “scorecards”; however, an existing “scorecard” must belong to only one “registered user” only. The relationship between “Question” and “Answer” implies that, one question can have one or many answers; however, each answer belongs to one question only. The other relationships can be interpreted in a similar manner.

3.2.3 Information Requirements

After a careful analysis of all the use cases and classes and their attributes, we concluded the information requirements of the project, which were shown in the following:

1) Web Interface: This is the only interface that a user can interact with the system anywhere in the world where an Internet connection is available. The web interface is built by using the latest web technology known as Active Server Pages (ASP) and Hyper Text Markup Language (html) version 4.0. Also, a software known as “Dreamweaver Ultra Dev 4.0” by Macromedia is used as the programming and development platform.

2) Relational Database: A relational database was built based on the class diagram so that data (performance metrics, user information, question, etc.) could be stored. We use the database to support our website so that web interface can be used to collect and display information. If the traffic to the web page and the utilization of the system are both high, we gave a good chance of having many transportation providers as the system users. Thus, the database will integrate the interests of many transportation providers, who are interested in benchmarking.

After the information requirements were identified, we began our implementation.

3.3 Step three: Implementation

The implementation of the system is mainly divided into three parts, which are the (i) Relational database design, and (ii) Web interface and (iii) Online performance analysis application based on SMART. In the following, we start our discussion by elaborating on the design of our database.

3.3.1 Part One: Relational Database Design

Andersen (1999) defines a database as an “organized collection of objects that is used to store, manipulate, and retrieve data for a specific purpose.” In the project, we used the relational database model that was first developed by Dr. E.F. Codd in 1970. The flexibility of the relational model contributes to the development of many database software packages that are available today, such as Microsoft Access, Oracle, SQL server, DB2, and Informix. In a relational database, data is stored in “tables” and the logical stored form of data is known as “record”. A table is made up of two elements, which are “records” and “fields”. “Record”, as we have mentioned earlier, is a logical stored form of data. Each information or data in a record belong to at least one “field”. For example, a metric’s record may consist of several fields, which are “metric name” and “property” and “definition”. Then, the information or data under “metric name” may be “Loading size”. On the other hand, the information or data under “property” may be “hi”. In each record, a metric can be identified by a unique identifier, known as “primary key”. In Figure 2, we set “PM ID” as our primary key in this table. Note that every table has to have one and only one unique primary key. The powerful aspect of the primary key is that it can be used to cross reference information that is stored in other tables in the same database.

Primary Key Field

Record Data/Information

Figure 2. Database Table

A relational database enables us to use “Structure Query Language (SQL)” to extract information that we needed from the table(s). By using SQL, we can construct any logical query as needed so that data can be retrieved and used for our purposes.

Benchmarking Database: Our Database

By using Microsoft Access, we created nine tables and defined the relationships among the tables. Tables are related so that we can create queries as needed to retrieve, store, and manipulate data. Figure 3 showed the architecture of the database with simplified table structure. Based on the design of the class diagram (Figure 1), the nine tables captured all the major elements as shown in the diagram. The lines that link the tables represent the relationship. By drawing a line between two tables, a relationship is defined. Note that we can interpret the relationships in the similar fashion as in the relationship interpretation within the class diagram.

As shown in the table, the name of the table is highlighted in gray. The fields of tables are listed below the table name. Note that the primary key of each table is highlight with “bold” text style.

Figure 3. Database Structure

As we have seen, Figure 3 showed the architecture of the database. However, the database is empty with no data at this point. In the following, we discuss the data collection of performance metrics so that the data can be used to populate the database.

Performance Metric Data Collection

In order to collect the data needed for the online analysis tools, we created online surveys so that performance data can be collected. The questionnaire was designed by referring to the performance metric list. The questions were mainly designed to obtain values for all the metrics in the list. The surveys were divided into five categories, which were “Carrier Categorization”, “Customer Perspective”, “Internal Business Perspective”, “Learning and Growth Perspective”, and “Financial Perspective”. Note that in addition to the four balance scorecard groups, “Carrier Categorization” was added to the survey so that an accurate type of carrier can be recognized. For example, a trucking company may be categorized as a less than truckload and regular-route carrier, who delivers hazardous material only. The following is a question taken from the Internal Business Process Survey (Appendix B).

Q12.What is your total mileage operated in intercity service annually?

Loaded Miles

Empty Miles

Additional questions were formatted for each performance metric. A complete list of survey questions can be found in Appendix B. Before we proceeded with a general survey of transportation providers, we wanted to ensure that we were asking the right questions. Therefore, we requested the assistance of transportation companies. We contacted the companies that were interested in the evaluation process and sent them an evaluation package. There were approximately 60 questions to review. We estimated that it would take one to two hours to review and comment on the survey.

The review of the questionnaire involved examining the survey from a transportation provider’s perspective and commenting on:

(i) Whether or not the question makes sense (categorized correctly with wording that is understandable to transportation professionnals)

(ii) The level of difficulty in gathering the data required for the questions.

(iii) The importance of the questions to transportation providers.

The questionnaire was reviewed by three experts in industry associated with WalMart, J. B. Hunt Transport, and ABF Freight System Inc. Please refer to Appendix D for the results of the industry evaluation on the questionnaire.

By using standardized evaluation form, feedback was collected and improvements or clarifications were made to the questionnaire. The following is an example of the evaluation questions on the evaluation form:

a) Do you understand the key terms of this question? Yes No

If No, please specify___________________________

b) Is this question is in a wrong or inappropriate category? (Operational) Yes No

If Yes, please recommend a category___________________________

c) Please rate the relevance of this question to your company’s operation:

Least 1 2 3 4 5 most relevant

Please state why: ________________________________________________________________________________ ________________________________________________________________________________

d) Will data be hard to obtain in order to answer this question? Yes No

If Yes, please indicate why?

(A) Data is too confidential	(B) Data is not collected by my company
(C) Data is not directly available to me. (someone else has the data, who? _______________________________)	(D) Data is available but will take significant resources or time to obtain

Other, please specify: ________________________________________________________________________________________________________________________________________________________

The final version of the questionnaire is hosted on the website and available to the general public. In fact, responses to the survey are very important. If sufficient responses are received from the public, we will be able to ensure the successful implementation of the application based on SMART. Specifically, participation in the survey process can yield the following benefits:

1) Accurate and representative performance analysis based on SMART: Online surveys collect performance data from the users. In turn, sufficient performance data ensures that the performance analysis will involve most participants of the surveys. Therefore, the result of the performance analysis will be more representative and accurate.

2) Identification of benchmarking partner(s): Participation in the survey contributes to the amount of performance data. A sufficient amount of available data improves the chances of identifying a suitable benchmarking candidate through the use of SMART based performance analysis.

3) Through the use of the website, the surveys are the essential tool that contributes to the dissemination of benchmarking knowledge and practices. In other words, if performance data is sufficient, many may be drawn to the site for benchmarking related information. Eventually, the website may become a tool to disseminate benchmarking practices.

In the following, we discuss Part Two, which is the implementation of the web interface.

3.3.2 Part Two: Introduction to Online Benchmarking System: The Web Interface

The website/interface was built by using both Internet scripting and programming tools, known as “Hyper Text Markup Language 4.0 (html)” and “Active Server Pages 2.0 (ASP)”. The following Figure 4 shows the homepage of the website.

Figure 4. Online Benchmarking Homepage

Further examples of the website pages are documented in Appendix G. From the homepage, a user can explore the resources provided in the page by clicking on the highlighted links. For example, a user can login, view performance metrics or participate in the surveys. The following flow diagram illustrates the structure and flow of the website.

User registration

Yes

Figure 5. Online Benchmarking System Structure

A new user has to register in order to gain access to web application resources such as the performance analysis based on SMART and surveys. Specifically, the user must provide “last name”, “first name”, “email address”, “username” and “password” in order to register. In addition, we included user’s contact address and phone number in the registration page; however, contact address and phone number are not necessary for registration.

Before one can use the system, the user has to login. After the user has logged in, he/she has full access to all the resources available in the website. For example, the user can take surveys or perform a performance analysis based on SMART. Then, the user can either stay in the website (to access more resources) or exit the website. Since the benchmarking system requires the enabling and tracking of cookies, all the sensitive information will be immediately destroyed once the user exits the website. In this case, the sensitive information is limited to username and password only. More information about the website and its code can be found in documentation of Chapter 4.

3.3.3 Performance Analysis Based on SMART

Transportation providers may be interested in comparing their performance to other providers by using the system. The online system provides an application that allows a trucking service provider to compare itself against others in terms of overall performance. The method of comparison is based on the method known as SMART method or “Simple Multi-Attribute Rating Techniques”. SMART is a rating technique that is capable of handling more than one attribute. The method requires the user to assign weights to attributes and calculations are performed to determine the overall “aggregate performance” of a company. The higher the value of aggregate performance, the better the performance of a company. The method based on SMART consists of the following two steps: 1) Identification of crucial performance indicators, and 2) Multi-criteria rating and ranking. In the following, we will discuss the two steps in detail.

Step1. Identification of crucial performance indicators

After all the necessary metrics are collected, we have enough attributes to perform analysis of performance. Note that not all metrics need to be used in the analysis. There are two reasons why we want to use a limited number of metrics:

a) Practicality: It may not be practical to use all the metrics because some companies simply do not collect the metrics.

b) Complexity: Having too many metrics increases the complexity, time and effort of the performance analysis by making it more difficult to assign and interpret weights.

During the process, the user provides performance data for the metrics and rates the metrics. Due to the diversity of trucking companies (e.g. truckload, less than truckload, regular route carrier, private carrier, etc.), some companies may consider different performance metrics as crucial in indicating their performance. Therefore, the metrics can be dynamically changed according to a company’s preference or situation. In the website, we allow the user to determine his/her own critical metrics (or attributes).

Step 2. Multi-criteria rating and ranking

There are two elements in this steps, which are (i) Swing Weight Assignment, and (ii) performance calculations. In the following, we will discuss each elements in details.

(i) Swing Weight Assignment

Swing weight:

The swing weight is a special weight to apply to a metric. In order to illustrate the concept of swing weight, the following imaginative scenario is used.

Scenario:

John is a CEO of an air conditioning company who needs to expand the company’s production capacity by building another manufacting plant in Arkansas. He has the following locations in mind, which are Fayetteville, Fort Smith, and Little Rock. Therefore, he needs to make a decision on which location is the best choice. In order to make the decision, he has a meeting with the executive managers. After the meeting is concluded, they decided to use the following metric as the selection criteria:

(1) Proximity: The average distance of the manufacturing facility to the nearest location of customer.

(2) Transportation convenience: The average distance of the facility to the closest disribution center.

(3) Size: The size limit of the facility in each location.

John decided to use SMART analysis to figure out the location problem. However, he does not know exactly how to assign swing weights to the three metrics. Therefore, an external consultant, Sam, is hired to advise on this matter. Sam starts by asking John to rank the importance of the three metrics. Specifically, if just one of these metrics could be moved to its best level, which metric would John chooses? In this case, John selects “Proximity”. After the change has been made, Sam continues to ask John to rank the other two metrics based on the respective importance. John’s rankings are:

1) Proximity (most important)

2) Size

3) Transportation convenience (least important)

Now, Sam assigns a weight of “100” to the most important metric, which is “proximity. The other weights are assesed as follows. John is asked to compare a swing from the smallest facility size to the biggest, with a swing from the most distant location from customers to the closest location (proximity). After a careful consideration, John decides that the swing in “size” is 90% as important as the swing in “proximity”. Therefore, “size” is given a weight of 90. Similarly, a swing from the worst “transportation convenience” to the best is considered to be 80% as important as a swing from the furthest to the closest customer location for “proximity”, so “transportation convenience” is assigned a weight of 80. As a result, the swing weights for “proximity”, “size” and “trasportation convenince” are 100, 90 and 80 respectively.

In order to summarize the swing weight assignment, we present a general guideline to assign swing weight in the following.

Swing weight assignment guide

After the performance metrics have been selected for performance analysis, the user must assign a value and a “swing weight” to each individual metric. In order to apply swing weights to metrics, we follow the following guidelines:

(1) Rank the “importance” of the metrics in descending or ascending order.

(2) Assign a weight of “100” to the most important metric.

(3) Assign a swing weight to the second most important metric by comparing the current metric with the previous metric (which is the most important metric). For example, the second metric may be 90% as important as the first metric. Therefore, the swing weight of the second metric is “90”.

(4) Similarly, the assignment of swing weight to the third important metric and the rest of the metrics can be determined by comparing them to the most important metric.

After the “swing weight” and value of metrics have been assigned, we perform the usual SMART calculation.

(ii) Performance calculations

The calculations can be divided into two parts, which are, (a) determine the value function of each attribute, and (b) determine the aggregate performance of each alternative. In the following, we will dicuss each part in details.

a) Determine the value function of each attribute: In this prototype, all the value functions were estimated by assuming a linear relationship. A linear function is used for simplicity and the linearity does not imply a specific preference structure. The linear relationship is found by taking the maximum value of metric and minimum value of metric from the database. For instance, we may take the maximum and minimum value of “on time delivery” from the database. Then, a linear equation is found by using the maximum and minimum value. In the following, we describe the process to find the linear relationship.

max

C

X

min

Figure 6. Value function of a metric (that preferably has its value maximized)

Figure 6 shows a typical linear value function of a metric. Note that this relationship only applies to those metrics that preferably have their values maximized. For example, if we want to maximize the value of “market share”, then, this type of value function can be used. In the following, we derive the general expression of linear value function.

Let y=mapped metrics value

x=Original metric value

max= Upper bound, or the maximum value of a metric

min= Lower bound, or the minimum value of a metric

Linear relationship:

y=mx+C (1)

Where,

(2)

From (1), we derive C,

C=y-mx (3)

Substitute X=max, Y=100 into (3), yields

C=100-m*max (4)

Substitute (2), (4) into (1), therefore,

y= (5)

Equation (5) is the general expression of metric value function. It is used to convert metrics that prefer to be maximized on the range of “0” to “100”.

On the other hand, Figure 7 shows a typical linear value function of a metric, which relationship only applies to those metrics that preferably have their values minimized. For example, if we want to minimize the value of “% distance driven empty”, then this type of value function can be used. In the following, we derive the general expression of linear value function.

Figure 7. Value function of a metric(that preferably has its value minimized)

Linear relationship:

y=mx+C (6)

Where,

(7)

C=y-mx (8)

Substitute x=max, y=0 into (8), yields,

C= - m(max) (9)

Substitute (7), (9) into (6), therefore,

y= (10)

Equation (10) is used to convert metrics that prefer to have their values minimized on the range of “0” to “100”.

After the value function of each metric has been determined, we can determine the aggregate performance of each company.

b) Determine the aggregate performance of each alternative: The aggregate performance of trucking companies (i.e. alternatives) can be computed at this step. The largest aggregate performance value indicates the best overall performer. The following is the general expression of aggregate performance.

Aggregate performance =AP = (11)

Where nw=normalized weight

n= number of metrics

In the online application, all the users who have the same set of metrics as those selected by the current user will have their aggregate performance calculated. For example, the calculations (for other users) will apply the swing weights assigned by the current user, based on the assumption that the current user is more interested in his/her own weighting system. Therefore, the approach of this method depends on the likelihood of different users having matching metrics.

In order to facilitate the “understanding” of SMART based system, the following example illustrates the mechanism of such system.

Example 3.1. In this example, we have a simple scenario: A SMART based system that only has two users (John and Jack). In this case, John tries to use the system to compare the performance of his company to his competitor’s (Jack). In other words, John is the current active user of the system, who tries to determine his company’s performance status against the competitor. Note that, incidentally, Jack’s metric profile matches John’s. Assumptions (1) both users understand the mechanism of SMART, (2) the database was populated with numerous performance data, even if there are only two system users.

Solution:

Current User: John

The user selected the following metrics and assigned both value and swing weight to each of the metrics. In John’s case, he ranked the three metrics according to their respective importance in descending order. As shown in the table below, John ranked “complaint rate” as the most important metric and “length if haul” as the least important metric. In addition, the “preference” in the following table indicates that the metric prefers to be either “maximized (max)” or “minimized (min)”.

Selected Metrics	Preference	Metric Value	Swing Weight Assigned
Complaints rate	min	20%	100%
Delivery date compliance	max	95%	90%
Length of haul	min	500 miles	75%

Table2. John’s metrics profile

Note: In the SMART based application, we created an interface in the website so that a user can assign swing weights to the selected metrics; however, a user must be familiar with the meaning of swing weights so that the weight can be assigned correctly. In addition, we provided an interface to the user so that a metric’s value could be assigned and recorded.

Other User: Jack

Jack was the user who incidentally selected the metrics that matches the metric profile of John.

Note: In the online performance analysis based on SMART, all user information is stored in the database. Therefore, we always check if there are any matching metric profiles in the database before the calculations execute. If there are no matching profiles available, we will terminate the application and inform the user about the situation.

In order to calculate the aggregate performance for Jack, we use John’s swing weight and impose it on Jack’s because John (current user) is more interested in comparing his aggregate performance with Jack’s by using his own weights. In other words, we always impose the current user’s weight on other users’ metric profile (that matches the current user’s metric profile). Table below shows John imposes his metric weights to Jack’s.

Selected Metrics	Metric Value	Original Swing weight	Imposed swing weight
Complaints rate	10%	100%	100%
Delivery date compliance	85%	80%	90%
Length of haul	300 miles	60%	75%

Table 3. Jon imposes his metric profile onto Jack’s.

Aggregate Performance

Before we can calculate the aggregate performance of both users, we need to determine the value function for all the metrics. The value function equation is obtained by deriving the linear relationship of x and y.

Metric value range= the maximum and minimum value of a specific metric that is currently available in the database.

Thus,

Metric value range: 0% to 100% (complaints rate)

Assign a weight of “100” for 0% complaints rate, and a weight of “0” for 100% complaints rate.

max=100, min=0, m=-1, C=100

Applying (10), the value function of “complaints rate” is y=-x+100

Metric value range: 0% to 100% (delivery date compliance)

Assign a weight of “100” for 100% delivery date compliance, and a weight of “0” for 0% delivery date compliance.

max=100, min=0, m=1, C=0

Applying (5), the value function of “Delivery compliance date” is y=x

Metric value range: 100 miles to 1000 miles (Length of haul)

Assign a swing weight of “100” for length under 100 miles, and a swing weight of “0” for length over 1000 miles.

max=1000, min=100, C=111.111

Applying (10), the value function of “Length of Haul” is

Then, we calculate the aggregate performance for both users:

John’s Aggregate Performance (Current user):

First, we normalize the swing weight.

Metrics	Swing Weight	Normalized Weight (nw)
Complaints rate	100%	37.7
Delivery date compliance	90%	33.9
Length of haul	75%	28.4

Table 4. John’s swing weight normalization

Second, we map each metric’s value by using the value function.

Metrics	Original Value	Mapped Value (mv)
Complaints rate	20%	80
Delivery date compliance	95%	95
Length of haul	500 miles	55

Table 5. John’s metric value mapping

Applying (11), AP_John=(37.7*80+33.9*95+28.4*55)/100 = 77.985

Jack’s Aggregate performance (other users):

First, we normalize the swing weight. Note that we impose current user’s weight to John’s.

Metrics	Swing Weight	Normalized weight (nw)
Complaints rate	100%	37.7
Delivery date compliance	90%	33.9
Length of haul	75%	28.4

Table 6. Jack’s swing weight normalization

Second, we map each metric’s value by using the value function.

Metrics	Original Value	Mapped Value (mv)
Complaints rate	10%	90
Delivery date compliance	85%	85
Length of haul	300 miles	78

Table 7. Jack’s metric value normalization

Applying (11), AP_Jack=(37.7*90+33.9*85+28.4*78)/100 = 84.89

Note that the value of aggregate performance of a company is directly related to the performance of the company. In this example, Jack’s company has a greater aggregate performance than John’s company. Therefore, we concluded that Jack’s company has been performing better than John’s company. By looking into the detail of the analysis, we notice that Jack’s company has a much higher value of “complaints rate” than John’s company. In addition, the assigned swing weight is 100% for “complaints rate”. Therefore, the aggregate performance of Jack’s company is higher than John’s company. In order to improve the performance of John’s company, John may focus on improving the value of “complaints rate”, which has the highest impact on the overall performance. The following chapter discusses the website documentation in detail.

CHAPTER 4: WEBSITE DOCUMENTATION

4.1 Purpose of documentation

The main purpose of the website documentation is to facilitate website improvement and modification. Since the website is built by the current Webmaster, the lack of proper website documentation may impose difficult website management task for future web administrator. Indeed, the main audience of this documentation is aimed at the future web administrator who is interested in maintaining website. Thus, the documentation will describe mainly the code or programming aspect of the website. The following knowledge is highly preferable for anyone who wishes to continually maintain and upgrade the website.

4.2 What you need to know

The following knowledge is helpful for maintaining the website:

1) Basic programming skill is required

2) Knowledge of Active Server Pages is required

3) Knowledge of HTML 4.0 is required

4) Knowledge of Structured Query Language (SQL) is highly desirable

5) Knowledge of Vbscript is highly desirable

6) Basic Client/Server concept is desirable

7) Knowledge of any database management system

8) Knowledge of Dreamweaver Ultra Dev 4 is preferable.

In the following, we will discuss the ASP code that we used to built the website.

4.3 ASP Code Structure:

Application: SMART

Filename: SM_S0.asp

The following code is an excerpt of the web page that is written with both html and asp. The web page is the first page that one will see if the user decided to use the SMART based application. This program can be divided into two parts:

In the first part, the program checks if “hid_check”=true, then carries out the process. Note that the value of “hid_check” is null the first time the program run. Therefore, nothing will be processed in this case and the program ignores the first part. However, when the program proceeds to run to second part, “hid_check” is set to “true” (see the code in bold). When user presses submit, the web page submits (value of hid_check” is submitted) to itself and the program re-runs. However, this time the value of “hid_check” is true so the code in first part is triggered and starts running. As we can see in first part, a user may be redirected to “newcard.asp” or “SM_S2.asp”, depend on the situation.

‘First part

If request.form(“hid_check”)=”true” then ‘Check the flag

If request.form(“chx_newcard”)=”new” then

response.redirect (“newcard.asp”)

end if

‘response.write (request.form(“hid_count”))

if request.form(“rd_SC_ID”)<>”” then

session(“scid”)=cint(request.form(“rd_SC_ID”))

response.redirect (“SM_S2.asp”)

end if

‘Second part

<tr>

<td width=”27%”><b>New Scorecard</b></td>

Create a new scorecard</td>

<p></p>

</tr>

<tr>

<td width=”27%”><b>Existed Scorecard</b></td>

If session(“scid”)=”” then

response.write “No record available”

Else

while Not rs_scorecard.EOF

<input type=”radio” name=”<%=”rd_SC_ID”%>” value=”<%=rs_scorecard(“SC_ID”)%>”>

<% response.write rs_scorecard(“cardname”) & “ “ & rs_scorecard(“Creation_date”)& “<br>”

rs_scorecard.movenext

wend

End if

<input type=”hidden” name=”hid_check” value=”true”> ‘This is a flag

</td>

</table>

<p>

</p>

</form>

Above scenario is very common in Client/Server development. It is very important that the Webmaster understands and masters this technique. Generally, the technique can be represented in the following Diagram.

Flag=true redirect

Page

Flag=false submit form

Diagram 1. Code Structure

(Diagram taken from “Foundation Dreamweaver Ultra Dev 4” by Paddock R. et al., pg 75)

The file “SM_S0.asp” will produce the web page similar to the following Figure:

Figure 8. Web page produced by SM_S0.asp

All the SMART based application web files are structured in the similar manner. The files that associated with SMART based application are listed in the following:

SM_S0.asp: The homepage of SMART. In this page, we can create, delete or edit a scorecard

SMART.asp: In this page, we can select the metrics that we wish to include in our scorecard.

SM_S1.asp: In this page, all the previously selected metrics are displayed. In addition, the user is given the option of deleting or adding additional metrics.

SM_S2.asp: In this page, the user can assign metric values and swing weights to the selected metrics.

SM_S3.asp: In this page, we can assign weights to each metric category

SM_S4.asp: In this page, the aggregate benefit is calculated and result is displayed.

The following files are used to display the error message an once error is detected in the application files:

err_noblank.asp: Error message is displayed if a user tries to submit an empty form.

err_numeric.asp: Error message is displayed if a user tries to enter a non numeric data into a specific text box.

err_login.asp: Error message is displayed if user tries to access a restricted web page.

err_nomatch.asp: Error message is displayed if there is no user in our database that has the same metrics that the current user has selected.

4.4 A walkthrough of Online SMART Application

Online SMART application can be accessed from the home page of online benchmarking system (OBS). Figure 7 show the home page of OBS.

Access SMART application by clicking this link

Figure 9. Home page of OBS

Note that one has to login in order to use the SMART application because the application requires the identification of every user so that a metrics profile or scorecard can be created and stored in the database. If the user has no user name, then he/she must register so that the user can create a user name and password. Once logged in, the system will automatically assign a user id for the current user. Note that the user id has been created and stored in the database when the user registers with the system. In the ASP code, the value of the user id for the current user is stored in the session variable “session(“userid”)”. Figure 8 shows the flows of the SMART based application.

Figure 10. Structure of SMART based performance analysis application

If the user has decided to continue the performance analysis, he/she must either create a new scorecard, continue to work on an existing scorecard, or delete an existing scorecard.

Case One: Create New Scorecard

Assume that the user wishes to create a new scorecard by selecting the “create a scorecard” option and presses the submit button. Then, the user will be brought to a scorecard creation page. In the scorecard creation page, the user is required to complete a simple form. Specifically, the user has to create a name for the current scorecard. Upon completion of the form, the user submits the form. The form submission does three things: (1) Stores the name of the scorecard and the creation date, (2) Assign a scorecard id for the user, and (3) redirect the page to “SM_S2.asp”. At this point, the importance of “scorecard id” exceeds the “user id” because we use scorecard id to identify each individual instead of user id. Since only users who decided to use SMART based application will be assigned a scorecard id, it is more convenient to use scorecard id as identification instead of user id. In the ASP code, the value of scorecard id is stored in the session variable “session(“scid”)”.

After a name had been assigned to the scorecard, the user must decide the metrics that he/she wishes to include in the scorecard by selecting at least one metric from the metric list page. Then, the user submits the form and the selected metrics will be stored in the user’s scorecard and the user will be brought to “metrics value/weight” page. The “metrics value/weight” page is a user interface that let the user assigns swing weights and metric values to the selected metrics. Note that if the user has participated in the benchmarking survey before, the metrics value text field will be automatically filled in. Then, the user can submit the form. The form submission does two things: (1) Stores both metric weights and values, and (2) redirect the page to “SM_S3.asp”.

In the next step, the user has to assign weight value to the categories, which are Internal Business Process, Learning and Growth, Financial, and Customer. Similarly, the user submits the form by clicking the submit button. The form submission does three things: (1) Stores the value of category weights, (2) store the weight, and (3) redirects the page to “SM_S4.asp”.

The “SM_S4.asp” page display results of analysis, which is in the form of aggregate performance. In this page, all the users who have the same metrics in their respective scorecards will have their aggregate performance calculated. (See “3.3.3 Performance Analysis Based on SMART” for more information).

In fact, “SM_S4.asp” is heavily programmed and a lot of calculations are performed so that aggregate performance can be found. Basically, the program on this page is divided into three parts:

The First Part:

The first part of the program checks if there is any user in the database who has the same metrics that has been selected by the current. Each metric in the metric profile of the current user is checked against other user’s metrics profile. Every time a matched profile is found, the aggregate performance of the matched profiled will be calculated in “The Second Part”.

The Second Part:

In this part, the program calculates the aggregate performance by following the steps below:

(i) Normalize the metric weight and store the normalized weight.

(ii) Mapped/normalize the value of each metric by using linear value function. The linear relationship is found by taking the maximum value of metrics value and minimum value of metrics from the database. Then, a linear equation is found by using these maximize and minimize value.

Then, the system calculates the user’s aggregate performance based on the data supplied by step (i) and (ii).

The Third Part:

In this part, the code displays the results of aggregate performance for the current user and other users (who have the matching performance metrics). “A picture is worth a thousand words.” Therefore, we want to be able to display the results graphically, so we created a bar chart.

The third part concludes the program as well as the SMART based application. Note that in order to manipulate data between and in the web pages, one has to create appropriate SQL statements and recordsets so that only specific data will be extract from the database.

Case Two: Edit an Existing Scorecard

On the other hand, if the user has decided to work on an existing scorecard, then, the user will be given the option to change part or all of the previous assigned weight and metric values. Then, aggregate performance will be calculated (based on the new values) as in Case One.

Case Three: Delete an Existing Scorecard

If the user has decided to delete an existing scorecard, then he/she must select the scorecard that is displayed in the page by selecting in the associate scorecard ID in a list menu. After the user has selected the scorecard and pressed the submit button, the user will be direct to a confirmation page that is used to confirm the delete action. After the confirmation, the scorecard will either be deleted or the action will be cancelled, depends on the user’s choice. After the delete action, the user will be redirect to the SMART homepage.

In the following, we direct our attention to the technology that we used to create our website, which is known as the Active Server Page (ASP).

4.5 ASP Databases

In this section, we wish to discuss the common ActiveX Data objects (ADO) that we used to create the website. In addition, we will discuss some Active Server Pages basics and its objects as well. We begin the discussion by introducing the ASP.

4.5.1 ASP

ASP allows the web programmer to connect database to the Internet so that information from the current state of database can be accessed in real time. Whenever the information in the database is changed, the associated web site will instantly reflect the changes without the need to change the HTML code.

There are six Active Server Objects, as shown in the following:

(1) Request

(2) Response

(3) Server

(4) Application

(5) Session

(6) Object Context

The following object descriptions are excerpt from “Beginning ASP Databases” by Kauffman et al (1999).

Request object: This is one of the most frequently used object in the website. The object used to deal with a request that a user might make of a site or application.

Response object: A frequently used object. This object is used to deal with the server’s response back to the browser.

Server object: The object provides several commonly used functions, such as setting timeout properties for scripts and converting text into HTML or URLs, by far the most important is its ability to create new objects or components. Again, another frequently used object.

Application and session object: These objects are used to manage information about the application that is currently running and the unique instances (versions) of the application, which individual users run, known as sessions. Note that session object is frequently used in the website.

Object Context: The object is used with Microsoft Transaction Server. This object was not used in the website.

4.5.2 ADO

ADO is the latest database interface technology developed by Microsoft that allows universal data access. In other words, ADO allows us access to information held in a database of one type from a database package to another type. As a result, we can access the data that held in “Oracle” from “Access” and vice versa. The lowest layer of this interface is known as ODBC, or Open Database Connectivity. However, the complicated structure of ODBC and limited data type access prevent many programmers from effectively using the full potential of the powerful interface. Recognizing the problem, Microsoft creates another database interface that sit on top of ODBC, which is known as OLE-DB. According to Microsoft, the new interface is more efficient and user-friendlier. In fact, OLE-DB allows program to access information in any type of data store, including spreadsheets, graphs and email. In addition, a database driver that known as OLE-DB provider allows us to use the ODBC driver for specific database.

Even though OLE-DB is user-friendlier than ODBC, it is still hard to learn and only a limited programming language can be used to manipulate it. Therefore, a higher level of database interfaces known as ADO is available. In fact, ADO is language independent that we can use many programming languages to manipulate it. In our website, we used Vbscript as our language of choice. Vbscript is used because of its simplicity, popularity, and myriad support sources on the Internet. Figure 10 showed the relationships of the old and new database interface.

Figure 11. Relationship between different levels of database interface

(Diagram adopt from “Beginning ASP Databases” by Kauffman et al. pp.17)

In the following, we discuss in general the ADO objects that I often used in the website.

4.5.3 The ADO object Model

There are three main objects on which the ADO object model is built:

· The Connection Object

· The Command Object

· The Recordset Object

(1) The Connection Object – We use this object to make a connection to the database. Once a connection is established, we can use it as many time as we wish.

(2) The Command Object – We use command object to run command against the database. However, the “command” is in SQL instead of ASP script. The command object is usually used to return information from a database, add new record, new tables or delete information within the database.

(3) The Recordset Object – This object is the most frequently used object in our code. The recordset object can be used to amend and view the information within the database. There are many methods and properties that are associated with the recordset. Specifically, we can use recordset to find record, delete record, move through the record, restrict view of record, update record, and sort record.

As one may find in the code, we exclusively use the connection object and the recordset object to extract the records that we want to manipulate.

4.6 System Evaluation

After the web site has been built, we conducted a system evaluation survey. In this survey, a total of five people have participated in the evaluation process. The five person were selected for their knowledge in such system. In other words, the participants must familiar with at least one of the following items:

(1) Web design,

(2) Know at least one programming language,

(3) Some knowledge about SMART application,

We assigned five tasks to the participant and each task must be performed by using the system. In the following, we described the tasks performed:

Task One: Browse the website freely for 5 minutes.

Task Two: Perform a user registration.

Task Three: Login to the system

Task Four: Take all five questionnaires

Task Five: Use SMART based online application

The evaluation process took less than one hour to complete. We asked the participant to record any problems encountered during the five tasks. In addition, the participants were required to record the web page that produced the error and the type of error. After the errors had been reported, we discussed the problems with the participant. The discussion was to verify the problem. In the following, we listed some of the verified problems found after the evaluation:

Type of Error: Broken links

Description: “Home” broken link in the banner.

Location: survey.asp

Type of Error: registration problems

Description: Error message appear when trying to register

Location: registration.asp

Type of Error: Wrong links

Description: “Home” links to another site called “PMRG” in the banner.

Location: faq.asp

After the problems were verified, they were corrected in the final version of the application.

CHAPTER 5. CONCLUSION

The Online Benchmarking System is intended to facilitate the benchmarking process. In this project, the following tasks were completed:

A data driven website that is used to facilitate the benchmarking process is created. The website contain tool and resources that may benefit transportation providers.
A comprehensive list of performance metrics are available online. All metrics are defined with formula and discussion given if available. In addition, the metrics are grouped according to the balance scorecard.
The online performance analysis application based on SMART provides transportation service providers the ability to benchmark against others and get a rough idea on the companies’ performance. The application is written by using ASP and supported by a relational database model. The performance of a company is indicated by an index known as the “aggregate performance”.
Five questionnaires were available online so that performance data can be collected. Since SMART based application uses the performance data that is supplied by either the user or online questionnaires, the questionnaires are extremely important to ensure the success of the SMART based application. In addition, all performance data collected online will be stored in the database.

However, the completion of the project does not necessary signify the end of the project. In fact, there is always room for improvements. In the following, we suggest some additional research.

Future Research:

(1) Metric relationship improvement: The relationship that we used to map performance metrics from zero to one hundred is always linear. Even though the mapping is sufficient in many cases, some metrics may have a non-linear relationship in terms of utility to the user. Future research can be done to (i) identify metrics with non-linear utility mappings, and (ii) to develop the appropriate value functions for metric mapping. The result will be a much more accurate mapped value for each metric.

(2) Website improvement: (i) Current website has only one online performance analysis tool (SMART). Future research can be done to integrate additional ranking and rating tools, for instance, ranking and rating based on Analytical Hierarchy Process into the website, (ii) Provide more useful and representative displays for comparing performance such as a graphical dashboard, (iii) More web functionalities can be added, for example, “chat room” is a very good web application that enables the user to chat in real time.

Appendix A: Performance Metrics List

I. Internal Business Measures:

Name: Loading Time per stop

Definition: Time needed to load goods onto a truck

Formula: Loading Time/Number of Trips

Units: Hour/stop

Purpose: To find out the time needed to load goods into a truck every time a truck stops for loading and unloading purposes.

Discussion: This metric may help to identify potential bottlenecks in delivery or shipping operations.

Name: Unloading Time per stop

Definition: Time needed to unload goods from a truck

Formula: Unloading Time/Number of Trips

Units: Hour

Purpose: To find out the time needed to unload goods into a truck every time a truck stops for loading and unloading purposes.

Discussion: This metric may help to identify potential bottlenecks in delivery or shipping operations.

Name: Loading Capacity

Definition: Maximum load per truck in ton

Formula: N/A

Units: Tons

Purpose: Used to identify the loading capacity of trailers.

Discussion: This metrics is needed to derive other metrics, such as “Average load of truck”.

Name: Average Load in Truck

Definition: Average load per truck in tons per trip

Formula: Load/Number of trucks

Units: Tons/Truck

Purpose: Used to track the average load of goods in a truck

Discussion: From a transportation service provider’s perspective, it is desirable to load as many goods as possible in a truck; however, government weight limit regulation must not be exceeded.

Name: Length of haul

Definition: Average distance traveled in miles per trip

Formula: Length of haul/number of trips

Units: Miles/trip

Purpose: To track the average length of haul of trucking operation.

Discussion: From a transportation service provider’s perspective, it is desirable to operate in a distance that is not too far away from the headquarter/maintenance office. Shorter distance may save on truck maintenance cost and reduce empty miles.

Name: Average Speed

Definition: Average speed of truck during operation

Formula: N/A

Units: Mph

Purpose: This metrics is used to track the average speed of truck.

Discussion: From a transportation service provider’s perspective, it is desirable to operate truck in high driving speed to meet customer goal. However, the trade off is government regulation and highway safety.

Name: Operating hour / Truck

Definition: Average hours in truck operation

Formula: Total operating hour / # of truck/week

Units: hour/truck/week

Purpose: This metric is used to track the average operating hour of truck in a week.

Discussion: Equipment utilization is one of the keys to better productivity. Equipments or trucks can be operated 24 hours a day if possible to boast productivity. The trade off is higher equipment maintenance cost.

Name: % distance driven empty

Definition: Percentage of miles that the truck is driven completely empty

Formula: (distance empty load/total distance) *100%

Units: %

Purpose: This metric is used to track the empty mileage of a truck.

Discussion: Empty mile has an adverse effect on the productivity of a company. This metric can be used to modify the schedule or network structure of the trucking operation so that empty mile can be reduced to the minimum.

Name: Time driven between 2 stops

Definition: Driving time between 2 stops

Formula: N/A

Units: Hour

Purpose: This metric is used to measure the average driving time travel between two stops.

Discussion: This metrics can be used to measure the transportation network density.

Name: Stop time

Definition: Average time that a truck spends in a stop

Formula: N/A

Units: Hour

Purpose: This metric can be used to track the idle time in the stop.

Discussion: Idle time should be reduced to the minimum. Even though a truck has to stop somewhere to refuel or take a break, the idle time should be keep to the minimum.

Name: Load factor

Definition: A ratio of average load over load capacity

Formula: Average load / Load capacity

Units: N/A

Purpose: This metric is used to measure the utilization of truck’s load capacity.

Discussion: The ideal situation is 100% load factor. This is another key metric for equipment utilization.

Name: # of stop per route

Definition: Average number of stops made in a route

Formula: Total number of stops / total number of routes

Units: Stops / Trip

Purpose: This metric is used to measure the number of stop that a driver has to make in order to complete the route.

Discussion: This metric can be used to measure the transportation network density. The more stops in a route, the higher the density. According to a study, company usually performs better with higher network density.

Name: % driven not empty

Definition: Percentage of miles that the truck is driven full or partially full

Formula: (Distance driven full or partially full load / Total driving distance)*100%

Units: %

Purpose: This metric is used to track the non-empty mileage of a truck.

Discussion: The non-empty mileage should be keep as high as possible. A high value of “% driven not empty” indicates well-planned route and efficient trucking schedule.

Name: Number of trucks active in company

Definition: Average number of trucks that is driven everyday

Formula: N/A

Units: Trucks

Purpose: This metric is used to track the number of trucks that is driven everyday.

Discussion: This is another metric that is used to measure the equipment utilization. Equipments can be operated 24 hours a day if possible to boast productivity. The trade off is higher equipment maintenance cost.

Name: Tracking Accuracy

Definition: Accuracy of shipment tracking

Formula: (The # of trucks with a known location / Total # of trucks)*100%

Units: %

Purpose: This metric is used to track the accuracy of shipment tracking.

Discussion: Effective shipment tracking keep mishandling to the minimum.

Name: Inventory record accuracy

Definition: Accuracy of availability of inventory

Formula: (# of correct records / Total # of records) *100%

Units: %

Purpose: This metric is used to measure the availability of inventory in a warehouse.

Discussion: Strictly for warehouse uses only. Some transportation company has warehouses. Accurate inventory record ensures efficient operation of the warehouse.

Name: Forecast accuracy

Definition: Accuracy of forecast on customer demand

Formula: N/A

Units: %

Purpose: This metric is used to track the forecast accuracy of customer demand

Discussion: The importance of customer demand cannot be understated. An accurate forecast ensures the customer demand will be met.

Name: Cycle time

Definition: Time needed to complete a route

Formula: N/A

Units: Hours

Purpose: This metric is used to measure the time needed to complete a route.

Discussion: A short cycle time is desirable in most cases. A short cycle time ensures goods to be delivered in time and increase the availability of fleet.

Name: Pick rate

Definition: The rate of a person to pick up lines of items

Formula: (# of lines picked per person per hour / lines for all orders) *100%

Units: %

Purpose: This metric can be used to measure the efficiency of picking up lines of items

Discussion: This metric has an impact on the “loading time” and “unloading time”. Therefore, it is very important to increase the pick rate to ensure efficient operation.

Name: Fill rate

Definition: Percentage of orders filled

Formula: (# of filled orders / Total # of orders) *100%

Units: %

Purpose: This metric is used to track the average rate to complete an order.

Discussion: A high fill rate is highly desirable. A high fill rate may indicate efficient trucking operation and high customer satisfaction.

Name: Maintenance efficiency

Definition: Efficiency of warehouse maintenance

Formula: maintenance cost / activity

Units: $/activity

Purpose: This metric is used to measure the maintenance efficiency of a warehouse in terms of cost per maintenance activity.

Discussion: It is highly desirable to keep the cost down. In this case, maintenance cost is one the costs that worth looking into.

Name: Material handling rate

Definition: It is defined as material handling expense over material handling asset value

Formula: (material handling expense / material handling asset value)*100%

Units: %

Purpose: This metric is used to measure the rate of material handling.

Discussion:

II. Customer

Name: Data entry accuracy

Definition: Accuracy of customer order entry process

Formula: (# of error