Background

Until recently, no consistent methodology existed for evaluating the effectiveness of transportation marketing in California. Promotions large and small, regional and corridor-specific were evaluated on a project by project basis, often without consideration of measurable objectives.

Also, a variety of narrowly focused, ad hoc, highly technical -- and mostly academic -- models measured pieces of commute behavior. These research efforts were inconsistent in methodology and focus, analyzing by distinctly separate means very different marketing activities and audiences, which resulted in findings that were not comparable. This piecemeal process made it virtually impossible to compare results even from similar promotions in a single region.

Little attention was given to measurement and evaluation of the interaction between TDM marketing, behavior intent and actual commute behavior. So focused were these attempted evaluations that the piecemeal data they provided was too narrow to apply in a broader marketing context, and rarely answered the questions central to marketing. Furthermore, many research methodologies were employed in a vacuum absent of "real world" information that is relevant to marketing TDM.

The Commuter Monitor

Pacific/West Communications Group, Inc., developed and implemented for Caltrans the Commuter Monitor to evaluate the effectiveness of corridor-specific and regional marketing activities. The goal is to refine and improve marketing activities in order to increase target consumers' trial and usage of products and services, and ultimately increase the market share of alternative modes among target audiences.

Utilizing the proven techniques of consistent marketing research methodology, the Commuter Monitor focuses on evaluating promotions in terms of the promotion's objectives. Though consistent, the evaluation methodology is flexible, adapting to specific marketing activities in the diverse markets that comprise California.

This approach to evaluations of marketing is rooted in the successful practices of consumer goods firms, where marketing is evaluated from both the consumer perspective as well as the sales perspective. Importantly, the Commuter Monitor includes "real world" data, including alternative mode usage and capacity.

Products, markets and dynamics change -- not always as the intended result of some marketing activities. Any intelligent evaluation of marketing must consider external factors -- "exogenous" factors -- and changes in addition to the intended consequences of the marketing. Systematically seeking and recognizing those changes is crucial to the marketing evaluation process.

The principal objective of the evaluation is to provide specific, meaningful feedback to transportation marketing managers and planners on marketing promotions by assessing the strengths and weaknesses of specific promotions in terms of:

1. Awareness and Perceptions of Modes

The issue of mode awareness is vital to marketing, and the lack of awareness of products and services leads managers to certain conclusions about marketing. Perceived mode attributes (e.g., reliability, flexibility, convenience, and safety to name a few) offer still other marketing opportunities and obstacles. Awareness and perceptions of products is the landscape of marketing.

• The Commuter Monitor's analysis of mode awareness and perception addresses directly the issue of product and service enhancement, further targeting and refining the placement of marketing resources.

1. Awareness and Perceptions of Marketing Promotions

The Commuter Monitor produces actionable information that translates directly into improved marketing decisions, which can be applied and tested in other regions under other conditions.

• The awareness of promotions addresses the need to evaluate the reach and targeting of promotions. Did the promotion find its intended target audience objectives as stated in the marketing plan?
  • The awareness and perceptions of promotions are analyzed in the context of their position in the process of decision-making and behavioral change. That is, how much do those issues contribute to
    • Likelihood to sample transportation information
    • Likelihood to sample alternative mode products (or "purchase intent")
    • Product selection
    • Actual trial and repeat experience of alternative modes
    • Permanent conversion.

• The awareness of specific promotions is compared to awareness of transportation promotions in general. The awareness of marketing activities in general serves as a constant -- a marketing environment or backdrop of sorts -- against which the effectiveness of specific promotions can be measured. For example, if the Monitor results yield higher awareness of non-specific marketing activities and news stories than highly specific promotions among the target audiences, then marketing managers will want to examine in greater detail the effectiveness of those targeted promotions.
• Because marketing transportation today often focuses on information-intensive promotions, analysis often centers on the actual key factors of information, such as recollection of specific details as well as the usefulness of the information and whether commuters actually use it.
• The Commuter Monitor, following the marketing focus, examines the effectiveness of marketing information (and its sources, e.g., ETCs, 800-COMMUTE, etc.) about alternatives.
• The perception of specific promotions is central to the evaluation of marketing activities. Perceptual issues -- and specific recall of promotion attributes -- are essential to understanding the relationship between those elements and their effect, if any, on purchase intent and ultimately product trial.

1. Stated Behavior of Target Audiences

These data are the centerpiece of the Commuter Monitor. Reliable data on commute behavior at the local level builds a foundation for understanding long-term transportation trends in the context of marketing effectiveness.

• Which marketing activities produced positive results in commuter behavior -- either in absolute mode-shift or frequency? (Working in coordination with Caltrans and other Caltrans contractors, Pacific/West is refining specific question methodologies to provide Caltrans with a more consistent data collection scheme for commute behavior.)

1. Stated Effect of the Marketing Promotions on Commuter Behavior

The stated effect of the promotion also addresses the marketer's need to understand the nature of "purchase intent," or the preliminary steps to actually sampling products and services.

• The Commuter Monitor analyzes the strength of the relationship between the stated effects of promotions and the likelihood to sample alternative modes and changes stated behavior. That analysis produces a clearer picture of the real effect of that promotion on stated behavior -- far beyond the reliability of commuters' responses to a single question.
• The stated effect analysis includes
  • Examination of the relationships among
    • Awareness and perceptions of specific promotions
    • Information sources
    • Likelihood to inquire about information
    • Likelihood to sample alternative mode products
    • Actual trial of modes after exposure to promotion
    • Likelihood to repeat use of alternative modes
    • Permanent conversion, and
    • Behavioral attrition.
• The measurement of each piece critical in establishing links between the
  • preliminary steps of awareness of modes and promotions ... to the
  • intermediate steps of building interest in those products (and information about them) ... to the
  • actual trial of those alternative mode products ... to the
  • repeat sampling ... and finally
  • the permanent conversion of commute behavior to alternative modes.

1. Real World Transportation Data

The Commuter Monitor frames the overall findings of the evaluation in the context of real-world transportation data, which is collected simultaneous to the marketing research.

• Because the evaluation incorporates real-world transportation data as well as proprietary survey research, the evaluation establishes standard parameters for the timely collection of the transportation data, especially as it relates to the timing of marketing activities. This practice reinforces the notion that marketing data and transportation data do not exist in separate vacuums.
• Rather, the two sets of information tell different parts of the same story, all of which will enable managers and planners to make surer marketing decisions.

Commuter Monitor -- Proven Marketing Research Methodology

The Commuter Monitor is based on consumer-oriented marketing research. Though marketing research methodology is extremely important to the development of a consistent evaluation process, it is from the perspective of an unbiased and neutral measurement tool that generates meaningful evaluations -- that is, constructive evaluations which contribute to better marketing planning and execution.

The Commuter Monitor applies proven marketing research techniques to the evaluation process. The Monitor creates a modular, consistent marketing research methodology that is implemented at the corridor and regional levels. Flexible enough to evaluate highly targeted marketing promotions as well as broader communications and promotions, the Monitor puts into place a consistent methodology that answers the need for standard evaluation of diverse marketing activities in a multi-market state.

The Commuter Monitor achieves this consistency through timing, sampling and questionnaire methodologies that are applied from corridor to corridor, region to region.

1. Timeliness

The Commuter Monitor establishes preliminary market evaluations prior to the marketing promotions. The preliminary research provides "clean baseline" data against which future waves of survey data are measured after the promotions. At that time, residual effects of various promotions may be analyzed. This pre/post methodology operates continuously on a three-times annually basis to accommodate and coordinate scheduled promotional activities.

• For example, a single wave of the Commuter Monitor serves as a preliminary (or baseline) evaluation for one promotion in a corridor or region, while providing post-evaluation for other concluding local promotions.
• Where commonalties exist (e.g., commute behavior and mode awareness and perceptions), data are collected for both promotions. Questions specific to the separate promotions, however, remain unique to the promotion (e.g., awareness, perception, and stated effect of promotion).
• Conducting the Commuter Monitor three times each year ensures that coordination and evaluation of the promotions are taking place in a timely fashion. This is an important feature of the Monitor. As a tool for evaluating marketing effectiveness, the Commuter Monitor must be executed around the promotion, yet be able to accommodate numerous promotional schedules. Also, regular evaluations allow testing of long-term residual effects of promotions and changes in behavior. This continuous process also allows RPMs to schedule promotions and evaluations with minimal disruption.
• Because consumer recall of behavior, promotions and the effects of those promotions deteriorates after a period of time, the preliminary and post-evaluations take place immediately prior and following (within days) the promotion to minimize erroneous recall.

1. Sampling Methodology

The sampling methodology follows marketing research industry standards for randomness and targeting. The samples in each region are large enough to accurately represent the region or corridor's attitudes, perceptions and behaviors, as well as support reliable analysis of sub-populations.

Commuter Monitor sampling methodology is built on three pillars:

1. A random selection of 300 commuting residents in a region or corridor.
2. A random selection of 300 respondents in a previous wave of that local Monitor (a "panel").
3. And, when possible, a random selection of 300 commuters who responded to the promotional activity.

For example, a given Commuter Monitor would sample at least 300 randomly selected resident commuters of the local population. This overview of the market is important for viewing the overall market environment of promotions. It provides a necessary perspective for evaluating the effect of specific promotions on the market as a whole.

Secondly, because it is important for the Commuter Monitor to examine changes and trends in attitudes, perceptions and behavior over time, every local Monitor draws upon previous Monitors for "panel" respondents. Through these respondents, marketers gain significantly greater insight into ever-changing perceptions and behaviors. As data is collected from one Monitor to the next, respondents may be paneled for future Monitors.

The Commuter Monitor also samples the targeted audience of the promotion. Whether the promotion generates responses to an 1-800 number, a mail reply or is merely targeted toward a very specific corridor, the Monitor sets aside 300 interviews to examine that targeted audience. This provides the detailed evaluation at the promotional level as opposed to the overall market.

Together, these three pillars of the sampling methodology support the objectives of evaluating the effect of promotions on the overall market, the effect on the specific target market, and the residual or continuing impact on the market.

When necessary, the Commuter Monitor also randomly samples the promotion's target audience in addition to the overall sample, an example of the flexibility of the Monitor's standard methodology for evaluating specific, highly targeted promotions. Any evaluation must accommodate such specific kinds of marketing activities, regardless of the target audience.

Specific target sampling supplements -- rather than replaces -- regular sampling methodologies. The additional target sampling and questioning would focus strictly on the specific promotion. This practice satisfies the need to evaluate highly targeted promotions in the context of the overall market, as well as avoiding unnecessary research complications.

1. Questionnaire Methodology

Marketing research questionnaire methodology and logic flows from the objectives. As an evaluation of transportation marketing effectiveness, the Commuter Monitor necessarily focuses questions on key issues discussed above:

• Awareness and perceptions of modes;
• Awareness and perceptions of the marketing promotions;
• Stated effect of the marketing on commute behavior; and
• Stated commute behavior itself.

In addition to those subject modules, the Commuter Monitor also incorporates a battery of valuable criteria questions from the original segmentation study. Those segmentation questions define the primary and secondary target classifications. As with all long-term research programs, the research program continually refines question modules and incorporates new thinking as pertinent research becomes available from other sources.

Consistency is a key objective to question structure and order. Every effort is made to standardize modules and questions -- especially segmentation schemes, demographics and commute behavior -- always with an eye toward improving the Monitor and the reliability of the data it generates.

The Commuter Monitor results do not exist in a marketing research vacuum. Rather, marketing research findings will be viewed in conjunction with two other sources of information:

1. Real-world transportation data collected simultaneously to the marketing research (discussed above); and
2. Non-marketing, external-- or "exogenous" -- factors that influence commuter decision-making and behavior. Presently under development, the roster of external factors might include -- but are not limited to -- changing economic conditions (e.g., business development patterns, economic growth, gas prices, etc.), transportation construction, changes in transportation services, changing regulations, other marketing activities, etc. As a part of the Commuter Monitor, Pacific/West analyzes the external factors for effects on commuter decision-making and behavior, in accordance with existing research from the transportation bibliography and other sources

The Commuter Monitor -- A Summary

The Commuter Monitor is much more than data collection for the sake of collecting marketing information. The real value of the Commuter Monitor is the analysis of local trends, disparities and variances in commuters' awareness, attitudes, perceptions and behaviors -- both stated and recorded at the bottom-line, real-world transportation data. The analysis of those factors will yield insights into 1) effectiveness of TDM marketing, 2) the quality of transportation products and services, 3) the dynamics of a competitive marketing environment, as well as 4) consumer motivations and marketing opportunities.

The Commuter Monitor's analysis focuses on direct findings about regional and corridor-specific promotions in the context of the market as a whole. The analysis speaks to the specific objectives and execution of the regional marketing plan. Because marketing does not exist in a vacuum, the Monitor broadens analyses to incorporate the possible impacts of aforementioned exogenous factors.