Assessment Results and Support Data

What follows is an analysis of just some of the most recent results from the latest assessment cycles in the department.  This analysis concentrates primarily on some of the quantitative assessment results.  For more complete results and a thorough description of the strategic planning process, the quality improvement plan, and all the quantitative and qualitative data collection, see the full ACCE Self Study posted under the “Program Mission and Goals” tab of this website – especially Section 9 and Appendices K, L, and M.

Graduating Senior Survey

Every year, each graduating senior in the CAED is solicited to participate in an academic assessment survey. The survey instrument poses eighteen questions on academic goals, four questions on support facilities quality, two questions on advising quality, and a self-reporting section on extracurricular activities.  (See Appendix L for a copy of this survey.)

Responses to both the support facilities quality and advising center quality questions are quantified as follows:

  1. Poor
  2. Fair
  3. Good
  4. Excellent

A Support Facilities Quality Chart is shown below. The chart reveals the highest student satisfaction with the Support Shop. However, it should be noted that these College facilities are not widely used by CM students which accounts for their relatively low impression of the venues.  The student perceptions correlate rather closely with their level of utilization. 

Support Facilities Quality Chart

Support Facilities Quality Chart
Click Here to open larger format

The Support Shop gets the highest use and the Photo Lab gets the lowest use by the CM students during their time at Cal Poly and their perceptions of quality reflect this.  As a department, these results did not concern the CM faculty and no actions were taken.  However, as a College, addressing the need to modernize all of these support facilities is stated as a high priority for the upcoming Cal Poly Capital Campaign.

It should be noted that this survey does not currently assess the graduating CM student impression of the main facilities used by the department.  As described in Section VI of this Self-Study, students in this department are enjoying new CM facilities completed in 2008 and 2010.  Up until now, polling student opinion on these new facilities was considered unnecessary.  However, it probably will prove useful in the near future to start including measures of student satisfaction with these new facilities in the Graduating Senior Survey.

An Advising Center Quality Chart is shown below. Student satisfaction with departmental advising is generally viewed more positively than that provided by the College Advising Center.  It also was noted in these results that student impression of the College advising services has declined somewhat over the past few years and their view of departmental advising was improving.  However, the results from the graduates in 2013 were quite alarming and reversed this trend in the department and declined even more sharply for the College.  Concentrating on the department, these results were the poorest during the review period – the impression of departmental advising went from an overall “Good” reaction to somewhere midway between “Good” and “Fair”.

While the quality of advising services provided by the department is an ongoing conversation among the faculty, waiting to see if this sudden change in the trend line represents just one outlier or a serious problem seemed warranted.  It was noted that the class of 2013 had a lower impression of every facility and service that was surveyed.  This may be due to a small, dissatisfied sample of graduates who took the survey in this year.  Clearly, this is an area to check again next year to determine if the trend line has reversed or whether an action plan is needed to address a problem related to advising. 

Two years ago, the faculty implemented the Professional Advancement for Construction Students (PACS) program to expand advising to areas of professional development in addition to academic concerns (see Section V.G.1. for a description of this program).  The impact of the PACS program may start to have an effect on the results of this survey question with the 2014 graduates.

Advising Center Quality Chart

Advising Center Quality Chart
Click Here to open larger format

With respect to the 18 questions related to academic goals, survey participants were asked to identify both their perceived level of “Attainment” of each goal and their perceived level of “Importance” for each. Responses within each category were quantified as follows:

Attainment

  1. Not attained
  2. Minimally attained
  3. Somewhat attained
  4. Attained
  5. Strongly attained

Importance

  1. Low value
  2. Little value
  3. Neutral
  4. Some value
  5. High value

The survey questions are provided in Appendix L as the CAED 2010 Graduating Student Survey.  For the past decade, both College and departmental reviewers felt that it was helpful to get a measure both of perceived “Attainment” and of the perceived “Importance” of each of these educational goals.  This enables the College to collect the same data for all of the graduates from all five of the departments in the College while allowing individual departments to interpret the results as appropriate for their students. 

For example, it would be expected that the educational goal related to Urban and Natural Systems is very important for City and Regional Planning (CRP) graduates, but less important for CM students.  In fact, this goal received the lowest rating for “Importance” among CM graduating Seniors.  Not surprisingly, their “Attainment” score was also the lowest for any academic goal, but this was not alarming to CM faculty because the question did not align with any particular educational goal for the department.  Similar results for CRP graduates, however, would be very alarming.  Collecting the data in this way enables comparisons among the departments, but also provides meaningful results within each department.  This use of both the “Attainment” and “Importance” scores also is demonstrated in the Gap Analysis described below.

The table presented below compares the scores obtained on this survey instrument for CM graduates over the past six years.  The six-year average and the Gap Score (described below) also are included on this table.  Of note, CM students consistently rate communication skills, critical thinking and problem solving skills, collaboration and leadership skills, computer skills, and lifelong learning as the most “important” parts of their education.  Of these, computer skills and written communication skills showed the biggest disparity between “Importance” and “Attainment”.  These data help to focus the department’s attention on key areas of the curriculum that still need ongoing attention.

Academic Goals Survey Results

Academic Goals Survey Results
Click Here for larger format

One of the best advantages for the CM department to collecting and analyzing these annual data from graduates is that these eighteen questions can be mapped to the six program goals stated for the department in the Q|A Framework and in Section I.C.2. of this Self-Study.  The questions map onto the CM program goals as follows:

 

 

Goal

Q#

Mean Gap Score

1.

Demonstrate a readiness and ability to perform in the construction industry.

2, 10, 12

0.070

2.

Demonstrate an ability to apply problem-solving skills and integrate technical knowledge.

5, 6, 7, 13

0.056

 

3.

Demonstrate an ability to participate successfully within an interdisciplinary team environment.

9, 11, 14, 15

0.043

4.

Demonstrate an understanding of professional behavior, standards, and leadership attributes.

8, 17

0.037

5.

Demonstrate an ability to communicate effectively, both orally and written, and professionally present ideas.

1, 3, 4

0.051

6.

Demonstrate a propensity for life long learning and service to the industry and community at large.  

16, 18

0.039

Looking at the data mapped against the department’s program goals shows interesting results for future initiatives.  Driven by the high disparity noted in the area of computer skills, the largest average Gap Score was associated with the first program goal related to industry readiness.  This is another triangulation point that has started to focus the faculty attention on the consistency of teaching these fundamental readiness skills across the curriculum.  These are being addressed currently.  On the other hand, these data indicate ongoing success at achieving program goals related to professional behavior and leadership.

An analysis of the gap between the perceived levels of attainment and the reported importance of each educational goal yields actionable insights. Resources are required to change curricula in response to any perceived shortcomings. These resources – faculty, software, facilities, etc. – are limited.  Allocations made must be made to maximum effect. A gap analysis has utility for determining how to best allocate resources or to otherwise validate allocations.

Gap analysis hypothesizes that optimization occurs when the level of importance for any goal equals the level of attainment for that goal. This is a balanced approach to the application of resources. It prioritizes the application of resources toward those goals that are furthest away from equilibrium while it limits attainment to only what is necessary to achieve equilibrium.

The Gap Analysis formula is as follows:

Gap Analysis Formula

Wherever the perception of Importance is equal to perception of Attainment, the gap formula equates to zero. Disparity between perception of Importance and perception of Attainment equates to a positive number whenever Importance is greater than Attainment. With rare exception during the six years of data summarized here, every educational goal that was measured equated to a positive number. In other words, Importance exceeded Attainment for every educational goal in the six-year averages. Large positive gaps infer that resources should be committed to improve Attainment.

The Goals Attainment/Importance Gap Analysis chart below summarizes the results of applying the gap formula to the six-year average of the means for the eighteen educational goals of the Graduating Senior Survey. Attainment and Importance measures are plotted as 100% bars with Attainment scores on the left and Importance scores on the right.  The line in the middle of this chart represents a Gap Score of zero.  Any bar with a change of color to the left of this line indicates a positive Gap Score and anything to the right (nothing in this example) indicates a negative score.

This stacked horizontal bar chart sorted by the magnitude of the Gap Score provides a useful visual representation.  Goals have been ordered on the chart to give a sense of priority. The goal at the top of the chart is the highest priority for the application of resources. The next lower goal is the next highest priority and so forth.  The top three priorities revealed in the gap analysis are Computer Skills, Sustainability, and Written Communication Skills.

As noted above, Computer Skills aligns with Department Goal #1: Demonstrate a readiness and ability to perform in the construction industry. Computer Skills have been addressed in various ways. Building Information Modeling (BIM) is an element of CM 115, Fundamentals of Construction Management, and has become a stand-alone, elective topics course. Discussions are ongoing on whether or not to make this elective course a required course in the future. Conceptual estimating software (DProfiler) has become an integral part of CM 415, Integrated Project Delivery.  Quantity Takeoff software (PlanSwift) is a software component of CM 214, Residential Construction Management, and On Screen Takeoff has been introduced in CM 313, Commercial Construction Management.  P6 Scheduling software is used in several courses alongside Microsoft Project.  Contracts software provided by the American Institute of Architects is an element of CM 334, Construction Law.

Goals Attainment/Importance Gap Analysis

Goals Attainment/Importance Gap Analysis
Click Here for larger format

  • Sustainability aligns with program goal # 2: Demonstrate an ability to apply problem-solving skills and integrate technical knowledge. Sustainability has been addressed in a major way with the introduction of CM 317, Sustainability and the Built Environment, as a new and required course in the curriculum in 2013.  This course has been approved to meet General Education requirements making it a popular course across majors.
  • Written Communication Skills aligns with program goal # 5: Demonstrate an ability to communicate effectively, both orally and written, and professionally present ideas. Written Communication Skills (as well as Critical Thinking Skills, and Problem Solving Skills, both aligned with program goal #2) are being subjected to further assessment with the introduction of the College Learning Assessment (CLA), discussed in Section IX.C.2.a above.

CM Program Goals are linked to the Learning Objectives of every course and are included in the syllabus for each course. (See individual course binders available to visiting team members and the syllabi in Appendix B.)

Finally, data on extracurricular educational experiences was obtained from the Graduating Senior Survey and the most recent results are summarized below:

 

2011

2013

Average

Co-Op

9.1%

8.6%

8.9%

Internship

81.8%

86.2%

84.0%

Off-Campus Experience

45.5%

56.9%

51.2%

Interdisciplinary Experience

45.5%

43.1%

44.3%

Community Based Projects

54.5%

27.6%

41.1%

Travel Experience

31.8%

46.6%

39.2%

Study Abroad

45.5%

31.0%

38.3%

Team Competition

59.1%

70.7%

64.9%

Student Clubs

59.1%

79.3%

69.2%

These data reflect several of the successful initiatives and priorities established by the department in recent years.  Of note is the very high percentage of students participating in Internships (not for academic credit) and Co-ops (for academic credit).  This is a direct result of the many industry partnerships maintained by the department and the extensive recruiting program conducted in-house (see the full description of the recruiting program in Section VII.F.1. of this Self-Study).  These data also confirm that most students are continuing to prefer non-credit internships over the co-op program in order to save on tuition costs.

The opportunities provided in recent years for student professional development are reflected in participation levels in off-campus and travel experiences and in study abroad.  For a program that does not require this participation as part of its curriculum, seeing that about half of the CM graduates had participated in these enrichments demonstrated the impact of investing private donations to support these experiences.  Similarly, the increasing number of students who are now participating in competitions and student clubs (about 3 out of 4 graduates) is another measure of success following an extensive departmental investment in this area.  The relatively high level of participation in interdisciplinary experiences is reflective of the close connections that the department enjoys with the other departments in the College of Architecture and Environmental Design.

 

Industry Advisory Committee Interviews

Some key external assessment instruments are aimed at our most knowledgeable and consistent stakeholders, our Industry Advisory Committee. All members of the committee recruit and employ CM students on a regular basis and many are alumni and financial supporters of the department.  The Industry Advisory Committee constitutes a focus group representative of the leading edge in the California construction industry. Their vision helps to prepare the department for the future.

The Industry Advisory Committee was formally surveyed twice during this review cycle: once in 2010 and once in 2013.  The 2010 survey summary is shown below.

2010 CM Industry Advisory Committee Meeting Summary

2010 Advisory Committee Meeting Symmary

Advisory Committee members were asked to rank order the importance of several statements related to the “purpose”, “aspirations”, and “goals” of the CM department.  What is reported in the table above is the number of times each statement was ranked as the highest priority by a committee member. 

Looking closer at the results for common themes, it was noted that student success, leadership, and collaboration were frequent elements of suggested priorities.  Within the summary of the board’s recommendations, there were 13 instances of “collaboration,” and 11 instances each of “leadership” and “success.”  The importance that the board placed on these collaboration and leadership skills correlates with the importance placed on those goals by the graduating seniors. Graduating seniors scored the importance of collaboration skills and leadership skills with scores of 4.68 and 4.65 respectively, at the top of the “Importance” attributes. Those scores were equaled or exceeded only by oral communication skills with a score of 4.79 and written communication skills with a 4.68.

The importance of communication skills is not lost on the committee, however. At the Industry Advisory Committee meeting in November 2013, the board was probed with the following questions to obtain qualitative data related to the success of CM graduates and departmental initiatives:

a.       In what ways should our program change to respond to new technologies?  What new technologies should be learned by our students to support your business success?

b.      The nature of the construction workforce is changing quickly, both in generational make-up and diversity.  In what ways can we adapt to the needs of these changes to the workforce?  How can we support your corporate goals in this area? 

c.       In what ways are professional relationships changing in your business?  What do we need to do to better prepare students for these changes?

d.      In what ways are required leadership skills changing?  How do we better prepare our students for these changes?

e.       What are the strengths of our program?

f.       What are its weaknesses?

g.      Are we missing any opportunities?

h.      If anything were possible, what would be your vision for our future?

The ensuing dialogue quickly centered on the topics of communication, leadership, the construction superintendent, and technology.

The sense of the board was that communication skills are the #1 issue. The expanding influence of constructors into the design process is driving a need for better communication skills. This need is going up in industry while communication skills are going down among graduates. The board reasons that communication skills are declining in parallel with an increased reliance on digital communication technology. Graduating seniors are at least aware of the importance of some communication skills: they rank written and oral communication skills highest in importance. Moreover, both written and oral communication goals exhibit a significant gap between importance and attainment. The CLA, as described previously, is being utilized along with other assessment instruments to evaluate these shortcomings and develop actionable plans.

There is an apparent disconnect when it comes to leadership. New project delivery methodologies are collaborative processes that rely on strong interpersonal relationships. The board holds that collaboration (i.e. interpersonal relationships) is critically important. They see “people skills” as a shortcoming in all graduates. The graduates, however, perceive that their collaboration skill attainment (4.38) is close to its importance (4.68), resulting in a gap score of only .033 and therefore second from the bottom in the Goals Attainment/Importance Gap Analysis. This will require further assessment so that actionable plans can be implemented.

The industry needs construction superintendents and the traditional route, upward through the trades and foremanship, is producing insufficient numbers. Moreover, technology is changing the skills required of a superintendent. There is an unfulfilled need for the university to produce graduates interested in becoming superintendents. There is an ongoing initiative to use internships and co-operative education to address this need. Further assessment is needed.

The board and graduating students are in lockstep on the ever-increasing importance of information technology. Computer skills are at the top of the Goals Attainment/Importance Gap Analysis and the board also placed great emphasis on this. The board sees BIM as a backbone technology. The board also implored the department to bring mobile technologies, now commonplace in the field, into the curriculum.

University Career Services Graduate Status Report

The Career Services Graduate Status Report tracks the reported salaries and other data related to graduates and makes median salary by class, major, and other information available to the public at http://careerservices.calpoly.edu/content/student/gsr_report.

With very few exceptions, CM graduates are employed in California. The median salary for these graduates held steady at or near $60,000 from 2007 through 2011. Graduate salaries in 2012 marked the beginning of a change with salaries starting to rise again to their current level of $64,000. This academic year, the department has witnessed robust recruiting activity along with reports of high and increasing levels of construction throughout California. The number of CM graduates has declined significantly from a peak in 2010, however: a lingering consequence of recession-induced declines in the applicant pool and CSU budget. Clearly, demand is now out of phase with supply, but recent recruiting efforts are starting to reverse these trends by attracting more applicants again.

Construction Management Graduate Status Report

                  

Employed
Full-Time

Employed
Part-Time

Graduate School

Other Responses

No Response

Median Salary

2006-07

45

0

0

2

32

$60,000

2007-08

60

2

3

1

35

$61,000

2008-09

35

0

0

0

38

$60,000

2009-10

68

0

1

5

45

$58,000

2010-11

31

0

1

1

41

$60,000

2011-12

44

1

0

1

53

$62,500

2012-13

24

0

0

1

38

$64,000

 

Related Content

Support CM

Support CM

Admissions

Image of student services window

Link to the Admissions Website for information on selection criteria, application deadlines, creating your VIP page and more

Admissions

Professional Certificates

Learn more about CM's Professional Certificates

Learn More

Remember:

25-35 hours/week of study time

Most College Students Don't Study Enough! Don't be left behind.