Evaluation update: Huddersfield

Joe Cullen
14th-15th February
Huddersfield
 What we’ve been doing and where we are now

‣ Continuous monitoring of project progress and direction of

travel (process dashboard)
‣ Third ‘wave’ of case studies (continuing to map cases’
evolution).
‣ Data mainly generated from (limited and qualitative) interviews
with key actors involved in the business cases.

‣ Development of Outcomes Survey for the outcome

evaluation (which will be the focus of the next months).
 CONTRIBUTION ANALYSIS

“assesses the contribution a programme is making to observed

results. It sets out to verify the theory of change behind a
programme and, at the same time, takes into consideration other
influencing factors"(Mayne, 2012)

IN OTHER WORDS…...

It tells the most plausible story about which combinations of factors lead to the
outcomes and impacts we identify for TCBL by:
1. Interrogating the project theory of change
2. Gathering evidence on whether the Theory of Change works, and what are the
plausible alternative explanations
3. Explaining how and why a result is caused by a particular sequence of events and
actions and specify the weaknesses in the story
 Create an A business
ecosystem service will
that evolve,
Business
Innovation is aggregates transforming increase in
Labs are set
not shared the best sector
up to create
knowledge- knowledge, outputs into capacity by
new business
driven innovation, service 5%,
models
creativity and concepts and
business value
innovations propositions

Presenting Solution Actions & Outcomes Impacts

Problem Theory Outputs
Actions and behaviours of different actors – REASONING
Influenced by ‘contextual RESOURCES’
Lead to certain outcomes

Mechanism
 TCBL mechanisms

TCBL Impact Mechanism

+5% manufacturing The’ alternative vision mechanism’..

The ‘’mobilizing untapped resources’ mechanism.
capacity The ‘win-win’ mechanism.
The ‘engagement and relationship’ mechanism.
The ‘leadership-followership’ mechanism.
The ‘internationalisation’ mechanism.

-20% eco footprint The ‘Lab as an Ecological Bridge’ mechanism:

New embedded Networking mechanism:

Knowledge Generation mechanism
services for c-driven Value change mechanism
supply chain
Novel supply network The ‘External Communication as pull factor’ mechanism.
The ‘Human relation/networks’ mechanism
of 1,000 organisations The ‘Mentality change’ mechanism
 Business Case Mechanisms
Business case TCBL impact Mechanism KPI

Natural fibre +5% ‘win-win’ mechanism: By matching like minded Proportion (%age)
manufacturing businesses, TCBL advisors help associates of matches made
capacity recognise win-win situations of new collaborations that lead to a
to (re-)create the natural fibre value chain. business outcome
Bacteria dye -20% eco footprint The lab as ecological bridge: by testing a range %age change of
of colours from bacteria dye in lab and enterprise waste produced
settings, knowledge and processes are created
that incentivise others to move from conventional
to bacteria dye hence creating environmental
savings
Short runs New embedded Networking mechanism: Successful networking % of participating
services for c- between TCBL stakeholders through successful businesses used
driven supply ‘matching’ of organisations in the TCBL ecosystem matchmaking
chain for business outcomes will provide concrete platform
examples of best practices and evidence of added
value of services
Independents Novel supply Human relations / networks: Creating spaces to % increase in
network of 1,000 bring people together, and supporting interactions
organisations independents to partner with each other and other between members
TCBL ecosystem members will increase of TCBL ecosystem
collaborations and create a novel supply network
 Outcomes Survey

‣ Crucial to collect evidence for contribution story

‣ Focuses on assessing whether the mechanisms we’ve
developed work
‣ Collect data on how the mechanisms work (how resources
are used to affect ‘reasoning’ and how this leads to
outcomes)
‣ Problems:
‣ Lots of mechanisms
‣ At 2 different levels
‣ Lead to a big survey instrument – 11 pages; 27 questions; 12
multi-item rating scales
 Block 1: Introduction and general questions

Introduction About you

Block 2: TCBL-level

Resources Immediate Outcomes Intermediate Outcomes

Block 3: Business pilot level

Resources Immediate Outcomes Intermediate Outcomes
Block 1 Introduction and general questions

Introduction

• Purpose of survey
• GDPR details

About you

• Type of TCBL member

• Time of joining
• Level of involvement
• Involvement in Business Pilots
 Block 2: TCBL Level
Resources

• Use of Knowledge resources and

technological services
• Use of Business services
• Use of Lab services
• Use of other resources and services

Immediate Outcomes

• Changes in awareness
• Changes in knowledge
• Changes in attitudes
• Changes in networks/business partners
• Changes in dissimination and other activities

Intermediate Outcomes

• Changes in behaviours
Block 3: Business pilot level

Resources

• Current involvement in the pilot

• Description of involvement
• List of pilot resources
• Motive for using resources
• Usefulness of resources

Immediate Outcomes

• Changes in awareness, knowledge and attitudes

• Rating of knowledge at start and end
• Rating of awareness at start and end

Intermediate Outcomes

• Changes in behaviours
• Rating of confidence level to sustain changes
• Contribution of external resources to changes
To what extent have the use of the resources stated
above supported you in changes in relation to your
dissemination and other activities?

Very Modera
great Great te Some Not at Unsu
extent extent extent extent all re
Disseminating
knowledge of ☐ ☐ ☐ ☐ ☐ ☐
ecological issues
Sharing knowledge in
online spaces ☐ ☐ ☐ ☐ ☐ ☐

Experimenting and
exploring new ways for
design/production ☐ ☐ ☐ ☐ ☐ ☐

Other ______________
☐ ☐ ☐ ☐ ☐ ☐
 Questions

‣ Do you support the overall approach?

‣ Block 2, Block 3 or both?
‣ How to reduce the instrument
‣ Any obvious gaps?
‣ Will people fill it in?
‣ What’s the best way of selling it to TCBL members?
(Businesses are key)
 What else?

‣ There’s datasets out there that have been done – or

planned – e.g. questionnaire for Associates on short Runs;
survey on what services do Associates want

‣ We need to make a list of what data re already available

that can help us identify outcomes

‣ We need to know what other data collection activities are

planned, so we can pool resources and make sure
respondents don’t get ‘survey fatigue’

‣ We need to deliver i) a financial outcome model ii) a

predictive ‘what works’ model – we need more data on
costs - and expertise e.g. on trade-off modelling