I.  BACKGROUND

SDGP TRACTIONS: The project aims to improve the livelihoods and wellbeing of Indonesian smallholders and their families by creating value addition through high-quality fermented cocoa beans, greater access to finance and markets, and a transparent, sustainable, and efficient fermented cocoa value chain. The project brings together the international and commercial expertise of the specialty chocolate sector, the robust experience of international and local NGOs in climate-smart agriculture, sustainable production, and community development, and an extensive network across the cocoa industry, including national and international (especially European) markets, supported by the government.

The project aims to create a transparent, sustainable, and efficient value chain for high-quality fermented cocoa beans produced by 3,400 smallholders in the Indonesian provinces of Bali, Sulawesi, and East Nusa Tenggara. High-quality fermented cocoa beans are used in premium chocolate, a niche market in which manufacturers pay premium prices for ingredients. The project provides smallholders, smallholder groups, and cooperatives with training, technical support, and awareness-raising, processing facilities, and access to inputs, finance, and markets. As such, the project aims to increase productivity and deliver high-quality fermented cocoa beans that meet the standards of premium chocolate producers. Beneficiary smallholders' income will increase as they benefit from the premium chocolate market's pricing advantage.

The TRACTIONS project is a six-year partnership programme (August 2020 – June 2026) funded by the Netherlands Enterprise Agency (RVO) through SDGP-2178RI, implemented by Rainforest Alliance in consortium with Valrhona, Kalimajari Rikolto and Jembrana district government. The project strengthens smallholder cocoa value chains across seven districts in Sulawesi, Bali, and Flores/NTT through climate-smart agriculture (CSA) training, consolidation of cooperative governance, and facilitated access to premium markets via buyers, including Valrhona as consortium members, as well as local chocolate makers in Indonesia.

 

Region

District

Partner Cooperative

Buyer

Sulawesi

East Kolaka

East Kolaka Cooperative

Fermented cocoa market – Onuka Chocolate

Sulawesi

North Luwu

Simultan Cooperative

PT Awina / MoU consortium

Sulawesi

Poso

KKB (Karya Bersama)

Valrhona — active supply

Bali

Jembrana

KSS (Kerta Semaya Samaniya)

Valrhona — active export

Bali

Tabanan

MAB (Manik Amerta Buana)

Valrhona — active supply

Flores/NTT

Ende

SIKAP Cooperative

Valrhona — sample stage

Flores/NTT

Sikka

NTT Partner Cooperative

IKAT Cacao Indonesia

 

A baseline study (October–December 2020) established indicator benchmarks across 455 smallholders and 7 cooperatives. The present endline evaluation measures achievement against those benchmarks, verifies the Theory of Change, and generates recommendations for successor programming.

II.  EVALUATION OBJECTIVES

• Measure achievement of SDGP overarching and project-specific indicators at project close, benchmarked against 2020 baseline values.
• Assess relevance, effectiveness, efficiency, impact, and sustainability across all seven districts with cross-regional comparative analysis.
• Document most significant changes for farmers, cooperative members, women, and youth using the MSC methodology.
• Evaluate project performance from a beneficial perspective using Importance Performance Analysis (IPA).
• Retrospectively verify the project’s Theory of Change — testing whether causal assumptions hold across regional contexts.
• Assess cooperative institutional capacity using the Endline CAT (9 Dimensions), producing comparable before-and-after scores against the 2020 CAT baseline.
• Provide strategic, evidence-based recommendations for successor programme design in Indonesia’s cocoa sector.

 

The evaluation applies to a mixed-method, theory-based approach that combines pre-post comparison with contribution analysis.

 

III.  EVALUATION QUESTIONS

 

Dimension

Key Evaluation Questions

 

 

Relevance

Is the project design aligned with the needs of the farmer and the cooperative? Is the cooperative model still fit for purpose, given current market conditions and EUDR requirements?

Effectiveness

To what extent have SDGP indicator targets been achieved? What enabled or constrained results across Sulawesi, Bali, and NTT?

Theory of Change

Did the CSA training → quality improvement → premium market access pathway hold? Which assumptions were confirmed, partially confirmed, or refuted?

IPA

Which interventions are most important to beneficiaries? Where are the largest gaps between importance and delivery quality?

MSC

What is the single most significant change experienced by farmers, women, and youth? In which domain — income, capacity, markets, environment — is change most felt?

Institutional Capacity

What is the current Endline CAT score across 9 dimensions for each cooperative? Which cooperatives face the highest risk of post-project regression?

Sampling Integrity

Is the endline sample statistically comparable to the 2020 baseline sample? Are results extrapolatable to the full beneficiary population?

Impact & Sustainability

What changes are attributable to TRACTIONS? Are cooperatives commercially viable and institutionally resilient enough to operate independently?

 

IV.  EVALUATION METHODOLOGY

The evaluation applies a seven-method mixed-methods design across two tiers. Tier 1 covers primary data collection; Tier 2 covers analytical and interpretive methods. All quantitative methods are benchmarked against the 2020 baseline study (455 smallholders, 7 cooperatives), which used a probability-based random sampling framework, to ensure statistical comparability.

 

#

Method

Purpose

Key Steps & Outputs

1

IPA

Measures beneficiary-perceived importance vs. delivery quality of each intervention; plots gaps on a 2×2 matrix to guide successor programe priorities.

•      Score each intervention (1–5) for Importance and Performance in FGDs

•      Plot weighted means on IPA matrix per region and respondent group

•      Output: IPA matrices + quadrant analysis per district

2

MSC

Collects, verifies, and interprets stories of change from beneficiaries, capturing unanticipated outcomes and the lived experience of change.

•      Prompt: “What is the single most significant change in your livelihood?”

•      Code stories by domain: income, capacity, governance, gender, markets, environment

•      Output: verified story archive + domain synthesis per cooperative

3

FGD

Primary in-field data collection vehicle integrating IPA and MSC across 5 homogeneous groups per district (35 sessions total).

•      5 groups × 7 districts = 35 FGDs: male farmers, female farmers, youth, coop management, local govt

•      Audio-recorded with consent; coded within 48 hours

•      Output: coded transcripts and discussion summaries

 

 

 

•       

4

ToC Verification

Retrospectively tests whether causal assumptions linking activities to outcomes are held in practice across all three regional contexts.

•      Reconstruct ToC from design documents and SDGP logical framework

•      Rate each assumption: Confirmed / Partly Confirmed / Refuted / Insufficient Evidence

•      Output: ToC Verification Matrix + narrative assessment

5

Random Sampling Survey

Statistically representative KPI measurement

•      Tier 1 — Primary Data Collection

6

KII

Semi-structured individual interviews with 26–37 key informants across RA staff, government, buyers, financiers, and cooperative leadership.

•      Respondents: RA/consortium staff, 7 district govts, buyers (Valrhona, PT Awina, PT. Adore Chocolate), banks, coop chairs

•      Thematic coding aligned to evaluation questions and ToC

•      Output: KII synthesis per respondent category

7

Endline CAT

Direct endline re-application of the 2020 baseline CAT — 9 identical dimensions, same 5-point scale — producing comparable before-after cooperative capacity profiles.

•      2.5–3 hr participatory workshop with coop management (3–5 persons) per cooperative

•      Self-scoring with evidence prompts; validated against documents and KII

•      Output: scorecards + radar diagrams (2020 vs. endline) + sustainability risk matrix

 

 

Endline CAT — 9 Assessment Dimensions

The Endline CAT replicates the exact 9 dimensions, 5-point scoring scale (1 = nascent → 5 = self-sustaining), and evidence-validation procedure of the 2020 TRACTIONS baseline CAT, enabling direct change-score comparisons per cooperative and per dimension. Data quality assurance for the endline CAT integrates four safeguards covering enumerator training, 10% spot checks baseline dimensions, daily data validation, and 3-tier field supervision.

 

#

Dimension

Key Endline Assessment Areas

Scale

1

Governance & Legal Compliance

Board structure, democratic decision-making, accountability, conflict resolution, and statutory compliance

1–5

2

Financial Management & Accounting

Bookkeeping quality, working capital, credit access, financial reporting and IMS self-financing progress

1–5

3

Member Services & Democratic Participation

Service delivery, savings/credit facilitation, pre-financing, member satisfaction, gender/youth in governance

1–5

4

Collective Marketing & Buyer Relationships

Contract management (Valrhona, PT Awina), pricing negotiation, sales volume, supply reliability

1–5

5

Fermentation Quality Control & Consistency

CCPU operations, post-harvest SOP adherence, fermentation management, quality grading

1–5

6

Input Supply Services to Members

Collective input procurement, organic input production, and advance payment systems

1–5

7

External Partnerships & Institutional Linkages

Government relations (Dinas Pertanian), private sector/CSR partnerships, community standing

1–5

8

Gender Inclusion & Equity

Women’s leadership representation, GALS follow-through, equitable access to services and benefits

1–5

9

Traceability, Certification & Sustainability

IMS functionality, RA certification readiness, EUDR polygon mapping, and sustainability SOP adherence

1–5

 

 

Baseline Reference — KPI Comparators (2020)

All endline findings are measured against the 2020 baseline (455 smallholders, 7 cooperatives). The endline must replicate baseline measurement logic to ensure comparability.

 

KPI Code

Indicator

Measurement Logic

IOM 1.1.1

Smallholders applying best crop production & post-harvest practices

Compound: agronomic + post-harvest practice survey questions

IOM 1.4.1

Smallholders applying best business management practices

Compound: business management sub-module questions

MOM 1.4

Smallholders with improved access to input markets

Compound: Q59, Q59a, Q48 — min. 2 of 3 sub-criteria met

MOM 1.5

Smallholder groups selling to sustainable premium markets

Q75: harvest sold in premium fermented cocoa market = Yes

MOM 3.1

Smallholders receiving government support under Decree 51 / 2012

Q136 = Yes AND at least one of Q137.6, Q137.8, Q137.9 = Yes

Resilience Index

Vulnerability level to climate change (CSA framework)

VCA questionnaire (40q) + BMKG climate data; Excel scoring model

Cooperative Capacity

Institutional capacity across 9 CAT dimensions

5-point scale per dimension; endline = Endline CAT

Circular Economy

Adoption of circular economy practices

R-Framework: 4 clusters × practice-adoption rates

 

Note: Sample proportions must be extrapolated to the 3,400-beneficiary universe using the same stratified projection method as in the baseline. All KPIs disaggregated by gender and age (yout

V.  EVALUATION LOCATIONS AND SAMPLING

All seven project districts are covered through two distinct sampling tracks. For the quantitative survey, a stratified random sampling approach will be applied, aligned with the 2020 baseline methodology (10% margin of error, 95% confidence interval), with a minimum sample size of 400-500 smallholder households disaggregated by gender, youth, and district, to ensure statistical comparability and valid extrapolation to the 3,400-beneficiary population. For the qualitative components (FGDs, KIIs, and MSC), participants will be purposively selected to capture diverse perspectives across active cooperative members; women with ≥2-GALS cycles; youth (<35) engaged in TRACTIONS activities; cooperative management and IMS staff; district government representatives (Dinas Pertanian, Dinas Perindustrian); and direct private-sector partners and buyers.

VI.  DELIVERABLES AND TIMELINE

 

Deliverable

Description

Due

Inception Report

Work plan, all instruments (FGD guide, IPA card, KII guide, Endline CAT scorecard, ToC map), and sampling plan

1 wk post-contract

All Instruments

FGD guides, KII protocol, IPA questionnaire, MSC guide, Endline CAT scorecard — Bahasa Indonesia & English

Before fieldwork

Interim Report

Preliminary findings: IPA matrices, draft MSC stories, KII synthesis, initial Endline CAT scores, emerging ToC findings

1 wk post-fieldwork

Draft Report

Full report: IPA, MSC, ToC Matrix, Endline CAT scorecards with 2020 comparison, KII, KPI achievement, recommendations

3 wks post-fieldwork

Findings Presentation

Presentation to RA, consortium, and RVO with min. 30 min Q&A

1 wk post-draft

Final Report

Revised report incorporating all stakeholder feedback

1 wk post-presentation

Executive Summary

4–6 page stand-alone summary in Bahasa Indonesia and English

With the final report

Full Dataset Package

Anonymised transcripts, IPA data, Endline CAT scorecards, MSC archive, KII summaries, all secondary data

With the final report

 

Indicative Schedule  (~10 weeks total)

Phase

Duration

Key Activities

Inception & Instrument Development

1 week

ToC reconstruction; Endline CAT finalisation; FGD/KII guide development; baseline review

Document Review & Secondary Data

1 week

Project reports, M&E data, SDGP indicators, IMS records, 2020 baseline annexes

Field Data Collection

4 weeks

35 FGDs + 26–37 KIIs + 7 Endline CAT workshops across 7 districts

Analysis & Report Writing

3 weeks

IPA matrix; MSC coding; Endline CAT scoring; ToC assessment; KPI tabulation

Review & Finalisation

1 week

Stakeholder feedback integration: final quality assurance

 

VII.  EVALUATOR QUALIFICATIONS

 

Minimum Requirements

Preferred / Added Advantage

•      7+ years in programme/project evaluation (development, agriculture, smallholder empowerment)

•      Demonstrated IPA application in programme evaluation

•      Demonstrated MSC methodology experience (collection, verification, selection)

•      Experience in baseline-endline comparability and advanced statistical analysis.

•      FGD facilitation with farming communities incl. women-only and youth groups

•      Semi-structured KII experience with government, private sector, and civil society

•      ToC framework experience and retrospective verification/contribution analysis

•      Cooperative or organisational capacity assessment tool experience (RA CAT, USAID OCA, or equivalent)

•      Physical presence in Indonesia; logistical capacity in Sulawesi, Bali, and NTT

•      Mixed-methods analysis; reports in Bahasa Indonesia and English

•      Cocoa, coffee, or tropical commodity programme evaluation

•      Indonesia cooperative sector and EUDR regulatory context familiarity

•      Experience with international NGOs or Netherlands/EU bilateral programmes

•      GALS methodology experience; gender and social inclusion in agriculture

•      SDGP / RVO reporting requirements and evaluation standards

•      Eastern Indonesia field experience (Sulawesi, NTT)

•      RA Sustainable Agriculture Standard 2020 and CSA framework familiarity

 

Required Team Composition

Role

Responsibilities

Min. Experience

Lead Evaluator / Team Leader

Overall methodology, ToC Verification, data integration, report writing, donor presentation

10+ yrs, senior

Qualitative Specialist / MSC Lead

MSC collection and selection, FGD facilitation, KII coding and synthesis

7+ years

Field Facilitators (2–3)

FGD facilitation, Endline CAT workshops, note-taking, local language

3+ yrs, field

Data Analyst

IPA analysis, Endline CAT scoring with 2020 comparison, KPI tabulation

5+ yrs, quant

Research / Logistics Assistant

Documentation, scheduling, data entry, logistics

2+ years

 

VIII.  PROPOSAL REQUIREMENTS

8.1  Technical Proposal

• Understanding of TOR: project context, six-method design, and RVO/RA reporting purpose.
• Proposed methodology: integration design for all six methods; draft FGD guide and Endline CAT scorecard aligned to 2020 CAT; sampling plan per district.
• Detailed work plan with timeline, milestones, and named team responsibilities.
• Team CVs for all proposed members.
• Portfolio: minimum 2 comparable endline/final evaluation reports.

8.2  Financial Proposal

• Detailed budget (IDR and/or EUR): team fees by role, travel across 7 districts, accommodation, FGD and Endline CAT workshop logistics, translation, overhead.
• Daily rates per role with reference to previously applied rates. Technical and financial proposals submitted as separate PDF documents.

8.3  Administrative Documents

• Certificate of incorporation; Tax ID (NPWP); domicile certificate or business licence.
• Signed declaration of no conflict of interest with TRACTIONS and its partners.

 

IX.  SELECTION CRITERIA

 

Evaluation Criteria

Weight

Notes

Quality of technical proposal and understanding of TOR and project context

30%

Includes familiarity with 2020 baseline methodology and instruments

Relevance and depth of prior evaluation experience (IPA, MSC, ToC, KII, CAT; agriculture; cooperatives)

25%

Tropical agriculture and Indonesia experience are preferred

Soundness of the six-method design and draft instruments with baseline alignment

20%

Endline CAT scorecard design and KPI comparator methodology evaluated here.

Team composition, qualifications, and field presence across Sulawesi, Bali, and NTT

15%

 

Financial proposal — value for money and budget transparency

10%

 

 

 

 

X.  ETHICAL PRINCIPLES AND DATA PROTECTION

• All FGD, KII, and Endline CAT data are treated as strictly confidential and anonymized in all outputs.
• Written informed consent obtained from all participants before each session or workshop.
• Photography and audio-visual documentation only with explicit participant consent.
• Evaluators must comply with Rainforest Alliance’s Child Protection and Safeguarding Policy.
• All raw data, transcripts, and Endline CAT scorecards transferred to RA upon completion; no distribution to third parties without written RA authorization.
• Evaluators must declare freedom from conflict of interest with TRACTIONS and its partners.
• All data stored in password-protected systems; personal identifiers removed from analytical files.

 

XI. PROPOSAL SUBMISSION INSTRUCTIONS

 

Item

Detail

Submission Email

[lukmansyah@ra.org — Lukmansyah - Rainforest Alliance Indonesia]

Email Subject Line

“Endline Evaluation Proposal — TRACTIONS — [Organisation Name]”

Submission Deadline

[Date — to be confirmed by RA]

File Format

File 1: Technical Proposal + Administrative Documents (PDF).  File 2: Financial Proposal (PDF, separate).

Enquiries

[RA contact email] — written questions accepted no later than [query deadline]

Finalist Notification

[Date — to be confirmed by RA]