

SDGP TRACTIONS: The project aims to improve the livelihoods and wellbeing of Indonesian smallholders and their families by creating value addition through high-quality fermented cocoa beans, greater access to finance and markets, and a transparent, sustainable, and efficient fermented cocoa value chain. The project brings together the international and commercial expertise of the specialty chocolate sector, the robust experience of international and local NGOs in climate-smart agriculture, sustainable production, and community development, and an extensive network across the cocoa industry, including national and international (especially European) markets, supported by the government.
The project aims to create a transparent, sustainable, and efficient value chain for high-quality fermented cocoa beans produced by 3,400 smallholders in the Indonesian provinces of Bali, Sulawesi, and East Nusa Tenggara. High-quality fermented cocoa beans are used in premium chocolate, a niche market in which manufacturers pay premium prices for ingredients. The project provides smallholders, smallholder groups, and cooperatives with training, technical support, and awareness-raising, processing facilities, and access to inputs, finance, and markets. As such, the project aims to increase productivity and deliver high-quality fermented cocoa beans that meet the standards of premium chocolate producers. Beneficiary smallholders' income will increase as they benefit from the premium chocolate market's pricing advantage.
The TRACTIONS project is a six-year partnership programme (August 2020 – June 2026) funded by the Netherlands Enterprise Agency (RVO) through SDGP-2178RI, implemented by Rainforest Alliance in consortium with Valrhona, Kalimajari Rikolto and Jembrana district government. The project strengthens smallholder cocoa value chains across seven districts in Sulawesi, Bali, and Flores/NTT through climate-smart agriculture (CSA) training, consolidation of cooperative governance, and facilitated access to premium markets via buyers, including Valrhona as consortium members, as well as local chocolate makers in Indonesia.
Region
District
Partner Cooperative
Buyer
Sulawesi
East Kolaka
East Kolaka Cooperative
Fermented cocoa market – Onuka Chocolate
Sulawesi
North Luwu
Simultan Cooperative
PT Awina / MoU consortium
Sulawesi
Poso
KKB (Karya Bersama)
Valrhona — active supply
Bali
Jembrana
KSS (Kerta Semaya Samaniya)
Valrhona — active export
Bali
Tabanan
MAB (Manik Amerta Buana)
Valrhona — active supply
Flores/NTT
Ende
SIKAP Cooperative
Valrhona — sample stage
Flores/NTT
Sikka
NTT Partner Cooperative
IKAT Cacao Indonesia
A baseline study (October–December 2020) established indicator benchmarks across 455 smallholders and 7 cooperatives. The present endline evaluation measures achievement against those benchmarks, verifies the Theory of Change, and generates recommendations for successor programming.
The evaluation applies to a mixed-method, theory-based approach that combines pre-post comparison with contribution analysis.
Dimension
Key Evaluation Questions
Relevance
Is the project design aligned with the needs of the farmer and the cooperative? Is the cooperative model still fit for purpose, given current market conditions and EUDR requirements?
Effectiveness
To what extent have SDGP indicator targets been achieved? What enabled or constrained results across Sulawesi, Bali, and NTT?
Theory of Change
Did the CSA training → quality improvement → premium market access pathway hold? Which assumptions were confirmed, partially confirmed, or refuted?
IPA
Which interventions are most important to beneficiaries? Where are the largest gaps between importance and delivery quality?
MSC
What is the single most significant change experienced by farmers, women, and youth? In which domain — income, capacity, markets, environment — is change most felt?
Institutional Capacity
What is the current Endline CAT score across 9 dimensions for each cooperative? Which cooperatives face the highest risk of post-project regression?
Sampling Integrity
Is the endline sample statistically comparable to the 2020 baseline sample? Are results extrapolatable to the full beneficiary population?
Impact & Sustainability
What changes are attributable to TRACTIONS? Are cooperatives commercially viable and institutionally resilient enough to operate independently?
The evaluation applies a seven-method mixed-methods design across two tiers. Tier 1 covers primary data collection; Tier 2 covers analytical and interpretive methods. All quantitative methods are benchmarked against the 2020 baseline study (455 smallholders, 7 cooperatives), which used a probability-based random sampling framework, to ensure statistical comparability.
#
Method
Purpose
Key Steps & Outputs
1
IPA
Measures beneficiary-perceived importance vs. delivery quality of each intervention; plots gaps on a 2×2 matrix to guide successor programe priorities.
• Score each intervention (1–5) for Importance and Performance in FGDs
• Plot weighted means on IPA matrix per region and respondent group
• Output: IPA matrices + quadrant analysis per district
2
MSC
Collects, verifies, and interprets stories of change from beneficiaries, capturing unanticipated outcomes and the lived experience of change.
• Prompt: “What is the single most significant change in your livelihood?”
• Code stories by domain: income, capacity, governance, gender, markets, environment
• Output: verified story archive + domain synthesis per cooperative
3
FGD
Primary in-field data collection vehicle integrating IPA and MSC across 5 homogeneous groups per district (35 sessions total).
• 5 groups × 7 districts = 35 FGDs: male farmers, female farmers, youth, coop management, local govt
• Audio-recorded with consent; coded within 48 hours
• Output: coded transcripts and discussion summaries
•
4
ToC Verification
Retrospectively tests whether causal assumptions linking activities to outcomes are held in practice across all three regional contexts.
• Reconstruct ToC from design documents and SDGP logical framework
• Rate each assumption: Confirmed / Partly Confirmed / Refuted / Insufficient Evidence
• Output: ToC Verification Matrix + narrative assessment
5
Random Sampling Survey
Statistically representative KPI measurement
• Tier 1 — Primary Data Collection
6
KII
Semi-structured individual interviews with 26–37 key informants across RA staff, government, buyers, financiers, and cooperative leadership.
• Respondents: RA/consortium staff, 7 district govts, buyers (Valrhona, PT Awina, PT. Adore Chocolate), banks, coop chairs
• Thematic coding aligned to evaluation questions and ToC
• Output: KII synthesis per respondent category
7
Endline CAT
Direct endline re-application of the 2020 baseline CAT — 9 identical dimensions, same 5-point scale — producing comparable before-after cooperative capacity profiles.
• 2.5–3 hr participatory workshop with coop management (3–5 persons) per cooperative
• Self-scoring with evidence prompts; validated against documents and KII
• Output: scorecards + radar diagrams (2020 vs. endline) + sustainability risk matrix
The Endline CAT replicates the exact 9 dimensions, 5-point scoring scale (1 = nascent → 5 = self-sustaining), and evidence-validation procedure of the 2020 TRACTIONS baseline CAT, enabling direct change-score comparisons per cooperative and per dimension. Data quality assurance for the endline CAT integrates four safeguards covering enumerator training, 10% spot checks baseline dimensions, daily data validation, and 3-tier field supervision.
#
Dimension
Key Endline Assessment Areas
Scale
1
Governance & Legal Compliance
Board structure, democratic decision-making, accountability, conflict resolution, and statutory compliance
1–5
2
Financial Management & Accounting
Bookkeeping quality, working capital, credit access, financial reporting and IMS self-financing progress
1–5
3
Member Services & Democratic Participation
Service delivery, savings/credit facilitation, pre-financing, member satisfaction, gender/youth in governance
1–5
4
Collective Marketing & Buyer Relationships
Contract management (Valrhona, PT Awina), pricing negotiation, sales volume, supply reliability
1–5
5
Fermentation Quality Control & Consistency
CCPU operations, post-harvest SOP adherence, fermentation management, quality grading
1–5
6
Input Supply Services to Members
Collective input procurement, organic input production, and advance payment systems
1–5
7
External Partnerships & Institutional Linkages
Government relations (Dinas Pertanian), private sector/CSR partnerships, community standing
1–5
8
Gender Inclusion & Equity
Women’s leadership representation, GALS follow-through, equitable access to services and benefits
1–5
9
Traceability, Certification & Sustainability
IMS functionality, RA certification readiness, EUDR polygon mapping, and sustainability SOP adherence
1–5
All endline findings are measured against the 2020 baseline (455 smallholders, 7 cooperatives). The endline must replicate baseline measurement logic to ensure comparability.
KPI Code
Indicator
Measurement Logic
IOM 1.1.1
Smallholders applying best crop production & post-harvest practices
Compound: agronomic + post-harvest practice survey questions
IOM 1.4.1
Smallholders applying best business management practices
Compound: business management sub-module questions
MOM 1.4
Smallholders with improved access to input markets
Compound: Q59, Q59a, Q48 — min. 2 of 3 sub-criteria met
MOM 1.5
Smallholder groups selling to sustainable premium markets
Q75: harvest sold in premium fermented cocoa market = Yes
MOM 3.1
Smallholders receiving government support under Decree 51 / 2012
Q136 = Yes AND at least one of Q137.6, Q137.8, Q137.9 = Yes
Resilience Index
Vulnerability level to climate change (CSA framework)
VCA questionnaire (40q) + BMKG climate data; Excel scoring model
Cooperative Capacity
Institutional capacity across 9 CAT dimensions
5-point scale per dimension; endline = Endline CAT
Circular Economy
Adoption of circular economy practices
R-Framework: 4 clusters × practice-adoption rates
Note: Sample proportions must be extrapolated to the 3,400-beneficiary universe using the same stratified projection method as in the baseline. All KPIs disaggregated by gender and age (yout
All seven project districts are covered through two distinct sampling tracks. For the quantitative survey, a stratified random sampling approach will be applied, aligned with the 2020 baseline methodology (10% margin of error, 95% confidence interval), with a minimum sample size of 400-500 smallholder households disaggregated by gender, youth, and district, to ensure statistical comparability and valid extrapolation to the 3,400-beneficiary population. For the qualitative components (FGDs, KIIs, and MSC), participants will be purposively selected to capture diverse perspectives across active cooperative members; women with ≥2-GALS cycles; youth (<35) engaged in TRACTIONS activities; cooperative management and IMS staff; district government representatives (Dinas Pertanian, Dinas Perindustrian); and direct private-sector partners and buyers.
Deliverable
Description
Due
Inception Report
Work plan, all instruments (FGD guide, IPA card, KII guide, Endline CAT scorecard, ToC map), and sampling plan
1 wk post-contract
All Instruments
FGD guides, KII protocol, IPA questionnaire, MSC guide, Endline CAT scorecard — Bahasa Indonesia & English
Before fieldwork
Interim Report
Preliminary findings: IPA matrices, draft MSC stories, KII synthesis, initial Endline CAT scores, emerging ToC findings
1 wk post-fieldwork
Draft Report
Full report: IPA, MSC, ToC Matrix, Endline CAT scorecards with 2020 comparison, KII, KPI achievement, recommendations
3 wks post-fieldwork
Findings Presentation
Presentation to RA, consortium, and RVO with min. 30 min Q&A
1 wk post-draft
Final Report
Revised report incorporating all stakeholder feedback
1 wk post-presentation
Executive Summary
4–6 page stand-alone summary in Bahasa Indonesia and English
With the final report
Full Dataset Package
Anonymised transcripts, IPA data, Endline CAT scorecards, MSC archive, KII summaries, all secondary data
With the final report
Phase
Duration
Key Activities
Inception & Instrument Development
1 week
ToC reconstruction; Endline CAT finalisation; FGD/KII guide development; baseline review
Document Review & Secondary Data
1 week
Project reports, M&E data, SDGP indicators, IMS records, 2020 baseline annexes
Field Data Collection
4 weeks
35 FGDs + 26–37 KIIs + 7 Endline CAT workshops across 7 districts
Analysis & Report Writing
3 weeks
IPA matrix; MSC coding; Endline CAT scoring; ToC assessment; KPI tabulation
Review & Finalisation
1 week
Stakeholder feedback integration: final quality assurance
Minimum Requirements
Preferred / Added Advantage
• 7+ years in programme/project evaluation (development, agriculture, smallholder empowerment)
• Demonstrated IPA application in programme evaluation
• Demonstrated MSC methodology experience (collection, verification, selection)
• Experience in baseline-endline comparability and advanced statistical analysis.
• FGD facilitation with farming communities incl. women-only and youth groups
• Semi-structured KII experience with government, private sector, and civil society
• ToC framework experience and retrospective verification/contribution analysis
• Cooperative or organisational capacity assessment tool experience (RA CAT, USAID OCA, or equivalent)
• Physical presence in Indonesia; logistical capacity in Sulawesi, Bali, and NTT
• Mixed-methods analysis; reports in Bahasa Indonesia and English
• Cocoa, coffee, or tropical commodity programme evaluation
• Indonesia cooperative sector and EUDR regulatory context familiarity
• Experience with international NGOs or Netherlands/EU bilateral programmes
• GALS methodology experience; gender and social inclusion in agriculture
• SDGP / RVO reporting requirements and evaluation standards
• Eastern Indonesia field experience (Sulawesi, NTT)
• RA Sustainable Agriculture Standard 2020 and CSA framework familiarity
Role
Responsibilities
Min. Experience
Lead Evaluator / Team Leader
Overall methodology, ToC Verification, data integration, report writing, donor presentation
10+ yrs, senior
Qualitative Specialist / MSC Lead
MSC collection and selection, FGD facilitation, KII coding and synthesis
7+ years
Field Facilitators (2–3)
FGD facilitation, Endline CAT workshops, note-taking, local language
3+ yrs, field
Data Analyst
IPA analysis, Endline CAT scoring with 2020 comparison, KPI tabulation
5+ yrs, quant
Research / Logistics Assistant
Documentation, scheduling, data entry, logistics
2+ years
Evaluation Criteria
Weight
Notes
Quality of technical proposal and understanding of TOR and project context
30%
Includes familiarity with 2020 baseline methodology and instruments
Relevance and depth of prior evaluation experience (IPA, MSC, ToC, KII, CAT; agriculture; cooperatives)
25%
Tropical agriculture and Indonesia experience are preferred
Soundness of the six-method design and draft instruments with baseline alignment
20%
Endline CAT scorecard design and KPI comparator methodology evaluated here.
Team composition, qualifications, and field presence across Sulawesi, Bali, and NTT
15%
Financial proposal — value for money and budget transparency
10%
Item
Detail
Submission Email
[lukmansyah@ra.org — Lukmansyah - Rainforest Alliance Indonesia]
Email Subject Line
“Endline Evaluation Proposal — TRACTIONS — [Organisation Name]”
Submission Deadline
[Date — to be confirmed by RA]
File Format
File 1: Technical Proposal + Administrative Documents (PDF). File 2: Financial Proposal (PDF, separate).
Enquiries
[RA contact email] — written questions accepted no later than [query deadline]
Finalist Notification
[Date — to be confirmed by RA]