APPENDIX III

 

 

 

 

 

Toward a Conceptual Framework for Evaluating International Social Change Networks.

NUNEZ, M. & WILSON-GRAU, R.

(2003)

 

 

 

 

Towards a conceptual framework for evaluating international social change networks[1]

 

 

By Martha Nu–ez[2] and Ricardo Wilson-Grau[3]

 

Introduction

During the 1990s, networks became an alternative form of social synergy because of their unique political and organisational potential. Through networks, diverse social actors pursue a common purpose based on personal and institutional relations. ÒSocial changeÓ networks aim to influence economic, political and cultural conditions in one or more societies. In these networks, the members are autonomous organisations—usually NGOs or community based organisations—and sometimes individuals. Furthermore, when the network is international, its aims and activities reflect heterogeneous contexts represented by its members.

 

Thus, an international social change network strives to link local efforts with global processes and build a movement that modifies power relations by:

á             Fortifying creativity and critical thinking through dialogue and exchange.

á             Sharing strategies and deepening understanding between diverse actors in complex situations.

á             Addressing global problems through knowledge of their local, national and regional contexts.

á             Strengthening a union of local forces in a global process.

á             Creating and reinforcing international consciousness, commitment and solidarity.

 

For these reasons, we believe that international networks represent today one of the best means to achieve social change in a globalised world.

 

With these notes, we intend to contribute to the discussion on evaluating this type of Network.[4] Conceptually, our reflections are based on the excellent if limited literature that we have been able to find (see References). Our most recent practical experience was the evaluation we did of the Oilwatch Network at the end of 2002, commissioned by HIVOS, one of their donor counterparts.

 

Understanding the political and operational complexity

A Network is an eminently political act. Its fundamental function is to configure the power and action of its members into a collective force for social change. Typically, the purpose of a Network is expressed in terms of modifying positions and relations of power; for example: protect and promote human and collective rights, conserve the environment or fortify equitable gender relations. Thus, the intended impact of a Network is expressed in its political purpose, rather than in institutional objectives. Said another way, Networks generally struggle for more or less intangible, even ideological, goals that have to do with the consciousness, behaviour, and empowerment of people and of societies. This does not mean, however, that the management bodies of a Network—general assembly, board of directors, secretariat—as well as each member, cannot set concrete and measurable operational objectives.  

 

Nevertheless, the management of a Network is also complex and unusual. Networks operate through facilitation and co-operation around the activities of its organisational components instead of by directing programmes and executing projects. The principal actors—the members—are autonomous organisations and not the employees or even the managers, as is the case in other types of organisations. The structure of a network is not hierarchical; commonly, the co-ordinating body or secretariat assumes responsibilities for communication, co-ordination and organisation to catalyse and carry-out activities. This is to say that in a Network the scope of authority is restricted, and there do not exist procedures for command and control common to NGOs, grassroots organisations, governments and many other forms of organisation. Consequently, a Network requires different processes for planning, monitoring and evaluation.

 

In the light of this reality, we believe that the evaluation of a Network is a special challenge, both for those to be evaluated and for the evaluators. We understand it is:

 

á             A means for learning about success and failure, more than as a mechanism of control.

á             An aid for planning, to measure progress and propose solutions to problems.

á             A transparent mechanism for accountability, based on indicators of effectiveness and efficiency, within the complexity of its operations.

á             An instrument for preserving the historical memory of the common processes that originated and sustain the Network.

á             A tool to register achievements and impacts, accepting that rarely will these be directly related to the activities of the Network, frequently the results will be collateral and unintentional, and almost always they will be the result of a broad effort with other social actors.

 

For these reasons, we consider that the evaluation of a Network should focus on two fundamental aspects: performance and results. 

 

Understanding performance

For the functioning of a Network, we take into account four performance criteria crossed by three operational dimensions. The four performance criteria[5] are:

 

Democracy

 

In addition to being a recognised value, democratic management is a necessity in a Network. Success depends on equity in the relations and exercise of power within the Network. The members are autonomous organisations and the best guarantee that a decision will be implemented in a Network is for the executors to participate in making it. Thus, the members must participate in decision-making as well as in decision-implementing.

 

Diversity

 

The strength of a network resides above all in the diversity of its membership. It draws on the distinct social, economic, political and cultural contexts represented by its members. Part of the genius of a Network is that its members have different conceptions and utilise a variety of strategies to achieve change, while at the same time sharing common values and a collective purpose. The challenge of Network is to enable each one of these actors to make a creative and constructive contribution.

 

Dynamism

 

The Network promotes and maintains dynamism to the extent it is able to balance the diverse contributions of members with joint, sustained collaboration. For this, the leadership must stimulate and strengthen democratic internal processes, the active participation of all members and working effectively in alliances. A Network must enhance the interaction between its members. It facilitates innovative proposals for action.

 

Excellence – The relationships between organisations and individuals engaged in purposeful action characterises a Network. The quality of the interaction is a result of the quality of organisational performance. In fact, the effectiveness and efficiency of the operations of a Network is often the best guarantee of political impact.

These four criteria run through three operational dimensions that we analyse in relation to the six principal components of a NetworkÕs functioning.

 

Political purpose and the strategies

 

This is the NetworkÕs capacity to nurture consensus amongst its membership for its political reason for being and for the avenues to fulfilling that purpose. The political purpose answers the questions: What social change does the Network aim to achieve? What values motivate its members? For other types of organisations, the answers would be found in their Òmission statementÓ o Òinstitutional objectivesÓ.

 

The strategies refer to the approaches the Network employs to achieve its political purpose: How does the Network propose to generate results that will fulfil its purpose? Since an international network is composed of organisations rooted in the reality of different countries, the strategies necessarily are of a general nature. Nonetheless, the relevance or not of the strategies that a Network develops is one of the elements that determines if its activities will have social impact or not.

 

Organisation and management

 

A Network operationalises its strategies through lines of action. These are systematic, continual processes that produce results on different levels and of varying importance, all of course to fulfil its purpose. The lines of action are similar to programmes and projects in other kinds of organisations. The difference in a Network is that the emphasis is more on the action or the process than on achieving pre-determined SMART objectives[6]. Furthermore, responsibility for the activities is more dispersed and the operational units—the members as well as the secretariat—operate with a high degree of autonomy.

 

In a Network, efficiency and effectiveness depend on structure, operational management, institutional capacity, and communication.

 

Within a Network structure, instead of an executive office there is a body whose function is co-ordination and facilitation. This entity steers the NetworkÕs strategies and actions, articulating them with the activities of individual members. The operation of this secretariat may include projects with precise objectives that can be readily evaluated. Nonetheless, it is important to keep in mind that the local activities, and the changes they bring about, are principally the responsibility of the individual members.

 

In contrast to other types of organisations, in a Network operational management focuses on enhancing collective, democratic, horizontal and diverse activities of members rooted in specific local realities. Management is guided by the common purpose, which is the basis for trust and gives coherence to the multiple activities.  Consequently, the secretariat, as the key component of operational management for the network, co-ordinates more than it administers programmes.

 

As in any organisation, the institutional capacity of a Network depends to some extent on the people in positions of responsibility. Decision-makers should be qualified for their specific tasks, just as the material and financial resources should be appropriate for the activities of the Network. Furthermore, the institutional capacity of a Network is based upon the capacity of its members. Consequently, a Network strives to empower and strengthen its members through training, exchange of information and mutual support. It develops and takes advantage of the resources and energy of all its organisational components, thus multiplying and compounding the effect of individual efforts.

 

For every social organisation, communication is important; in a Network, it is vital. A Network is essentially a complex of human relations, and their quality and characteristics determine success. Due to its character, a Network promotes social mobilisation, generates technical, political and financial support and involves external actors. Therefore, it must create complementarity, synergy and strategic alliances.  Consequently, communication is as much an organising and management function as it is one of information exchange. Furthermore, an international network is intercultural, requiring understanding across great distances and social and cultural differences. For all these reasons, the communication function is central to success or failure in a Network.

 

Leadership and participation

 

For a Network, everything related to leadership and participation is as important as its strategies, organisation, and management, because the values of democracy are intrinsic to its nature. A Network aims to be more that an association of like-minded organisations. Common agreement on the strategies is as important as selecting the right strategies. Similarly, the Network action should be more than the sum of the individual activities of its members. To achieve this Òadded valueÓ, decision-making processes must be characterised by a democratic leadership and the active involvement of the members. Also, there must be many opportunities for all members to participate in the activities of the Network and collaborate with each other. More concretely:

 

á             Decision-making requires as much agreement about who should participate in which decisions as it does broad participation in making specific decisions.

 

á             The participation of those who make up a Network is fundamental for its sustenance and endurance; it is a source of for enrichment and strengthening the Network. Effective participation depends on a mix of different factors—the opportunities, funding, time available, interest, commitment and above all trust.

 

á             Co-ordination is basic to a healthy Network that generates synergy. This depends heavily on a leadership that enhances internal management and the presence and influence in the wider world.

 

Initial proposal of indicators

 

For greater clarity about the potential inter-relations between the performance criteria and the operational dimensions, as well as to order the analytical framework, we propose a set of possible indicators for evaluating a Network. We realise that these are tentative and rough. They are presented in the following table.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

POSSIBLE INDICATORS FOR THE EVALUATION OF AN INTERNATIONAL SOCIAL CHANGE NETWORK

 

 

OPERATIONAL DIMENSIONS

 

 

Political purpose and strategies

Organisation and management

Leadership and participation

PERFORMANCE CRITERIA

Democracy*

á     Vision and mission are shared.

á     Members have a sense of belonging

á     The Network focuses on the essential—fulfilling its political purpose.

á     All the members collaborate in the activities.

á     The members contribute and have equitable access to the resources (people, funds, goods and services) and reputation of the Network.

á     The structure is not hierarchical.

á     The decision-making process is considered just, inclusive and effective.

á     Participation is generalised and voluntary.

Diversity*

á     The political positions and ideological contributions of all members are reflected in the strategies of the Network.

á     The range of opinions and ideas of the members have a place in the activities of the Network.

á     Importance is given to building relationships of trust internally and externally.

á     The human and financial base of the Network is sufficiently broad to avoid the dependence of many on a few.

á     The diversity of members is appropriate for the NetworkÕs purpose and strategies.

á     The strategies are developed with the contribution of all of the members.

á     The interaction between the members is creative and constructive.

Dynamism*

á     The priority is to act.

á     Goals are pursued seizing the opportunities and adjusting to obstacles without losing sight of the political purpose.

á     The Network learns from experience. Achievements serve as a basis for reformulating the strategies. 

á     The responsibility and authority is effectively balanced between that vested in the secretariat or co-ordinating entity and that decentralised to other bodies of decision-making and execution.

á     The structure is light, facilitative and supportive. The rules are minimal.

á     The resources expand and contract, quantitatively and qualitatively, according to the strategic needs.

á     The members take initiative and influence the development of the Network.

á     The co-ordination between the members is constant and effective.

á      Co-ordination with other networks on common action issues is effective.

á     All the members contribute to and benefit from the achievements.

á     The effect and impact are more than the sum of the activities of the individual members.

Excellence

á     The social changes that are pursued are clearly defined.

á     The strategies are based on an up-to-date analysis of the environment.

á     The strategies and lines of action are coherent with the social changes the Network seeks.

á     There is a clear organisational identity embraced internally and externally.

á     The Network achieves results at the local and international levels.

á      The Network has impact—it achieves structural, long-term changes.

á     Work is planned, monitored and evaluated.

á     Policies on how the Network should and should not function are followed.

á     The financial function is well structured.

á     Internal communication is effective.

á     The qualifications of the staff of the Network are suitable to their responsibilities.

á     The assets—material or immaterial—are appropriate for the requirements of the strategic lines of action.

á     A financial strategy is pursued and the financial resources are adequately managed.

á     The Network is autonomous—it decides on and defines its own paths.

á     Learning is a basis for innovation

á     The membership is active.

á     Members participate as much as they desire, and their contribution is recognised.

á     There is sufficient opportunity to participate in processes of reflection and decision-making.

á     Conflicts are resolved.

á     Decision-making processes are solid.

á     The Network is able to involve and lead other social actors.

á     Alliances contribute to the implementation of the lines of action.

á     Alliances lead to the formulation of new strategies.

á     Members become more effective and committed actors and protagonists.

á     The Network effectively dialogues and negotiates with other social actors.

Source: Martha Nu–ez and Ricardo Wilson-Grau.     *Based on Madeline Church, et al 2003


Assessing the results

This methodological proposal for evaluating the performance of an international social change network is a tentative, initial step. We recognise that our reflections about assessing the results are even more rudimentary and, undoubtedly, debatable.

 

We consider that it is of fundamental importance for a Network to identify and comprehend its internal and external achievements, including its impact—understood as durable structural change. Notwithstanding, the evaluation of the impact of a Network becomes extremely complex, as complex as the structure and operation of an international political movement.

 

For example, concerning the internal achievements, one of the principal results of great validity and importance is the NetworkÕs existence and permanence over time. We know that this is an unconventional criteria for evaluation. A for-profit business can rarely justify itself by the number of employees; its principal results are measured by a margin of profit and return on investment. Sometimes the major achievement of a government may be simply to have finished its term of office, but usually its results are evaluated in terms of the quantity and quality of its contribution to the common good. An NGO cannot exist to exist; the NGO must benefit other people. In a Network, however, the results are the fruit of a collective effort of all the members. With the support of the Network, they reinforce each other and advance together with joint strategies to achieve their common purpose. If the Network functions effectively and efficiently, it strengthens and develops the web of organisations and relationships that make it up. That is, the existence of the Network is itself an inter-active, innovative process with added value for its members.

 

The success of a Network also is a function of its external achievements. Of course, we are evaluating the results of a political movement. How do you evaluate external achievements when they are intangibles such as awareness-raising, empowerment, solidarity, and equity between people and in society? We believe, on the one hand, that these results are difficult to measure, at least with any conventional methods and instruments. On the other, they can only be measured indirectly, through the results that flow from them. That is, greater awareness, empowerment, solidarity or equity are identified through the actions, the processes and the changes they bring about.

 

For example, the achievement of empowerment can be recognised in the manner in which people more actively assume responsibilities, propose and build their relationships, and present and defend their demands before authorities and other actors. An external evaluator, however, rarely can, directly and independently, perceive greater power in another organisation, group of people or a community. With limited time, he or she can at best obtain signs of change by examining documents or interviewing people. In contrast, the members of the organisation, the group or community, through their common struggle, and by reflecting before and after the event, can formulate a judgement, systematize events and measure progress. Thus, evaluating these external achievements of a Network requires a different exercise.

 

Regarding impact, we believe that to achieve it, and to be aware of having achieved it, is vital for a Network. It is, at the end of the day, a NetworkÕs political reason for being. Nevertheless, the evaluation of durable social change achieved by any organisation is a complex endeavour currently subject to intense debate.[7] To begin, the definition of what is ÒimpactÓ is controversial, and much more so is the discussion about how to measure it. The complexity grows when we try to evaluate the impact of a Network, because of its political nature and organisational characteristics.

 

Since a Network is an association with the aim of changing relations of power, made up of diverse national organisations with their own missions and objectives, the problems in evaluating impact are double-edged. First, how do you measure changes in the power processes of societies that are indefinite in time, occur in heterogeneous contexts and do not depend on the decisions of the network or its members? Second, when there is a change that represents impact: Who can assume credit for the change? Who is accountable, and to whom and how?

 

In summary, there appears to be an emerging consensus that it is illusory to attempt to identify relations of cause and impact, at least for an international network. It is very difficult to determine which impact can be attributed to one organisation and which not. It is even less possible to know to what degree one or another organisation has contributed to lasting structural results.[8] To date, it seems to us the most objective criteria is maintained by authors[9] who argue that the best guarantee of impact in a Network is the relevance of its strategies and their coherence with the activities. That is, perhaps the closest we can get to understanding the impact of a Network is by evaluating its performance.

 

REFERENCES

Chapman, Jennifer and Wameyo, Amboka, Monitoring and Evaluating Advocacy: A Scoping Study, ActionAid, January 2001

Church, Madeline, et al, Participation, Relationships and Dynamic Change: New thinking on evaluating the work of international networks, Development Planning Unit, University College London, 2003

Karl, Marilee editor, ÔMeasuring the Immeasurable: Planning, Monitoring and Evaluation of NetworksÕ, WFS, 1999

Roche, Chris, Impact Assessment for Development Agencies: Learning to Value Change, Oxfam GB, 1999

 

 

 

 

POSSIBLE INDICATORS FOR THE EVALUATION OF AN INTERNATIONAL SOCIAL CHANGE NETWORK – edited version, Church and Kiriwandeniya

 

 

OPERATIONAL DIMENSIONS

 

 

Political purpose and strategies

Organisation and management

Leadership and participation

PERFORMANCE CRITERIA

Democracy*

á               Vision and mission are shared.

á               Members have a sense of belonging

á               The Network focuses on the essential—fulfilling its political purpose.

á               All the members have the opportunity to collaborate in activities that make best use of their skills and contribution.**

á               The members have equitable access to the resources (people, funds, goods and services) and reputation of the Network.

á               The structure is not hierarchical.

á               The decision-making process is considered just, inclusive and effective.

á               Participation is generalised and voluntary.

Diversity*

á               The strategies of the network reflect the range of political positions in the network.**

 

á               The range of opinions and ideas of the members have a place in the activities of the Network.

á               Importance is given to building relationships of trust internally and externally.

á               Conflicts donÕt paralyse the networkÕs capacity to act**

á               The human and financial base of the Network is sufficiently broad to avoid the dependence of many on a few.

á               The diversity of members is appropriate for the NetworkÕs purpose and strategies. Members are enriched by the difference**.

á               The strategies are developed with the contribution of all of the members.

á               The interaction between the members is creative and constructive.

Dynamism*

á               The priority is to act.

á               Goals are pursued seizing the opportunities and adjusting to obstacles without losing sight of the political purpose.

á               The Network learns from experience. Achievements serve as a basis for reformulating the strategies. 

á               The responsibility and authority is effectively balanced between that vested in the secretariat or co-ordinating entity and that decentralised to other bodies of decision-making and execution.

á               The structure is light, facilitative and supportive. The rules are minimal.

á               Organisational culture is  in tune with network principles – it ÔthinksÕ and ÔactsÕ as a network, not an institution**

á               The resources expand and contract, quantitatively and qualitatively, according to the strategic needs.

á               The members take initiative and influence the development of the Network.

á               The co-ordination between the members is effective.

á               All the members contribute to and benefit from the achievements.

á               The effect and impact are more than the sum of the activities of the individual members.

Excellence

á               The social changes that are pursued are clearly defined.

á               The strategies are based on an up-to-date analysis of the environment.

á               The strategies and lines of action are coherent with the social changes the Network seeks.

á               There is a clear organisational identity embraced internally and externally.

á               The Network achieves results at the local and international levels.

á               The Network is a key player in the work ** to achieve structural, long-term change.

á               Work is planned, monitored and evaluated.

á               Policies on how the Network should and should not function are followed.

á               The financial function is well structured.

á               Internal communication is effective.

á               The Network understands what qualities and skills are needed in the Coordinating Function, and people are managed in ways that allow those skills and qualities to be utilised. **

á               The assets—material or immaterial—are appropriate for the requirements of the strategic lines of action.

á               A financial strategy is pursued and the financial resources are adequately managed.

á               Learning is a basis for innovation

á               Leadership combines co-ordination, facilitation,, new ideas, and encourages innovation, and  focus,**

á               Leadership is not just vested in the coordination function, but emerges around the network where appropriate to activities or issues. **

á               The Network is able to involve and lead other social actors.

á               Alliances contribute to the implementation of the lines of action and lead to the formulation of new strategies.

á               Members become more effective and committed actors and protagonists.

 

 

 

[1] Also available in Spanish from the authors.

[2] Ecuadorian anthropologist and consultant; member of consultancy and advisory groups for national and international institutions; formerly, co-ordinator of the Latin American Forests Network. Martha Nu–ez [ghelix@pi.pro.ec].

[3] Senior advisor with Novib and independent consultant; formerly, co-ordinator of the International Toxics Campaign of Greenpeace International and director of development, educational, research and journalistic programs and organisations in Latin America. Ricardo.Wilson-Grau@inter.nl.net

[4] From here on we use the term ÒNetworkÓ to refer specifically to an international social change network.

[5] The first three are based on Madeline Church, et al.

[6] Specific, Measurable, Achievable, Realistic and Time-bound.

[7] See, for example, Chris Roche, and Jennifer Chapman and Amboka Wameyo.

[8] See: Madeline Church, et al and Marilee Karl.

[9] Marilee Karl.