#ocTEL MOOC (week 4 A42) Why would the student do or say this rather than that?

The second activity of this week on "Producing Engaging and Effective Learning Materials" is about the evaluation of resources in our area. So, it means, in my case, evaluating a resource for the learning of mathematics. However, I will start from a more general perspective. Whatever is the targeted learning, the first thing to check is the validity of the content the resource claims providing the learners with respect to the referent discipline. Then only, I will assess it from a learning perspective. Indeed, there are many issues to consider from accessibility to usability, motivation and autonomy. But, three questions have a hight priority in driving my evaluation:

Why would the student do or say this rather than that?
What must happen if she does it or doesn’t do it?
What meaning would the answer have if she had given it?

I borrow these formulations from the Theory of Didactical Situations (Brousseau 1997 p.65), but the questions are very pragmatic. The theory works here as a driver of our thinking; it is a tool to anticipate what could be the learning outcome, its likeliness, the possible limits and hence the needed intervention of a teacher. Depending on the responses, one may have to stage the use of the resource in one way or another.

Interaction and feedback are the main objects of the evaluation. The issue is not that students will do that or this, but why they do it,  because the constructed piece of knowledge must appear as the best adapted to the situation. Knowledge is something you reconstruct for yourself and appropriate because of its use value. The next issue is to verify, if the resource is interactive in some way, that it can feedback students so that they have a chance to realize that something went wrong and then react to that. If the resource is not interactive, then the issue is whether it is possible to figure out any thing about the activity (possibly, just reading) of students and find the appropriate support to bring. Eventually, the stake of this inquiry is the meaning possibly constructed by the student.

All this means that there is enough documentation about the resource, otherwise one has to guess or invent... just having a resource without information about its design, the intention of the designer and indications about its use, it is hardly possible to make a proper evaluation. This may be the reason why I couldn't do it for the proposed resource. But, anyway, I will make the exercise when achieving the third task of the week.

"e-assessment", a new entry of the TEL Dictionary

Assessment is one of the key functions of a learning environment, be it formative or summative. I would even claim that implicitly or explicitly, consciously or not, it drives its design. The definition of e-assessment prepared by Valerie Shute and Yoon Jeon Kim for the TEL Dictionary reminds us the expectation that technology enhanced assessment would be more efficient than assessment, thanks to the power of the computer. I suggest to add: thanks to the possibility of getting more personalised and accurate information on the learner activity.

As a matter of fact, e-assessment, from a research perspective, might not be far from learner modelling, a term which is not mentioned as one of the related terms Valerie and Yoon enumerate. In your opinion, what ontological or semantic relation can be established between "e-assessment" and "learner modelling"?

They play. So far, so good. But, what do they learn?

Retrieved from a post of Nicolas Balacheff on the SOA scientific portal on February the 26th, 2010
 
Martin Oliver and Caroline Pelletier contribution to the edited book "Digital Generations" is quite interesting and stimulating, taking up the challenge of contributing to our effort to understand "what, if anything, people are learning by playing games" (p.69) Their contribution is based on activity theory, referring primarily to Vygotsky, built on the system formed by the Tool as a mediator between the Subject and the Object (the latter meaning the intention of the subject) and its contemporary extension by Engeström and others which takes into account the social determination of both the Subject and (his/her) Object(ive). The authors make the relevant remark that taking a systemic perspective means that properties which may be identified cannot be ascribed to the Subject as an isolated part of that system. What raises a theoretical and methodological difficulty when the problématique is to understand learning (or Subject semantic/meaning attached to a behaviour). Meeting this difficulty with the cK¢ model [*], I solved it (if I may say so) by considering what could be seen as the projection of the system model onto one of its components, the learner (or onto the Tool). In the case of cK¢ it leads me to propose the (P, R, L, Σ) quadruplet to model the learner conception (what could be mirrored by a quadruplet of the same kind to model the Tool). So, it is clear that I am interested in the method of analysis which the authors propose in order to operationalize the theory.

Then, looking precisely at the proposed methodology, I see a few issues which may be interesting to discuss: contraction, action/operation and in the end the reference to learning and the related question "what is learned?"

Contradiction is a difficult concept to manipulate from a methodological point of view. As Piaget analysed it, contradiction exists if there is a witness of its existence and it can be noticed only if there is an explicit awareness of an objective or an expectation. So there may be a contradiction from the point of view of the observer and not from the point of view of the Subject. How to decide on that? Which observed behaviours can inform the observer? These are difficult questions but critical ones when learning is at stake (as pointed by the authors). So we cannot diagnose a contradiction if there is not an evidence that it is the case for the Subject and hence if we cannot state what is the Object from the Subject point of view. This points a new question: is the Object what the designers or the researchers or the observers claim to be? This question which is important to model the game-playing activity is indeed critical from a learning perspective (it is directly related to identifying learning outcomes). The authors identified in the discussion section, in relation to the interpretation and classification of observed behaviours, the "such claim are difficult to justify without assuming (rather than knowing) the intention of the player" (p.83). My own position is that this is a central issue for learning and that our research effort must start from that point : an explicit hypothesis on the learner intention.

The delicate distinction between action and operation could be better addressed if it was contextualised by such a claim about the intention of t he Subject or the Object of the activity. The authors express their expectations of a progress if a finer grained reading of the actions or the behaviours (eg eye tracking) was possible. My claim is that it may be of no help if the observer cannot relate it to an intention or an objective. Actually it is the identification of the Object in the system and/or the intended learning outcomes (at least as research hypotheses) which will determined the reasonable level of granularity we have to reach.

Eventually, in my opinion, the question "what is learned?" cannot be answered without responding to the question of the objective, intention, aim of the game and the situation which contextualises it. If we do not start from that point, we will progress as blind researchers and in the end respond "they learn how to play" (p.86), which may be a disappointing and quite unhelpful answer. We may agree that this applies also to the problem of understanding the Subject intention, then the nature of the Object and in the end the whole question of learning in a game environment. This issue may be peripheral from a strict game-play perspective, where whatever is learned the motivation and the interest in the game is the thing which counts, but it is critical from an educational point of view.

Note: - Piaget et al. (1974) Recherches sur la contradiction Paris: Presses univ. de France, 2 volumes. - (P, R, L, Σ) stands for Problem, Operators (in French "règles"), Representation (semiotic system), Control structure

A note after the reading of: Oliver M., Pelletier C. (2006) Activity theory and learning from digital games: developing an analytical methodology. In: Burckingham D., Willett R. (eds)  Digital generations (pp. 67-92). Mahwah, NJ: Lawrence Erlbaum.

The fascination of research

Retrieved from the TEL opinion blog, December the 17th, 2005

The word “Research” evokes the fascination of knowledge as well as the expectation of the mastery of the unknown. Research outcomes are expected to be innovative by nature and reliable by construction. Everything works as if being based on research, actions and decisions should be less risky than being based on any other grounds; namely, opinions and beliefs. Indeed, “opinion” is an intellectual category hazardous and anything but reliable, while “belief” is as contingent as opinion with the worse characteristic that facing failures it does not leave room for much revision.

The strength of research results lies in their justification, ruled argumentation (proof) or systematic empirical evidence, and their accessibility to revision under the pressure of refutation. Research results have the epistemic characteristic of knowledge; they are products of a human activity which transcend the historical and anecdotal context marking their origin. However, from a scientific perspective, a piece of knowledge is not a statement, but the complex “object” shaped by the relations between a statement, a proof and a theory—all framed by an accepted problématique that informs about the relevance of a question. The return of investment in research is the reliability, universality and openness of its outcomes, its cost is theory, proofs and dealing with refutations. This has two meanings: (i) research is not about the so called “reality”, but phenomena identified through the lenses of a problématique; (ii) the dialectic of proofs and refutation is not empirical but of a theoretical nature, possibly addressing not a result but its rationale or even its underlying problématique.

Nothing new there, but something to bring back to the fore when we question the role and the contribution of research to the development of TEL. Something which has been forgotten (or lost) with the emergence of “acadustry”!

Bad news from the West!

Retrieved from the TEL opinion blog. Originally published on April the 14th, 2007

The Washington Post has recently drawn our attention to the release of a Report to Congress by the US Department of Education , which --­shortly said--demonstrates that educational software have no significant impact on student performance. This is a good news, to some extend, since it seems that in general no one can demonstrate whatever in our domain. Counter-examples are much more common than proofs and here again some have found that “other research trials have proven that the technology works”… Indeed, this is not a response but at best one possible argument to fight the conclusions of the report. Other responses could be either that the quality of the software might be questioned, or that all this is due to the fact that “teachers were not prepared or properly trained to use the technology”. However, having a look at the report itself shades a different light on the results the Washington post comments on. It appears that the use of the software represents about 10% of the instructional time, what is quite limited. It is also said that teachers felt trained enough, that the software didn’t present major problem of use, and eventually that teachers tended to be more supporting the students than lecturing, and the students were more likely to engage in individual practice. By the way, some may conclude also that supporting students instead of lecturing, and engaging individually in learning have no significant impact on students performance… indeed it is a joke. The lessons to be learned may be more complex that those the media would jump on. The scope of a statement like “products did not affects test scores” is, in my opinion, limited if first we don’t know what was the use of these products, second if we don’t inform the way the scores relate to what the said products may impact. For the first issue, this is not easy: how to characterize the use of an educational software and its impact? And moreover, once we have such a characterization can we ensure that the impact can be reproduced? In other words: is the claim about the software impact (or non impact) based on a serious knowledge of the learning and teaching phenomena at stake? These questions are of a theoretical nature. Only once researchers will have reached a consensus about them, will we be able to engage in comparative studies and seriously address issues like those raised by the Report to Congress. Otherwise our discussions and arguments will only be offering stories against stories (even academic stories as we publish so many). For the time being the report is serious and deserve attention. Its strength comes from the rigour of the tools used and the size of the sample, at least at first glance.  Its weakness is that we don’t know the meaning of what is measured, and we have every difficulty to discuss the claimed results. This is not a new situation. We have the same difficulty with many research reports we read in academic journals and we hear in conferences. It is time to engage in the building of a common scientific quality reference in our field so that (i) we can support our claims about technology enhanced learning, (ii) we can inform properly research results and (iii) we can reproduce them where ever they come from, and eventually (iv) we can critically and scientifically evaluate research outcomes. Until now we are more often ascertaining that proving (validating). Before closing this short note, I would like to draw your attention to an article published by El Pais , the Spanish newspaper, reporting that thanks to the use of technology students (a sample of 1800) have much better achievement  than without in geometry. They score 25% higher…

En este artículo se presenta una investigación sobre la incorporaci´on de las Nuevas Tecnologías de la Información y Comunicación en el aula de Matemáticas en los niveles de ESO y Bachilleratos. La metodología que hemos seguido consiste en formar a los profesores con los mismos materiales y de la misma forma que ellos lo aplicarían en el aula. El proyecto, entre la fase investigación y la de generalización, ha tenido una duración de 6 cursos, desde el año 2000 al actual, en la que han participado más de 400 profesores y 15000 alumnos. Los resultados de la investigación confirman una mejora en el rendimiento de los exámenes “tradicionales” escritos del alumnado, del 11,2%, con una mejora global del 24,39%. Teniendo en cuenta que el informe Pisa 2003 otorga a España el puesto 23 con 485 puntos en capacidades matemáticas, si sobre esta puntuación aumentamos un 11,2%, tendrámos 539 puntos que nos elevaría al tercer puesto.

The complete version of the Report to Congress has been made available on the TeLearn Open Archive  [click here ]