Abstract
This works focuses on the problem of confidence in decisions made. Existing studies of prognosis calibration (defined as the ratio between the correct response frequency and the subjective evaluation of response correctness) have revealed that people tend to overestimate the probability of correct responses ("overconfidence effect"). The authors hypothesized that subjective estimates of confidence in decisions made and the accuracy of prognosis calibration depend on the method used to measure confidence.

An experimental study was performed in a sample of school students (N=50) and university students (N=36). The participants answered common knowledge questions and evaluated the degree of confidence in the correctness of their answers on verbal scales, using either a standard scale or a duplex scale (rating their confidence in both chosen and rejected answer options). Finally, the participants made bets that the chosen alternatives would be correct.

The results indicated that the "overconfidence effect" was weaker in the duplex scale condition, confirming the hypothesis. We explain this finding by the notion of residual uncertainty that remains when two answer options are compared, but not in the standard scale setting, when the respondents' attention is concentrated only on the chosen answer option. The association between the level of confidence and prognosis quality had an inverse linear shape in the standard scale setting and a parabolic shape in a duplex scale setting. Finally, we found that in cases when the decisions were rational and the confidence ratings were consistent, bets predicted prognosis quality more reliably than verbal estimates did.

Details
This work started at Teachers College, Columbia University, New York in 2012 as research for my Master thesis and was finished in collaboration with Prof. Alexis Belianin at National Research University Higher School of Economics, Moscow in 2013.

Description
The work on this paper consisted of the following stages

1. Based on Social-Cognitive Career Theory the framework for vocational interests questionnaire in Data Science was constructed. Popular occupations in Data Science were decomposed into 4 dimensions: math, programming, analytical thinking, and leadership. Potential career choice was modelled as a combination of self-efficacy, outcome expectations, and occupational interests.
2. 44 statements were developed based on the framework. These statements were prepared for both Likert-type and ipsative questionnaire.
3. Based on three cognitive laboratories, 40 statements were chosen for the next stage, numerous statements were re-written.
4. Questionnaires were piloted. 84 respondents participated in the pilot study. Classical Test Theory was used to assess the quality of the items formulated for Likert-type questionnaire. Based on this analysis, statements were re-written once again.
5. New versions of the questionnaires were tested in 5 cognitive laboratories. They were re-written.
6. Randomised design was developed for the next round of surveys exposure. The last step will be to gather new dataset for both questionnaires. The analysis will include IRT for item quality assessment for the Likert-type survey and transformed data for the ipsative survey. Then SEM will be used to test whether the data fits the theoretical framework. Finally, prediction models will be built to test whether the questionnaire could be useful for prediction of one's vocational interests.

Supervisors
Maxim Skryabin
Denis Federyakin

Further details
This work has been developed during my Master's program in Educational Measurement at National Research University Higher School of Economics, Moscow

Description
It was necessary for the online school to understand whether its students develop during their studies. For the pilot study two courses were chosen: 12-month program on Data Science and 6-month program on Analytics. The work on this project consisted of the following steps:

1. Pedagogical specifications for both courses were revised
2. Based on the specifications, two knowledge tests were designed in collaboration with domain experts (15 items in the Analytics test, 25 items in the Data Science test)
3. 47 and 255 respondents participated (Analytics and Data Science correspondingly), they represented 16 and 29 cohorts who had started studying at different time
4. CTT and IRT were used for the analysis of Analytics and Data Science tests respectively
5. It was shown that, at the Data Science program, students who started to learn earlier showed significantly higher knowledge level than more recent cohorts. This difference was not so significant for the students of the Analytics course. Some knowledge gaps were discovered. The tests were revised based on the item quality analysis

Details
I initiated this research while working in SkillFactory as a product manager at Quality Lab. The goal of our team was to raise the quality of our educational content

Description
The idea was to create games for 21st century skills assessment. All games were based on the famous experiments from behavioral economics or cognitive science. As a result 4 online games were created from scratch just in two weeks with a team of 4 people. Games were piloted with a 200+ children in cooperation with an educational platform InternetUrok.

Four competencies that games were aimed to measure:
– Ability to learn (the game was based on the theory of concept formation; Voiklis, J., & Corter, J. E. (2012). Conventional wisdom: Negotiating conventions of reference enhances category learning. Cognitive Science, 36(4), 607-634)
– Resource management (the game was based on the ability of respondents to act rationally; Shin, J., & Ariely, D. (2004). Keeping Doors Open: The Effect of Unavailability on Incentives to Keep Options Viable. Management Science, 50(5), 575-586)
– Ability to concentrate (the game was based on the classical n-back task; https://en.wikipedia.org/wiki/N-back)
– Ability to manage risks (this game was more about personal preferences than abilities; it was based on the Tanaka, T., Camerer, C. F., & Nguyen, Q. (2010). Risk and time preferences: Linking experimental and household survey data from Vietnam. American economic review, 100(1), 557-71)

This project doesn't have any scientific value, as I understand right now. However, it is because of this initiative that I decided to dive deep into psychometrics.

Further details
Try a prototype:
http://128.199.32.231/#/login
Login: testphd
Password: passphd20

Description
I participated in data collection for public goods experiments in Perm, Tomsk & Moscow. In particular we gathered data to explore the phenomenon of spite. The spite could arise when respondents saw that someone contributed more points to the public good than they did. This effect was found to be especially strong in Middle East, Russia and Eastern Europe (Herrmann, Gachter, Thoni, 2009).

The goal of the experiments conducted by our lab was to explore true motives of the people who showed such anti-social behaviour. It was found that such motives are numerous: availability of the option to punish, competitiveness, preemption, etc.

The conclusions were based on 300 paid participants. 128 from Moscow, 76 from Perm, 96 from Tomsk. The experiment was realised in the software z-tree.

Supervisor
Prof. Alexis Belianin

The initial goal was to create an opportunity for every child to explore the world around. 3.5 years ago we started with online-meetings for teenagers living in distant orphanages, today we work with both young people from regional orphanages and high-school students from all over Russia and CIS. By working with two groups of participants we are able to be financially stable. Currently our focus is on offline activities, for example city camps.

• 10 products in career counselling (online meetings, trainings, school courses, camps, etc.)
• More than 18k participants in online course
• More than 500 hours of events
• More than 400 unique participants from all over Russia in summer camps
• 17 regions including Armenia and Ukraine
• Won the competition for funding from federal government, 2018; from Moscow government, 201

SkillFactory (Data Science School, more than 4,000 students, 30 online-courses). Launched Quality Lab - the department for quality control. As a product-manager increased gross profit from 30% to 55%, quality from 23% to 60% (NPS). Launched 'New products' department (free experimental products).

The idea was to show that Economics as a science doesn't study only finance and business but analyses human behaviour under different settings: from shopping to dating. This project is currently on hold, though I conduct some rare events on non-standard economics and teach classes in the school on weekly basis.

2010-2016 results

• More than 800 participants overall
• From 7 to 60 years old
• 20 events in 1 year
• 2 year-courses for schools are developed and conducted

Taught Economics for school children in High School, 2015-2017 School №1533,
Taught Behavioral Economics for school children in Middle School, School Intellectual, 2014
Designed and taught 10+ workshops on Statistics and Economics, 7-12 year old children, Polytechnic Museum, Moscow 2015-2018 (Independant)

Developed a board game about communication for top-managers, Masterskaya O. Zamyshlyaeva

Developed free online course about MyStory approach to career counselling and published it at Stepik.org. It includes 6 chapters on how to organise engaging meetings for young people and professionals from various occupational spheres at school.