You will be presented with a series of hypothetical events. For each event, you will submit a probability \(p \in [0, 1]\) that the event occurs before the end of the year.
Your score is the sum of your scores across all events. Your score on one event is defined according to an adjusted logarithmic scoring rule.
\[ \mathsf{score}(p) = \begin{cases} \ln p - \ln 0.5 & \text{if event occurs} \\ \ln (1 - p) - \ln 0.5 & \text{otherwise} \end{cases} \]
Define “scaling \(p\) by confidence \(c\)” as
\[ \mathsf{scale}_c(p) = \frac{p^c}{p^c + (1 - p)^c} \]
Let \(\mathsf{score}_c(p)\) be your score if you scale \(p\) by confidence \(c\). \(\mathsf{score}_c(p)\) is concave in \(c\), so there exists some \(c\) that maximizes it. Your confidence is the multiplicative inverse of this optimal \(c\).
A confidence greater than \(1\) means you are overconfident, a confidence of \(1\) means you are perfectly confident, and a confidence less than \(1\) means you are underconfident. A confidence less than \(0\) also indicates that you would have been better off submitting probabilities in the opposite direction.
For example, if your score would have been maximized by scaling your probabilities by \(2\), then your confidence is \(0.5\) and you were underconfident.
The methodology is described in my blog post Scaling Probability by Confidence.