Annotations are associated with the zoom level of the document. However, you can set the default font size by opening the calculator and clicking "Set Default Font".