Meanwhile, in a medium bowl, whisk egg yolks and sugar until smooth. Gradually whisk cream mixture into egg-yolk mixture until smooth. Return mixture to saucepan and cook over medium-low heat, stirring constantly, until it thickens and coats the back of a spoon (do not boil or it will curdle).
Remove saucepan from heat; stir in buttermilk. Strain custard through a sieve into a large bowl; add almonds. Refrigerate custard until well chilled, at least 2 hours but no more than overnight.
Freeze in ice cream maker as directed by manufacturer.