Meanwhile, finely chop remaining lavender sprig and set aside with remaining teaspoon orange zest. In a medium stainless steel bowl, combine yolks and sugar, whisk for 2 to 3 minutes until it reaches a pale yellow color, and sugar is dissolved.
Reheat milk mixture until very hot but not boiling, and slowly pour about a third of it into eggs, whisking constantly. Then pour this mixture back into the saucepan with remaining milk. Over medium heat, stir constantly with a wooden spoon until thick and can coat the back of the spoon. Do not let it boil. Remove bean pods and strain mixture into a large bowl. Add remaining lavender, orange zest, and orange juice and allow the mixture to cool to room temperature, stirring occasionally.
Refrigerate for at least 2 hours. (Ice cream can also be made one to two days in advance.) Using an ice cream machine, process according to manufacturer's instructions. Serve when firm enough to scoop.