Preheat oven to 425˚F (220˚C). Line 8 large or 12 standard muffin tins with liners or lightly grease.
In a large bowl, whisk together the flour, crushed graham crackers, brown sugar, granulated sugar, baking powder, baking soda, salt, and nutmeg.
Make a well in the center and add the buttermilk, vegetable oil, eggs, and vanilla extract.
Whisk together just until combined without any lumps.
Gently fold in the milk chocolate chips and miniature marshmallows.
Transfer the batter into each prepared muffin tin until 3/4 full. If desired, top each muffin with a few more chocolate chips and miniature marshmallows.
Bake in preheated oven for 5 minutes, then reduce heat to 350˚F (180˚C) and bake for another 10-15 minutes, until a toothpick inserted in the center comes out clean (melted chocolate is fine). It will be closer to 10-12 minutes for regular-sized muffins and 12-15 minutes for large.
Cool in the pan for 5 minutes before removing to a wire rack.
These S'mores Muffins are best served the day they are made, especially within a couple of hours of baking.