Preheat the oven to 350F, and spray a 6-cavity mini bundt pan very well with baking spray. You can also use a 12-cavity cupcake pan. Set aside.
In the bowl of a stand mixer, combine the flour, baking powder, baking soda, salt, sugar, and chai tea. Add in the butter, and beat on low until it's incorporated and has a sandy texture.
In a small bowl, whisk together the oil, eggs, sour cream, and vanilla, and add them to the dry ingredients. Mix until combined.
Finally, pour in the milk, and beat on medium-low until smooth and creamy.
Divide the batter evenly between the cakes, and bake until golden brown and a toothpick inserted in the center comes out with just a few crumbs, 20-25 min. Set aside to cool completely.
To make the glaze, pour the powdered sugar in a medium bowl. Add in the vanilla, then add in the heavy cream, a little at a time, until a thick glaze forms. You may not need all the cream.
Pour the glaze on top of the cooled chai tea latte bundt cakes, and enjoy!