In a medium saucepan, whisk together the egg yolks and sugar until well combined. Whisk in the milk, and cook over medium heat, stirring constantly, until it gets thick. Remove from heat, and add the butter. Place the custard in a bowl, and press wrap against the surface to prevent a skin from forming. Let cool to room temperature.
Meanwhile, beat the heavy whipping cream and vanilla extract until stiff peaks form. Whisk the mascarpone cream into the custard, and fold in the whipping cream until fluffy. Set aside.
In a small bowl, brew the 2 Tablespoons of lavender earl grey with 1 cup of hot water. Brew for about 4-5 minutes, until very strong. Let cool slightly. Mix in the brandy (or orange liqueur if you prefer!)
If your lady fingers are whole, cut them in half length-wise. (Mine came cut in half). Dip each half into the tea mixture, and place them at the bottom of a 8x8 baking dish (the dish I used was 8.5x5.5). Break the lady fingers in half to fill the bottom if necessary. Some gaps are ok. Spread half of the mascarpone filling over the lady fingers, and then repeat with the remaining lady fingers and cream. Refrigerate for at least four hours, preferably overnight.
Before serving, dust the top of the surface with a couple Tablespoons of cocoa powder. Serve cold with earl grey tea!