Preheat the oven to 350ºF (180ºC). Butter a 9-inch (23cm) springform cake pan.
Halve, pit, and cut the plums into 8 slices.
In a small bowl, make the topping by mixing together the sliced almonds, 2 tablespoons of flour, light brown sugar, 1/2 teaspoons (each) cinnamon and cardamom, and melted butter, until everything is evenly mixed. Set aside.
In the bowl of a stand mixer fitted with the paddle attachment, or by hand in a medium bowl, beat the butter and sugar until light and fluffy, about 3 to 5 minutes.
In a separate bowl, whisk together the flour, baking powder, baking soda, 1 1/2 teaspoons of ground cardamom, and salt.
Add the eggs to the creamed butter one at a time, scraping down the sides of the bowl after each addition to make sure they’re mixed in. Add the vanilla extract. Stir in half of the dry ingredients, then the buttermilk, then the rest of the dry ingredients, mixing just until combined. Do not overmix.
Spread the batter into the prepared cake pan. Strew the plums over the top in an even layer, then spread the almond topping over the plums. Bake until the center just feels set, about 55 minutes to one hour. A toothpick inserted into the center should come out clean of any cake batter. If the top is browning too fast, drape a sheet of aluminum foil over the cake pan and finish baking. Let the cake cool completely then remove the sides of the springform pan.
To make the toffee glaze, mix the butter, dark brown sugar, cream, salt, and vanilla in a small saucepan. Bring to a boil, stirring gently. Reduce the heat and cook at a low boil for 1 minute. Remove from heat and drizzle the toffee over the cake.