Place the sliced plums, berries and sugar in a saucepan. Using a sharp knife, slice the vanilla beans open lengthwise and scrape the seeds into the saucepan. Bring the mixture to a boil and then simmer until the sugar dissolves, about 10 minutes.
Slice one of the oranges and leave the other whole.
Off heat, stir in the brandy, wine, orange juice and orange slices. Transfer to pitchers and chill in the refrigerator, preferably overnight. When ready to serve, slice the remaining orange and use it as garnish.