Preheat the oven at 375 degrees F. Line a 8 1/2 x 4 1/4 loaf pan with parchment paper and lightly spray or brush with oil. Set aside.
Combine gluten-free flour, almond flour, potato starch, sugar, baking powder, and salt in a large bowl and mix well.
Using a whisk or mixer, add milk, oil, vanilla, and mix until the batter is smooth and creamy.
Divide the batter equally into two separate bowls.
Add the carob powder or cocao powder to one bowl and mix until smooth.
Pour alternate layers of each color batter into the prepared loaf pan. Using a butter knife, make swirls.
Make swirls, using a butter knife, making a zig-zag motion around the pan twice.
Bake loaf for 1 hour and 20 minutes or until a fork or skewer inserted in the center comes out clean
Transfer the loaf to a cooling rack and let it cool in the pan for 10 minutes, then remove from the pan and cool completely before slicing.