Remove or change the color of the "file display border"

How can I remove or change the color of the "file display border" as shown in the picture below?

I know I have been able to do this previously, but I cannot remember where I did it. Please advise.

Or do you mean the thin border/edge around the File Display Border toolbar?

Yeah I think I meant the latter, I already managed to change the background color, but not the borders/shadows around.

Those use the Preferences / Display / Colors and Fonts / Pane borders: Highlight and Shadow colors.