Mark, if anybody has an "issue" with you wearing that patch, they will have a "problem" if any of us hear about it.
Whatever you or your family decide is right to honor your brother's memory, that is the right thing to do, period. Everybody else's opinion is in second place. There's no third place.
Go for it.
Bob