Short answer: This will not end well for you, my friend.
Slightly more serious answer: I have my Xbox in my living room so I can play it on my 55". Nowadays I mostly play when she's gone, asleep, or has work to do and wouldn't be paying attention to the TV anyway. It works for us. We enjoy a lot of the same shows, so I'd feel kind of rude sitting in the living room playing video games when we could otherwise be watching something we both like. Especially after we've both gotten home after a long day of work. I will say, though, that it has led to me playing much less.
Very serious answer: If you value being able to play on your own time for however long you want, leave your setup downstairs. My wife has never complained about me playing video games, but I have a feeling it would reach that point if I took up the living room TV for a number of hours while she sat and watched. I guess she could always go watch TV in the bedroom, but I'd never ask her to do that.
At this time 4 years ago, my younger brother lived with me and I had a 55" and a 42" side by side in the living room. My xbox was hooked up to the 55" and his to the 42", and we played Call of Duty online with our college friends for 3-5 hours every night. On Saturdays we'd have 2 college football games on at the same time the entire day. Ahh, the good old days