I'm no theologian, and I don't pretend to be. I probably have less biblical knowledge than you. But here's my two cents:
We are created with a purpose. Our purpose is to glorify God. To paraphrase: "Whether you eat, or drink, or whatever you do; do it all for the glory of God".
We are commanded to go, preach the gospel, "making disciples of all nations, baptizing them in the name of the Father, the Son, and the Holy Spirit".
Does this mean that when are not actively in pursuit of God's glory or out talking to people about Jesus that we are sinning? To be overly-legalistic about it (to the EXTREME), then yes. We have put something in the place of God - whether it be TV, video games, sleep, the need to relieve oneself, whatever. Unless you make a conscious decision to fertilize God's creation for His glory when you just gotta go, then you have placed your own needs as more important than God.
Now let's be realistic. Does God expect that we do nothing but read the Bible, stand and preach on street corners, and spend every other minute of our day in face-on-the-floor prayer with Him? Absolutely not. If we spend 8 hours a day working at a secular job (writing computer programs, teaching math in a public school, cleaning toilets, etc), are we saying that we need to do so for a paycheck because we do not trust God to provide for us? Absolutely not.
I believe the key word is Moderation.
Do we allow video games to TAKE THE PLACE OF prayer time or Bible study? Do we choose not to go reach a lost world because we want to get that last 1% map completion in Cursed Shore? That's where we get into trouble.
Maybe I'm rationalizing (I'm good at that!). Maybe I'm missing the point. But here's my take on it:
1. Spend time with God each day. Talk to him (pray without ceasing). Study His Word.
2. Make an effort to develop relationships, both with Christians for mutual edification, and with lost people for their salvation.
3. Be a good steward with what He has given. Provide for your family, as you are able. Bless others. Give back to God out of the abundance (and yes, we ALL have abundance).
4. Enjoy life, in whatever way gives you happiness, but do so in a manner that glorifies God and does not create problems for others.
Romans 14:13-18
Therefore let us stop passing judgment on one another. Instead, make up your mind not to put any stumbling block or obstacle in the way of a brother or sister. I am convinced, being fully persuaded in the Lord Jesus, that nothing is unclean in itself. But if anyone regards something as unclean, then for that person it is unclean. If your brother or sister is distressed because of what you eat, you are no longer acting in love. Do not by your eating destroy someone for whom Christ died. Therefore do not let what you know is good be spoken of as evil. For the kingdom of God is not a matter of eating and drinking, but of righteousness, peace and joy in the Holy Spirit, because anyone who serves Christ in this way is pleasing to God and receives human approval.
I'm not sure if that verse is necessarily appropriate in this situation (remember, I'm no scholar), but what I get out of it is this: 1) behave in a way that does not cause problems for others, 2) nothing in and of itself is evil, and that 3) it's not about what we do - it's about the spirit we do it in.
I've been in the same struggle as you more times than I can count, and I continue to look for answers.